2007 August — Security Bloggers Network

Having discussed the importance of security training and really its criticality – without security training most software security programs are doomed to failure – I wanted to spend a little bit of time talking about how to go about creating such a program. Really the expert at this is one of my co-workers Roman Hustad who hopefully will write in detail about this at some point (for now he is on a long hike to nowhere way up North J - lucky him!). Anyways most of this post is really inspired and attributed to things I have learnt from Roman.

Here's what we will talk about in this post:

Who needs to be trained?
How do you scale?
How to be effective and efficient?
How to measure success?

Who needs to be trained?

In a word "everyone". The bottom line is it is not just the developers who play a critical role in building software, there are a bunch of other roles involved in the process and all too often companies tend to ignore or entirely forget about the impact these other folk can have on your software security.

Let's start in the beginning: well the business – whether this is the product management teams in an independent software vendor or it is the business units who need the specific application being developed in an IT organization. The business must be trained on why this security thingie is important in the first place. Ultimately in some way they pay the bills and decide the schedules – they will therefore need to buy off on the concept that during the development process we will engage in security activities that seemingly provide no direct value at least in terms of functionality.
Secondly, the business analysts, requirements specialists or whatever it is that role is called – the people who write software requirements. It is essential that they understand how to document security requirements. They need to understand what the different organizational drivers for security are – whether these are specific regulations that must be complied with, industry specific standards, company policies and then security features themselves.
Next are the architects, designers and developers. This is where the bulk of the training investment will be made. Most of the other groups discussed here are relatively small and the quantum of information that they need to be provided with is not that large either. With this group really the sky is the limit and the training has to constantly be evolving as the security industry itself evolves. This group needs to be trained into secure design and architecture principles, secure coding practices (in each of the languages that they work with) as well as in performing threat models, doing code reviews and security unit testing. As you can see this could become a one year graduate program in itself J so we will discuss some strategies later in this post of how to deal with this.
Testers. For some reason everyone tends to forget these poor souls. Given that their charter is to test software, why not provide them with the training and tools to test the security of software?
Deployment and operations engineers are the next group. Responsible for installing and maintaining the application in production, software security training is critical to teach them how to configure and run the application securely. This is everything from the need to patch servers and third party components to the SSL configuration on the web server or from the secure configuration settings in the web.config or web.xml files to the service contexts and credentials.
Finally the users themselves. Security awareness is critical for the users in general because guess what even if their account for instance is compromised due to their mistake (perhaps they provided their password to someone claiming to be from your helpdesk), they will most likely still blame the application ("well no one told me that I was NOT supposed to give my password out! And they did say they were from your helpdesk!").

How do you scale?

One challenge that any company that is looking to implement a security training program runs into is how to scale? A lot of the development groups I run into have hundreds or even thousands of developers – and like I said above this generally tends to be an issue only with developers; other groups above tend to be smaller or don't require as broad of a training regimen. So how do we train each and every one of those while still ensuring we deliver products on budget and on schedule? Do we need to train every single developer? Most training classes tend to be spread along the space of a week so how can we afford to take all of our developers off their work tasks for a week? It's not just about the money but about the time as well?

In my experience this is best dealt with by creating a tiered training program. Essentially, for every team (depending obviously on the size of the team) create a new role called a software security architect. This person should become the "go to guy or gal" for anything and everything related to software security.

Once you have this type of a development organization in place, the task if providing training becomes significantly easier. The Software Security Architect is obviously provided with elaborate training and education. This could take the form of one to two weeks of training on security and the development technologies and how the two relate to each other.

The average developer on the other hand can be provided with 4-8 hours of software security awareness that covers the importance of security, security principles, and common mistakes that development teams make and how to avoid them. They don't have to for example attempt to become experts at cryptography or authentication protocols. Anytime they need help with such things they can go up to their Software Security Architect. In fact chances are their Software Security Architect might have built a nice little API for all of these complex security functions.

This tiered program allows you to both achieve a goal of having all of the development staff being trained in software security while also letting you do this and indeed scale across the hundreds or thousands of developers you might have without putting your entire business on hold while the training is being disbursed.

Finally another strategy I have seen successful albeit on a smaller scale is to provide training as part of annual developer summits or geek clubs if your teams already have those. This is a good opportunity to impart training or more likely at least awareness when a large collection of representatives from your development teams happen to be in the same room for perhaps other purposes even.

How to be effective and efficient?

Once the decision to disseminate training has been made, there are a number of things that can be considered to make the entire process both effective and efficient i.e. how do we make sure we are successful both at truly increasing the average knowledge level from a software security perspective as well as do so at the minimal cost. Firstly for the shorter duration classes (4-8 hours) consider self paced computer based training (CBT) classes. Obviously with CBTs, you do gain the advantage of being able to start and stop at any time and incurring minimal cost per student. On the other hand you perhaps lose the hands on experience and interactivity with a live instructor. CBTs therefore work well for training that does not have significant hands on component – for example talking about the fundamental principles of software security or the common mistakes such as avoiding buffer overflows and SQL injection. Live training is much better suited for the intricacies of implementing cryptography or an authentication protocol. You might also prefer live training when your development teams are not too large and are comprised of highly experienced individuals who are likely to have tones of questions.

Another thing to consider is how to position this training. All too often we see companies mandating training in response to a compliance objective and pushing it down to the employees in a similar fashion. Instead it is always better to sell training internally as an investment in the employees. Security is an increasingly popular component of the skill set for developers and thus any formal or informal training they might have on this aspect adds value to their career and helps in its progression. This is especially true for the software security architects. As mentioned in the sidebar it is vital that the person that fills this role be part of the development team and not be supplanted from a security team. Moreover, this role can be made a track in the developer career path wherein senior developers and development leads can fill the role after gaining sufficient development experience with the technologies in use within their teams. This is an excellent way to sell the role to development teams with the carrot rather than the stick approach – the carrot being the tremendous investment in terms of personal training that will be made into this individual.

One aspect of training that is often forgotten is that the training itself can be forgotten! It is therefore vital to have at least yearly refreshers where not only is the material covered again but is also updated to account for the research that came out in the previous year. This is especially important since the world of software security is evolving at a rapid pace wherein problems that were considered purely reliability issues suddenly allow an attacker to remotely exploit them. There is also the opportunity to impart some of the continuous training through the threat modeling and code review processes albeit in an informal manner. By discovering bugs and flaws and discussing how to fix them we are subconsciously training the development teams to avoid them in the future.

If you considering bringing in an outside vendor for training, also consider having them customize their stock classes with examples, references to policies and procedures and contacts for people within your own environments. Ensure that their instructors are developers in the languages that they are going to be teaching. The last thing you want is someone who has never programmed in that language to teach people based on knowledge gained from a textbook. Finally, and along similar lines it is also helpful if the trainer has experience of working with similar enterprise applications as your own.

Finally, another effective strategy is to ensure that at least the security awareness training is part of the developer on boarding process. This means that such training must be integrated with the new hire training and package in general. This ensures that before a new developer even touches any code within their respective development teams they have at the very least gone through the fundamentals of software security.

How to measure success?

As Lord Kelvin once famously said "If you cannot measure it, you cannot improve it." So the question obviously always arises how do we measure the success or effectiveness of our software security training if you will. For that matter that is something that we can ask of any form of training. There are a number of mechanisms I have seen to be useful for this. Perhaps the easiest way is to tie the measurement into the training itself. For example, a quiz or test at the end of the training or perhaps after each module or component of the training. This is the quick and dirty way if you will but multiple choice quizzes are not always the most effective at measuring success – they are more likely measure the memory of the individual than the true understanding of the concepts. Of course you could take this further and make them more detailed assessments where perhaps in a lab format, students have to actually write code that corrects a software security problem in an existing code base.

At perhaps the other extreme is measuring actual improvement on the ground. This is typically a much longer assessment and at first look appears that it would cost a whole lot more besides just the cost of training itself. However, if you think about the assessment not just tied to the training but to the larger software security program, the assessment might not appear to be as heavy. To perform such an assessment, it is important to first create a baseline based on the current state of affairs. Consider performing threat models, code reviews and penetration tests of 5 to 10 of your most critical applications. The results from these security activities will form the current score if you will. Then go through the training program you have chosen to adopt. Don't worry about new individuals joining the team and other such changes too much since they will in reality reflect changes to your team as they would normally occur and hence don't necessarily represent an anomaly. Once the first run of training is completed – perhaps a year to 18 months down the line – run a secondary series of assessments possibly on the next versions of the same applications. If your training and software security program was truly effective you will hopefully see an improvement in the results from your current score from earlier. If you see a slip downward obviously you have a problem and have to question why there was no improvement or indeed negative improvement if you will. On the other hand even if the improvement does not meet your expectation as well you can question which activities are not having the desired effect and why that might be the case.

Hopefully the longish post above will provide you with something useful as you look to create your own software security training program. In fact the more I think about it the more I realize that this will perhaps help in any type of training program. I would also be very interested to hear from you of any other valuable lessons you might have learnt from your own experiences – what worked and what did not for instance. Good luck J.

{ Comments on this entry are closed }

A few months ago, the software security folks at Microsoft put up a pretty insightful post on security trainings. Over the last few years I have had the opportunity to do a number of security assessments and I must agree that time and again, this fact has been reiterated for me. A lot of organizations and indeed developers make the mistake of thinking software security can be a silver bullet or shiny red easy button if you will. While we would love if that were the sadly, this is not the case. As Edmund Hillary who was the first man to climb Mount Everest, once said "There is no virtue in easy victory". While I don't think software security is that high a mountain to climb but by no means is it a quick fix either. Ultimately it takes a significant amount of time and effort, and really continuous improvement to make sure we are staying at the top of our game and keeping the bad guys out.

Software security is a function of your people, your processes and the technologies involved. Efforts must be made in all of those directions before true "zen" if you will can be achieved. One aspect no doubt about the people is having an informed and trained workforce. This includes your developers but also your testers, your architects, requirements specialists or business analysts, project managers and last but by no means the least your users.

Let me illustrate with an example. All too often when doing application testing we find cross site scripting problems in web applications. Typically, for illustration and documenting evidence we will use something simple like alert('XSS');. The idea being this is proof that the site is vulnerable to cross site scripting. Of course the recommendation goes on to suggest data validation and output filtering and such. Every once in a while when we get the application for retesting – the assumption being all the vulnerabilities (or at least some chosen subset of them) have been fixed – we will see something that shows why just doing security testing (or any one single software security activity) by itself is not enough. We see code that looks like this (in pseudo code):

...

if(input == "<script>alert('XSS');</script>") {

print "Invalid data";

exit;

}

...

As a security professional you cannot help but go "Aargh!". But then when you step back and think you realize we are not backing the security testing with training and awareness, policies and processes.

I really think this is where as an industry we have failed. We like selling silver bullets to our customers – "hey buy my product and its all you need" or "buy my services and you will need nothing else". Same goes for things like security industry conferences – there must be more focus on all aspects of software security not just the I found a new security bug or as has been happening more recently – I found an old bug but I am going to give it a new name ;). Of course I know it makes it a harder sell but I think therein lies the challenge. Just like there are many aspects to staying healthy e.g. genetics, food habits, exercise …, similarly with keeping our applications healthy so to speak.

P.S. In a future post I will discuss how to setup a software security training program that is both effective and helps keep the costs down by using people's time efficiently.

P.P.S. I plan to expand on the general "Software Security – A Holistic View" on this blog with other aspects that should be critical components of a effective and efficient software security program.

{ Comments on this entry are closed }

August 2007

Fortinet FortiGate Devices Reinstated Following Network IPS Tests

Implementing a Security Training Program

Risk Assessment Resources (from the SAGE mailing list)

Why Software Security Must Be Holistic

Black Hat USA 2007 / DEFCON15

Contributors