In my most recent post, I identified the direction and state-of-the-art in application security. We all know of the importance of application security in today’s environments. However, finding out where to fit application security policies and programs into an overall security program (or organizational security plan) is as difficult (or more difficult) than integrating mandatory regulations, compliance standards, secure enterprise architectures, and many other risk management activities.
Building a continually improving security program is an important and common topic. For many CISOs and other directors of security programs — this has been their day job since they earned their titles. There still exists huge gaps between IT/Operations, Application Development, and Information Security Management organizations and how they work together. There are gaps in communication between departments, and even within departments. The challenges of finding and retaining talent are not unique only to appsec, as suggested in my last post.
I’ve only spoken about building a security plan once before on this blog, but it’s a popular conversation making the rounds. securitymetrics.org (the blog, mailing-list, Metricon conferences, and book) resurfaced a lot of my interest, as well as the work that Mike Rothman did with the Pragmatic CSO, Michael Santarchangelo with his book and the SecurityCatalyst blog/podcast/forums, and numerous others.
Not all security programs and bloggers have picked up on these resources. Take Creating a Solid Security Program from Accuvant’s new blog called Insight from Kirk Greene. He appears to be familiar with some of the above resources, but I think there is a lot more out there. After you read my comment (which never got “approved”), be sure to check out the new material I’ve been reading on the state-of-art in information security management, especially including the human element.
Comment gone wrong #2
I think what you wrote here is a great example of a vulnerability management program, but not a security program. Even then, it’s actually more operational (like a compliance initiative) because it gives little strategic or tactical advice.
Starting with awareness is probably the worst way to build a vulnerability management or security program. Maybe we just disagree, but I’d like to see some evidence or metrics demonstrating that this technique has any value, if you can point me to the literature.
Capital planning based on current or mock Strategy Maps and Scorecards/Dashboards is really the first step for building a security program. It is often best to first work with risk management (an operational activity) that can feed metrics up to the strategy, although this should be done along with compliance, regulatory requirements, and potential liability factors. Risk assessments, especially ones done with data classifications, can be the tactical metrics to pull into a risk management report. Simple risk assessments can be done using business tools such as 5 Forces, PESTEL, and/or SWOT anlaysis — although in security we have various others including FAIR, FMEA, and PRA.
I also like the concept of drilling down another strategic metric platform via Enterprise Architecture, in particular an Enterprise Architecture Blueprint (such as the one from Gunnar Peterson). Enterprise Architecture can bring metrics down to the operational level with security policy and certification standards. These can be turned into server and application hardening standards at the tactical level.
Finally, asset/inventory management is another strategic activity that can be conducted to build a proper security program. When combined with the risk analysis data, asset management will provide guidance on where to scan & patch, pen-test, and perform exploit development activities at the tactical level. These tactical procedures can then provide more metrics up to risk management, and back again up to more strategic activities.
On second or further iteration, a balanced scorecard can easily be created to include compliance metrics (operational) along with a strategic direction (suggested as a strategy map). The balanced scorecard could then include metrics from incident management, which in turn could feed back into risk management and liability factors. SABSA could be used to build a governance program to keep the capital planning and security program alive and running with the rest of the business. Additional qualitative metrics based on organizational development and organizational behavior could be included in a hybrid platform such as business scorecards very easily, including Six Sigma metrics such as Voice of the Customer, et al. Simple, isn’t it?
Your notion of using Application Security Scanners in a vulnerability management program disturbs me — especially in the way you have suggested it. Maybe you’re not familiar with these tools or how an application assessment is best performed to today’s standards.
First of all, the surface coverage for even the best app scanners is 94%, with many getting less than 1% surface coverage. Even IBM/Rational AppScan was only showing 74% surface coverage using modern link extraction application drivers.
Secondly, the false negative rate of app scanners is approaching 92%, often more. The false positive rate varies between tools, testers and apps, but I’ve seen figures as high as 40%. App scanners must be properly configured and utilized by an expert in order to be effective at all. Even then, black-box app scanners need to be combined with static analysis and manual expert review for a significant majority of applications falling under “most-risky” data classifications such as PII (PCI-DSS, HIPAA, state performance auditing, etc) or financial data (SOX, GLB, et al). Even middle-of-the-road risky data classifications (e.g. proprietary information that has yet to be patented) should probably have more done to them than a simple black-box app scanner.
When I say manual review + static analysis, I really mean it. The automated tools pay for themselves by the amount of time saved — but can never be used alone. Security review tools that implement static analysis techniques, such as Fortify, Ounce, Checkmarx, Parasoft, Grammatech, DevInspect, AppScan DE, Coverity, Klocwork, and SciTools have better false negative rates than black-box scanners, but much worse false positive error rates. FN is usually between 65-85% (the tool FAILS to find vulnerabilities this often); FP is 85-99%, you’ll often see more “vulnerabilities found” than lines of code averaged across apps. This is why manual expert review with full-knowledge remains the best application assessment technique.
I don’t mean to harsh on you too hard, but it does appear that you need to do more homework before making prescriptions for building a security program — let alone a vulnerability management program. You seem to be capable of providing this information accurately (based on your last blog post and the great blogroll you’ve setup so far), so I expect better out of future blog posts.
Aftermath and reasoning
The consulting companies that I work with (and other colleagues, often consultants from other consulting companies that have been on the same or similar engagements with me) have all taken a strong interest in building trusted advisory adjuncts to the “too busy IT manager” or Mascot CISO/CSO. We have to in order to remain relevant and respected. However, I’ve always viewed consultants as “the colostomy bag of a very ill organization”. Fix the organization and the technology advancements (or whatever else is needed) become agile and sustainable.
Rafal Los recently had me on his 31337 Spotlight: Andre Gironda for his Digital Soapbox blog. BTW – Thanks Rafal — hope you and nearly everyone else are having fun in Vegas right now! There are a few links which may have got lost in my nonsensical chatter, so I wanted to specifically point them out. I said:
I like the idea that I can use my hacking skills for good and cause organizational change through discovery of organizational management and behavior. A real “hack” to me is to take a disfunctional organization and turn it into something awesome.
There are very few state-of-the-art resources on organizational theory combined with information security management. Allow me to point you to the few that I’m familiar with and highly recommend. After you check them out, you may find yourself coming to similar or related conclusions as I did with the above comment.
- David Lacey, author of Managing the Human Factor in Information Security: How to Win over Staff and Influence Business Managers
- Krag Brotby, author of Information Security Metrics: A Definitive Guide to Effective Security Monitoring and Measurement and Information Security Governance
- Ron Person, author of Balanced Scorecards & Operational Dashboards With Microsoft Excel — one of many books on Balanced Scorecards, but very recently written and caught my attention.
- Ian Gorrie, blogger of Bad Penny, with posts such as the most recent The Trials of Toorcamp where he kindly provided the slides to his talk entitled “Hacking HR”. He has even posted earlier on information security management (or as he calls it security information management, an interesting but perhaps confusing twist there). My favorite was a presentation he did at ITCi 2007 that is a must read.
- Kevin Nassery, (@knassery, who spoke at LayerOne 2009 on Diplomatic Security Consulting, with video and slides available.
I have at least one more of these “comments gone X” posts, but the next ones should both begin and end on more positive notes. If you have any suggestions of comments you’ve seen from me that you would like to see turned into a blog post, let me know!
Following up their bleeding edge research on bios resident malware at CanSecWest the ultra-smart guys (Alfredo and Sacco) from CoreSecurity have disclosed a significant issue with the laptop recovery software LoJack.
I’ll leave it up to you to figure who’s who [I'm the one with the 'good' accent,] but Craig Balding from 
I wanted to get this post up before I leave for DefCon since it will be hard to have time to blog in Vegas. In a nutshell, I started a new web site called 
If you would like to contribute anything (guides, videos, research, tools, blog on the site) or have feedback let me know by sending me an email (tom[aT]spylogic.net). There are a few other researchers and volunteers working on some really cool stuff for the web site. Far too many ignore the security and privacy issues of social media. We welcome your participation to help make a difference!