Measuring Software Security: BSIMM2 and Beyond
The new BSIMM2 provides the data-driven measurements that allow organizations to objectively assess their software security maturity level against their peers and themselves.
You are not a special snowflake.
This is how Dr. Gary McGraw, author of Software Security: Building Security In, Exploiting Online Games: Cheating Massively Distributed Systems, and CTO of the software security company Cigital distills the findings from his Building Security In Maturity Model (BSIMM) and recently launched BSIMM2 projects. Quick translation: the measurement of whether or not the software meets quantifiable security levels is applicable to all software, regardless of what unique vertical, industry, or purpose it was written for. Although each firms process is unique, the measurement of a software security initiative is not.
Measurements are what we use to determine how well were doing and gauge improvement (or decline) over time. Measurements are particularly helpful when assessing the relative effectiveness of different methods. Consider a runner who wants to increase her one-mile speed record. A quick read through of various runners magazines and Websites brings up a variety of methodologies that athletes could employ like: splits and sprints, ultra-long distance training, and high-altitude hill running. Yet the ability to measure the effectiveness stays the same. One method may work better, or worse, for a specific runner; but it is the clocking of the stop-watch that is quantifiable when assessing improvement no matter which method of training the runner chooses.
BSIMM is one way organizations can find a stop-watch for software security assessment. It is a measurement tool that organizations and companies can use to understand how a piece of software compares to relative security levels of software written in-house, by other organizations in the same vertical, and against the entire portfolio of software applications being deployed by organizations that have chosen to take part in the BSIMM project.
Model not methodology
With the plethora of software development methodologies available, its not surprising that some people mistake BSIMM for yet another methodology among the many. But BSIMM is not a software development lifecycle methodology like Cigitals Touchpoints, The Microsoft SDL, IBM Rational or OWASPs Comprehensive, Lightweight Application Security Process (CLASP). BSIMM is a descriptive model that observes data associated with software development and security - and records that information in as objective a manner as possible.
Being objective with the model and reporting observations means staying away from a lot of the judgment calls and Father Knows Best-type prescription found in many best practices documents. That doesnt mean organizations dont need to consider and define their own best practices or recommended procedures as part of their overall software development program. In fact, having internal and external prescriptive guidance is a necessary aspect of software compliance described as one of the 109 activities in the BSIMM2 model itself.
BSIMM complements prescriptive guidance. It is the data-driven, observational stop-watch that can help companies understand if their time (or in software security parlance their risk management and exposure windows), is improving or declining and how they stack up against their peers. The authors and participants of BSIMM observe and report on the characteristics common within organizations that have successfully deployed software initiatives and use those data as the basis for measurability and comparative assessment. To date the BSIMM2 project has aggregated software security measurement data from over 30 software security initiatives with over ten more underway. The over 30 organizations providing the data include: .Adobe, Aon, Bank of America, Capital One, The Depository Trust & Clearing Corporation (DTCC), EMC, Google, Intel, Intuit, Microsoft, Nokia, QUALCOMM, Sallie Mae, Standard Life, SWIFT, Symantec, Telecom Italia, Thomson Reuters, VMware, and Wells Fargo.
If we dive a little deeper into BSIMMs measurements, the authors have divided measurement points regarding 109 activities into 12 core practices:
1. Strategy and Metrics
2. Compliance and Policy
4. Attack Models
5. Security Features and Designs
6. Standards and Requirements
7. Architecture Analysis
8. Code Review
9. Security Testing
10. Penetration Testing
11. Software Environment
12. Vulnerability Mgmt and Change Management
Using these practice areas as measurement aggregators, the authors can graph average maturity levels as shown in the spider graph below:
Source: BSIMM2, May 2010
In addition to graphing (or timing) maturity by practice, BSIMM2 also provides 109 maturity timers, or software measurements, for clock points like: identification gates, known PII obligations, and awareness training.
Back to the special snowflake analogy, one of the most interesting observations by the BSIMM authors is that SDLC milestones and clock points are similar across verticals.
The cross-vertical consistency of the milestones shows that a measurement tool like BSIMM can be effective regardless of organizational size or the software applications purpose. But software security consistency cross-enterprise and purpose may sound shocking to some professionals. Dont the requirements and the approach change based on business unit, regulatory oversight, and purpose? Well, yes and no. Some of the requirements may change, but some, as shown by real BSIMM data, may not.
Consider language itself. Writing a technically oriented article for an online security site is different from writing the script for Stephen Colberts The Word segment. But the basic rules of English grammar apply in both instances, otherwise neither this article nor Colberts jokes would make sense. Or to put it another way, theres isnt a Java for Healthcare, a Java for Financial Services, and a Retail-Only Java. While there is quite a lot of flexibility within the language that allows Java developers to write many disparate software applications, there are some hard and fast rules about writing Java code that dont change no matter who or why the application is being written.
What, then, does BSIMM2 tell us about what successful organizations are doing well in the software security and deployment space? First and foremost, every one of the successful organizations has a software security group (SSG) in place. This group doesnt do double duty, time-slicing between security and development; it is a dedicated team that, for BSIMMers has been measured at 1% of the total development team size. However, the similarities of SSGs only go so far while some SSGs are centralized others are highly distributed. Some work closely on policy and strategy, while others focus more effort on penetration testing and code review. While its easy to get caught up in the differences, the critical point is that having an SSG was a universal check point for successful software security. For more information, McGraw expands on the concept of the SSG and the benefits of having one in the article You Really Need an SSG.
Another key take-away from BSIMM2 are the maturity times (or levels) reported within the 12 practice areas. By reviewing the maturity levels, organizations can assess where they are with their own software security maturity and determine how balanced their approach is relative to others. For example, in the area of Code Review (CR) there are 3 maturity levels which are paraphrased below:
1. Does code review
2. Enforces standards through mandatory automated code review
3. Automated code review with customized (tailored) rules
Each level has sub-activities or milestones that clarify what occurs at BSIMM participant organizations, at each maturity level. When using the BSIMM2 document, keep in mind that activities at one maturity level may still be appropriate at higher levels. In code review level 1, automated and manual review are noted. Manual review does not appear in the higher maturity levels, but is definitely a critical part of the code review process even though the organization has advanced to level 3.
Although each software development has elements of uniqueness, there are also useful commonalities that apply to all software projects. BSIMM2 provides the data-driven measurements of these commonalities that allow organizations to objectively assess (or time) their software security maturity level against their peers and themselves. The model and research also provide useful insights into specific tasks and milestones that organizations can implement to improve overall maturity. Is BSIMM a replacement for everything else thats being done in your SDLC? Absolutely not! But it is an excellent way to measure where you are and if youre improving.