Often, the people asking questions about application security (executives, regulators, auditors, customers) are not application security experts. They may ask the “wrong” questions unintentionally, because they’re trying to use success criteria that they are familiar with from other fields. It’s the responsibility of an application security professional to recommend the “right” questions to ask and to provide data-driven answers that show progress towards meaningful objectives.
For example, a technology executive might ask, “How secure are we?” An application security leader might respond by asking different questions like “Do we have a complete software inventory and has each application been assigned a risk ranking?” and “Has each application received the appropriate security controls for their assigned level of risk?” These are questions which can be answered with data, and provide the organization with information that can be used to take action or support decisions about how to proceed with regards to future investment and strategy.
The RSA Conference 2017 Peer2Peer Session Application Security Metrics provided the opportunity for a passionate group of information security professionals to talk to each other in a small session about the use of metrics in changing the conversation about the value of application security activities and investments.
At the start of the session, I went around the room and asked each attendee to introduce themselves, tell us their role, and share the reason they decided to attend this particular session. The three most common reasons for attending were:
- I’m new to application security and curious about where to start with metrics
- I’ve been in application security for some time and curious about what other organizations are doing about security metrics
- I’ve been doing application security metrics for some time and I want to see if I’m on the right track / get some new ideas about how to refine my approach
Various attendees volunteered useful information sources, including:
- BSIMM research study and strategic planning tool, describing 113 actual application security activities: https://www.bsimm.com/
- Pen Test Metrics report, containing lessons learned from hundreds of pen test programs and 2016 data from a crowdsourced pen test platform: https://resource.cobalt.io/pentest-metrics-booklet
- Security Metrics, A Beginner’s Guide: https://www.amazon.com/Security-Metrics-Beginners-Guide-Caroline/dp/0071744002
The most interesting takeaway from the session was regarding a practical approach to application security metrics. Many application security teams have some kind of defect discovery in place (architecture analysis, code review, pen testing, etc.) and some kind of developer training in place (computer-based or instructor-led). We talked about the importance of tracking the results of defect discover activities to determine an organization’s Top 3 Vulnerability Types and tailoring coding standards, static analysis tools, and developer training accordingly. Over time, as developers are educated about how to fix and prevent their organization’s Top 3 Vulnerability Types, the normalized percentage of instances of each of the Top 3 Vulnerability Types should go down over time. This approach can be easily understood when shown in a visualization and used to demonstrate real value and ROI of application security activities.
Application security professionals should ensure that the activities they are directing are aligned with business objectives and security requirements. They can also position security metrics in G-Q-M (Goal-Question-Metric) format. For example,
- Goal: An organization should conduct a penetration test on every critical web application, mobile application, and API in its software portfolio.
- Question: What percentage of critical applications have been penetration tested in the last 12 months?
- Metric: % = # critical applications tested / total # critical applications in the portfolio.