After reading an excellent article by Ernie Hayden of Verizon that discusses the new Framework for Improving Critical Infrastructure Cybersecurity, a few things popped into my mind.  While I’ve expressed my opinions on this framework and frameworks in general several times, I thought Ernie’s information could definitely be useful to small, unregulated businesses that he notes would benefit the most.  However, he makes reference to the concept of performance-based objectives and suggests that this new framework could be considered performance-based.  He notes that while the distinction between typical compliance requirements and performance-based may seem subtle, the differences in practice are substantial.  I would disagree and argue that it is really a matter of degrees.  But since this issue is just a small part of Ernie’s article, I don’t mean to use this post as a critique of what is a great article.  Instead, I want to delve further into the myth of performance-based cybersecurity guidance as many have proclaimed not only its existence, but its tremendous value.  So, let’s break this down further.

What Does Performance-Based Mean?

The first point of confusion seems to be in defining performance-based to a sufficient level of specificity that one could easily determine whether a particular requirement is performance-based.  One method of divining that answer is to ask whether the requirement actually describes a desired condition or is simply a means people believe will achieve the desired condition.  For example, avoiding accidents on the road is a desired condition that people nearly universally agree with.  By contrast, a speed limit set at a particular level is viewed as a means to help to avoid accidents.  However, speed limits are not the only way to achieve the desired objective of fewer accidents, and some believe that it is not a very effective way to achieve that as evidenced by the existence of the German Autobahn and, for a time, the lack of speed limits on certain Montana highways.  Most would agree, therefore, that speed limits are not performance-based.  They are one of many regulations designed to reduce accidents.  For most of human history, our rules were simple and largely performance-based, boiling down to a simple premise that one could do whatever he wanted as long as he didn’t harm (or perhaps annoy) anyone else.  While the definition of harm could vary, the principle was fairly straightforward.  In many ways, the outgrowth of this principle was the English Common Law that has been adopted by the United States and most former British colonies.  As society became more complicated, we needed more rules to regulate behavior as the eventual outcome of an action may not be readily apparent and waiting for harms to occur is not desirable in many cases.  For example, perceived victimless crimes like prostitution, illegal drug use, and insider trading are illegal because the aggregate result of that behavior on a large scale could lead to very bad consequences such as sex slavery and abuse; an epidemic of addiction and overdoses; and a sort of crony capitalism where the well-connected reap all the rewards, with average citizens not having a chance (i.e., you know, a situation that would be completely different than the current environment :-)). 

Nonetheless, we still find plenty of places where performance-based objectives are alive and well.  In business, chief executive officers and many of their direct reports often draw the majority of their compensation through performance-based measures such as company earnings.  However, before someone suggests that I’m violating my own rules, I would readily acknowledge that higher earnings are not necessarily the end desired condition.  In public companies, shareholders don’t benefit until those earnings translate into either dividends or a higher stock price.  That said, there is nearly unanimous agreement that continuous growth in earnings over time will produce one or both of those phenomena.  The same is true for commissions awarded to salespeople who grow revenue even though it is still a bit removed from the ultimate objectives.  By contrast, other supposed performance-based metrics in business have a much more tangential linkage to the ultimate business objectives.  Instead, they are more analogous to the speed limit in our example above.  They may include things like the number of inventions generated, number of hours worked when time is not directly billed to the customer, the number of absences or frequency someone arrives late to work, or some subjective measure of productivity.  For most large companies, it is next to impossible to draw a direct causal relationship between an individual employee’s behavior and the bottom line.  In most cases, it is the collective contributions of many that produce the desired result.  Consequently, supposed performance-based objectives become approximations at best that are prone to cronyism, personal biases, and just random judgments. 

Performance-Based Cybersecurity Objectives

So, given the somewhat mixed experience we’ve had with performance-based objectives in other fields, one may wonder whether cybersecurity is any different.  To me, cybersecurity is even less mature.  So, do we have performance-based cybersecurity objectives?  To some extent we do.  For example, the absence or existence of successful cybersecurity attacks is certainly a performance-based metric assuming an organization actually knows that it has been attacked.  And it is analogous to the harms we discussed above that the Common Law is so known for.  However, the general conclusion seems to be that waiting for a successful compromise is not ideal, as it often produces insufficient information to improve.  Along the same lines, judging an organization by how quickly it can withstand or recover from an attack is also a worthwhile measure, but perhaps also too late.  Going a bit further, realistic simulations like penetration testing and tabletop exercises can be informative and provide some sense of how well an organization could respond.  However, those can be easily gamed to demonstrate the holes that someone with a compliance bent wants to highlight.  As we move further away from the measurement of an actual cybersecurity compromise, the more tortured and speculative the requirements are.  For example, the framework actually cites one objective (or requirement depending upon your perspective) that states that “malicious code is detected.”  To me, that’s a bit of a stretch when you consider that the objective references the National Institute of Standards and Technology’s (NIST’s) Special Publication 800-53 SI-3 requirement dealing with malware protection …  in other words, that anti-virus software should be running.  But instead of listing that as the objective, which sounds too prescriptive, the objective uses the more fanciful claim that is nearly impossible to prove.  Penetration testing may offer some use cases, but suggesting that such an objective can be met and proven is a rather arrogant statement to make.

Before wrapping up our discussion about the true existence of performance-based objectives for cybersecurity, I want to commend Ernie’s use of Institute of Nuclear Power Operations’ (INPO’s) Performance Objectives and Criteria (PO&C) as an example of useful performance-based objectives.  The underlying metrics are sound.  However, it also demonstrates how much further along we are with safety.  For cybersecurity, our challenge is not so much that we cannot word requirements in a way to make them sound like performance-based objectives.  Instead, it is that the relationship between the supposed objective and the actual desired condition of fewer successful cybersecurity attacks is so lacking in any substantiated causal connection that the value is minimal at best.  Moreover, I would argue that the distinction between something being a compliance requirement and a so-called performance-based objective for cybersecurity is often nothing more than the difference between a vague, high-level requirement and one offering more specifics.  Neither is helpful to those with limited cybersecurity expertise who simply want to avoid being hacked.  I agree that flexibility is important, but eventually the “what” has to become the “how,” and organizations are scrambling to figure out the “how.”  And besides offering a mapping to more detailed guidance, the implication of the framework is that what these organizations need is to hire an expert to help them prioritize which controls to choose and how best to customize them for their environment.  It is advice that could have saved the government a whole lot of time and money if it had simply pointed to existing guidance and called for hiring those who do this sort of thing for a living.  By analogy, when someone is having chest pains, you don’t give that person an entire framework of things to consider, you offer a few pointers and then tell the person to call 911.