By Col. John Burger

This session focused on the need for continuous testing and followed with a discussion on testing approaches, best practices, and lessons learned from the collective group.

The participants provided a great mix of both commercial and government industries. Commercial sectors represented included energy, automotive, retail, financial and automotive. Government entities represented included several military services (U.S. Navy, U.S. Marine Corps and U.S. Army) and a few intelligence agencies. Countries represented included the United States, Australia, Singapore, and South Africa.

Several main points and themes that emerged:

• The effectiveness of most security programs is based on several assumptions that, in most cases, have not been validated. These include, but are not limited to:
  • We assume that technology we purchased works as vendors say it will;
  • We assume the products we purchased have been fully deployed;
  • We assume the products we purchased have been properly configured;
  • We assume our controls continue to work despite changes within the environment;
  • We assume our controls will detect and alert us to adversary activity;
  • We assume our analysts will not miss a legitimate alert;
  • We assume that if our analysts address a legitimate alert, they will properly qualify it, and respond to it.

There was consensus around a need for a comprehensive and realistic testing program to validate these assumptions, justify a security spend (i.e. something to show that our investments actually work) and identify gaps that require additional investments.

• Most current testing venues suffer several weaknesses. That is, traditional pen testing and ad-hoc Red Teaming exercises are narrowly focused, “point-in-time” exercises that do not account for the rapid pace of change of threat capabilities and ever-changing network vulnerabilities. And perhaps most disappointing, objectives and incentives are seldom aligned between Red Teams (Attackers) and Blue Teams (Defenders). Red Teams are incentivized to bypass as many controls as possible to produce a “big scary” report. Blue Teams are incentivized to stop them. There is seldom a collaborative effort to “get better together.” The overarching theme? Significant opportunities to meaningfully improve the organization’s security posture are lost in typical testing methods.
• Defining Purple Teaming. The group discussed that with a slight shift in mindset and approach, an organization can significantly improve its defenses with essentially the same level of effort. Purple Teaming aligns Red and Blue Teams through a collaborative “test- learn- fix-repeat” process of conducting focused testing operations with the sole purpose of improving an organization’s security posture. Best practice purple teaming programs are continuous, comprehensive, collaborative and consist of a series of scoped and focused campaigns that accelerate the pace of improvements in an organization’s security posture. In many ways, the fundamentals of the program are closely aligned with the same principles in the “agile” development methodology.

Purple Teaming Best Practices discussed during the session:

• Continuous. A continuous program ensures:
  • That ever-changing adversary Tactics, Techniques, and Procedures, and network vulnerabilities are accounted for.  In other words, there should be no false “sense of security” based on “last year’s results”.
  • Increased proficiency on both sides. Blue Teams gain confidence that their controls work against realistic adversaries, which in turn requires Red Teams to continuously use more advanced techniques. Thus, the test-learn-fix- repeat cycle accelerates improvements on both sides.
• Comprehensive. Purple Teaming programs should:
  • Assess all phases of the kill chain. This includes all controls that would detect or prevent an attacker from exploiting and gaining a foothold via technical, social, physical, and wireless methods. It also includes testing all those controls that would detect internal reconnaissance or prevent an attacker from escalating privileges and moving laterally.
  • Test all types of controls. This includes testing for intelligence exposures, technical vulnerability and security controls, and detection and monitoring controls. These are all necessary defense in depth controls needed to prevent or disrupt modern adversaries.
    
• Collaboration
  
  
  • Planning: This was viewed by several leaders in the group as an essential component for success. Purple Team objectives all have one thing in common:  to improve the security posture of the organization –  not just testing for the sake of testing.
  • Teaching: “Everything is a finding.” Red Team members must document every attempt they make to bypass a control or compromise a system, which is counter to what most are used to (normally only document their successful compromises and bypasses), but essential to the success of the program. The rationale? To prioritize work efforts, security leaders must equally understand which controls are working and which controls are failing. For all controls, detailed technical documentation in the form of an “attack narrative” enumerates the chronology of the testing campaign. On the Blue side, it’s equally important that the Blue Team inform the group which activities were detected as monitoring controls are just as important as technical security/prevention controls.
  • Notification: This was a spirited topic of conversation. Notifying the Blue Team when the Red Team is starting an exercise was a controversial topic. Some believed that prior notification detracts from the realism of the exercise and will not provide a realistic assessment of Blue Team readiness. Others, including this author, contend that it does not matter, as the success or failure of detection is usually dependent on the level of precision instrumentation, more than the attentiveness of the analyst. The level of detail in the notification can vary from providing a time window only (i.e. within the next 5 days,) or getting more granular with time windows and exploitation vectors. In the end, this is a leadership call.
• Scoped and focused

An effective Purple Team effort should not test all phases of the kill chain in one exercise (i.e. establishing a foothold to exfiltration). To facilitate meaningful improvement while the exercise is fresh in both teams’ minds, it’s better to design shorter exercises with smaller, more manageable objectives. For example, to test for a common technique for lateral movement and pivoting, the purple team objective might simply be: “Implement/improve security controls that prevent and detect the unauthorized use of PSExec.” The Blue Team first creates controls and alerts to identify unauthorized use of PsExec in the enterprise.  The Red Team then attempts to use PsExec, and if successful, watches to see if alerts fire that allow the Blue Team to quickly determine which hosts were accessed or from which hosts they pivoted. The Red Team could also make use of many PsExec alternatives (winexe, msf, impacket, etc.) so the Blue Team could continue to refine and improve their monitoring and alerting for this kill chain phase.

• Low/No Risk

There was a discussion about testing in a “lab” or production environment. Purple Team Operations are best conducted on a live production network. While most risk can be mitigated in the planning phase with appropriate scoping and rules of engagement, there are some basic TTPs that Red Team Operators should follow in order not to disrupt business functions. Most Red Team Operators follow a “do no harm” approach to testing, and further, their requirement for stealth is consistent with this philosophy. Some don’ts include no disruption of services and no exfiltration of live sensitive data. Many use surrogate techniques for these objectives to include planting flags, taking screen shots, leaving marker files, and exfiltrating dummy data to demonstrate level of infiltration or compromise

Additional lessons learned from both military and commercial Purple Teaming programs

• Purple Teaming Exercises are a great way to drive improvements in the production of SIEM content and other detection instrumentation. This might seem self-evident, but absent a continuous purple teaming program, these improvements typically meander or are driven by the latest article that leadership read, or the latest technology that has been implemented.  

• Analysts on the Blue Team got better at recognizing real attacks which increased their confidence. Most analysts have not seen what an attack looks like – the “misuse case” that is evident in the logs, or series of logs. A purple team iterative approach allows them to not only see these artifacts, often they find ways to ensure they are accentuated. 

• Initially, we see significant acceleration of improvements in security posture without much investment. Organizations fix low hanging fruit and indiscipline’s such as reducing attack surface, removing default credentials, properly storing API keys or improving password discipline (length, reuse, storage etc.). However, after this initial 6-9-month period, the fixes may take longer, especially if they require IT/infrastructure participation or additional technology spend. The best organization embrace all improvements, regardless of the level of effort, while the weaker ones don’t like to be “overrun” with findings and opt to slow down the pace of testing. 

• A testing portfolio with a mix of human and automated controls testing is becoming more appealing with new automated platforms on the market. Human-based testing, while more realistic, provides more narrow coverage as Red Teamers, just as real attackers, take path of least resistance. In other words, Red Teamers will aggress and test controls until they find a weakness to exploit and then move to the next phase of the kill chain. Typically, there are large portions of infrastructure that never get tested. To satisfy this shortfall, the use of automated testing platforms has become a compelling compliment to human testing. Thus, a testing portfolio with a mix of human and automated controls testing is becoming more appealing. 

• Notifying the blue team when they are being tested does not substantively changed the outcome. We tend to see more engagement and interest when we notify, and tend to get more out of them. 

• Rotating members between Red and Blue teams improves the overall performance of the program. Rotating a blue team member to the Red team and vice versa significantly improves collaboration and the resulting ideation of how to improve the overall posture of the organization.  Many Red Team members lack an understanding of the capabilities of many of the defensive tools just as many Blue Team members lack an adversary mindset and an understanding of exploitation techniques. The rotation helps each side appreciate the other, but more importantly improves ideation and innovation in security posture improvements.  

• Purple Team campaigns make for great boardroom content, if used correctly. Everyone likes a story!
  Security Leaders must continue to justify their security spend. If they can show the results of realistic threat testing and how that maps to continued maturity and risk reduction, they are better postured to brief:
  • Return on security investments, essentially validating the spend on previous investments
  • Security gaps and where additional investments should be made
  • User Security Shortfalls. When poor user security practices are a key finding in a testing effort, briefing the campaign at the board level is a great way to solicit leadership for funding to implement additional controls or authority to discipline poor practices. The “story” of the testing helps them “see” the impacts.
• Scenario trends we are seeing include:
  • Use of Purple Teaming for Third Party Risk testing of suppliers, vendors, partners, or partners with privileged access.
  • Use of “point in time” Red Team testing to aggress and assess potential new acquisitions.
• Most common findings
  • New employees, recruiters, and human resource personnel are the best targets for social engineering
  • Developers and admins are very in disciplined in credential sensitive information management (establishment, use, and storage). They are always the targets of post exploitation.
  • If organizations do not have an advanced endpoint solution deployed, they will seldom detect malicious behavior. The endpoint is truly the new perimeter
  • Most organizations are not capable of monitoring the use of privileged credentials (local and domain admins). As such, they are not able to detect irregular use when credentials have been compromised. 

Is it the right time for my organization to explore Purple Teaming?

It is insightful to explore the reasons why organizations are not employing continuous Purple Teaming to better understand why they should. The reasons are varied depending on perception of the threat or the maturity of the organization. Here are few common reasons:

“We are not the target of sophisticated adversaries; we have not had a breach in years.”

Perhaps not, but some of the most damaging breaches were targets of opportunity versus precision targeting. Most criminals don’t care where they get credit card or health care data, they just want it. 

“We already know we have a lot of work to do, so adding a Red Team report isn’t going to help.”

More than any other investment, Purple Teaming best accelerates improvement in an organization’s security posture. Further, it improves the security posture from the most relevant perspective – that of an attacker. 

“We work in a highly-regulated industry, and we are compliant. So, it’s not necessary.”

Enhanced security readiness from an attacker’s perspective is a significantly different and more effective approach than readiness from a compliance perspective. While compliance is necessary, it is often insufficient, as typical standards lag the TTPs of modern adversaries. Do you want to be compliant or secure? The answer should be “both.”

We covered a lot of ground in a short period, and deferred some conversations to after the session.