For many years, signature-based detection was the hallmark of finding and eliminating security threats in the enterprise. While antivirus and similar products were successful against single-vector attacks, the fact is, we're seeing more and more major security breaches where traditional approaches to security no longer work. To address these new threats, intelligence-driven security is needed.

What makes intelligence-driven security different from what we've mostly had available to us over the past 20 or so years of security products? The biggest difference is that intelligence-driven security can detect things that may not match a specific, known pattern. Instead, they detect behavioral abnormalities, or "what's not normal." There's a big difference between the two. Traditional, signature-based threat detection is based on the premise that something exists about that threat that can be detected (and hopefully blocked, removed, or otherwise mitigated). Here's the problem with that approach: What happens if the threat doesn’t have something tangible that uniquely identifies it? What if, for example, a user went from accessing several documents and files per day to hundreds of files per day? What changed would only be his behavior. There is no malware or something else on his workstation that a signature-based system could detect.

Of course, today we have a large number of security technology vendors who recognize this problem and are attempting to implement some sort of anomaly detection within their products. In most cases, this detection is designed to detect abnormalities in certain types of security data: unusual flow records, for example (but not necessarily in violation of a defined security policy), or perhaps unusual patterns of user access to specific data (again, by detecting activity that is not necessarily specifically against policy). This is a good start, but it's not enough.

What is needed — and where some pioneering vendors may soon find themselves by leveraging Big Data technologies — is a macro-level view across all types of security data, at (or close to) real time, with the ability to take action immediately when these anomalies are detected. This really requires two things:

• All Security Data. Detecting a modern security threat — say, the ubiquitous "advanced persistent threat" (APT) — can require access to a lot of different types of data: event information (traditionally the realm of SIEM and log management tools) culled from across every conceivable network device; operating system, database, and application; system configuration change detection; file integrity monitoring; layer 3/4 traffic information (e.g., flow data); and even content analysis (such as DLP or other layer 5 through layer 7 technologies). Getting all of this data is not easy. It consumes a lot of bandwidth, and more importantly, there aren't (yet) any tools that can truly ingest all these data types at scale, let alone analyze it.
• Close to Real Time. Security organizations have been using business intelligence (BI) and data warehousing tools for years to detect abnormalities in everything from data to user behavior. However, these tools aren't intended for real-time analysis. Detecting an APT is most useful before something bad happens — not afterward. A truly effective intelligence-driven security solution needs to be able to correlate and detect anomalies before data is compromised.

So where does that leave security practitioners, who are trying desperately to protect their data and systems from the bad guys? Well, the bad news is, we're not yet at a point where security and data technology can provide organizations with truly holistic, intelligence-driven security, regardless of context. However, the good news is that with many of the current advances in data analysis, we're looking at a hopeful future in which security goes from a reactive, forensic operation to an adaptive — and possibly even predictive — discipline that greatly reduces the likelihood of advanced threats.