I began my computer security career in 1988 when I was given the task of creating the original Network Security Monitor (NSM). By 1991 we had a version that was literally detecting attacks and compromises every single day. I don’t think anyone was aware how pervasive these attacks were before we had the tools to detect and investigate them.

The 1990s became the Golden Age for network-based intrusion detection. New companies were being created. New products were coming to market. And the open source systems Bro, Snort, and Wireshark were started.

Even today network analysis seems to be the dominate method for detecting compromises in an organization.

But the security analyst’s job of today is much harder than it was in the 1990s. The following video (with lots of examples) captures many of the technology and business changes that have made things so much more challenging. I recommend you click through to watch the video on YouTube in a larger window.

In a nutshell:

Ubiquitous encryption has taken away much of the data we used for signals for modeling as well as human investigations of suspicious activity.

DHCP and NAT have made it much harder to identify individual client devices for modeling or device forensics when there is reason to believe one might be compromised.

REST means that the same server port, 443, is being used by virtually all new software.

And a whole raft of technologies and business practices have made it virtually impossible to identify the real destination of outbound connections from our organizations.

Now mix in the fact that the C-suite often doesn’t authorize the budget to instrument the network to analyze East-West traffic within an organization and employees are taking their computers offsite all the time, and much of the organization’s traffic is no longer visible to the detection tools or the human analysts.

Taken together, these changes mean that building effective detection models and being able to quickly investigate suspicious activity to confirm or refute that the traffic is associated with an attack is much harder. today.