Help! Is there a doctor in the network???

July 23rd, 2010 | Ari Gesher

Cyber security is a hot topic, especially in national security circles. The world has witnessed a number of high-profile incidents in the past two years that have been notable for sharing three very important aspects:

they were targeted attacks, carried out against specific institutions
they were politically motivated, and, inconclusively, appear to be state-sponsored
they used multiple-step, multi-vectors attacks and managed to evade existing security countermeasures

This deviates from the types of attacks that IT-centric approaches have sought to defend networks against. Traditional approaches neutralize the perceived threats against a network with a host of countermeasures: firewalls, malware scanners, automated network vulnerability scanning, patch policies, and intrusion detection systems. The network defenses can learn new tricks when the administrators update the signatures, or, for certain types of data, employ a Bayesian inference strategy (as has been employed to fight spam). This approach does a good job of protecting against untargeted attacks as well as weak targeted attacks.

Full network defense requires human analysts looking at anomalies at a level above the automated countermeasures. Check out the rest of this post to take a look at how human-driven, computer-aided analysis is a game changer in cyber security.

A classic doctrine: the immune system

If you’ve worked in network security, you’re undoubtedly familiar with most (if not all) of the countermeasure systems listed above. The question we don’t often ask is:

What is the defensive doctrine being employed by this security architecture?

Classic network security can be summed up as this philosophy:

Become unattractive as a target-of-opportunity to the legions of script kiddies and somewhat more sophisticated opportunists who search for network defenses they can easily breach.

The goal of the IT-based approach is to be a tougher nut to crack than the network next door. Attackers throw themselves against the defenses, find no exploitable vulnerabilities and move on to the next target-of-opportunity.

As the old joke goes: when a tiger attacks your safari group, you don’t have to run faster than the tiger, you just need to run faster than your friends. We might rewrite that today as: when the ‘l33t h4cker comes a’knocking in your network neighborhood, just make sure that you’re less of a n00b than the next guy and you’ll probably avoid getting pwned too hard.

And so we’re faced with this reality: today’s state-of-the-art network defense is a patchwork system of automated countermeasures designed to stop dumb, undirected, automated attacks. This architecture is not unique to cyber security — it has a close analog in biology.

The human immune system produces antibodies that recognize and defend against specific attacks; it learns over time through successful defense of the organism and, more recently, vaccinations. Millions of bacteria and viruses are foiled every day by immune systems. We can observe this same pattern in cyberspace: hijacked systems tirelessly scour the Internet’s address space, looking for hapless networks ripe for takeover. The Pentagon is probed something like 250,000 times a day.

It would be insanity to connect a network to the modern Internet without security countermeasures in place to defend against these sort of attacks. However, while they are necessary to the task of securing a network, they are certainly not sufficient.

Targeted attacks: slipping past the immune system

The Lifecycle of Hookworm

The countermeasures discussed thus far are essential but not infallible and can be bypassed by things like never-before-seen viruses or carefully crafted penetration attempts. In the biological domain a targeted attack might come in the form of HIV (evolved to slip past the immune defenses), a toxin (non-biological, nothing the immune system can do), or a parasite.

The original crafty adversary

A parasite can survive and thrive inside its host while evading or suppressing the normal immune response to invaders . They take up comfortable residence inside the body of their host, using it as source of food and protection; finally, they use the host as a place to reproduce and spread to other individuals in the host species. Parasites don’t generally kill or gravely harm their hosts (or at least they don’t do it quickly), as it’s in their own self-interest to have the host continue living.

Targeted parasite networks: GhostNet and the Shadow network

Cyber analog? You betcha: Vint Cerf was quoted just last week, “The hackers don’t want to destroy the network. They want to keep it running, so they can keep making money from it.”

The Citizen Lab, a University of Toronto-based non-profit that does in-depth, hands-on, technical research in the cyber security domain had this to say:

Crime and espionage form a dark underworld of cyberspace. Whereas crime is usually the first to seek out new opportunities and methods, espionage usually follows in its wake, borrowing techniques and tradecraft.

That’s in the foreword from their recent report, “Shadows in the Cloud: Investigating Cyber Espionage 2.0“. The report details their experiences tracking down the size, scope, and tradecraft behind a massive cyber-espionage botnet, dubbed GhostNet:

Tracking GhostNet: Investigating a Cyber Espionage Network [their first report on this botnet] was the product of a ten-month investigation and analysis focused on allegations of Chinese cyber espionage against the Tibetan community. The research entailed field-based investigations in India, Europe and North America working directly with affected Tibetan organizations, including the Private Office of the Dalai Lama, the Tibetan Government-in-Exile, and several Tibetan NGOs in Europe and North America. The fieldwork generated extensive data that allowed us to examine Tibetan information security practices, as well as capture evidence of malware that had penetrated Tibetan computer systems. We also engaged in extensive data analysis and technical investigation of web-based interfaces to command and control servers that were used by attackers to send instructions to, and receive data from compromised computers.

The report documented a wide ranging network of compromised computers, including at least 1,295 spread across 103 countries, 30 percent of which we identified and determined to be “high-value” targets, including ministries of foreign affairs, embassies, international organizations, news organizations, and a computer located at NATO headquarters.

These attacks used carefully forged email attacks, known as spearphishing, to entice their targets to unknowingly infect themselves with remote control software. The infections allowed the attackers to exfiltrate data from compromised machines and use them as springboards to attack other systems using similar targeted attacks. Sound familiar?

A New Doctrine: The Doctor

Without an immune system, we’d be dead within hours; our immune system is absolutely necessary but, again, not sufficient to keep us healthy. For those things that the immune system can’t take care of, we use doctors. Doctors are adaptive adversaries to disease: they can run tests, they can talk to the patient, they can apply insights learned from other patients or diseases. Most importantly, a doctor has a much more omniscient view of the patient than the immune system.

Network Security – a 10,000 ft. discipline

Applying this approach to the network enables security responses that can actually counter targeted attacks. A security officer (our network’s “doctor”) starts an investigation with some sort of anomalous event, a unexpected IP address in a log, an alert from intrusion detection system.

Remember that a runny nose or flagged packet is not an illness or a network compromise, it’s a symptom. Symptoms suggest causes, but are only clues. Taken in isolation, they don’t often offer conclusive information on the health of the patient. In fact, finding the root cause of a symptom (a diagnosis) requires the synthesis of multiple sources of data into a complete, coherent picture of the network or patient. This often includes things that you can’t see in the blood or packet stream, like understanding where the patient or user has travelled, what environmental factors might be present in their home, existing allergies, open wireless networks, insecure web apps, drug use, etc.

Node health vs. network health

A node gets an infection on your network? Re-image it, the symptoms go away. In the domain of human medicine, re-imaging of humans when they get sick has not yet gained FDA approval – something doctors have been uttering oaths about since way before the days of Hippocrates.

But it’s not the symptoms we’re after, it’s the root cause. Couple that with how easy it is to treat the symptoms via re-imaging, and security officers are more akin to public health officials, more concerned about the overall health of the network than the health of a single node. This broader concern manifests as an instant list of begged questions about any security anomaly on the network:

How did this happen? Was it a machine (network exploit) or human vector (somebody clicked on something they shouldn’t have)?
What is the extent of this infection? Is it limited to a single node? Why does this small moon appear to have a tractor beam locked on to our ship?
Is this part of a larger attack? What is the true target of this attack? Is this a trap?
Do the tracks lead out of or deeper into my network? Was this an inside job? Did I find an intermediary node in a multi-node penetration?
Who is behind this attack and why do they want in? Can I match this modus operandi with any other known attacks on this or other networks?
How do I prevent this sort of attack in the future? Do I need to deploy new countermeasures, re-architect parts of the network, and/or teach my people to be more careful?

The answer to any of these questions does not appear in a single log file on your network, no more than any single antibody can tell you that the H1N1 flu you’re now infected with came from the grocery clerk who got it from her boyfriend who, in turn, acquired it on his recent trip to Mexico.

The trees don’t know how big the forest is.

Cyber security doctors

A doctor examines a boy looking for hookworm.

The way to find the answers to these questions is to give a skilled, experienced analyst powerful tools to use against all the data about the attack on all of the systems on your network mashed up with relevant data about the messy meatspace that contains the computers, users, and attackers in question.

You need firewall logs, intrusion detection system logs, malware detection logs, badge logs to determine who had physical access to the network, travel records of where you expect your employees to be logging into the VPN from, and a dozen other sources of data that are unique to this network.

The data is not enough — they must to be accessible in a way that enable expedient analysis. In most shops, many of the aforementioned data sources exist, but accessing and cross-referencing them requires a high-level of technical fluency in the storage systems themselves, even for a user that has strong grasp of the story that the data are telling. Some combination of SQL, shell, grep, awk, sed, perl, and Mk I Eyeball are used to suss out answers from the data. It’s a slow, fragile, error-prone game, and the bar is high to even begin playing.

Whenever computers are recording information about the activities of other computers, the data gets big and it gets big fast. For example, grep is a very powerful and flexible tool, but its linear search through data starts to falter as the data size exceed about 10 GB on rotational media.

In order address and solve these sorts of problems, the world needs a platform with the following properties: