Cyber security is a hot topic, especially in national security circles. The world has witnessed a number of high-profile incidents in the past two years that have been notable for sharing three very important aspects:

• they were targeted attacks, carried out against specific institutions
• they were politically motivated, and, inconclusively, appear to be state-sponsored
• they used multiple-step, multi-vectors attacks and managed to evade existing security countermeasures

This deviates from the types of attacks that IT-centric approaches have sought to defend networks against. Traditional approaches neutralize the perceived threats against a network with a host of countermeasures: firewalls, malware scanners, automated network vulnerability scanning, patch policies, and intrusion detection systems. The network defenses can learn new tricks when the administrators update the signatures, or, for certain types of data, employ a Bayesian inference strategy (as has been employed to fight spam). This approach does a good job of protecting against untargeted attacks as well as weak targeted attacks.

Full network defense requires human analysts looking at anomalies at a level above the automated countermeasures. Check out the rest of this post to take a look at how human-driven, computer-aided analysis is a game changer in cyber security.

A classic doctrine: the immune system

If you’ve worked in network security, you’re undoubtedly familiar with most (if not all) of the countermeasure systems listed above. The question we don’t often ask is:

What is the defensive doctrine being employed by this security architecture?

Classic network security can be summed up as this philosophy:

Become unattractive as a target-of-opportunity to the legions of script kiddies and somewhat more sophisticated opportunists who search for network defenses they can easily breach.

The goal of the IT-based approach is to be a tougher nut to crack than the network next door. Attackers throw themselves against the defenses, find no exploitable vulnerabilities and move on to the next target-of-opportunity.

As the old joke goes: when a tiger attacks your safari group, you don’t have to run faster than the tiger, you just need to run faster than your friends. We might rewrite that today as: when the ‘l33t h4cker comes a’knocking in your network neighborhood, just make sure that you’re less of a n00b than the next guy and you’ll probably avoid getting pwned too hard.

And so we’re faced with this reality: today’s state-of-the-art network defense is a patchwork system of automated countermeasures designed to stop dumb, undirected, automated attacks. This architecture is not unique to cyber security — it has a close analog in biology.

The human immune system produces antibodies that recognize and defend against specific attacks; it learns over time through successful defense of the organism and, more recently, vaccinations. Millions of bacteria and viruses are foiled every day by immune systems. We can observe this same pattern in cyberspace: hijacked systems tirelessly scour the Internet’s address space, looking for hapless networks ripe for takeover. The Pentagon is probed something like 250,000 times a day.

It would be insanity to connect a network to the modern Internet without security countermeasures in place to defend against these sort of attacks. However, while they are necessary to the task of securing a network, they are certainly not sufficient.

Targeted attacks: slipping past the immune system

The Lifecycle of Hookworm

The countermeasures discussed thus far are essential but not infallible and can be bypassed by things like never-before-seen viruses or carefully crafted penetration attempts. In the biological domain a targeted attack might come in the form of HIV (evolved to slip past the immune defenses), a toxin (non-biological, nothing the immune system can do), or a parasite.

The original crafty adversary

A parasite can survive and thrive inside its host while evading or suppressing the normal immune response to invaders . They take up comfortable residence inside the body of their host, using it as source of food and protection; finally, they use the host as a place to reproduce and spread to other individuals in the host species. Parasites don’t generally kill or gravely harm their hosts (or at least they don’t do it quickly), as it’s in their own self-interest to have the host continue living.

Targeted parasite networks: GhostNet and the Shadow network

Cyber analog? You betcha: Vint Cerf was quoted just last week, “The hackers don’t want to destroy the network. They want to keep it running, so they can keep making money from it.”

The Citizen Lab, a University of Toronto-based non-profit that does in-depth, hands-on, technical research in the cyber security domain had this to say:

Crime and espionage form a dark underworld of cyberspace. Whereas crime is usually the first to seek out new opportunities and methods, espionage usually follows in its wake, borrowing techniques and tradecraft.

That’s in the foreword from their recent report, “Shadows in the Cloud: Investigating Cyber Espionage 2.0“. The report details their experiences tracking down the size, scope, and tradecraft behind a massive cyber-espionage botnet, dubbed GhostNet:

Tracking GhostNet: Investigating a Cyber Espionage Network [their first report on this botnet] was the product of a ten-month investigation and analysis focused on allegations of Chinese cyber espionage against the Tibetan community. The research entailed field-based investigations in India, Europe and North America working directly with affected Tibetan organizations, including the Private Office of the Dalai Lama, the Tibetan Government-in-Exile, and several Tibetan NGOs in Europe and North America. The fieldwork generated extensive data that allowed us to examine Tibetan information security practices, as well as capture evidence of malware that had penetrated Tibetan computer systems. We also engaged in extensive data analysis and technical investigation of web-based interfaces to command and control servers that were used by attackers to send instructions to, and receive data from compromised computers.

The report documented a wide ranging network of compromised computers, including at least 1,295 spread across 103 countries, 30 percent of which we identified and determined to be “high-value” targets, including ministries of foreign affairs, embassies, international organizations, news organizations, and a computer located at NATO headquarters.

These attacks used carefully forged email attacks, known as spearphishing, to entice their targets to unknowingly infect themselves with remote control software. The infections allowed the attackers to exfiltrate data from compromised machines and use them as springboards to attack other systems using similar targeted attacks. Sound familiar?

A New Doctrine: The Doctor

Without an immune system, we’d be dead within hours; our immune system is absolutely necessary but, again, not sufficient to keep us healthy. For those things that the immune system can’t take care of, we use doctors. Doctors are adaptive adversaries to disease: they can run tests, they can talk to the patient, they can apply insights learned from other patients or diseases. Most importantly, a doctor has a much more omniscient view of the patient than the immune system.

Network Security – a 10,000 ft. discipline

Applying this approach to the network enables security responses that can actually counter targeted attacks. A security officer (our network’s “doctor”) starts an investigation with some sort of anomalous event, a unexpected IP address in a log, an alert from intrusion detection system.

Remember that a runny nose or flagged packet is not an illness or a network compromise, it’s a symptom. Symptoms suggest causes, but are only clues. Taken in isolation, they don’t often offer conclusive information on the health of the patient. In fact, finding the root cause of a symptom (a diagnosis) requires the synthesis of multiple sources of data into a complete, coherent picture of the network or patient. This often includes things that you can’t see in the blood or packet stream, like understanding where the patient or user has travelled, what environmental factors might be present in their home, existing allergies, open wireless networks, insecure web apps, drug use, etc.

Node health vs. network health

A node gets an infection on your network? Re-image it, the symptoms go away. In the domain of human medicine, re-imaging of humans when they get sick has not yet gained FDA approval – something doctors have been uttering oaths about since way before the days of Hippocrates.

But it’s not the symptoms we’re after, it’s the root cause. Couple that with how easy it is to treat the symptoms via re-imaging, and security officers are more akin to public health officials, more concerned about the overall health of the network than the health of a single node. This broader concern manifests as an instant list of begged questions about any security anomaly on the network:

• How did this happen? Was it a machine (network exploit) or human vector (somebody clicked on something they shouldn’t have)?
• What is the extent of this infection? Is it limited to a single node? Why does this small moon appear to have a tractor beam locked on to our ship?
• Is this part of a larger attack? What is the true target of this attack? Is this a trap?
• Do the tracks lead out of or deeper into my network? Was this an inside job? Did I find an intermediary node in a multi-node penetration?
• Who is behind this attack and why do they want in? Can I match this modus operandi with any other known attacks on this or other networks?
• How do I prevent this sort of attack in the future? Do I need to deploy new countermeasures, re-architect parts of the network, and/or teach my people to be more careful?

The answer to any of these questions does not appear in a single log file on your network, no more than any single antibody can tell you that the H1N1 flu you’re now infected with came from the grocery clerk who got it from her boyfriend who, in turn, acquired it on his recent trip to Mexico.

The trees don’t know how big the forest is.

Cyber security doctors

A doctor examines a boy looking for hookworm.

The way to find the answers to these questions is to give a skilled, experienced analyst powerful tools to use against all the data about the attack on all of the systems on your network mashed up with relevant data about the messy meatspace that contains the computers, users, and attackers in question.

You need firewall logs, intrusion detection system logs, malware detection logs, badge logs to determine who had physical access to the network, travel records of where you expect your employees to be logging into the VPN from, and a dozen other sources of data that are unique to this network.

The data is not enough — they must to be accessible in a way that enable expedient analysis. In most shops, many of the aforementioned data sources exist, but accessing and cross-referencing them requires a high-level of technical fluency in the storage systems themselves, even for a user that has strong grasp of the story that the data are telling. Some combination of SQL, shell, grep, awk, sed, perl, and Mk I Eyeball are used to suss out answers from the data. It’s a slow, fragile, error-prone game, and the bar is high to even begin playing.

Whenever computers are recording information about the activities of other computers, the data gets big and it gets big fast. For example, grep is a very powerful and flexible tool, but its linear search through data starts to falter as the data size exceed about 10 GB on rotational media.

In order address and solve these sorts of problems, the world needs a platform with the following properties:

• Has access to all known information about a given incident
• Makes querying and exploring relationships conceptual and interactive
• Scale to handle large data sizes

It probably looks something like this:

Palantir is an intense place to work. There are people here around the clock (since developers set their own schedules) and folks and equipment arriving and leaving all the time. We’re a very focused bunch, trying to change the world as fast as we can by creating a whole new class of tools.

However, we’re not just people who build software; we’re sons and daughters, mothers and fathers and citizens of our community. As we headed into the holidays, Palantir employees decided to give something back: we signed up with a local organization call The Family Giving Tree, a now national charity that started as an MBA project out of San Jose State University.

The Family Giving Tree is unique in that it allows children to request the presents that they want. In this way, rather than putting money into a black box of a charity, you purchase the gift itself and donate that.

The people of Palantir Technologies purchased over 100 gifts, fulfilling the holiday wishes of the children that asked for them as well as cash donations that will buy gifts for at least 40 more.

The pile of presents collected by Palantir employees for The Family Giving Tree.

Happy Holidays, everyone! We’ll be back next year with more technical articles and information about Palantir.

On Oct. 9th, Palantir hosted our quarterly Government Conference in the DC area. The idea was to bring together customers of Palantir Government from across the defense and intelligence community to create a forum for them to:

• Talk candidly about their experiences using Palantir
• Discuss the many different domains they apply our technology against, everything from cyber defense to counter-terrorism to counter-proliferation
• Share experiences deploying our large distributed systems
• Learn about and see what new features and capabilities are in the pipeline for our next quarterly release

Most of the conference time is allocated to our government customers to present information on how they are using Palantir to provide deep mission impact. While this is only the second conference we have held using this open, customer-focused forum, nearly 200 people attended.

The speakers included:

• Lt. Col. Robert “Pic” Piccerillo (ret), from the Counter IED Operations Integration Center (COIC)
• David Arsenault, Assistant Department Head at MITRE
• Mike Jennings, an intelligence analyst from the FBI

In addition to presentations/demonstrations from our customers, there were several presentations of new functionality and demos by us—including demonstrations of our:

• Application platform, which allows customers to easily extend Palantir’s frontend by writing applications and helpers that embed in our platform framework
• New geospatial capabilities, including geosearch, geotagging, and other integrated workflows not seen elsewhere
• PalantirWeb—the new Palantir thin client/web frontend for expanded organizational integration

We also had a very special presentation from Jeff Carr, author of the IntelFusion blog. Jeff launched an open-source intelligence effort to analyze the actors and nature of the cyber war launched against Georgia that paralleled the Russian invasion called Project Grey Goose. Jeff presented some very compelling analytic tradecraft used in and some preliminary results from Project Grey Goose. The iteration 1 report comes out next week!

All in all, the conference went extremely well: it was gratifying for the Palantir team to see some of the innovative uses of the product. When your users are surprising and delighting you with the depth and quality of analysis they’re presenting back to you, you know you’re building and selling the right platform to truly change the way that people relate to data.

We’re witnessing the end of the data age and the first sparks of the age of analysis.

The following is my attempt at describing what we do at a high level without oversimplifying. I hope that after reading this a candidate will ‘get’ what we’re about, or at least understand enough not to apply tiny labels to our expansive vision.

The problem: implementing analysis

At Palantir we specialize in analysis.

Yes, that’s painfully abstract, and I’ll get to it in a second.

In real-world terms, we are building a software platform that enables people to take whatever data is relevant to them and understand it more easily and thoroughly than ever before, using concepts that they already understand. And we are applying this vision, at first, to solving problems in the finance sector and the government intelligence community.

The first important thing to note is that we don’t actually do the analysis ourselves. We don’t devise winning trading strategies and we don’t catch terrorists. We write software that enables other people to pull off these feats. These people, experts in their respective fields, are called analysts.

So what exactly do analysts do? What is analysis?

Analysis is everything necessary to extract insight from information.

Let’s break that down a bit.

Information is easy: It’s data. It lives in a relational database or as files indexed on a hard drive, and you can easily run queries against it. It comes in two forms, structured and unstructured. And there is a lot of it in the modern world – too much, actually, for current tools to make sense of.

Insight is trickier. Insight is something only a person can generate, and understanding this is critical for any organization that wants to do analysis right. Thus the challenge of data analysis is how to bring vast amounts of information into productive contact with human intelligence. In other words, the challenge is how to enable the analyst.

From the analyst’s perspective there are five essential features of an analysis platform:

1. First, and most important, the analyst should be in control. In other words, the primary way of interacting with an analysis tool should be human-driven queries. While automated approaches can complement a human-driven approach, there simply is no substitute for human intelligence. Unless you put a person behind the wheel, the system can never be flexible or creative enough to uncover truly original insight. Artificial Intelligence just isn’t there yet.
2. Ability to summarize large data sets. Some of this is what has traditionally been called data mining: the largely automated approach—using machine learning or other statistical techniques—of processing lots of data at once and extracting nuggets that capture something interesting about the data. Unlike Palantir, traditional approaches have focused almost exclusively on this aspect of analysis.
3. Ability to visualize large data sets. Here the analyst wants interesting and informative ways of viewing data graphically, to make it easier for him to digest. The analyst wants more than just a summary of the data; he wants a nuanced view of what’s going on inside these data sets: What’s the overall shape of the distribution? What are the outliers? What are important structures within the data?
4. Ability to iterate rapidly. This means enabling the analyst to ask a question, get the answer, and then quickly ask either a variant on the initial question or a follow-up question that depends on the answer to the initial question. This rapid, iterative process allows the analyst to quickly test out hypotheses and develop theories about what’s going on in the data, and by extension to discover what’s going on in the world.
5. Ability to collaborate with other analysts. Getting a handle on a terabyte of data, especially when it comprises multiple data types, is definitely more than a one-person job. Any organization that’s serious about understanding the world needs a team of analysts that can work together as more than the sum of its parts. This requires the ability for one analyst to effortlessly share the results of his analysis with his colleagues.

The Palantir approach

That’s what analysis looks like to the analyst, or rather what it should look like in an ideal world. (Current tools fall far short of this vision.) So what do we do at Palantir in order to make analysis this smooth and easy?

You could say that we help summarize large data sets, in the sense that we have to provide the analyst with a rich library of techniques and algorithms. You could also say that we do visualization, in the sense that we have to provide the analyst with a set of interesting and informative ways of visualizing their data. We do both of these things, and we have to be creative and solve hard problems in order to add value in these areas. But we do a lot more than that.

Probably the most central hard problem that we address in trying to enable the analyst is data modeling, the process of figuring out what data types are relevant to a domain, defining what they represent in the world, and deciding how to represent them in the system. At Palantir we make sure our data model (ontology) is both flexible and dynamic, and that it mirrors the concepts people naturally use when reasoning about the domain. This is no small challenge, but we’re already making it a reality. In finance our basic data types include financial instruments, dates, portfolios, indices, and strategies—the same things that financial researchers think about, talk about, and reason with. In the intelligence product our basic data types include people, places, and events (all with associated properties), which is exactly the way we all represent the world in our minds.

Data modeling, data summarization, and data visualization are the core disciplines for approaching large data sets. Human-driven queries, rapid iteration, and collaboration are multipliers, taking the power unlocked by the core disciplines to the next level. When these pieces are brought together in a coherent system, the result is in an analysis platform both very generic and very powerful.

This is what we mean when we say that we’re changing the way people approach data. Welcome to the future of analysis.