Late last year, we were honored to be invited to talk at Reflections|Projections, ACM@UIUC’s annual student-run computing conference. We decided to bring a talk about Horizon, our system for doing aggregate analysis and filtering across very large amounts of data. The video of the talk was posted a few weeks back on the conference website.

Horizon started as research project / technology demonstrator built as part of Palantir’s Hack Week – a periodic innovation sprint that our engineering team uses to build brand new ideas from whole cloth. It was then used by the Center For Public Integrity in their Who’s Behind The Subprime Meltdown report. We produced a short video on the subject, Beyond the Cloud: Project Horizon, released on our analysis blog. Subsequently, it was folded into our product offering, under the name Object Explorer.

In this hour-long talk, two of the engineers that built this technology tell the story of how Horizon came to be, how it works, and show a live demo of doing analysis on hundreds of millions of records in interactive time.

Cyber security is a hot topic, especially in national security circles. The world has witnessed a number of high-profile incidents in the past two years that have been notable for sharing three very important aspects:

• they were targeted attacks, carried out against specific institutions
• they were politically motivated, and, inconclusively, appear to be state-sponsored
• they used multiple-step, multi-vectors attacks and managed to evade existing security countermeasures

This deviates from the types of attacks that IT-centric approaches have sought to defend networks against. Traditional approaches neutralize the perceived threats against a network with a host of countermeasures: firewalls, malware scanners, automated network vulnerability scanning, patch policies, and intrusion detection systems. The network defenses can learn new tricks when the administrators update the signatures, or, for certain types of data, employ a Bayesian inference strategy (as has been employed to fight spam). This approach does a good job of protecting against untargeted attacks as well as weak targeted attacks.

Full network defense requires human analysts looking at anomalies at a level above the automated countermeasures. Check out the rest of this post to take a look at how human-driven, computer-aided analysis is a game changer in cyber security.

A classic doctrine: the immune system

If you’ve worked in network security, you’re undoubtedly familiar with most (if not all) of the countermeasure systems listed above. The question we don’t often ask is:

What is the defensive doctrine being employed by this security architecture?

Classic network security can be summed up as this philosophy:

Become unattractive as a target-of-opportunity to the legions of script kiddies and somewhat more sophisticated opportunists who search for network defenses they can easily breach.

The goal of the IT-based approach is to be a tougher nut to crack than the network next door. Attackers throw themselves against the defenses, find no exploitable vulnerabilities and move on to the next target-of-opportunity.

As the old joke goes: when a tiger attacks your safari group, you don’t have to run faster than the tiger, you just need to run faster than your friends. We might rewrite that today as: when the ‘l33t h4cker comes a’knocking in your network neighborhood, just make sure that you’re less of a n00b than the next guy and you’ll probably avoid getting pwned too hard.

And so we’re faced with this reality: today’s state-of-the-art network defense is a patchwork system of automated countermeasures designed to stop dumb, undirected, automated attacks. This architecture is not unique to cyber security — it has a close analog in biology.

The human immune system produces antibodies that recognize and defend against specific attacks; it learns over time through successful defense of the organism and, more recently, vaccinations. Millions of bacteria and viruses are foiled every day by immune systems. We can observe this same pattern in cyberspace: hijacked systems tirelessly scour the Internet’s address space, looking for hapless networks ripe for takeover. The Pentagon is probed something like 250,000 times a day.

It would be insanity to connect a network to the modern Internet without security countermeasures in place to defend against these sort of attacks. However, while they are necessary to the task of securing a network, they are certainly not sufficient.

Targeted attacks: slipping past the immune system

The Lifecycle of Hookworm

The countermeasures discussed thus far are essential but not infallible and can be bypassed by things like never-before-seen viruses or carefully crafted penetration attempts. In the biological domain a targeted attack might come in the form of HIV (evolved to slip past the immune defenses), a toxin (non-biological, nothing the immune system can do), or a parasite.

The original crafty adversary

A parasite can survive and thrive inside its host while evading or suppressing the normal immune response to invaders . They take up comfortable residence inside the body of their host, using it as source of food and protection; finally, they use the host as a place to reproduce and spread to other individuals in the host species. Parasites don’t generally kill or gravely harm their hosts (or at least they don’t do it quickly), as it’s in their own self-interest to have the host continue living.

Targeted parasite networks: GhostNet and the Shadow network

Cyber analog? You betcha: Vint Cerf was quoted just last week, “The hackers don’t want to destroy the network. They want to keep it running, so they can keep making money from it.”

The Citizen Lab, a University of Toronto-based non-profit that does in-depth, hands-on, technical research in the cyber security domain had this to say:

Crime and espionage form a dark underworld of cyberspace. Whereas crime is usually the first to seek out new opportunities and methods, espionage usually follows in its wake, borrowing techniques and tradecraft.

That’s in the foreword from their recent report, “Shadows in the Cloud: Investigating Cyber Espionage 2.0“. The report details their experiences tracking down the size, scope, and tradecraft behind a massive cyber-espionage botnet, dubbed GhostNet:

Tracking GhostNet: Investigating a Cyber Espionage Network [their first report on this botnet] was the product of a ten-month investigation and analysis focused on allegations of Chinese cyber espionage against the Tibetan community. The research entailed field-based investigations in India, Europe and North America working directly with affected Tibetan organizations, including the Private Office of the Dalai Lama, the Tibetan Government-in-Exile, and several Tibetan NGOs in Europe and North America. The fieldwork generated extensive data that allowed us to examine Tibetan information security practices, as well as capture evidence of malware that had penetrated Tibetan computer systems. We also engaged in extensive data analysis and technical investigation of web-based interfaces to command and control servers that were used by attackers to send instructions to, and receive data from compromised computers.

The report documented a wide ranging network of compromised computers, including at least 1,295 spread across 103 countries, 30 percent of which we identified and determined to be “high-value” targets, including ministries of foreign affairs, embassies, international organizations, news organizations, and a computer located at NATO headquarters.

These attacks used carefully forged email attacks, known as spearphishing, to entice their targets to unknowingly infect themselves with remote control software. The infections allowed the attackers to exfiltrate data from compromised machines and use them as springboards to attack other systems using similar targeted attacks. Sound familiar?

A New Doctrine: The Doctor

Without an immune system, we’d be dead within hours; our immune system is absolutely necessary but, again, not sufficient to keep us healthy. For those things that the immune system can’t take care of, we use doctors. Doctors are adaptive adversaries to disease: they can run tests, they can talk to the patient, they can apply insights learned from other patients or diseases. Most importantly, a doctor has a much more omniscient view of the patient than the immune system.

Network Security – a 10,000 ft. discipline

Applying this approach to the network enables security responses that can actually counter targeted attacks. A security officer (our network’s “doctor”) starts an investigation with some sort of anomalous event, a unexpected IP address in a log, an alert from intrusion detection system.

Remember that a runny nose or flagged packet is not an illness or a network compromise, it’s a symptom. Symptoms suggest causes, but are only clues. Taken in isolation, they don’t often offer conclusive information on the health of the patient. In fact, finding the root cause of a symptom (a diagnosis) requires the synthesis of multiple sources of data into a complete, coherent picture of the network or patient. This often includes things that you can’t see in the blood or packet stream, like understanding where the patient or user has travelled, what environmental factors might be present in their home, existing allergies, open wireless networks, insecure web apps, drug use, etc.

Node health vs. network health

A node gets an infection on your network? Re-image it, the symptoms go away. In the domain of human medicine, re-imaging of humans when they get sick has not yet gained FDA approval – something doctors have been uttering oaths about since way before the days of Hippocrates.

But it’s not the symptoms we’re after, it’s the root cause. Couple that with how easy it is to treat the symptoms via re-imaging, and security officers are more akin to public health officials, more concerned about the overall health of the network than the health of a single node. This broader concern manifests as an instant list of begged questions about any security anomaly on the network:

• How did this happen? Was it a machine (network exploit) or human vector (somebody clicked on something they shouldn’t have)?
• What is the extent of this infection? Is it limited to a single node? Why does this small moon appear to have a tractor beam locked on to our ship?
• Is this part of a larger attack? What is the true target of this attack? Is this a trap?
• Do the tracks lead out of or deeper into my network? Was this an inside job? Did I find an intermediary node in a multi-node penetration?
• Who is behind this attack and why do they want in? Can I match this modus operandi with any other known attacks on this or other networks?
• How do I prevent this sort of attack in the future? Do I need to deploy new countermeasures, re-architect parts of the network, and/or teach my people to be more careful?

The answer to any of these questions does not appear in a single log file on your network, no more than any single antibody can tell you that the H1N1 flu you’re now infected with came from the grocery clerk who got it from her boyfriend who, in turn, acquired it on his recent trip to Mexico.

The trees don’t know how big the forest is.

Cyber security doctors

A doctor examines a boy looking for hookworm.

The way to find the answers to these questions is to give a skilled, experienced analyst powerful tools to use against all the data about the attack on all of the systems on your network mashed up with relevant data about the messy meatspace that contains the computers, users, and attackers in question.

You need firewall logs, intrusion detection system logs, malware detection logs, badge logs to determine who had physical access to the network, travel records of where you expect your employees to be logging into the VPN from, and a dozen other sources of data that are unique to this network.

The data is not enough — they must to be accessible in a way that enable expedient analysis. In most shops, many of the aforementioned data sources exist, but accessing and cross-referencing them requires a high-level of technical fluency in the storage systems themselves, even for a user that has strong grasp of the story that the data are telling. Some combination of SQL, shell, grep, awk, sed, perl, and Mk I Eyeball are used to suss out answers from the data. It’s a slow, fragile, error-prone game, and the bar is high to even begin playing.

Whenever computers are recording information about the activities of other computers, the data gets big and it gets big fast. For example, grep is a very powerful and flexible tool, but its linear search through data starts to falter as the data size exceed about 10 GB on rotational media.

In order address and solve these sorts of problems, the world needs a platform with the following properties:

• Has access to all known information about a given incident
• Makes querying and exploring relationships conceptual and interactive
• Scale to handle large data sizes

It probably looks something like this:

This is a response to Ari’s awesome post on human-computer symbiosis. Ari and I were chatting about the equation he developed and I was wondering if there were some further refinements that are possible… let’s take a look:

We are attempting to understand the total analytic capability for a given task a of a human-computer team. Analytic capability in this case probably means:

(1)

Where A is the answer to the analytic problem in question and t_A is the time needed to arrive at the answer based on the inputs available. In the case of chess, A could be the optimum next move given all previous information and t_A would be how long it takes to decide on this move.

Read on for a look at how this generalizes in human-computer symbiotic systems.

In the case of the human-computer team, we know that a is going to be a function of both the human’s analytical capability h and the computer’s analytical capability c (where both h and c have units of answers/time). In the limit case we know that:

(2)

(3)

Or in plain English, if there is no human present, the total analytic capability is simply the analytic capability of the computer. So the naïve solution would be that:

(4)

(4) clearly meets the limiting cases described in (2) and (3). Kasparov noticed a mixing function where the ability of the human and computer to work together becomes the dominant term — we might call this the mixing capability for the given task or m. Including this phenomenon, the total analytic capability (4) would be re-defined as:

(5)

where m has the property that:

(6)

Thus maintaining the limits expressed in (2) and (3) and adhering to the observation that if there is no human or computer component then there will be no mixing advantage. A naïve solution to this constraint would be simple linear mixing:

(7)

where M (units of time per answer) is the mixing efficiency and will be primarily based on the type of task being solved — some analytical tasks lend themselves to a combined process more than others (for example, multiplying 20 digit numbers does not really benefit from the intuition of a human so the ability of a human and computer to perform this task is merely their additive ability).

What Kasparov noticed is that the mixing was primarily based on the quality of the process rather than the analytical power of either the human or computer separately. This seems to imply that we must somehow account for the fact that the quality of the human-computer interface is responsible for the quality of the mixing. This can be modeled as a unitless friction of interaction f_i that impedes the ability of the human and computer to work together.

Equation (7) can thus be re-written as:

(8)

In this case, the maximum value for the mixing capability is realized when the friction of interaction goes to zero. This mixing capability is the same as the equation Ari developed (less the coefficient which is necessary to maintain consistent units throughout).

We can now re-write our analytic capability in (5) as:

(9)

Below, see a plot of this function over a range of values for h, c and f_i:

As can clearly be seen from this functional plot (note the vertical scale), the effect of interface friction dominates over the other terms whenever both the human and computer can make important contributions to the task at hand. The conclusion can be drawn that the most effective way to solve analytical problems is to minimize the friction of the human-computer interface; or to put it another way: optimal analytical systems are those that are built specifically to maximize the ability of the human to leverage the ability of the computer.

I am certain there is still the possibility for further refinement, for example:

(10)

Visualizing SMS hotspots in days following the earthquake in Palantir.

Screenshot courtesy of Palantir Technologies

[Editor's Note: an edited version of this post first appeared on O'Reilly's Radar blog.]

The prologue was an earthquake of unexpected magnitude and location that left 250,000 dead.

As computer scientists and technologists, we’re used to dealing with large numbers in the abstract. Expressed in human terms, the mind-boggling numbers of 250,000 dead, 300,000 injured and over 1 million people left homeless are hard to comprehend.

Hit the link to read more about how effective data management and analysis is crucial to recovery efforts and see specific examples of data about the situation in Haiti modeled in Palantir Government.

Chapter One: Rescue

There was one glimmer of hope in this sea of tragedy: the world’s reaction. In the early hours and days after the quake, the focus was on pinpointing, triaging, and rescuing those in grave danger. Since those first harrowing hours, the world has made plain its willingness to help the people of Haiti. Supplies of money, food, medicine, fresh water, and volunteers have been pouring into Haiti and fundraising efforts are on-going around the world.

Technology also played an early, crucial role, with Mission 4636, InSTEDD and Ushahidi reacting lighting-fast to create a data collection system that enabled people in trouble to quickly communicate their urgent needs to rescuers and relief workers . If you haven’t already read it, Lukas Biewald’s piece, How crowdsourcing helped Haiti’s relief efforts, is a great look at those first, and most urgent efforts to collect data and synthesize information about the situation on the ground.

Chapters Two Through Many: Recovery

The extent of the devastation in Haiti.

Photo courtesy of Marko Kokic / ICRC / American Red Cross

Unfortunately, even partial recovery in Haiti will take years at the bare minimum. U.S. Vice President Joe Biden stated on 16 January that President Obama “does not view this as a humanitarian mission with a life cycle of a month. This will still be on our radar screen long after it’s off the crawler at CNN. This is going to be a long slog.”

Building the Deep, Big Picture

The recovery from a disaster of this magnitude presents some important tasks in the sphere of information technology: coordination of effort, triaging those most in need, and getting good data into the hands of decision makers and aid workers.

Here’s a partial list of aid, relief, and rescue organizations currently in Haiti, gleaned from Wikipedia:

• An Argentine military field hospital
• The Red Cross/Crescent, in various forms
• The US military
• Multiple UN agencies
• Remnants of the Haitian government
• The French navy
• Sri Lankan relief workers
• At least 2000 rescuers from 43 different groups (along with 161 search dogs)

A wealth of collaborators like this presents some unique challenges around information fusion: unlike business competitors or opposing sides of a war, the different groups want to share as much information as possible to achieve their common goal. A unified organization, like a single national military will have pre-existing methods to model and share their situational awareness. That is not the case in Haiti: this is a collection of groups coming together to form an ad-hoc relief force. Everything from differences in human languages, database schema, collection methodology, and problem domain make most of the datasets seemingly disjoint from the others.

However, each organization has a produced a fairly detailed picture of the parts of Haiti that they are interacting with. Each organization also wants to consume every other’s organization’s detailed knowledge of the situation. To act effectively, they need to integrate that knowledge into a common operating picture that accurately models the situation on the ground yesterday, today, and tomorrow.

Analyzing the Haiti situation using Palantir Government

Our reaction to the earthquake was to try to help in the best way we knew how. We set up a publicly available instance of our Palantir Government product, already loaded with relevant data, for use by aid workers and organizations working in Haiti. Using relevant, open-source data we’ve started modeling a picture of what’s going in Haiti.

Our first cut was to include the locations and names of collapsed buildings, Internally Displaced People (IDP) camps, and Misson 4636 SMS messages, among others. We also added in map layers that let us see what administrative zone any point on the map is located in.

Having mapped the data into this model, users have access to it through a suite of visualization, analysis, querying, and collaboration tools that allow them to get useful answers to practical questions. Here are some examples:

• Which administrative sectors have had the most SMS requests for food in the past 24 hours?
• What collapsed buildings are there that may contained hazardous materials that will require special cleanup?
• Are any IDP camps near enough to these hazmat sites to warrant special precautions or moving the residents?

We’ve created a video showing all the pieces put together into a seamless whole, using live data in our publicly available Haiti instance:

The Next Chapter: Flooding

A view of the water point at the Citee Renault camp in Port-au-Prince, Haitii

Photo courtesy of Joe Lowry / IFRC / American Red Cross

From the Red Cross website:

“We’re racing against the clock with hurricane season just around the corner,” said Jean Pierre Taschereau, a Red Cross disaster expert just back from Haiti. “Getting semi-permanent structures in place as well as trenches for sanitation latrines will be critically important.”

From “Quake-ravaged Haiti faces flooding”:

The UN wants to move 200,000 people out of overcrowded camps like this one. The Haitian government is trying to find land. It’s identified five sites outside of the Haitian capital, but those five sites are about 200 hectares and by the UN’s estimates 600 hectares will be needed to house the people it plans to move safely to have proper drainage when the rainy season finally arrives.

Haiti’s rainy season is notorious for causing flooding. Now, with the infrastructure of the country destroyed, flood season will be more dangerous than usual. Not only are the normal structures that protect people from the waters gone, but they’ve moved out of the ruins of Port-au-Prince to hastily constructed IDP camps, some of which are sitting in the flood plains of Haiti’s waterways.

The essential question facing relief workers: Which of the approximately 2500 IDP camps are most at risk from flooding?

In a place like the United States, an earthquake response and recovery team could engage the services and expertise of the US Geological Survey, which maintains the National Water Information System, a warehouse of detailed information about all things water in this country. No such luck in Haiti, where the closest thing to the USGS is the Centre National de l’Information Géo-Spatiale. A quick look at their website shows that they didn’t really make it through the earthquake. (In the video, we feature a picture of what’s left of their facility — it’s not pretty).

Since we’re starting from square one we put together data from the Army Geospatial Center, the UN, NOAA, Haiti-based NGOs, a number of academic papers, and even geo-tagged photos from Flickr. The time it took to integrate this data? About six hours. Time it took to do the analysis? About seven minutes. Amount of that work that is reusable? All of it.

Check out this video for a walk-through of the analysis:

The best way to improve this analysis will be to add more detailed information about flooding, gathered from the field. We’re looking into getting new conduits of information into the Haiti instance to make this a reality as the rains really pick up.

A Call To Action

If you’d like to help us, we’re accepting new data sources, analyses, and contact with relief organizations.

Volunteers, supplies, and goodwill are only the raw ingredients to recovery; it’s the efficient and timely application of those resources to Haiti’s most pressing problems that will change lives and make recovery a reality instead of just a good intention.

As we build our platforms and applications following a human-computer symbiosis approach, we keep an ear to the ground for interesting examples that illuminate new techniques or validate our approach in some empirical way.

One of the areas that we’re interested is in the overall friction of analysis systems. The systems that we build are built on commodity hardware — we’re not building faster computers and yet we can deliver orders-of-magnitude better performance on analysis tasks than existing solutions. How do we do this? By building software in such a way that it reduces the friction experienced at the boundaries between the computing power, the analyst, and the source data.

Chess as analysis laboratory

Chess is, at its heart, a predictive venture. The player attempts to anticipate their opponent’s moves, planning their own moves accordingly, with the straightforward goal of finding a sequence of piece moves that force checkmate.

This game is, in its ideal form, analysis. (The moves made are the logical extension of the analysis.) The data are clean, the problem is well-defined and everyone plays by the same rules. There are even well-defined metrics for ranking chess players by skill — a better chess player is a better chess-game analyst.

In the realm of evaluation of analysis systems, this is as about as good as it gets in terms of designing controlled experiments to study the relative strengths of different analysis systems.

Garry Kasparov, widely considered to be the greatest chess player of all time, recently wrote a review of Diego Rasskin Gutman’s book, Chess Metaphors: Artificial Intelligence and the Human Mind.

The review is excellent and covers a lot of ground. However, one particular anecdote stood out as a very interesting example of human-computer symbiosis (emphasis added):

In 2005, the online chess-playing site Playchess.com hosted what it called a “freestyle” chess tournament in which anyone could compete in teams with other players or computers. Normally, “anti-cheating” algorithms are employed by online sites to prevent, or at least discourage, players from cheating with computer assistance. (I wonder if these detection algorithms, which employ diagnostic analysis of moves and calculate probabilities, are any less “intelligent” than the playing programs they detect.)

Lured by the substantial prize money, several groups of strong grandmasters working with several computers at the same time entered the competition. At first, the results seemed predictable. The teams of human plus machine dominated even the strongest computers. The chess machine Hydra, which is a chess-specific supercomputer like Deep Blue, was no match for a strong human player using a relatively weak laptop. Human strategic guidance combined with the tactical acuity of a computer was overwhelming.

The surprise came at the conclusion of the event. The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.

After the jump, we look at this finding in a more generalized way and map it onto the Palantir approach.

The cyborg Grandmaster: a fearsome opponent

The tournament Kasparov recalls was a showcase of chess talent, human-computer symbiosis, and raw computing power. Among those entered in the tournament were a purpose-made chess machine (similar to Deep Blue) named Hydra and a team of Grandmasters assisted by computer programs.

One losing participant had this to say about the computer-aided Grandmasters:

Secondly, I have learned that a Grandmaster armed with a chess engine is a killer combination against a plain Engine. Engines see everything via brute force, Grandmasters use their intuition and are able to see “obvious” moves at once. So the two of them together are a mighty force.

This is just as Licklider predicted 50 years ago — quoting Man-Computer Symbiosis (if I could put it better, I would):

Men will set the goals and supply the motivations, of course, at least in the early years. They will formulate hypotheses. They will ask questions… In general, they will make approximate and fallible, but leading, contributions, and they will define criteria and serve as evaluators, judging the contributions of the equipment and guiding the general line of thought.

…

In addition, the computer will serve as a statistical-inference, decision-theory, or game-theory machine to make elementary evaluations of suggested courses of action whenever there is enough basis to support a formal statistical analysis. Finally, it will do as much diagnosis, pattern-matching, and relevance-recognizing as it profitably can, but it will accept a clearly secondary status in those areas.

So in classic intelligence amplification fashion, having computer programs that can quickly evaluate a move’s likelihood of success can amplify the power of the Grandmaster.

While empirically true, it does beg the question: how much does it amplify the power of the Grandmaster?

One approximation might be product as a simple linear amplification. Let’s imagine a function, a(h,c), in which the analytic power (a) is the product of power of the human (h) and the computing power of the chess engine being used (c). This gives us the equation:

One term to dominate them all: friction-of-interface

Does this simple approximation hold up? It does not. The team that won the PAL/CSS Freestyle Tournament in 2005 was composed of two amateur chess players that were able to best a computer-assisted Grandmaster.

How did they accomplish this feat? It was not through superior compute power. Instead, they did so by more effectively feeding insights to their three chess engines. They played so well that a large number of people actually assumed that it was actually Kasparov himself playing:

Many speculated that it might be Garry Kasparov, who was the initiator of this kind of computer assisted chess matches. When we asked him Kasparov confirmed that was not the case. But he reminded us that it doesn’t really matter. The guiding principle of Freestyle Chess: anything is allowed. “Even if they were assisted by the devil, that would probably be covered by the rules,” he joked. “Only the moves they played count.”

What does this mean for our simple equation? Well, it looks it’s missing a term, one we’ll call f, that describes the efficiency or friction of the interface between human and computer.

Quoting Kasparov again:

Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.

The implication being that the equation actually looks like this:

So as the friction of the interface goes to zero, the full amplification of the chess engine is brought to bear. A quick gut-check in the opposite direction agrees: one can imagine the world’s most powerful chess engine with the world’s worst interface; spending the time it would take to express commands to this theoretically awful program would actually be worse than playing without it.

Palantir: a low-friction interface to data

As analysis problems go, chess resembles a spherical cow in a vacuum. Analysis problems in the real world are orders of magnitude messier.

Let’s reframe the terms of our equation above into a more general approach to analysis:

• H – this is power of the analyst. In chess, the value of this terms varies widely between players; in designing real-world data analysis systems, this is more or less a constant (which is why h above becomes H below). Of course there are differing levels of expertise, training, and raw ability amongst the user population, but when we design systems, it’s with the average case in mind.
• c – computing power. How fast are the machines? How well do they scale? How efficiently do they perform the data tasks at hand? Palantir spends significant engineering effort on optimizing the c term, but most of the growth in this term comes from the layers we depend on, built by companies like Intel, Sun, Oracle, etc.
• f – friction. How easy is it to bring c to bear on the problem? Note that when we talk about friction of interface, this is not exclusively referring to user interface. More generally, friction can be present at any interface between two systems: data-software, software-software, human-software, etc. The f that we consider in this simple model is sum total system friction.

So our final formulation is just in terms of c and f (holding H as a constant):

When we discuss friction in real-world analysis systems, the friction actually exists at multiple levels:

1. Creating an analysis model that will enable answering the questions that need to be explored
2. Integrating the data into a single coherent view of the problem
3. Enabling analysis tools to efficiently query and load the data
4. Exposing APIs that allow developers to develop custom solutions quickly and efficiently for modeling and analysis tasks not covered by general tools
5. User interface that makes the tools easy, enjoyable, and quick to use

Minimizing f: Haiti Flooding Predictions

If this is starting to sound very similar to Palantir’s marketing information, this is no accident. While some of our backend engineers are concerned with things like scaling and speed-of-querying, the overall innovation that we’re bringing to the field is not simply about faster data processing systems (even if they are) but reducing the friction at every interface inside a complex human-computer symbiotic system.

You want an example that ties it all together? It starts with a simple question: which of the many displaced-person camps in Haiti are most at risk for flooding as the rainy season approaches? Easy to ask, but not so simple to answer.

The original introduction to this video:

As we enter the beginning of the rainy season in Haiti, one of the biggest problems facing relief organizations today is the spectre of flooding and mudslides destroying Internally Displaced Persons (IDP) Camps. In this video, we integrate data from many sources to determine high risk aid locations.

The data integration for this video took about six hours, using sources of data that had never before been fused. The analysis itself takes a few minutes and quickly comes to an actionable answer to the original question.

If you’ve taken the time to peruse the Palantir Government analysis blog, you’ve seen numerous examples of Palantir Government as applied to interesting problems; they are recorded screen captures of our analysis desktop client. It’s a showcase of useful, meaningful, and compelling visual and semantic tools being used to do analysis on a wide range of datasets.

What enabled this analysis? Aside from the obvious hard work of our UI and analysis tools teams, it’s the flexibility and power of the Palantir data platform. More than just a scalable datastore, the Palantir data platforms act as robust and clean abstractions on top of data.

One of the early architecture decisions that we made when building both Palantir Government and Palantir Finance was to separate the respective data platforms from the end-user applications used to actually perform analysis. More than just following the client-server model, this separation made the data servers in both products into generic intelligence infrastructure for analytic problems, with our clients acting as analysis applications on top of those platforms.

And so, one way to look at our data platform is as an operating system for analytic applications. In this post we’ll explore the history of operating systems, understand why they’re so important and see how the Palantir data servers deliver the same potential to revolutionize the writing of analysis software that operating systems did to the writing of general programs for computers.

The OS: abstraction that begat a paradigm

In the early days of computing, when a programmer wanted to write a program, they had to understand the inner workings of the machine. Writing a program required understanding things like the bus interface of a specific model of hard drive when all that was needed by the program was the clean abstraction of a filesystem. The upshot of this is that much of the time and effort put into a given task was spent writing code to interface with the “physical” minutiae of the machine rather than implementing the solution to the problem that the programmer was trying to solve with their software.

This pattern was observed by J.R. Licklider and noted in his influential paper, Man-Computer Symbiosis (emphasis added):

About 85 per cent of my “thinking” time was spent getting into a position to think, to make a decision, to learn something I needed to know. Much more time went into finding or obtaining information than into digesting it. Hours went into the plotting of graphs, and other hours into instructing an assistant how to plot. When the graphs were finished, the relations were obvious at once, but the plotting had to be done in order to make them so.
…
Throughout the period I examined, in short, my “thinking” time was devoted mainly to activities that were essentially clerical or mechanical: searching, calculating, plotting, transforming, determining the logical or dynamic consequences of a set of assumptions or hypotheses, preparing the way for a decision or an insight. Moreover, my choices of what to attempt and what not to attempt were determined to an embarrassingly great extent by considerations of clerical feasibility, not intellectual capability.

This description of his time as a researcher was echoed in the work of the early programmers: they spent much of their programming time re-inventing the wheel and writing routines that were doing essentially clerical or mechanistic work related to the functioning of the hardware rather the core functions of their programs.

The operating system changed all that: suddenly (and by that I mean: with years of hard work, research, and incremental change) that noisy, inconsistent pile of hardware was transformed into a set of clean abstractions. The programmer was finally freed to spend time and energy on the problem they were really trying to solve.

And so we come to the modern era: dealing with the messy details of hardware has been replaced by the clean and robust abstraction of the operating system.

Three important properties of modern operating systems:

• Hard boundaries between OS functions and process functions – in modern operating systems, this is usually accomplished with system calls. The process places the inputs to the system call in a known location and then asks the OS to perform some operation, like writing to a file or making a network connection. The OS may or may not perform the function, based on things like permissions, availability of resources, etc.

  The most important feature here is that the process never has direct access to the true resources of the machine — instead, all access to the machine’s resources are brokered by the OS.

• Extensions of the abstraction in every direction – An OS like Linux is really, at its core, a kernel that does process scheduling and lifecycle, manages memory, and services system calls. Everything else is handled by some sort of driver. A driver might also be called, more generically, a plugin or extension. Drivers exist for everything from block devices (like hard drives), network cards, and filesystems to input devices and displays.
• Designed as a general purpose framework – the operating system doesn’t actually do any computing; rather, it’s a set of services to facilitate processes using the resources of the computer. To that end, they’re not designed with a specific process in mind, but rather to serve a large class of programs, each designed and written to accomplish a different task using a similar set of resources.

Analysis: the modern computing task

The first computer, ENIAC, was conceived to do calculation of ballistics tables for artillery pieces — it was a glorified calculator. Lacking anything even resembling an operating system, it would just run its program. Its compiler? A group of six women who would configure the machine by hand with the program logic. The input for its first test run, a calculation related to the hydrogen bomb project, was approximately one million punch cards.

Times have changed: 40 or so years of the unrelenting march of Moore’s Law in computing power has given us something like an eight order of magnitude increase in the amount of computing power available per unit cost. Coupled with similar, more recent gains in storage capacity and network bandwidth, this has produced a world awash in data, crying out for analysis.

So the situation today is that we now expect to bring these considerable computing resources to bear on larger, more complex problems in the world. I’m talking about things like the spread of food-borne illnesses, understanding the connection between genes and protein expression, understanding terrorist networks, finding botnets in network traffic logs, and exploring influence networks in government.

These problems, while spanning a widely disparate areas of analysis, share some common traits:

The data is spread out

They are described by multiple data sources. Just to make things more interesting: the data sources don’t agree on their native representations of the real-world data. And finally, the real-world objects that the data are describing are actually described in multiple data sources, with no single source giving a complete and accurate representation.

The data schema are not human-conceptual

Rather than representing the data in some schema that maps easily into how the experts on a given problem think about said problem, the data stores in question tend to model data in whatever way was convenient for the creators of that particular data store. Put another way: people don’t think in tables, rows, columns, and XML snippets. These first-class data storage elements don’t usually map to real-world objects.

The data is sensitive

Whether it’s patient information, mortgage data, a law enforcement investigation, or sensitive foreign intelligence, there is often the need for foolproof access controls on the data.

Palantir: an operating system-class abstraction for analysis

A Palantir data server provides a similar class of services that an operating system does but focused on the specific needs of analytic tasks. Here I’ll focus on the model used by Palantir Government; Palantir Finance uses a similar but significantly different approach to delivering these services.

As you might imagine, however, they both start at a somewhat higher level than punch cards.

It starts with an ontology

The Palantir approach to analysis begins with a task-specific ontology: essentially, a human-conceptual description of the real-world problem that’s being analyzed.

It’s roughly composed of three pieces:

• A hierarchical type system of the real-world objects that human experts use to think about this problem. We call these PTObjects, short for “Palantir Objects”.
• A type system of properties that will contain the data describing these PTObjects. PTObjects are essentially typed containers for properties. This is where most of the detail of the ontology lies.
• A type system of possible relationships between different types of PTObjects.

Within the ontology, there are numerous extension points that allow the customization of how data is imported, retrieved, and displayed (following the principle of extending the abstraction in all directions).

The data server takes the ontology as input and is agnostic to its content. This is where the principle of building a general purpose framework comes into play.

The data sources are mapped into the ontology

This part of the Palantir data server is a pattern that is very similar to an operating system’s notion of block device drivers. The difference? Instead of low-level storage systems like hard drives, we’re dealing with complex databases describing the problem at hand.

In an operating system, every block device can read and write blocks of data. In the Palantir data server, everything becomes a source of PTObjects.

Our data importer plugins, by analogy, fulfills the same role as a block device driver:
we build glue code to map the data source’s schema into the ontology and the connectors to surface the data itself wrapped up in PTObjects.

The data are composed into real-world objects.

Part of this mapping is composing real-world objects into composite PTObjects by resolving PTObjects together.

The operation of resolving is pretty straightforward: we basically union the properties of the two PTObjects into a new PTObject. The end result is a single PTObject that completely represents all the data about something in the real-world from all the available data sources.

As we do this composition, we keep track of where each property came from, down to the record level, in each of its original sources. (Note that most composed PTObjects will usually have at least one property that comes from two sources). By preserving the original identity of every atom of data, it allows us to later decompose these PTObjects into their constituent parts or, more importantly, censor a client’s view based what permissions they have for each of the original data sources.

This a fundamental operation in our system that doesn’t have an exact analog in operating systems — it’s sort of similar to taking multiple filesystems and mounting them inside a virtual filesystem tree, like Unix does. However, if each data source is like a filesystem, what we’re doing is essentially composing individual files from their fragments stored on multiple block devices.

Another analogy: at a level below the block device in the OS, this is also sort of similar to what a RAID0 device does, the difference being that our composition is based on the contents of the data itself rather than some previously applied, content-agnostic, decomposition function. The other difference being motivation: a RAID0 does it for performance, while Palantir is composing data to make it correspond to the real-world objects it represents.

The server exposes Palantir “system calls”

The interface that the Palantir data server exposes can be boiled down to two essential operations:

• The client can download copies of PTObjects from the server. It may request them by id or perform some sort of search/query to specify a set of PTObjects. This is roughly analogous to the open() and read() system calls on Unix.

  Note that each client only sees the subset of properties for a given PTObject that it is authenticated for. This censorship of full PTObjects into projected slices is something done by the server on every load of PTObjects.

• The client can send new or updated PTObjects to the data server for storage. This is roughly analogous to the write() system call in Unix. It, of course, entails a check as to whether the given client has permission to write to the given PTObject.

The server’s responsibility is the same as the operating system: only let the client do what it has been granted permission to do. In an operating system, the OS uses hardware features like protected mode to keep lower-privileged processes from accessing machine resources. Palantir uses network calls to achieve the same separation, by placing the client and server on different logical machines. The effect is the same: the client basically requests (rather than commands) that certain operations are performed by the server. The server uses its own rules to decide if the access or change is allowed and responds accordingly. And so the principle of hard boundaries is implemented.

The clients do the analysis

When an operating system yields to a process, that’s the time when the true processing begins. By the same token, in Palantir, it’s not until a client connects and starts searching, visualizing, and manipulating PTObjects that analysis actually starts taking place (even if the server is doing a lot of the heavy lifting).

The wide open future

So why is this exciting? I’m glad you asked!

It’s about taking analysis to the next level.

Let’s say you’re someone who wants to write an analytic task. Let me ask you a series of rhetorical questions:

• Do you want to start with three disparate sources of data or with the data already mapped into a Palantir data server?
• Which one is a better use of your time as a programmer?
• Which one allows you to not repeat mistakes that other programmers have already made and fixed?
• Which one is more like writing a program than an operating system?

Operating systems took us to a new level of expressiveness when it came to writing computing processes to run on computing hardware. It inverted that 85/15 ratio that Licklider talked about so that programmers spent more time writing the code that did the thing they were trying to create and less time mucking around with hardware.

More programmer time == better analytic tasks.

It’s about making machine learning easier.

Now consider machine learning as a field. Pretty much every machine learning task could benefit from starting with its data in something that looks like a Palantir data server. I’ve taken an informal survey of machine learning researchers and they agree: the 85/15 ratio still holds for machine learning.

Simply put: most of the time and effort in machine learning is spent getting the data into a form that you can actually apply an algorithm to! Now imagine if the starting point for that was a Palantir data server — now the machine learning implementer has a world of expressiveness open to them and time and energy are spent on the task at hand instead of the overhead of messing with the data.

Now, we don’t think that we’re building Skynet. Quite the contrary: we believe that platforms like the one we’ve built will allow machine learning techniques to be put in the hands of experts to augment their ability to look at the world come to conclusions about complex real-world problems by asking questions of the data we’ve collected. It’s about Intelligence Augmentation, which can use machine learning techniques and algorithms to build better tools, not creating Strong AI.

It’s about creating new markets

Let’s go back to the well of operating systems and look back at the history of MS-DOS: the first “killer” application on MS-DOS was VisiCalc (that screenshot at the top of this post), a text-based spreadsheet. As you know, VisiCalc was not the end of the story but just the introduction. MS-DOS, evolved into Windows, allowed application writers an (arguably) clean abstraction on top of commodity hardware in order to build the applications that users actually wanted. Today, we have things like web browsers, multimedia authoring software, virtual machines, and IDEs built on top of what is, essentially, the same set of abstractions that VisiCalc was built on.

However, the most important thing to note is that VisiCalc is credited with creating the market for commercial operating systems — businesses needed VisiCalc so they paid Microsoft for MS-DOS (and IBM for a PC). Without VisiCalc, there was no market for MS-DOS (most people, unsurprisingly, didn’t want to buy a BASIC interpreter).

We’re in the business of selling software and we agree with our customers: the Palantir approach has tremendous value. We’ve just started tapping the potential of this market. Think about what Oracle looked like in 1979, think what Microsoft looked like in 1980 — that’s Palantir in 2009.

It’s about the start of the analysis age

It can be argued that the operating system is the innovation that ushered in the “information age“. Without the operating system, there is no software explosion, which allows computing technology to actually be used on data in the world.

We think that we’re on the cusp of the analysis age, as imagined by Vernor Vinge in Rainbow’s End. It was something foreseen by Licklider in 1960, albeit with a timeline that was off by at least a few decades:

“…it seems worthwhile to avoid argument with (other) enthusiasts for artificial intelligence by conceding dominance in the distant future of cerebration to machines alone. There will nevertheless be a fairly long interim during which the main intellectual advances will be made by men and computers working together in intimate association. A multidisciplinary study group, examining future research and development problems of the Air Force, estimated that it would be 1980 before developments in artificial intelligence make it possible for machines alone to do much thinking or problem solving of military significance. That would leave, say, five years to develop man-computer symbiosis and 15 years to use it. The 15 may be 10 or 500, but those years should be intellectually the most creative and exciting in the history of mankind.”

It’s a golden age of analysis and we’re just getting started: we’ve got a lot of work to do, so if this sort of thing excites you, please come and join us.

[A number of weeks ago, we published a post on the search technology used by Palantir. That post covered raising the memory efficiency of a couple of operations. This is part two of that series.]

The most familiar use of search engines is to index documents made available on the Internet via the hypertext transfer protocol. Forgotten names like AltaVista, names not-yet-really-learned like Bing, and, of course, Google come to mind.

This one, massive use case has a couple of properties that I’d like to highlight:

• Asynchronous indexing and querying – web search engines tend to use crawlers and indexers to build up an index of the web. After each crawl is finished, the new index is brought online for use by the query engine.
• Lack of access controls – all the data in the index is available to any query. In fact, most queries are (from the standpoint of the index) completely anonymous.

Palantir: not a web search engine

Search technology is just one part of what makes up a Palantir system. For us, it’s a way to quickly retrieve Palantir objects in a Palantir system, it’s not the whole of the application.

I’d like to highlight a couple of differences from the web search engine case. A Palantir system needs the following properties:

• Realtime indexing and querying – we need information to be available immediately as it changes in the system.
• Leak-proof access controls – we need the search engine to help us make sure that we don’t have information leaking across access control boundaries.

Hit the link to read more about these topics.

Realtime indexing

The Palantir platforms implement realtime indexing: as soon as an analyst changes an object in the system, it needs to be available to query. This could be a change to data in the object or a change to the security tags on the object.

From a programming perspective, this is pretty straightforward: a Palantir transaction will not commit until the search engine is finished indexing the new data.

From a search engine operational perspective, this induces some challenges. Asynchronous indexing allows the search engines to bring online a highly optimized static form of the index. Contrast that with realtime indexing, where every cycle spent optimizing the index is removing cycles from serving other queries and there is likely a human waiting for the optimizing process to finish.

When using the static index, a query only accesses one, optimized index file which then points to the documents containing the results. However, as changes and additions are indexed into the system, there is a lot of overhead to merging them into the master index.

Instead of merging and optimizing on every change, Lucene can keep around a number of smaller indexes that hold all the fresh entries. These are fixed-size append-only segments that are much cheaper to write to than the optimized and merged form of the index. So basically, these ‘dynamic’ indexes are linear lists of single-document indexes. When the search engine goes to run a query it has to follow this simple (yet expensive) algorithm:

• Query the static, merged index, accumulating results. (this part is reasonably fast)
• For each of the dynamic indexes:
  • Open the file, incurring IO overhead.
  • Query each single-document index and look for additional records or newer records that supersede one of the existing found results.

You can see how the overhead of this can quickly get pretty large as the number of dynamic indexes grows: it grows linearly with number of new indexed records. Compare that with the optimized index, which should be close to constant time for any given query.

To get around this, the indexer will only allow a certain number of these dynamic indexes to accumulate before it kicks off a background merge job. During the merge job, we take a noticeable performance hit, but by batching up the merge run we amortize the overhead away for an overall performance win. This hybrid mode didn’t require us to write any new code, but just to tune Lucene to give us the performance profile we wanted.

Preventing Information Leaks

The Palantir data platform has a fairly sophisticated security model baked in (see The White Videos for a more in-depth look at the security model). One of the features that we have implemented is the ability to show a narrower view of an object based the user’s permissions: the user only sees the slice of the data that they have been granted access to. Part of the complexity in implementing this was that we can’t even hint that the other, hidden data exists at all.

Search engines ranks their results by relevance, showing the matches to the query that it believes to be most relevant first. One common way to make these relevance calculations is by comparing the length of the search term or phrase to the length of the term that it matched. Consider the search term ‘king’: it will match the following phrases:

• “I’m the king of the world!”
• “King salmon are often found in the Pacific Northwest and are also known as Chinook salmon.”
• “Yes, my king.”

Using a length-computed relevance, the phrase, “Yes, my king.” is the most relevant.

Getting back to the Palantir object model: for each distinct set of permissions that an object has, we compute a different object label based on the properties that are visible to that particular slice. These multiple titles all go into the search engine. If we were to compute relevance based on the length of the phrase that matched, and the shortest match on the object is shorter than the match that is actually visible to us, we could return the object with a higher-than-obvious relevance. If we were to do that, we’d be leaking information, namely that there’s data on this object that the user making the query is not privy to. (Note that filtering of objects that aren’t at all visible to the user is done in a higher layer after the results have been accumulated and ranked by the search engine.)

Given this problem, there are two approaches one can take:

1. Store all the information needed to decide which labels are visible to the user running the query and then use only the visible labels when calculating the relevance of a match. Note that is a pretty expensive operation.
2. Don’t use the length of match to compute relevance. Note that skipping a relevance calculation is, obviously, a very cheap thing do.

Which do we do? Both.

When matching against object labels, the length metric actually lets us discern between better and worse matches. So in that case, we incur the cost of this calculation in order to return higher quality results.

However, when matching against things like document bodies, the ratio of the size of the match to the size of the search term starts to have less meaning but still has the possibility of leaking information in the query results. For fields like this, we turn off the relevance calculations based on length of match. The upshot is the we don’t have to store the permissions information in the index nor incur the cost of the permissions/views calculation for these fields.

A heartfelt thank you

To be clear, this post highlights the ways in which our search code diverges from the main Lucene code base. We’re huge fans of Lucene and have great respect for the developers that built and maintain what is probably the world’s greatest open-source search engine.

We put up a post last year on the 2008 VAST Grand Challenge. Well, the IEEE VAST Challenge 2009 is over and the awards are in. We had another strong year, scoring two awards:

• Grand Challenge: Analyst’s Tool Choice (Of 48 submissions, only 3 Grand Challenge awards were given)
• Intuitive Traffic Visualization and Video Description of the Analysis Process

Some background on the event: three years ago, the IEEE began an annual conference called VAST (Visual Analytics in Science and Technology). The VAST symposium focuses on the fundamental research contributions and real-world application of visual analytics. As a part of the conference, the VAST Challenge allows teams to compete on delivering analytic solutions against a synthetic real-world dataset.

A selection of choice quotes from the judges:

• An award for “highly usable integrated exploration environment”, “efficient analytic exploration platform” or something along these lines would be appropriate.
• Survey Question: How much novelty do you see in this submission (data processing, visualization, interaction, hypothesis generation or evaluation, overall process, etc.)? Answer: More so than novelty was the extremely efficient solution approach to this challenge, much more so than other solutions.
• The submission shows two things very clearly: One, it shows the analytical process as being a multi-faceted, simultaneous processing of different information that is quite common among analysts. Two, it shows how multiple perspectives can be displayed on a single monitor, enabling the analyst to visualize what his mind is analyzing. Outstanding!

Our submission

And finally, our submission to the Grand Challenge. Here we have our overview video, with a link to the full video below:

For an in-depth look at the data and techniques used to make this a reality, check out our full submission in Finding a Mole: Cyber Counter Intelligence on the Palantir Analysis Blog.

At Palantir, we work in Silicon Valley, read High Scalability, and think of web companies like Facebook and Google as our peers. Most of the time, this is exactly the right recipe for bringing disruptive innovation into the intelligence community. Sometimes, though, it’s misleading – when discussing a design decision, it’s received knowledge that “Disk is cheap.” or “CPU is cheap”. For a web company with a deployment in a commercial data center (or its own data center), this received knowledge is correct. But for a company that ships distributed systems instead of hosting them, and for whom the deployment environment is the kind of locked-down server room in which classified data can reside, these assumptions couldn’t be more false.

At Palantir, we are almost never able to host our customers’ data – typically, as the data is very sensitive, we are not even allowed to see it! Our customers’ highly sensitive data has to reside in a Secure Compartmented Information Facility or SCIF – a building which has been built to be resistant to attempts to access the information within, whether through active or passive measures. The network inside a SCIF is physically separated – “airgapped” – from the public Internet to prevent information leakage. As the entire rationale for such facilities is to prevent information leakage, moving information into or out of one is a tightly regulated process, almost always requiring a human to be in the loop.

Bandwidth is narrow

Bandwidth in and out of a data center is cheap. Bandwidth in and out of a SCIF is not – and this manifests in surprising ways. First off, what does it take to get data into a SCIF? First, the data has to be downloaded from wherever it’s hosted and burned to a CD. Then, someone has to carry it into the SCIF and find a security officer to approve adding it to the network. Finding the security officer can take anywhere from 10 minutes to an entire day. Once you’ve found the security officer, he has to run a virus scan on the CD, which can run at a rate of roughly 20 minutes per 100MB.

If you look at the entire process, you can model our connection into the SCIF as averaging about an 8 hour latency and 640 Kbps bandwidth. That’s about the bandwidth of a slow DSL line and the latency of a radio connection to Pluto. (Actually, it’s somewhat slower.) There’s also a big non-linearity at 700MB, which is the amount of data that fits on a single CD. For instance, this non-linearity is the big reason why we prefer to send patches to our customers rather than full distributions, which are slightly less than a gigabyte including dependencies – and thus why it’s worth it to us to build a system for automating patch application rather than simply replacing jar files by hand.

Disks are expensive

Similarly, if you are running a data warehouse, disk is cheap. You can buy a 1 TB, 7200 RPM disk for about $100, which is perfect for the kind of large, serial reads or writes that a data warehousing workflow requires. However, Palantir uses disk for our database and our search engine, both of which have an OLTP-style usage pattern. As opposed to a data warehouse access pattern, which emphasizes full table scans, OLTP emphasizes random access and therefore requires fast disk. To get 1TB at 15k RPMs costs about $1000, and requires a disk array rather than a single disk. In order to keep the disk fast, you also want to leave it only about 20% full, which overall makes fast disk about 50 times more expensive than slow disk. Most importantly, however, installing a disk array requires trained personnel, a special approval process, and reconfiguring the system to use the new disks, which is a fairly complicated and error-prone process.

CPUs are hot

Finally, in a commercial data center, CPU is the cheapest resource of all. In a secure server room, however, it can be quite expensive. Each CPU or additional box requires more power and cooling. If the room is nearly full, adding that extra box may require building out an entirely new server room, which can cost months and hundreds of thousands of dollars just for an office building. Building a server room in a SCIF is much more expensive and prohibitively time-consuming.

RAM to the rescue

On the other hand, some things in a SCIF are comparatively cheap. We never use boxes with less than 32GB of memory, and, in fact, lots of sites use 128GB of memory. RAM requires negligible power and cooling, and compared to disk, it’s relatively simple to install. It’s also easy to reconfigure the setup to use the additional memory.

The upshot

The design guidelines that follow from this are simple: build a system that is as autonomous as possible and scales down as well as it scales out.

All these statistics are compiled from our day-to-day experiences in the office environment of a SCIF. Deploying to soldiers in the field makes the issues involved in deploying to a SCIF seem minor. Of course, that’s what makes what we do fun.

At Palantir, we build distributed software. When deployed at a customer site, our platform consists of several servers running on, and distributed across, a cluster of machines. When I first joined the company, deploying and managing our platform was tedious and time consuming. Need to install servers? One by one, login to the machines where they need to go, lay down their requisite files and manually configure them such that they can work together. Have to bring down a deployment for scheduled maintenance? One by one, and in the correct order, login to the machines where the servers reside and shut them down. Want to change the private keys and certificates used to secure communication between servers? Well, you get the point.

From a customer perspective, the complexity associated with the administration of distributed software represents a significant challenge. Not providing tools to help reduce that complexity impacted the overall usability of our platform. Furthermore, from a Palantir perspective, a non-trivial portion of our resources were being devoted to deploying and managing instances of our platform, both externally (by Forward Deployed Engineers working directly with our customers) and internally (by development, QA and support staff working to maintain and improve our product). Could we be more efficient? No doubt. Given our intense focus on customer satisfaction and the desire to grow / scale our business, action was necessary.

To see how we solved this problem, read on.

We stepped back a bit, taking time to reflect on our situation and understand the problem. Based upon our experience, what key areas would a solution need to address? We settled on the following:

1. Lifecycle management.
   1. Ease initial deployment and upgrade.
   2. Handle coordinated starting, stopping and restarting.
2. Configuration management.
   1. Track which servers are installed on what machines.
   2. Provide centralized management of server configuration information.
3. Automation.
   1. Support encoding common management tasks based on best practices.

In addition to those three key areas, we also identified several important requirements. A couple that definitely warrant mention:

1. Security.
2. Extensibility.

After getting a good sense of what needed to be accomplished, we put effort into investigating if an existing solution would fit the bill. For a variety of reasons (i.e., available feature set, licensing constraints, etc.), we never found a good match. We did, however, come across several open source building blocks that could, when composed appropriately, combine to form the foundation of a homegrown solution. The Config Server was born.

Architecture

The Config Server works with remote agents to enable centralized deployment management. The diagram presented above provides an overview of our management infrastructure. Below is a brief discussion of each key component of our architecture.

• Agent – Agents are installed on every machine in a deployment. They are lightweight background processes that sit around waiting to execute commands submitted by the Config Server, interacting directly with the services installed on a given machine. Instead of implementing our own agent solution, we decided to leverage existing technology, the open source peer-to-peer Software Testing Automation Framework (STAF). From its homepage:
  

  The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework designed around the idea of reusable components, called services (such as process invocation, resource management, logging, and monitoring). STAF removes the tedium of building an automation infrastructure, thus enabling you to focus on building your automation solution. The STAF framework provides the foundation upon which to build higher level solutions, and provides a pluggable approach supported across a large variety of platforms and languages.

  We added support for two-way SSL to STAF to enhance the security of our management infrastructure (specifically, to allow us to implement authorization based on self-signed certificates). But beyond that, no modification was necessary. STAF provides us with a robust solution for remote process invocation and file management, both absolutely essential for centralized deployment management.

• Agent Manager – The Agent Manager provides lifecycle and configuration management functionality for the agents in a deployment. It interacts with remote machines through SSH, using the open source Trilead SSH for Java library.
• Config Registry – The Config Registry maintains and provides access to all of the information the Config Server has about a deployment. It consist of the following:
  • Agent Registry – The Agent Registry contains information about all of the agents in a deployment.
  • Service Registry – The Service Registry keeps track of all of the services in a deployment.
  • Config Repository – The Config Repository is a central store for configurations of the agents and services in a deployment.
  • Package Repository – The Package Repository holds all of the service packages that can be installed in a deployment.
  • Plugin Repository – The Plugin Repository houses all of the plugins that are available for use in the Config Server. Plugins are used by the Security Manager, Service Manager and Task Manager.
• Security Manager – We secure our servers and management infrastructure using public key cryptography. The Security Manager handles the generation and packaging of private keys and certificates. We perform private key and certificate generation using the Bouncy Castle Crypto APIs for Java. Packaging is taken care of by plugins in the Plugin Repository. For example, one plugin packages private keys and certificates into JKS files for use with Java, while another packages them into PEM files for use with OpenSSL.
• Service – Services represent the software installed on the machines in a deployment that drive our platform. They correspond to the servers we’ve built and the 3rd party offerings on which they depend (i.e., databases, entity extractors, etc.).
• Service Manager – The Service Manager interacts with agents to provide lifecycle and configuration management functionality for the services in a deployment. The actual mechanics of lifecycle and configuration management vary from to service to service. For example, starting service A might require invoking one script, while starting service B might require invoking another. For each type of service in a deployment, the Plugin Repository contains a corresponding plugin that embeds the necessary management logic. The Service Manager works with those plugins to get its job done.
• Task Manager – Managing a deployment requires performing tasks that go beyond lifecycle and configuration management for its constituent agents and services (i.e., log aggregation, database user creation, etc.). Such tasks are implemented as plugins. They make things happen by communicating with agents and / or directly with machines via SSH. The Task Manager interacts with the Plugin Manager to load tasks and coordinate their execution.

Functionality

How did we do with respect to our stated needs?

• Lifecycle management – The Agent Manager and Service Manager provide centralized lifecycle management. Initial deployment and upgrades, as well as starting, stopping and restarting servers, can all be handled directly through the Config Server.
• Configuration management – The Config Repository of the Config Server maintains information about deployments and provides centralized configuration management. The Agent Manager and Service Manager support the remote retrieval and application of agent and service configuration.
• Automation – The Config Server’s functionality is exposed via a clean and consistent Java API. Common management tasks can be automated by writing code against that API.

And what about some of our more important requirements?

• Security – All communication in our management infrastructure is secured using two-way SSL. A simple authorization mechanism, implemented using self-signed certificates, ensures that only the authorized entities (most notably, the Config Server), can execute commands through agents. Client access to the data maintained, and functionality exposed, by the Config Server requires password-based authorization.
• Extensibility – The Config Server can be extended to support new types of services and perform new tasks by implementing plugins and dropping them in the Plugin Repository.

Future

In the space of a few months, we built the Config Server to address several key needs and requirements related to the management of our platform. Our work has already begun to pay dividends. Looking ahead, there are several things we would like to do:

• Add support for low-level system management and configuration related to our platform (i.e., user and group management, firewall configuration, etc.).
• Implement multi-deployment management with support for features like staging, mirroring and migration.
• Autonomic Computing, integrating with our monitoring solution to implement platform self-management.

While we’ve accomplished a fair amount, plenty of work remains. We look forward to enhancing our Config Server and its associated infrastructure as we strive to make our platform one that is not only powerful and a pleasure to use, but also easy to manage and maintain.