“You’re asking us to test our platform’s programming language? How am I supposed to do that?”

My head itches from trying to recall the bits and pieces of what I learned in high school about programming, specifically the semantics of a programming language. Sure, I did a bit of programming for homework assignments in college, but I was no CS major. This was a much different challenge for a QA engineer to test. Compared to an application, a programming language is completely open ended; there are no specifications to test, guidelines to follow, or limits to break.

The Hedgehog language had the basic set of tools laid out for me already: I could declare variables, create data structures, and use loops for iteration. As I was trying out individual usage examples, such as how to structure if statements or how to cast an object to a different type, I realized that this was no way to test something as powerful and flexible as an entire language. It would be like a doctor who claims that since each individual organ works fine, there are no problems with the entire system. This is insufficient: one needs to look at the system as a whole, including examining the interactions between each component. I decided I needed to create much larger and elaborate code samples in order to test the Hedgehog language in a larger scope.

Using the Hedgehog language, I had programmed several algorithms, solving puzzles that would output a number. This was getting bit boring, since once the output value was matched the expected number, there was nothing more to be done. I wanted to create something more dynamic, a toy I could play around and experiment with. And opportunity presented itself in the form of one of our newest tools: the spreadsheet application. With the capability of setting the value of each cell programmatically and then coloring them depending on their value… hmm what could I do with this?

Hedgehog is a powerful tool in coding functions and workflows that directly interact with our applications. Most of the time, the language is used to write expressions for an input value, create custom metrics that return values after a set of calculations, or even to set inputs, calculate, and save documents. Given the language’s ability to integrate with Spreadsheet, the capabilities of the Hedgehog language can literally be visually shown to the user, resulting in some stunning displays. Below are three examples I’ve coded in Palantir Finance’s own language: calculating and drawing the Mandelbrot fractal, simulating Conway’s Game of Life, and solving a Nonograms puzzle.

The Mandelbrot fractal

The Mandelbrot set is one of the most recognized images among fractals. Fractals are unique images due to their self-similarity, such that they have infinite detail: as you zoom in on a fractal border, detailed structures will continue to appear indefinitely.

Since the Mandelbrot set is an example of an escape time fractal, we can write a function that counts the number of iterations it takes for a given point to escape a threshold value. Applying this function to each point within a grid of specified range and resolution, we can generate a field of values to be set in a spreadsheet. After resizing the spreadsheet cells into small squares and applying conditional formatting, specifically a heat map that colors the cell based on its value, the below images can be generated:

Fig M1. Mandelbrot Fractal drawn from (-1.5, -1) to (0.5, 1), with the top left quadrant visible

Fig M2. Mandelbrot Fractal drawn from (0.27205, 0.00451) to (0.27505, 0.00751)

Fig M3. Mandelbrot Fractal with manually created rainbow heat map

The Game of Life

John Conway’s Game of Life is a simple cellular automaton that shows the evolution of an initial state of cells governed by a set of rules. In the classic scenario, cells can be alive or dead, indicated by a filled or empty cell respectively. When evaluating the next generation of cells, each cell checks the number of live neighbors among its adjacent 8 cells. If the cell is alive with 2 or 3 live neighbors, it remains alive. If a dead cell has 3 live neighbors, it becomes alive. All other scenarios, the cell evolves into a dead cell.

In this example, we rely heavily on spreadsheet’s cell dependency tree. Cells are able to reference the value of other cells within their expression, creating a directed acyclic graph. If the value of a cell changes, all cells that point towards the changed cell need to recalculate their value. In this specific case, the user can change the value of which generation to view, which causes the recalculation of the grid of cells representing the game.

In the spreadsheet below, the initial state, or generation 0, of “Noah’s Ark” is shown. The initial data for Noah’s Ark is derived from an external spreadsheet document containing a grid of 0’s and 1’s.

Below is an animated image of the evolution of Noah’s Ark from generation 0 to generation 10.

One problem encountered is that it takes increasingly longer to calculate a larger generation number. Because the only generation that is saved is the initial generation, calculations must always start from generation 0 when the user changes the generation number. This triggers a lot of redundant calculations: if I am at generation 10 and want to see generation 11, the calculation starts from generation 0, repeats all the same calculations up to generation 10, and finally calculates one more iteration to reach generation 11. To prevent unnecessary calculations, the results of previous calculations can be written to a cache file: a separate spreadsheet document containing the generation data that was calculated previously. When caching is enabled, previous generation calculations can be retrieved so the same calculation never happens twice.

Below is an animated image of a “Gospel Glider Gun” from generation 0 to generation 40, shown in increments of 2 generations. The glider gun will continue to oscillate over time while producing a 5 cell glider object that floats away from the gun diagonally down to the right. Without the caching functionality, the 40^th generation would take approximately 4 times as long as the 10^th generation. By enabling caching, each animation frame takes approximately the same time to calculate since only 2 generations are computed per frame.

Nonograms Solver

Nonograms, sometimes better known as Picross, are a type of logic puzzle where the user is given a blank grid with a set of numbers for each row and column. The numbers represent the number of consecutively filled blocks in the final solution within that row or column. Using these numbers as clues, it can be deduced whether or not a space is filled or not, until the entire grid simultaneously satisfies all given conditions. Solving puzzles is a NP-complete problem: it is very easy to verify a solution by checking if each row and column satisfies its set of numbers, but becomes increasingly difficult to solve larger puzzles. Time it takes to check a puzzle increases linearly with puzzle size, but the time to solve increases much faster due to the increase in number of iterations over each row or column crossed with an increased number of permutation possibilities for each set of numbers per iteration. Solving a 25×25 nonogram puzzle by hand usually takes about 30 minutes for an experienced solver.

Given the initial set of values for each row and column, the algorithm iterates through each row or column individually. The simplest method to solve is to find all possible permutations of the given numbers over the length of the row or column and check if there are any spaces that are consistently filled or unfilled across all possibilities. Let’s consider a simple example: In a row of 5 spaces, the set of values given is [2,1]. Thus the possible permutations are shown on the right, where 1 is a filled space, -1 is an unfilled space, and 0 is undetermined.

Additional optimizations, such as queuing the order of rows and columns to solve, trimming the solved ends of a range of spaces, and avoiding unnecessary calculations, were made to improve the performance of the algorithm such that the average time to solve a 25×25 puzzle is 5 minutes. This is fairly impressive given that the Hedgehog language is by no means optimized to handle such calculations and array manipulations. Below are some examples of solved puzzles:

Fig N1. 10x10 nonogram solution, solved in less than a second

Fig 2. 20x20 nonograms solution, solved in 20 seconds

Fig N3. 25x25 nonograms solution, solved in 3 minutes

The initial spark that ignited this series of code examples started from just a simple curiosity of, “I wonder if it would be possible to write this in Hedgehog?” In a way, it was also an exercise for me to learn more about the Hedgehog language, extending the use of its capabilities and libraries of metrics, in order to inject more complicated expressions while testing other applications on our platform. Throughout the process of developing the finished product, I was able to expose issues with the language that may have gone unnoticed with smaller, simpler test cases.

But of course, the driving motivation behind all this was to have fun, which is a big part of what it means to be working at Palantir.

A few months back, Kevin introduced us to the Hedgehog Programming language – (here’s the post if you missed it).

The Palantir Finance programming language — Hedgehog as we know it — is an interpreted, statically typed, object-oriented language. With a syntax that’s based loosely on Java, it mixes roughly Java-style semantics and a few idiosyncrasies that make it a really interesting case study in language design. It’s built to be extremely efficient for batch operations on time series, which is the heavy lifting in financial analysis.

In this video, Eugene and Dave, two of the engineers that work on the language and platform features needed to support it, give a talk that goes into a number of areas around the Hedgehog language, including why we needed to build a language, how it makes the platform more powerful, how we built dev tools into the UI to make debugging easier, and a bunch of the nitty-gritty features that go into the strange (but fitting) beast that is the Hedgehog Language.

As a final note: this is one of things that I love about working at Palantir Technologies. We study a problem pretty hard before we decide that we need to re-invent the wheel – and then when we do, we go all out. It’s one of the benefits of working with the incredibly talented and motivated folks here. When someone says, “well, we need to build a programming language. No, we’re sure,” we just roll up our sleeves and do it. We can add it to the list of: JMX monitoring system, refined Lucene search engine, speeding up Map-Reduce-like systems to interactive time, and implementing our own GIS platform.

One thing about being a developer on the Palantir Finance product that doesn’t get nearly enough publicity is the fact that we have our own programming language. I’m pretty excited about it so let me repeat, with emphasis: we have our own programming language. Yeah, it’s awesome. All those late hours you spent in the lab working on your final project in compilers: turns out they’re actually good for something other than getting into grad school.

Building this language ourselves — as opposed to, say, using an existing language that already just works — wasn’t an easy decision. In fact, it wasn’t even a single decision. We wracked our collective brain dozens of times trying to think of a better approach. But every which way we sliced it, the problems we needed to solve always pointed to building our own language. I still question this decision sometimes, but on the whole I’m very happy with how things have turned out.

The Palantir Finance programming language — Hedgehog as we know it — is an interpreted, statically typed, object-oriented language. With a syntax that’s based loosely on Java, it mixes roughly Java-style semantics and a few idiosyncrasies that make it a really interesting case study in language design. It’s built to be extremely efficient for batch operations on time series, which is the heavy lifting in financial analysis. It also allows you to dynamically add methods to a class from outside the class itself (conceptually similar to Ruby’s Mixins) — you define the function and its input type, and when you type the dot operator, your new method is auto-completed alongside all the “native” methods. Hedgehog also has a vast number of effectively global constants: all the stocks, bonds, and other financial instruments that are essential to the user experience, but that make for quite a design challenge.

I’m not a language guy myself, so instead of continuing to geek out over the core language features, I want to geek out about an emergent property that’s truly unique to the Hedgehog language. But first I’m going to back up and talk about something else that’s really important to us at Palantir: user experience. (I’ll get back to languages I promise.)

There’s a UX principle that says your interface should be “low threshold, high ceiling”. That is, it should be easy for the user to get started, but also able to do powerful things. This is actually a corollary of a more general principle: that your interface should strive for the optimal learning curve. My first CS professor explained this with a set of three diagrams, each representing one of the major OS families. I don’t remember exactly how he drew these diagrams at the time, but an updated version of them might look like this:

The x-axis of each curve represents “wizardry,” a measure of the user’s technical sophistication. The y-axis represents the power of the system — how much the user can accomplish at a given level of wizardry.

The best of the three curves, my prof argued, was the third curve. The first learning curve is great for providing incentives to learn. Each unit of effort spent to increase your wizardry yields an appropriate amount of reward or power. The drawback is that it’s hard for new users to do anything useful; its reward threshold is too high. The middle curve has a lower threshold and is better for novice users, but will frustrate an intermediate user because of the great plateau in the middle. (This might represent a place where the GUI isn’t powerful enough for the tasks you want to accomplish but scripting is still too difficult, leaving no way to express your commands) The third curve, however, is the best of both worlds: a low threshold and a smooth trajectory to the top.

Now let’s apply this back to our topic at hand, programming languages. Specifically, what does the learning curve look like for learning a first language? (Once you’ve learned one, of course, the rest come pretty easily.)

If your experience of learning to program was anything like mine, the first few projects in your first language were painful. You could sense the power further up the curve — it’s what convinced you to stick with CS — but simple tasks took a lot more effort than they should have, at the beginning.

Hedgehog on the other hand — our little homebrew that will someday have its own Wikipedia page — has the smoothest learning curve I’ve ever seen in a programming language. That’s the emergent property I wanted to talk about, because it’s a thing of beauty. You can get started with Hedgehog right away and accomplish quite a bit — without even knowing that you’re “programming” and the slope on the curve stays relatively constant throughout your trajectory.

We didn’t realize it at the time, but we were probably destined to create a low-threshold, high-ceiling language with a smooth learning curve, due to the nature of our user base. Financial analysts are impatient, and they still need to perform many kinds of complicated analysis. They definitely don’t have the time or inclination to spend a semester learning how to program. The solution to their problem is Hedgehog.

Allow me to illustrate with one of the earliest things a user might type into the expression bar:

And that’s it. The user types a ticker symbol and he gets a chart of IBM’s stock price. At no point did he have to wonder about variables or types or #includes. This experience is so frictionless he probably doesn’t even realize he’s writing code in a programming language. He just starts with what he knows, and the system gives him what he wants.

It starts to get interesting as you move further up the curve. Take this user input:

Of course that innocent dot between “IBM” and “volume” means a method invocation to anyone who’s familiar with C++ or Java. But to a new Palantir Finance user it simply means, “Let me access all the types of data associated with IBM.” Conceptually painless.

Or how about this one?

The volume/1000 expression is an anonymous method acting in the scope of a Stock object; it’s syntactic sugar for return this.volume()/1000;. But by allowing the user to strip away all the unnecessary syntax, we make learning the language that much easier.

I could go on tracing the curve here (I’ve only scratched the surface), but I hope I’ve made my point: we coax new users into writing code by making it look as much as possible like performing operations that they already intuitively understand. This is one of the benefits of creating a domain-specific language — we got the richness of the domain for free, and all the understanding that comes with it — and then we went above and beyond the simplification of a traditional DSL to really pare down the complexity of the language for novice users.

From simple beginnings like the ones I’ve shown here, it doesn’t take our users long at all to cross the threshold to more intermediate-level work, such as chaining function calls together or creating their own methods. As far as the high ceiling goes, we’re still working on it, but the language is currently capable of producing not only a quine, as one of our candidates showed us (yes, we ended up hiring him), but also code that can generate studies like the one below:

So Hedgehog has a low threshold and a smooth learning curve, and the ceiling is high enough that our users can do some really serious information processing with it — tasks that would make their other tools break down and cry. But there’s still a lot of interesting work for us to do, especially in pushing the language’s ceiling higher (developing better interactive debugging; working with large objects efficiently) — and as always, making it faster.

If you’d like to see the Hedgehog Programming Language in action, you can sign up for an account at Palantir JoyRide. the Palantir Finance public demo.

This is a response to Ari’s awesome post on human-computer symbiosis. Ari and I were chatting about the equation he developed and I was wondering if there were some further refinements that are possible… let’s take a look:

We are attempting to understand the total analytic capability for a given task a of a human-computer team. Analytic capability in this case probably means:

(1)

Where A is the answer to the analytic problem in question and t_A is the time needed to arrive at the answer based on the inputs available. In the case of chess, A could be the optimum next move given all previous information and t_A would be how long it takes to decide on this move.

Read on for a look at how this generalizes in human-computer symbiotic systems.

In the case of the human-computer team, we know that a is going to be a function of both the human’s analytical capability h and the computer’s analytical capability c (where both h and c have units of answers/time). In the limit case we know that:

(2)

(3)

Or in plain English, if there is no human present, the total analytic capability is simply the analytic capability of the computer. So the naïve solution would be that:

(4)

(4) clearly meets the limiting cases described in (2) and (3). Kasparov noticed a mixing function where the ability of the human and computer to work together becomes the dominant term — we might call this the mixing capability for the given task or m. Including this phenomenon, the total analytic capability (4) would be re-defined as:

(5)

where m has the property that:

(6)

Thus maintaining the limits expressed in (2) and (3) and adhering to the observation that if there is no human or computer component then there will be no mixing advantage. A naïve solution to this constraint would be simple linear mixing:

(7)

where M (units of time per answer) is the mixing efficiency and will be primarily based on the type of task being solved — some analytical tasks lend themselves to a combined process more than others (for example, multiplying 20 digit numbers does not really benefit from the intuition of a human so the ability of a human and computer to perform this task is merely their additive ability).

What Kasparov noticed is that the mixing was primarily based on the quality of the process rather than the analytical power of either the human or computer separately. This seems to imply that we must somehow account for the fact that the quality of the human-computer interface is responsible for the quality of the mixing. This can be modeled as a unitless friction of interaction f_i that impedes the ability of the human and computer to work together.

Equation (7) can thus be re-written as:

(8)

In this case, the maximum value for the mixing capability is realized when the friction of interaction goes to zero. This mixing capability is the same as the equation Ari developed (less the coefficient which is necessary to maintain consistent units throughout).

We can now re-write our analytic capability in (5) as:

(9)

Below, see a plot of this function over a range of values for h, c and f_i:

As can clearly be seen from this functional plot (note the vertical scale), the effect of interface friction dominates over the other terms whenever both the human and computer can make important contributions to the task at hand. The conclusion can be drawn that the most effective way to solve analytical problems is to minimize the friction of the human-computer interface; or to put it another way: optimal analytical systems are those that are built specifically to maximize the ability of the human to leverage the ability of the computer.

I am certain there is still the possibility for further refinement, for example:

(10)

As we build our platforms and applications following a human-computer symbiosis approach, we keep an ear to the ground for interesting examples that illuminate new techniques or validate our approach in some empirical way.

One of the areas that we’re interested is in the overall friction of analysis systems. The systems that we build are built on commodity hardware — we’re not building faster computers and yet we can deliver orders-of-magnitude better performance on analysis tasks than existing solutions. How do we do this? By building software in such a way that it reduces the friction experienced at the boundaries between the computing power, the analyst, and the source data.

Chess as analysis laboratory

Chess is, at its heart, a predictive venture. The player attempts to anticipate their opponent’s moves, planning their own moves accordingly, with the straightforward goal of finding a sequence of piece moves that force checkmate.

This game is, in its ideal form, analysis. (The moves made are the logical extension of the analysis.) The data are clean, the problem is well-defined and everyone plays by the same rules. There are even well-defined metrics for ranking chess players by skill — a better chess player is a better chess-game analyst.

In the realm of evaluation of analysis systems, this is as about as good as it gets in terms of designing controlled experiments to study the relative strengths of different analysis systems.

Garry Kasparov, widely considered to be the greatest chess player of all time, recently wrote a review of Diego Rasskin Gutman’s book, Chess Metaphors: Artificial Intelligence and the Human Mind.

The review is excellent and covers a lot of ground. However, one particular anecdote stood out as a very interesting example of human-computer symbiosis (emphasis added):

In 2005, the online chess-playing site Playchess.com hosted what it called a “freestyle” chess tournament in which anyone could compete in teams with other players or computers. Normally, “anti-cheating” algorithms are employed by online sites to prevent, or at least discourage, players from cheating with computer assistance. (I wonder if these detection algorithms, which employ diagnostic analysis of moves and calculate probabilities, are any less “intelligent” than the playing programs they detect.)

Lured by the substantial prize money, several groups of strong grandmasters working with several computers at the same time entered the competition. At first, the results seemed predictable. The teams of human plus machine dominated even the strongest computers. The chess machine Hydra, which is a chess-specific supercomputer like Deep Blue, was no match for a strong human player using a relatively weak laptop. Human strategic guidance combined with the tactical acuity of a computer was overwhelming.

The surprise came at the conclusion of the event. The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.

After the jump, we look at this finding in a more generalized way and map it onto the Palantir approach.

The cyborg Grandmaster: a fearsome opponent

The tournament Kasparov recalls was a showcase of chess talent, human-computer symbiosis, and raw computing power. Among those entered in the tournament were a purpose-made chess machine (similar to Deep Blue) named Hydra and a team of Grandmasters assisted by computer programs.

One losing participant had this to say about the computer-aided Grandmasters:

Secondly, I have learned that a Grandmaster armed with a chess engine is a killer combination against a plain Engine. Engines see everything via brute force, Grandmasters use their intuition and are able to see “obvious” moves at once. So the two of them together are a mighty force.

This is just as Licklider predicted 50 years ago — quoting Man-Computer Symbiosis (if I could put it better, I would):

Men will set the goals and supply the motivations, of course, at least in the early years. They will formulate hypotheses. They will ask questions… In general, they will make approximate and fallible, but leading, contributions, and they will define criteria and serve as evaluators, judging the contributions of the equipment and guiding the general line of thought.

…

In addition, the computer will serve as a statistical-inference, decision-theory, or game-theory machine to make elementary evaluations of suggested courses of action whenever there is enough basis to support a formal statistical analysis. Finally, it will do as much diagnosis, pattern-matching, and relevance-recognizing as it profitably can, but it will accept a clearly secondary status in those areas.

So in classic intelligence amplification fashion, having computer programs that can quickly evaluate a move’s likelihood of success can amplify the power of the Grandmaster.

While empirically true, it does beg the question: how much does it amplify the power of the Grandmaster?

One approximation might be product as a simple linear amplification. Let’s imagine a function, a(h,c), in which the analytic power (a) is the product of power of the human (h) and the computing power of the chess engine being used (c). This gives us the equation:

One term to dominate them all: friction-of-interface

Does this simple approximation hold up? It does not. The team that won the PAL/CSS Freestyle Tournament in 2005 was composed of two amateur chess players that were able to best a computer-assisted Grandmaster.

How did they accomplish this feat? It was not through superior compute power. Instead, they did so by more effectively feeding insights to their three chess engines. They played so well that a large number of people actually assumed that it was actually Kasparov himself playing:

Many speculated that it might be Garry Kasparov, who was the initiator of this kind of computer assisted chess matches. When we asked him Kasparov confirmed that was not the case. But he reminded us that it doesn’t really matter. The guiding principle of Freestyle Chess: anything is allowed. “Even if they were assisted by the devil, that would probably be covered by the rules,” he joked. “Only the moves they played count.”

What does this mean for our simple equation? Well, it looks it’s missing a term, one we’ll call f, that describes the efficiency or friction of the interface between human and computer.

Quoting Kasparov again:

Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.

The implication being that the equation actually looks like this:

So as the friction of the interface goes to zero, the full amplification of the chess engine is brought to bear. A quick gut-check in the opposite direction agrees: one can imagine the world’s most powerful chess engine with the world’s worst interface; spending the time it would take to express commands to this theoretically awful program would actually be worse than playing without it.

Palantir: a low-friction interface to data

As analysis problems go, chess resembles a spherical cow in a vacuum. Analysis problems in the real world are orders of magnitude messier.

Let’s reframe the terms of our equation above into a more general approach to analysis:

• H – this is power of the analyst. In chess, the value of this terms varies widely between players; in designing real-world data analysis systems, this is more or less a constant (which is why h above becomes H below). Of course there are differing levels of expertise, training, and raw ability amongst the user population, but when we design systems, it’s with the average case in mind.
• c – computing power. How fast are the machines? How well do they scale? How efficiently do they perform the data tasks at hand? Palantir spends significant engineering effort on optimizing the c term, but most of the growth in this term comes from the layers we depend on, built by companies like Intel, Sun, Oracle, etc.
• f – friction. How easy is it to bring c to bear on the problem? Note that when we talk about friction of interface, this is not exclusively referring to user interface. More generally, friction can be present at any interface between two systems: data-software, software-software, human-software, etc. The f that we consider in this simple model is sum total system friction.

So our final formulation is just in terms of c and f (holding H as a constant):

When we discuss friction in real-world analysis systems, the friction actually exists at multiple levels:

1. Creating an analysis model that will enable answering the questions that need to be explored
2. Integrating the data into a single coherent view of the problem
3. Enabling analysis tools to efficiently query and load the data
4. Exposing APIs that allow developers to develop custom solutions quickly and efficiently for modeling and analysis tasks not covered by general tools
5. User interface that makes the tools easy, enjoyable, and quick to use

Minimizing f: Haiti Flooding Predictions

If this is starting to sound very similar to Palantir’s marketing information, this is no accident. While some of our backend engineers are concerned with things like scaling and speed-of-querying, the overall innovation that we’re bringing to the field is not simply about faster data processing systems (even if they are) but reducing the friction at every interface inside a complex human-computer symbiotic system.

You want an example that ties it all together? It starts with a simple question: which of the many displaced-person camps in Haiti are most at risk for flooding as the rainy season approaches? Easy to ask, but not so simple to answer.

The original introduction to this video:

As we enter the beginning of the rainy season in Haiti, one of the biggest problems facing relief organizations today is the spectre of flooding and mudslides destroying Internally Displaced Persons (IDP) Camps. In this video, we integrate data from many sources to determine high risk aid locations.

The data integration for this video took about six hours, using sources of data that had never before been fused. The analysis itself takes a few minutes and quickly comes to an actionable answer to the original question.

If you’ve taken the time to peruse the Palantir Government analysis blog, you’ve seen numerous examples of Palantir Government as applied to interesting problems; they are recorded screen captures of our analysis desktop client. It’s a showcase of useful, meaningful, and compelling visual and semantic tools being used to do analysis on a wide range of datasets.

What enabled this analysis? Aside from the obvious hard work of our UI and analysis tools teams, it’s the flexibility and power of the Palantir data platform. More than just a scalable datastore, the Palantir data platforms act as robust and clean abstractions on top of data.

One of the early architecture decisions that we made when building both Palantir Government and Palantir Finance was to separate the respective data platforms from the end-user applications used to actually perform analysis. More than just following the client-server model, this separation made the data servers in both products into generic intelligence infrastructure for analytic problems, with our clients acting as analysis applications on top of those platforms.

And so, one way to look at our data platform is as an operating system for analytic applications. In this post we’ll explore the history of operating systems, understand why they’re so important and see how the Palantir data servers deliver the same potential to revolutionize the writing of analysis software that operating systems did to the writing of general programs for computers.

The OS: abstraction that begat a paradigm

In the early days of computing, when a programmer wanted to write a program, they had to understand the inner workings of the machine. Writing a program required understanding things like the bus interface of a specific model of hard drive when all that was needed by the program was the clean abstraction of a filesystem. The upshot of this is that much of the time and effort put into a given task was spent writing code to interface with the “physical” minutiae of the machine rather than implementing the solution to the problem that the programmer was trying to solve with their software.

This pattern was observed by J.R. Licklider and noted in his influential paper, Man-Computer Symbiosis (emphasis added):

About 85 per cent of my “thinking” time was spent getting into a position to think, to make a decision, to learn something I needed to know. Much more time went into finding or obtaining information than into digesting it. Hours went into the plotting of graphs, and other hours into instructing an assistant how to plot. When the graphs were finished, the relations were obvious at once, but the plotting had to be done in order to make them so.
…
Throughout the period I examined, in short, my “thinking” time was devoted mainly to activities that were essentially clerical or mechanical: searching, calculating, plotting, transforming, determining the logical or dynamic consequences of a set of assumptions or hypotheses, preparing the way for a decision or an insight. Moreover, my choices of what to attempt and what not to attempt were determined to an embarrassingly great extent by considerations of clerical feasibility, not intellectual capability.

This description of his time as a researcher was echoed in the work of the early programmers: they spent much of their programming time re-inventing the wheel and writing routines that were doing essentially clerical or mechanistic work related to the functioning of the hardware rather the core functions of their programs.

The operating system changed all that: suddenly (and by that I mean: with years of hard work, research, and incremental change) that noisy, inconsistent pile of hardware was transformed into a set of clean abstractions. The programmer was finally freed to spend time and energy on the problem they were really trying to solve.

And so we come to the modern era: dealing with the messy details of hardware has been replaced by the clean and robust abstraction of the operating system.

Three important properties of modern operating systems:

• Hard boundaries between OS functions and process functions – in modern operating systems, this is usually accomplished with system calls. The process places the inputs to the system call in a known location and then asks the OS to perform some operation, like writing to a file or making a network connection. The OS may or may not perform the function, based on things like permissions, availability of resources, etc.

  The most important feature here is that the process never has direct access to the true resources of the machine — instead, all access to the machine’s resources are brokered by the OS.

• Extensions of the abstraction in every direction – An OS like Linux is really, at its core, a kernel that does process scheduling and lifecycle, manages memory, and services system calls. Everything else is handled by some sort of driver. A driver might also be called, more generically, a plugin or extension. Drivers exist for everything from block devices (like hard drives), network cards, and filesystems to input devices and displays.
• Designed as a general purpose framework – the operating system doesn’t actually do any computing; rather, it’s a set of services to facilitate processes using the resources of the computer. To that end, they’re not designed with a specific process in mind, but rather to serve a large class of programs, each designed and written to accomplish a different task using a similar set of resources.

Analysis: the modern computing task

The first computer, ENIAC, was conceived to do calculation of ballistics tables for artillery pieces — it was a glorified calculator. Lacking anything even resembling an operating system, it would just run its program. Its compiler? A group of six women who would configure the machine by hand with the program logic. The input for its first test run, a calculation related to the hydrogen bomb project, was approximately one million punch cards.

Times have changed: 40 or so years of the unrelenting march of Moore’s Law in computing power has given us something like an eight order of magnitude increase in the amount of computing power available per unit cost. Coupled with similar, more recent gains in storage capacity and network bandwidth, this has produced a world awash in data, crying out for analysis.

So the situation today is that we now expect to bring these considerable computing resources to bear on larger, more complex problems in the world. I’m talking about things like the spread of food-borne illnesses, understanding the connection between genes and protein expression, understanding terrorist networks, finding botnets in network traffic logs, and exploring influence networks in government.

These problems, while spanning a widely disparate areas of analysis, share some common traits:

The data is spread out

They are described by multiple data sources. Just to make things more interesting: the data sources don’t agree on their native representations of the real-world data. And finally, the real-world objects that the data are describing are actually described in multiple data sources, with no single source giving a complete and accurate representation.

The data schema are not human-conceptual

Rather than representing the data in some schema that maps easily into how the experts on a given problem think about said problem, the data stores in question tend to model data in whatever way was convenient for the creators of that particular data store. Put another way: people don’t think in tables, rows, columns, and XML snippets. These first-class data storage elements don’t usually map to real-world objects.

The data is sensitive

Whether it’s patient information, mortgage data, a law enforcement investigation, or sensitive foreign intelligence, there is often the need for foolproof access controls on the data.

Palantir: an operating system-class abstraction for analysis

A Palantir data server provides a similar class of services that an operating system does but focused on the specific needs of analytic tasks. Here I’ll focus on the model used by Palantir Government; Palantir Finance uses a similar but significantly different approach to delivering these services.

As you might imagine, however, they both start at a somewhat higher level than punch cards.

It starts with an ontology

The Palantir approach to analysis begins with a task-specific ontology: essentially, a human-conceptual description of the real-world problem that’s being analyzed.

It’s roughly composed of three pieces:

• A hierarchical type system of the real-world objects that human experts use to think about this problem. We call these PTObjects, short for “Palantir Objects”.
• A type system of properties that will contain the data describing these PTObjects. PTObjects are essentially typed containers for properties. This is where most of the detail of the ontology lies.
• A type system of possible relationships between different types of PTObjects.

Within the ontology, there are numerous extension points that allow the customization of how data is imported, retrieved, and displayed (following the principle of extending the abstraction in all directions).

The data server takes the ontology as input and is agnostic to its content. This is where the principle of building a general purpose framework comes into play.

The data sources are mapped into the ontology

This part of the Palantir data server is a pattern that is very similar to an operating system’s notion of block device drivers. The difference? Instead of low-level storage systems like hard drives, we’re dealing with complex databases describing the problem at hand.

In an operating system, every block device can read and write blocks of data. In the Palantir data server, everything becomes a source of PTObjects.

Our data importer plugins, by analogy, fulfills the same role as a block device driver:
we build glue code to map the data source’s schema into the ontology and the connectors to surface the data itself wrapped up in PTObjects.

The data are composed into real-world objects.

Part of this mapping is composing real-world objects into composite PTObjects by resolving PTObjects together.

The operation of resolving is pretty straightforward: we basically union the properties of the two PTObjects into a new PTObject. The end result is a single PTObject that completely represents all the data about something in the real-world from all the available data sources.

As we do this composition, we keep track of where each property came from, down to the record level, in each of its original sources. (Note that most composed PTObjects will usually have at least one property that comes from two sources). By preserving the original identity of every atom of data, it allows us to later decompose these PTObjects into their constituent parts or, more importantly, censor a client’s view based what permissions they have for each of the original data sources.

This a fundamental operation in our system that doesn’t have an exact analog in operating systems — it’s sort of similar to taking multiple filesystems and mounting them inside a virtual filesystem tree, like Unix does. However, if each data source is like a filesystem, what we’re doing is essentially composing individual files from their fragments stored on multiple block devices.

Another analogy: at a level below the block device in the OS, this is also sort of similar to what a RAID0 device does, the difference being that our composition is based on the contents of the data itself rather than some previously applied, content-agnostic, decomposition function. The other difference being motivation: a RAID0 does it for performance, while Palantir is composing data to make it correspond to the real-world objects it represents.

The server exposes Palantir “system calls”

The interface that the Palantir data server exposes can be boiled down to two essential operations:

• The client can download copies of PTObjects from the server. It may request them by id or perform some sort of search/query to specify a set of PTObjects. This is roughly analogous to the open() and read() system calls on Unix.

  Note that each client only sees the subset of properties for a given PTObject that it is authenticated for. This censorship of full PTObjects into projected slices is something done by the server on every load of PTObjects.

• The client can send new or updated PTObjects to the data server for storage. This is roughly analogous to the write() system call in Unix. It, of course, entails a check as to whether the given client has permission to write to the given PTObject.

The server’s responsibility is the same as the operating system: only let the client do what it has been granted permission to do. In an operating system, the OS uses hardware features like protected mode to keep lower-privileged processes from accessing machine resources. Palantir uses network calls to achieve the same separation, by placing the client and server on different logical machines. The effect is the same: the client basically requests (rather than commands) that certain operations are performed by the server. The server uses its own rules to decide if the access or change is allowed and responds accordingly. And so the principle of hard boundaries is implemented.

The clients do the analysis

When an operating system yields to a process, that’s the time when the true processing begins. By the same token, in Palantir, it’s not until a client connects and starts searching, visualizing, and manipulating PTObjects that analysis actually starts taking place (even if the server is doing a lot of the heavy lifting).

The wide open future

So why is this exciting? I’m glad you asked!

It’s about taking analysis to the next level.

Let’s say you’re someone who wants to write an analytic task. Let me ask you a series of rhetorical questions:

• Do you want to start with three disparate sources of data or with the data already mapped into a Palantir data server?
• Which one is a better use of your time as a programmer?
• Which one allows you to not repeat mistakes that other programmers have already made and fixed?
• Which one is more like writing a program than an operating system?

Operating systems took us to a new level of expressiveness when it came to writing computing processes to run on computing hardware. It inverted that 85/15 ratio that Licklider talked about so that programmers spent more time writing the code that did the thing they were trying to create and less time mucking around with hardware.

More programmer time == better analytic tasks.

It’s about making machine learning easier.

Now consider machine learning as a field. Pretty much every machine learning task could benefit from starting with its data in something that looks like a Palantir data server. I’ve taken an informal survey of machine learning researchers and they agree: the 85/15 ratio still holds for machine learning.

Simply put: most of the time and effort in machine learning is spent getting the data into a form that you can actually apply an algorithm to! Now imagine if the starting point for that was a Palantir data server — now the machine learning implementer has a world of expressiveness open to them and time and energy are spent on the task at hand instead of the overhead of messing with the data.

Now, we don’t think that we’re building Skynet. Quite the contrary: we believe that platforms like the one we’ve built will allow machine learning techniques to be put in the hands of experts to augment their ability to look at the world come to conclusions about complex real-world problems by asking questions of the data we’ve collected. It’s about Intelligence Augmentation, which can use machine learning techniques and algorithms to build better tools, not creating Strong AI.

It’s about creating new markets

Let’s go back to the well of operating systems and look back at the history of MS-DOS: the first “killer” application on MS-DOS was VisiCalc (that screenshot at the top of this post), a text-based spreadsheet. As you know, VisiCalc was not the end of the story but just the introduction. MS-DOS, evolved into Windows, allowed application writers an (arguably) clean abstraction on top of commodity hardware in order to build the applications that users actually wanted. Today, we have things like web browsers, multimedia authoring software, virtual machines, and IDEs built on top of what is, essentially, the same set of abstractions that VisiCalc was built on.

However, the most important thing to note is that VisiCalc is credited with creating the market for commercial operating systems — businesses needed VisiCalc so they paid Microsoft for MS-DOS (and IBM for a PC). Without VisiCalc, there was no market for MS-DOS (most people, unsurprisingly, didn’t want to buy a BASIC interpreter).

We’re in the business of selling software and we agree with our customers: the Palantir approach has tremendous value. We’ve just started tapping the potential of this market. Think about what Oracle looked like in 1979, think what Microsoft looked like in 1980 — that’s Palantir in 2009.

It’s about the start of the analysis age

It can be argued that the operating system is the innovation that ushered in the “information age“. Without the operating system, there is no software explosion, which allows computing technology to actually be used on data in the world.

We think that we’re on the cusp of the analysis age, as imagined by Vernor Vinge in Rainbow’s End. It was something foreseen by Licklider in 1960, albeit with a timeline that was off by at least a few decades:

“…it seems worthwhile to avoid argument with (other) enthusiasts for artificial intelligence by conceding dominance in the distant future of cerebration to machines alone. There will nevertheless be a fairly long interim during which the main intellectual advances will be made by men and computers working together in intimate association. A multidisciplinary study group, examining future research and development problems of the Air Force, estimated that it would be 1980 before developments in artificial intelligence make it possible for machines alone to do much thinking or problem solving of military significance. That would leave, say, five years to develop man-computer symbiosis and 15 years to use it. The 15 may be 10 or 500, but those years should be intellectually the most creative and exciting in the history of mankind.”

It’s a golden age of analysis and we’re just getting started: we’ve got a lot of work to do, so if this sort of thing excites you, please come and join us.

One of the big challenges in Palantir Finance comes when integrating data from multiple data providers. When the server is launched, it needs to create a coherent model of the financial world based on data coming from potentially dozens of data providers. Each data provider defines a set of “models” that it supports. These models can be things like equities, currencies, futures, options, or even new types that the providers themselves define.

The major challenge occurs when multiple providers define models that represent the same real-world entity. Provider A might know about Google, have basic open/high/low/close data for the stock, and know its ticker, country, and ISIN. Provider B might also provide a Google model, have balance sheet data, and know its country, exchange, and ISIN. We want to expose only one Google model to the user, however, and so we need a means of resolving the two Googles together – recognizing that they’re the same instrument – and adding just one equity to the system that encompasses both.

Resolution logic can be fairly complicated. For equities, for example, there are several different ways in which resolution can take place. If two equities have identical ISINs, we can be pretty confident they match, since those identifiers are declared as globally unique. If two equities have the same ticker and the same country of exchange, we might also consider that a match, though perhaps of weaker quality. Two models resolve to each other if any form of resolution considers them equal (with errors being thrown if other forms of resolution contradict the form that considers them equal…i.e. provider A and provider B agree on an instrument’s ISIN but disagree on its ticker).

Read on for the details of how we solve this seemingly n² problem with a linear solution.

Given N models across providers of a given asset class (say, equities), there are N² potential checks that I need to do to properly “resolve” all models, since any model can resolve to any other model in the system (and I potentially do want to attempt to resolve a model from provider A to other models from provider A to do error checking, since I may consider it invalid for a provider to provide the same model twice). Obviously we would like to do better than this, and we can, assuming that most models do not resolve to each other.

Envision the set of all (model, provider) pairs as the set of nodes on a graph. Two models from different providers that resolve to each other can be represented by an edge between two nodes in the graph. If the number of providers k is small relative to N, the number of resolution forms for a given asset class is small, and our data is valid, we can come up with an algorithm that solves our problem in N time as follows:

1. For every form of resolution, ask the data providers for all the data necessary for resolution to take place. For ticker/country resolution, with our data provider interfaces, this gives us a map from every (model, provider) pair to its ticker and country.
2. We can then invert this map, giving us a map from (ticker, country) pairs to a set of (model, provider) pairs. Note that the values in the inverted map do have to be sets, since there can be multiple (model, provider) pairs with the same ticker and country (indeed, this is expected if ANY models can be resolved between providers).
3. Then, for every model, for each resolution form, we can look up the relevant properties for that model, and then look up in the inverse map any models that are equivalent to it. This tells us what edges to add to our (model, provider) graph.

We’re essentially building up an in-memory, inverted index of the relevant data each model is giving us. The amortized O(1) lookups that the hashtable-backed maps provide allows us to trade the O(N²) complexity for something more like O(N).

N checks, rather than N² (assuming that k is trivial compared to N).

Once we’ve done this for every model each connected component of our graph should correspond to one model to be added into our final system. Since the connected components of a graph can be computed in time linear to the number of nodes, we can compute all the final models in linear time. And what is nice is that the maps give us the ability to quickly post-process our data to look for errors. If any two models in a given connected component come from the same provider, this is an error (either the provider has incorrect data, or it is modeling the data improperly). If two models from two different providers resolve, but have conflicting data for a given resolution form, this is also an error. Note that since providers do not have to provide data for every resolution form, it is possible that k models from different providers that resolve together do not form a k-clique on the graph.

Writing data providers is not always easy. There are many data sources out there that are messy, and properly modeling real world data in code can be quite challenging. That’s why it is important to come up with sound, efficient resolution logic that fails noisily, and tells the engineer building the provider when they are and are not playing nicely with the rest of the system.

We talk often with our contacts in finance and intelligence, and an increasingly common subject is the U.S. Government’s Troubled Assets Relief Program (TARP — part of the Treasury Department). Our friends see the large problems facing the TARP and the Federal Reserve, and have been asking how our technology can help.

Some of the problems are out of our hands, but many others are solvable with the proper analytics. Taking a closer look at the task before TARP, we noticed that many challenges mirror those facing the intelligence community:

• Entity and relationship data is scattered across many sources in a wide variety of formats; some are structured, some are unstructured.
• Entity structure and relationships are not always known upfront, so the solution must adapt to new data structures on the fly.
• It is costly, time-consuming, and unnecessary to impose one structure on the entire industry.
• Scalability is a must: millions of mortgages have been securitized into hundreds of thousands of entities.
• Sensitive, private data requires sophisticated access control and knowledge management — understanding who is accessing which data, what the organization knows, when it was known, and how it was discovered.
• Specialists from different fields and geographical regions must be able to collaborate effectively.

Palantir’s technology already solves these problems for the intelligence community. Our dynamic ontology makes it easy to import TARP data and entities, so we’ve created a short video using Palantir that shows the power of our approach. We analyze individual mortgage loans, mortgage-backed securities comprising these loans, and institutions holding tranches of the securities:

For more detail on the similarities, click the link to see a detailed breakdown of intelligence vs. TARP workflows.

Workflows

The types of questions the TARP and the Federal Reserve need to answer successfully are similar to those in the intelligence community. In essence, TARP is performing the sort of analysis performed at intelligence agencies: making sense of large amounts of data to create a coherent and accurate picture of the world. TARP is performing analysis on domestic financial data rather than global intelligence data, and using those insights to craft solutions to the current financial crisis. Our breakdown and comparison of the different aspects of the workflows along the same broad lines looks like this:

Strategic: Mission Planning and Policy Design

Operational: Asset Class Level Management and Tactical Planning

Tactical: Asset Targeting, Program Implementation, Specific Action Support.

Mission

We believe that the TARP’s success is critical to the global financial markets and the health of our nation. We’ve said from the beginning that our mission is to change the way the world approaches data, and today Palantir is a technology leader in both intelligence and finance. As we begin work on this new challenge we’re excited to be making a difference where it’s needed most.

The value of data analysis as a career was recently emphasized by Hal Varian in the Freakonomics blog in The New York Times. Hal is an internationally known economist who is currently serving as Google’s Chief Economist while on leave from his three professorships at the University of California at Berkeley.

Q: Your job sounds extremely interesting. What jobs would you recommend to a young person with an interest, and maybe a bachelors degree, in economics?

A: If you are looking for a career where your services will be in high demand, you should find something where you provide a scarce, complementary service to something that is getting ubiquitous and cheap. So what’s getting ubiquitous and cheap? Data. And what is complementary to data? Analysis. So my recommendation is to take lots of courses about how to manipulate and analyze data: databases, machine learning, econometrics, statistics, visualization, and so on. [emphasis added]

The following is my attempt at describing what we do at a high level without oversimplifying. I hope that after reading this a candidate will ‘get’ what we’re about, or at least understand enough not to apply tiny labels to our expansive vision.

The problem: implementing analysis

At Palantir we specialize in analysis.

Yes, that’s painfully abstract, and I’ll get to it in a second.

In real-world terms, we are building a software platform that enables people to take whatever data is relevant to them and understand it more easily and thoroughly than ever before, using concepts that they already understand. And we are applying this vision, at first, to solving problems in the finance sector and the government intelligence community.

The first important thing to note is that we don’t actually do the analysis ourselves. We don’t devise winning trading strategies and we don’t catch terrorists. We write software that enables other people to pull off these feats. These people, experts in their respective fields, are called analysts.

So what exactly do analysts do? What is analysis?

Analysis is everything necessary to extract insight from information.

Let’s break that down a bit.

Information is easy: It’s data. It lives in a relational database or as files indexed on a hard drive, and you can easily run queries against it. It comes in two forms, structured and unstructured. And there is a lot of it in the modern world – too much, actually, for current tools to make sense of.

Insight is trickier. Insight is something only a person can generate, and understanding this is critical for any organization that wants to do analysis right. Thus the challenge of data analysis is how to bring vast amounts of information into productive contact with human intelligence. In other words, the challenge is how to enable the analyst.

From the analyst’s perspective there are five essential features of an analysis platform:

1. First, and most important, the analyst should be in control. In other words, the primary way of interacting with an analysis tool should be human-driven queries. While automated approaches can complement a human-driven approach, there simply is no substitute for human intelligence. Unless you put a person behind the wheel, the system can never be flexible or creative enough to uncover truly original insight. Artificial Intelligence just isn’t there yet.
2. Ability to summarize large data sets. Some of this is what has traditionally been called data mining: the largely automated approach—using machine learning or other statistical techniques—of processing lots of data at once and extracting nuggets that capture something interesting about the data. Unlike Palantir, traditional approaches have focused almost exclusively on this aspect of analysis.
3. Ability to visualize large data sets. Here the analyst wants interesting and informative ways of viewing data graphically, to make it easier for him to digest. The analyst wants more than just a summary of the data; he wants a nuanced view of what’s going on inside these data sets: What’s the overall shape of the distribution? What are the outliers? What are important structures within the data?
4. Ability to iterate rapidly. This means enabling the analyst to ask a question, get the answer, and then quickly ask either a variant on the initial question or a follow-up question that depends on the answer to the initial question. This rapid, iterative process allows the analyst to quickly test out hypotheses and develop theories about what’s going on in the data, and by extension to discover what’s going on in the world.
5. Ability to collaborate with other analysts. Getting a handle on a terabyte of data, especially when it comprises multiple data types, is definitely more than a one-person job. Any organization that’s serious about understanding the world needs a team of analysts that can work together as more than the sum of its parts. This requires the ability for one analyst to effortlessly share the results of his analysis with his colleagues.

The Palantir approach

That’s what analysis looks like to the analyst, or rather what it should look like in an ideal world. (Current tools fall far short of this vision.) So what do we do at Palantir in order to make analysis this smooth and easy?

You could say that we help summarize large data sets, in the sense that we have to provide the analyst with a rich library of techniques and algorithms. You could also say that we do visualization, in the sense that we have to provide the analyst with a set of interesting and informative ways of visualizing their data. We do both of these things, and we have to be creative and solve hard problems in order to add value in these areas. But we do a lot more than that.

Probably the most central hard problem that we address in trying to enable the analyst is data modeling, the process of figuring out what data types are relevant to a domain, defining what they represent in the world, and deciding how to represent them in the system. At Palantir we make sure our data model (ontology) is both flexible and dynamic, and that it mirrors the concepts people naturally use when reasoning about the domain. This is no small challenge, but we’re already making it a reality. In finance our basic data types include financial instruments, dates, portfolios, indices, and strategies—the same things that financial researchers think about, talk about, and reason with. In the intelligence product our basic data types include people, places, and events (all with associated properties), which is exactly the way we all represent the world in our minds.

Data modeling, data summarization, and data visualization are the core disciplines for approaching large data sets. Human-driven queries, rapid iteration, and collaboration are multipliers, taking the power unlocked by the core disciplines to the next level. When these pieces are brought together in a coherent system, the result is in an analysis platform both very generic and very powerful.

This is what we mean when we say that we’re changing the way people approach data. Welcome to the future of analysis.