This is a response to Ari’s awesome post on human-computer symbiosis. Ari and I were chatting about the equation he developed and I was wondering if there were some further refinements that are possible… let’s take a look:

We are attempting to understand the total analytic capability for a given task a of a human-computer team. Analytic capability in this case probably means:

(1)

Where A is the answer to the analytic problem in question and t_A is the time needed to arrive at the answer based on the inputs available. In the case of chess, A could be the optimum next move given all previous information and t_A would be how long it takes to decide on this move.

Read on for a look at how this generalizes in human-computer symbiotic systems.

In the case of the human-computer team, we know that a is going to be a function of both the human’s analytical capability h and the computer’s analytical capability c (where both h and c have units of answers/time). In the limit case we know that:

(2)

(3)

Or in plain English, if there is no human present, the total analytic capability is simply the analytic capability of the computer. So the naïve solution would be that:

(4)

(4) clearly meets the limiting cases described in (2) and (3). Kasparov noticed a mixing function where the ability of the human and computer to work together becomes the dominant term — we might call this the mixing capability for the given task or m. Including this phenomenon, the total analytic capability (4) would be re-defined as:

(5)

where m has the property that:

(6)

Thus maintaining the limits expressed in (2) and (3) and adhering to the observation that if there is no human or computer component then there will be no mixing advantage. A naïve solution to this constraint would be simple linear mixing:

(7)

where M (units of time per answer) is the mixing efficiency and will be primarily based on the type of task being solved — some analytical tasks lend themselves to a combined process more than others (for example, multiplying 20 digit numbers does not really benefit from the intuition of a human so the ability of a human and computer to perform this task is merely their additive ability).

What Kasparov noticed is that the mixing was primarily based on the quality of the process rather than the analytical power of either the human or computer separately. This seems to imply that we must somehow account for the fact that the quality of the human-computer interface is responsible for the quality of the mixing. This can be modeled as a unitless friction of interaction f_i that impedes the ability of the human and computer to work together.

Equation (7) can thus be re-written as:

(8)

In this case, the maximum value for the mixing capability is realized when the friction of interaction goes to zero. This mixing capability is the same as the equation Ari developed (less the coefficient which is necessary to maintain consistent units throughout).

We can now re-write our analytic capability in (5) as:

(9)

Below, see a plot of this function over a range of values for h, c and f_i:

As can clearly be seen from this functional plot (note the vertical scale), the effect of interface friction dominates over the other terms whenever both the human and computer can make important contributions to the task at hand. The conclusion can be drawn that the most effective way to solve analytical problems is to minimize the friction of the human-computer interface; or to put it another way: optimal analytical systems are those that are built specifically to maximize the ability of the human to leverage the ability of the computer.

I am certain there is still the possibility for further refinement, for example:

(10)

As we build our platforms and applications following a human-computer symbiosis approach, we keep an ear to the ground for interesting examples that illuminate new techniques or validate our approach in some empirical way.

One of the areas that we’re interested is in the overall friction of analysis systems. The systems that we build are built on commodity hardware — we’re not building faster computers and yet we can deliver orders-of-magnitude better performance on analysis tasks than existing solutions. How do we do this? By building software in such a way that it reduces the friction experienced at the boundaries between the computing power, the analyst, and the source data.

Chess as analysis laboratory

Chess is, at its heart, a predictive venture. The player attempts to anticipate their opponent’s moves, planning their own moves accordingly, with the straightforward goal of finding a sequence of piece moves that force checkmate.

This game is, in its ideal form, analysis. (The moves made are the logical extension of the analysis.) The data are clean, the problem is well-defined and everyone plays by the same rules. There are even well-defined metrics for ranking chess players by skill — a better chess player is a better chess-game analyst.

In the realm of evaluation of analysis systems, this is as about as good as it gets in terms of designing controlled experiments to study the relative strengths of different analysis systems.

Garry Kasparov, widely considered to be the greatest chess player of all time, recently wrote a review of Diego Rasskin Gutman’s book, Chess Metaphors: Artificial Intelligence and the Human Mind.

The review is excellent and covers a lot of ground. However, one particular anecdote stood out as a very interesting example of human-computer symbiosis (emphasis added):

In 2005, the online chess-playing site Playchess.com hosted what it called a “freestyle” chess tournament in which anyone could compete in teams with other players or computers. Normally, “anti-cheating” algorithms are employed by online sites to prevent, or at least discourage, players from cheating with computer assistance. (I wonder if these detection algorithms, which employ diagnostic analysis of moves and calculate probabilities, are any less “intelligent” than the playing programs they detect.)

Lured by the substantial prize money, several groups of strong grandmasters working with several computers at the same time entered the competition. At first, the results seemed predictable. The teams of human plus machine dominated even the strongest computers. The chess machine Hydra, which is a chess-specific supercomputer like Deep Blue, was no match for a strong human player using a relatively weak laptop. Human strategic guidance combined with the tactical acuity of a computer was overwhelming.

The surprise came at the conclusion of the event. The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.

After the jump, we look at this finding in a more generalized way and map it onto the Palantir approach.

The cyborg Grandmaster: a fearsome opponent

The tournament Kasparov recalls was a showcase of chess talent, human-computer symbiosis, and raw computing power. Among those entered in the tournament were a purpose-made chess machine (similar to Deep Blue) named Hydra and a team of Grandmasters assisted by computer programs.

One losing participant had this to say about the computer-aided Grandmasters:

Secondly, I have learned that a Grandmaster armed with a chess engine is a killer combination against a plain Engine. Engines see everything via brute force, Grandmasters use their intuition and are able to see “obvious” moves at once. So the two of them together are a mighty force.

This is just as Licklider predicted 50 years ago — quoting Man-Computer Symbiosis (if I could put it better, I would):

Men will set the goals and supply the motivations, of course, at least in the early years. They will formulate hypotheses. They will ask questions… In general, they will make approximate and fallible, but leading, contributions, and they will define criteria and serve as evaluators, judging the contributions of the equipment and guiding the general line of thought.

…

In addition, the computer will serve as a statistical-inference, decision-theory, or game-theory machine to make elementary evaluations of suggested courses of action whenever there is enough basis to support a formal statistical analysis. Finally, it will do as much diagnosis, pattern-matching, and relevance-recognizing as it profitably can, but it will accept a clearly secondary status in those areas.

So in classic intelligence amplification fashion, having computer programs that can quickly evaluate a move’s likelihood of success can amplify the power of the Grandmaster.

While empirically true, it does beg the question: how much does it amplify the power of the Grandmaster?

One approximation might be product as a simple linear amplification. Let’s imagine a function, a(h,c), in which the analytic power (a) is the product of power of the human (h) and the computing power of the chess engine being used (c). This gives us the equation:

One term to dominate them all: friction-of-interface

Does this simple approximation hold up? It does not. The team that won the PAL/CSS Freestyle Tournament in 2005 was composed of two amateur chess players that were able to best a computer-assisted Grandmaster.

How did they accomplish this feat? It was not through superior compute power. Instead, they did so by more effectively feeding insights to their three chess engines. They played so well that a large number of people actually assumed that it was actually Kasparov himself playing:

Many speculated that it might be Garry Kasparov, who was the initiator of this kind of computer assisted chess matches. When we asked him Kasparov confirmed that was not the case. But he reminded us that it doesn’t really matter. The guiding principle of Freestyle Chess: anything is allowed. “Even if they were assisted by the devil, that would probably be covered by the rules,” he joked. “Only the moves they played count.”

What does this mean for our simple equation? Well, it looks it’s missing a term, one we’ll call f, that describes the efficiency or friction of the interface between human and computer.

Quoting Kasparov again:

Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.

The implication being that the equation actually looks like this:

So as the friction of the interface goes to zero, the full amplification of the chess engine is brought to bear. A quick gut-check in the opposite direction agrees: one can imagine the world’s most powerful chess engine with the world’s worst interface; spending the time it would take to express commands to this theoretically awful program would actually be worse than playing without it.

Palantir: a low-friction interface to data

As analysis problems go, chess resembles a spherical cow in a vacuum. Analysis problems in the real world are orders of magnitude messier.

Let’s reframe the terms of our equation above into a more general approach to analysis:

• H – this is power of the analyst. In chess, the value of this terms varies widely between players; in designing real-world data analysis systems, this is more or less a constant (which is why h above becomes H below). Of course there are differing levels of expertise, training, and raw ability amongst the user population, but when we design systems, it’s with the average case in mind.
• c – computing power. How fast are the machines? How well do they scale? How efficiently do they perform the data tasks at hand? Palantir spends significant engineering effort on optimizing the c term, but most of the growth in this term comes from the layers we depend on, built by companies like Intel, Sun, Oracle, etc.
• f – friction. How easy is it to bring c to bear on the problem? Note that when we talk about friction of interface, this is not exclusively referring to user interface. More generally, friction can be present at any interface between two systems: data-software, software-software, human-software, etc. The f that we consider in this simple model is sum total system friction.

So our final formulation is just in terms of c and f (holding H as a constant):

When we discuss friction in real-world analysis systems, the friction actually exists at multiple levels:

1. Creating an analysis model that will enable answering the questions that need to be explored
2. Integrating the data into a single coherent view of the problem
3. Enabling analysis tools to efficiently query and load the data
4. Exposing APIs that allow developers to develop custom solutions quickly and efficiently for modeling and analysis tasks not covered by general tools
5. User interface that makes the tools easy, enjoyable, and quick to use

Minimizing f: Haiti Flooding Predictions

If this is starting to sound very similar to Palantir’s marketing information, this is no accident. While some of our backend engineers are concerned with things like scaling and speed-of-querying, the overall innovation that we’re bringing to the field is not simply about faster data processing systems (even if they are) but reducing the friction at every interface inside a complex human-computer symbiotic system.

You want an example that ties it all together? It starts with a simple question: which of the many displaced-person camps in Haiti are most at risk for flooding as the rainy season approaches? Easy to ask, but not so simple to answer.

The original introduction to this video:

As we enter the beginning of the rainy season in Haiti, one of the biggest problems facing relief organizations today is the spectre of flooding and mudslides destroying Internally Displaced Persons (IDP) Camps. In this video, we integrate data from many sources to determine high risk aid locations.

The data integration for this video took about six hours, using sources of data that had never before been fused. The analysis itself takes a few minutes and quickly comes to an actionable answer to the original question.

If you’ve taken the time to peruse the Palantir Government analysis blog, you’ve seen numerous examples of Palantir Government as applied to interesting problems; they are recorded screen captures of our analysis desktop client. It’s a showcase of useful, meaningful, and compelling visual and semantic tools being used to do analysis on a wide range of datasets.

What enabled this analysis? Aside from the obvious hard work of our UI and analysis tools teams, it’s the flexibility and power of the Palantir data platform. More than just a scalable datastore, the Palantir data platforms act as robust and clean abstractions on top of data.

One of the early architecture decisions that we made when building both Palantir Government and Palantir Finance was to separate the respective data platforms from the end-user applications used to actually perform analysis. More than just following the client-server model, this separation made the data servers in both products into generic intelligence infrastructure for analytic problems, with our clients acting as analysis applications on top of those platforms.

And so, one way to look at our data platform is as an operating system for analytic applications. In this post we’ll explore the history of operating systems, understand why they’re so important and see how the Palantir data servers deliver the same potential to revolutionize the writing of analysis software that operating systems did to the writing of general programs for computers.

The OS: abstraction that begat a paradigm

In the early days of computing, when a programmer wanted to write a program, they had to understand the inner workings of the machine. Writing a program required understanding things like the bus interface of a specific model of hard drive when all that was needed by the program was the clean abstraction of a filesystem. The upshot of this is that much of the time and effort put into a given task was spent writing code to interface with the “physical” minutiae of the machine rather than implementing the solution to the problem that the programmer was trying to solve with their software.

This pattern was observed by J.R. Licklider and noted in his influential paper, Man-Computer Symbiosis (emphasis added):

About 85 per cent of my “thinking” time was spent getting into a position to think, to make a decision, to learn something I needed to know. Much more time went into finding or obtaining information than into digesting it. Hours went into the plotting of graphs, and other hours into instructing an assistant how to plot. When the graphs were finished, the relations were obvious at once, but the plotting had to be done in order to make them so.
…
Throughout the period I examined, in short, my “thinking” time was devoted mainly to activities that were essentially clerical or mechanical: searching, calculating, plotting, transforming, determining the logical or dynamic consequences of a set of assumptions or hypotheses, preparing the way for a decision or an insight. Moreover, my choices of what to attempt and what not to attempt were determined to an embarrassingly great extent by considerations of clerical feasibility, not intellectual capability.

This description of his time as a researcher was echoed in the work of the early programmers: they spent much of their programming time re-inventing the wheel and writing routines that were doing essentially clerical or mechanistic work related to the functioning of the hardware rather the core functions of their programs.

The operating system changed all that: suddenly (and by that I mean: with years of hard work, research, and incremental change) that noisy, inconsistent pile of hardware was transformed into a set of clean abstractions. The programmer was finally freed to spend time and energy on the problem they were really trying to solve.

And so we come to the modern era: dealing with the messy details of hardware has been replaced by the clean and robust abstraction of the operating system.

Three important properties of modern operating systems:

• Hard boundaries between OS functions and process functions – in modern operating systems, this is usually accomplished with system calls. The process places the inputs to the system call in a known location and then asks the OS to perform some operation, like writing to a file or making a network connection. The OS may or may not perform the function, based on things like permissions, availability of resources, etc.

  The most important feature here is that the process never has direct access to the true resources of the machine — instead, all access to the machine’s resources are brokered by the OS.

• Extensions of the abstraction in every direction – An OS like Linux is really, at its core, a kernel that does process scheduling and lifecycle, manages memory, and services system calls. Everything else is handled by some sort of driver. A driver might also be called, more generically, a plugin or extension. Drivers exist for everything from block devices (like hard drives), network cards, and filesystems to input devices and displays.
• Designed as a general purpose framework – the operating system doesn’t actually do any computing; rather, it’s a set of services to facilitate processes using the resources of the computer. To that end, they’re not designed with a specific process in mind, but rather to serve a large class of programs, each designed and written to accomplish a different task using a similar set of resources.

Analysis: the modern computing task

The first computer, ENIAC, was conceived to do calculation of ballistics tables for artillery pieces — it was a glorified calculator. Lacking anything even resembling an operating system, it would just run its program. Its compiler? A group of six women who would configure the machine by hand with the program logic. The input for its first test run, a calculation related to the hydrogen bomb project, was approximately one million punch cards.

Times have changed: 40 or so years of the unrelenting march of Moore’s Law in computing power has given us something like an eight order of magnitude increase in the amount of computing power available per unit cost. Coupled with similar, more recent gains in storage capacity and network bandwidth, this has produced a world awash in data, crying out for analysis.

So the situation today is that we now expect to bring these considerable computing resources to bear on larger, more complex problems in the world. I’m talking about things like the spread of food-borne illnesses, understanding the connection between genes and protein expression, understanding terrorist networks, finding botnets in network traffic logs, and exploring influence networks in government.

These problems, while spanning a widely disparate areas of analysis, share some common traits:

The data is spread out

They are described by multiple data sources. Just to make things more interesting: the data sources don’t agree on their native representations of the real-world data. And finally, the real-world objects that the data are describing are actually described in multiple data sources, with no single source giving a complete and accurate representation.

The data schema are not human-conceptual

Rather than representing the data in some schema that maps easily into how the experts on a given problem think about said problem, the data stores in question tend to model data in whatever way was convenient for the creators of that particular data store. Put another way: people don’t think in tables, rows, columns, and XML snippets. These first-class data storage elements don’t usually map to real-world objects.

The data is sensitive

Whether it’s patient information, mortgage data, a law enforcement investigation, or sensitive foreign intelligence, there is often the need for foolproof access controls on the data.

Palantir: an operating system-class abstraction for analysis

A Palantir data server provides a similar class of services that an operating system does but focused on the specific needs of analytic tasks. Here I’ll focus on the model used by Palantir Government; Palantir Finance uses a similar but significantly different approach to delivering these services.

As you might imagine, however, they both start at a somewhat higher level than punch cards.

It starts with an ontology

The Palantir approach to analysis begins with a task-specific ontology: essentially, a human-conceptual description of the real-world problem that’s being analyzed.

It’s roughly composed of three pieces:

• A hierarchical type system of the real-world objects that human experts use to think about this problem. We call these PTObjects, short for “Palantir Objects”.
• A type system of properties that will contain the data describing these PTObjects. PTObjects are essentially typed containers for properties. This is where most of the detail of the ontology lies.
• A type system of possible relationships between different types of PTObjects.

Within the ontology, there are numerous extension points that allow the customization of how data is imported, retrieved, and displayed (following the principle of extending the abstraction in all directions).

The data server takes the ontology as input and is agnostic to its content. This is where the principle of building a general purpose framework comes into play.

The data sources are mapped into the ontology

This part of the Palantir data server is a pattern that is very similar to an operating system’s notion of block device drivers. The difference? Instead of low-level storage systems like hard drives, we’re dealing with complex databases describing the problem at hand.

In an operating system, every block device can read and write blocks of data. In the Palantir data server, everything becomes a source of PTObjects.

Our data importer plugins, by analogy, fulfills the same role as a block device driver:
we build glue code to map the data source’s schema into the ontology and the connectors to surface the data itself wrapped up in PTObjects.

The data are composed into real-world objects.

Part of this mapping is composing real-world objects into composite PTObjects by resolving PTObjects together.

The operation of resolving is pretty straightforward: we basically union the properties of the two PTObjects into a new PTObject. The end result is a single PTObject that completely represents all the data about something in the real-world from all the available data sources.

As we do this composition, we keep track of where each property came from, down to the record level, in each of its original sources. (Note that most composed PTObjects will usually have at least one property that comes from two sources). By preserving the original identity of every atom of data, it allows us to later decompose these PTObjects into their constituent parts or, more importantly, censor a client’s view based what permissions they have for each of the original data sources.

This a fundamental operation in our system that doesn’t have an exact analog in operating systems — it’s sort of similar to taking multiple filesystems and mounting them inside a virtual filesystem tree, like Unix does. However, if each data source is like a filesystem, what we’re doing is essentially composing individual files from their fragments stored on multiple block devices.

Another analogy: at a level below the block device in the OS, this is also sort of similar to what a RAID0 device does, the difference being that our composition is based on the contents of the data itself rather than some previously applied, content-agnostic, decomposition function. The other difference being motivation: a RAID0 does it for performance, while Palantir is composing data to make it correspond to the real-world objects it represents.

The server exposes Palantir “system calls”

The interface that the Palantir data server exposes can be boiled down to two essential operations:

• The client can download copies of PTObjects from the server. It may request them by id or perform some sort of search/query to specify a set of PTObjects. This is roughly analogous to the open() and read() system calls on Unix.

  Note that each client only sees the subset of properties for a given PTObject that it is authenticated for. This censorship of full PTObjects into projected slices is something done by the server on every load of PTObjects.

• The client can send new or updated PTObjects to the data server for storage. This is roughly analogous to the write() system call in Unix. It, of course, entails a check as to whether the given client has permission to write to the given PTObject.

The server’s responsibility is the same as the operating system: only let the client do what it has been granted permission to do. In an operating system, the OS uses hardware features like protected mode to keep lower-privileged processes from accessing machine resources. Palantir uses network calls to achieve the same separation, by placing the client and server on different logical machines. The effect is the same: the client basically requests (rather than commands) that certain operations are performed by the server. The server uses its own rules to decide if the access or change is allowed and responds accordingly. And so the principle of hard boundaries is implemented.

The clients do the analysis

When an operating system yields to a process, that’s the time when the true processing begins. By the same token, in Palantir, it’s not until a client connects and starts searching, visualizing, and manipulating PTObjects that analysis actually starts taking place (even if the server is doing a lot of the heavy lifting).

The wide open future

So why is this exciting? I’m glad you asked!

It’s about taking analysis to the next level.

Let’s say you’re someone who wants to write an analytic task. Let me ask you a series of rhetorical questions:

• Do you want to start with three disparate sources of data or with the data already mapped into a Palantir data server?
• Which one is a better use of your time as a programmer?
• Which one allows you to not repeat mistakes that other programmers have already made and fixed?
• Which one is more like writing a program than an operating system?

Operating systems took us to a new level of expressiveness when it came to writing computing processes to run on computing hardware. It inverted that 85/15 ratio that Licklider talked about so that programmers spent more time writing the code that did the thing they were trying to create and less time mucking around with hardware.

More programmer time == better analytic tasks.

It’s about making machine learning easier.

Now consider machine learning as a field. Pretty much every machine learning task could benefit from starting with its data in something that looks like a Palantir data server. I’ve taken an informal survey of machine learning researchers and they agree: the 85/15 ratio still holds for machine learning.

Simply put: most of the time and effort in machine learning is spent getting the data into a form that you can actually apply an algorithm to! Now imagine if the starting point for that was a Palantir data server — now the machine learning implementer has a world of expressiveness open to them and time and energy are spent on the task at hand instead of the overhead of messing with the data.

Now, we don’t think that we’re building Skynet. Quite the contrary: we believe that platforms like the one we’ve built will allow machine learning techniques to be put in the hands of experts to augment their ability to look at the world come to conclusions about complex real-world problems by asking questions of the data we’ve collected. It’s about Intelligence Augmentation, which can use machine learning techniques and algorithms to build better tools, not creating Strong AI.

It’s about creating new markets

Let’s go back to the well of operating systems and look back at the history of MS-DOS: the first “killer” application on MS-DOS was VisiCalc (that screenshot at the top of this post), a text-based spreadsheet. As you know, VisiCalc was not the end of the story but just the introduction. MS-DOS, evolved into Windows, allowed application writers an (arguably) clean abstraction on top of commodity hardware in order to build the applications that users actually wanted. Today, we have things like web browsers, multimedia authoring software, virtual machines, and IDEs built on top of what is, essentially, the same set of abstractions that VisiCalc was built on.

However, the most important thing to note is that VisiCalc is credited with creating the market for commercial operating systems — businesses needed VisiCalc so they paid Microsoft for MS-DOS (and IBM for a PC). Without VisiCalc, there was no market for MS-DOS (most people, unsurprisingly, didn’t want to buy a BASIC interpreter).

We’re in the business of selling software and we agree with our customers: the Palantir approach has tremendous value. We’ve just started tapping the potential of this market. Think about what Oracle looked like in 1979, think what Microsoft looked like in 1980 — that’s Palantir in 2009.

It’s about the start of the analysis age

It can be argued that the operating system is the innovation that ushered in the “information age“. Without the operating system, there is no software explosion, which allows computing technology to actually be used on data in the world.

We think that we’re on the cusp of the analysis age, as imagined by Vernor Vinge in Rainbow’s End. It was something foreseen by Licklider in 1960, albeit with a timeline that was off by at least a few decades:

“…it seems worthwhile to avoid argument with (other) enthusiasts for artificial intelligence by conceding dominance in the distant future of cerebration to machines alone. There will nevertheless be a fairly long interim during which the main intellectual advances will be made by men and computers working together in intimate association. A multidisciplinary study group, examining future research and development problems of the Air Force, estimated that it would be 1980 before developments in artificial intelligence make it possible for machines alone to do much thinking or problem solving of military significance. That would leave, say, five years to develop man-computer symbiosis and 15 years to use it. The 15 may be 10 or 500, but those years should be intellectually the most creative and exciting in the history of mankind.”

It’s a golden age of analysis and we’re just getting started: we’ve got a lot of work to do, so if this sort of thing excites you, please come and join us.

One of the big challenges in Palantir Finance comes when integrating data from multiple data providers. When the server is launched, it needs to create a coherent model of the financial world based on data coming from potentially dozens of data providers. Each data provider defines a set of “models” that it supports. These models can be things like equities, currencies, futures, options, or even new types that the providers themselves define.

The major challenge occurs when multiple providers define models that represent the same real-world entity. Provider A might know about Google, have basic open/high/low/close data for the stock, and know its ticker, country, and ISIN. Provider B might also provide a Google model, have balance sheet data, and know its country, exchange, and ISIN. We want to expose only one Google model to the user, however, and so we need a means of resolving the two Googles together – recognizing that they’re the same instrument – and adding just one equity to the system that encompasses both.

Resolution logic can be fairly complicated. For equities, for example, there are several different ways in which resolution can take place. If two equities have identical ISINs, we can be pretty confident they match, since those identifiers are declared as globally unique. If two equities have the same ticker and the same country of exchange, we might also consider that a match, though perhaps of weaker quality. Two models resolve to each other if any form of resolution considers them equal (with errors being thrown if other forms of resolution contradict the form that considers them equal…i.e. provider A and provider B agree on an instrument’s ISIN but disagree on its ticker).

Read on for the details of how we solve this seemingly n² problem with a linear solution.

Given N models across providers of a given asset class (say, equities), there are N² potential checks that I need to do to properly “resolve” all models, since any model can resolve to any other model in the system (and I potentially do want to attempt to resolve a model from provider A to other models from provider A to do error checking, since I may consider it invalid for a provider to provide the same model twice). Obviously we would like to do better than this, and we can, assuming that most models do not resolve to each other.

Envision the set of all (model, provider) pairs as the set of nodes on a graph. Two models from different providers that resolve to each other can be represented by an edge between two nodes in the graph. If the number of providers k is small relative to N, the number of resolution forms for a given asset class is small, and our data is valid, we can come up with an algorithm that solves our problem in N time as follows:

1. For every form of resolution, ask the data providers for all the data necessary for resolution to take place. For ticker/country resolution, with our data provider interfaces, this gives us a map from every (model, provider) pair to its ticker and country.
2. We can then invert this map, giving us a map from (ticker, country) pairs to a set of (model, provider) pairs. Note that the values in the inverted map do have to be sets, since there can be multiple (model, provider) pairs with the same ticker and country (indeed, this is expected if ANY models can be resolved between providers).
3. Then, for every model, for each resolution form, we can look up the relevant properties for that model, and then look up in the inverse map any models that are equivalent to it. This tells us what edges to add to our (model, provider) graph.

We’re essentially building up an in-memory, inverted index of the relevant data each model is giving us. The amortized O(1) lookups that the hashtable-backed maps provide allows us to trade the O(N²) complexity for something more like O(N).

N checks, rather than N² (assuming that k is trivial compared to N).

Once we’ve done this for every model each connected component of our graph should correspond to one model to be added into our final system. Since the connected components of a graph can be computed in time linear to the number of nodes, we can compute all the final models in linear time. And what is nice is that the maps give us the ability to quickly post-process our data to look for errors. If any two models in a given connected component come from the same provider, this is an error (either the provider has incorrect data, or it is modeling the data improperly). If two models from two different providers resolve, but have conflicting data for a given resolution form, this is also an error. Note that since providers do not have to provide data for every resolution form, it is possible that k models from different providers that resolve together do not form a k-clique on the graph.

Writing data providers is not always easy. There are many data sources out there that are messy, and properly modeling real world data in code can be quite challenging. That’s why it is important to come up with sound, efficient resolution logic that fails noisily, and tells the engineer building the provider when they are and are not playing nicely with the rest of the system.

We talk often with our contacts in finance and intelligence, and an increasingly common subject is the U.S. Government’s Troubled Assets Relief Program (TARP — part of the Treasury Department). Our friends see the large problems facing the TARP and the Federal Reserve, and have been asking how our technology can help.

Some of the problems are out of our hands, but many others are solvable with the proper analytics. Taking a closer look at the task before TARP, we noticed that many challenges mirror those facing the intelligence community:

• Entity and relationship data is scattered across many sources in a wide variety of formats; some are structured, some are unstructured.
• Entity structure and relationships are not always known upfront, so the solution must adapt to new data structures on the fly.
• It is costly, time-consuming, and unnecessary to impose one structure on the entire industry.
• Scalability is a must: millions of mortgages have been securitized into hundreds of thousands of entities.
• Sensitive, private data requires sophisticated access control and knowledge management — understanding who is accessing which data, what the organization knows, when it was known, and how it was discovered.
• Specialists from different fields and geographical regions must be able to collaborate effectively.

Palantir’s technology already solves these problems for the intelligence community. Our dynamic ontology makes it easy to import TARP data and entities, so we’ve created a short video using Palantir that shows the power of our approach. We analyze individual mortgage loans, mortgage-backed securities comprising these loans, and institutions holding tranches of the securities:

For more detail on the similarities, click the link to see a detailed breakdown of intelligence vs. TARP workflows.

Workflows

The types of questions the TARP and the Federal Reserve need to answer successfully are similar to those in the intelligence community. In essence, TARP is performing the sort of analysis performed at intelligence agencies: making sense of large amounts of data to create a coherent and accurate picture of the world. TARP is performing analysis on domestic financial data rather than global intelligence data, and using those insights to craft solutions to the current financial crisis. Our breakdown and comparison of the different aspects of the workflows along the same broad lines looks like this:

Strategic: Mission Planning and Policy Design

Operational: Asset Class Level Management and Tactical Planning

Tactical: Asset Targeting, Program Implementation, Specific Action Support.

Mission

We believe that the TARP’s success is critical to the global financial markets and the health of our nation. We’ve said from the beginning that our mission is to change the way the world approaches data, and today Palantir is a technology leader in both intelligence and finance. As we begin work on this new challenge we’re excited to be making a difference where it’s needed most.

The value of data analysis as a career was recently emphasized by Hal Varian in the Freakonomics blog in The New York Times. Hal is an internationally known economist who is currently serving as Google’s Chief Economist while on leave from his three professorships at the University of California at Berkeley.

Q: Your job sounds extremely interesting. What jobs would you recommend to a young person with an interest, and maybe a bachelors degree, in economics?

A: If you are looking for a career where your services will be in high demand, you should find something where you provide a scarce, complementary service to something that is getting ubiquitous and cheap. So what’s getting ubiquitous and cheap? Data. And what is complementary to data? Analysis. So my recommendation is to take lots of courses about how to manipulate and analyze data: databases, machine learning, econometrics, statistics, visualization, and so on. [emphasis added]

The following is my attempt at describing what we do at a high level without oversimplifying. I hope that after reading this a candidate will ‘get’ what we’re about, or at least understand enough not to apply tiny labels to our expansive vision.

The problem: implementing analysis

At Palantir we specialize in analysis.

Yes, that’s painfully abstract, and I’ll get to it in a second.

In real-world terms, we are building a software platform that enables people to take whatever data is relevant to them and understand it more easily and thoroughly than ever before, using concepts that they already understand. And we are applying this vision, at first, to solving problems in the finance sector and the government intelligence community.

The first important thing to note is that we don’t actually do the analysis ourselves. We don’t devise winning trading strategies and we don’t catch terrorists. We write software that enables other people to pull off these feats. These people, experts in their respective fields, are called analysts.

So what exactly do analysts do? What is analysis?

Analysis is everything necessary to extract insight from information.

Let’s break that down a bit.

Information is easy: It’s data. It lives in a relational database or as files indexed on a hard drive, and you can easily run queries against it. It comes in two forms, structured and unstructured. And there is a lot of it in the modern world – too much, actually, for current tools to make sense of.

Insight is trickier. Insight is something only a person can generate, and understanding this is critical for any organization that wants to do analysis right. Thus the challenge of data analysis is how to bring vast amounts of information into productive contact with human intelligence. In other words, the challenge is how to enable the analyst.

From the analyst’s perspective there are five essential features of an analysis platform:

1. First, and most important, the analyst should be in control. In other words, the primary way of interacting with an analysis tool should be human-driven queries. While automated approaches can complement a human-driven approach, there simply is no substitute for human intelligence. Unless you put a person behind the wheel, the system can never be flexible or creative enough to uncover truly original insight. Artificial Intelligence just isn’t there yet.
2. Ability to summarize large data sets. Some of this is what has traditionally been called data mining: the largely automated approach—using machine learning or other statistical techniques—of processing lots of data at once and extracting nuggets that capture something interesting about the data. Unlike Palantir, traditional approaches have focused almost exclusively on this aspect of analysis.
3. Ability to visualize large data sets. Here the analyst wants interesting and informative ways of viewing data graphically, to make it easier for him to digest. The analyst wants more than just a summary of the data; he wants a nuanced view of what’s going on inside these data sets: What’s the overall shape of the distribution? What are the outliers? What are important structures within the data?
4. Ability to iterate rapidly. This means enabling the analyst to ask a question, get the answer, and then quickly ask either a variant on the initial question or a follow-up question that depends on the answer to the initial question. This rapid, iterative process allows the analyst to quickly test out hypotheses and develop theories about what’s going on in the data, and by extension to discover what’s going on in the world.
5. Ability to collaborate with other analysts. Getting a handle on a terabyte of data, especially when it comprises multiple data types, is definitely more than a one-person job. Any organization that’s serious about understanding the world needs a team of analysts that can work together as more than the sum of its parts. This requires the ability for one analyst to effortlessly share the results of his analysis with his colleagues.

The Palantir approach

That’s what analysis looks like to the analyst, or rather what it should look like in an ideal world. (Current tools fall far short of this vision.) So what do we do at Palantir in order to make analysis this smooth and easy?

You could say that we help summarize large data sets, in the sense that we have to provide the analyst with a rich library of techniques and algorithms. You could also say that we do visualization, in the sense that we have to provide the analyst with a set of interesting and informative ways of visualizing their data. We do both of these things, and we have to be creative and solve hard problems in order to add value in these areas. But we do a lot more than that.

Probably the most central hard problem that we address in trying to enable the analyst is data modeling, the process of figuring out what data types are relevant to a domain, defining what they represent in the world, and deciding how to represent them in the system. At Palantir we make sure our data model (ontology) is both flexible and dynamic, and that it mirrors the concepts people naturally use when reasoning about the domain. This is no small challenge, but we’re already making it a reality. In finance our basic data types include financial instruments, dates, portfolios, indices, and strategies—the same things that financial researchers think about, talk about, and reason with. In the intelligence product our basic data types include people, places, and events (all with associated properties), which is exactly the way we all represent the world in our minds.

Data modeling, data summarization, and data visualization are the core disciplines for approaching large data sets. Human-driven queries, rapid iteration, and collaboration are multipliers, taking the power unlocked by the core disciplines to the next level. When these pieces are brought together in a coherent system, the result is in an analysis platform both very generic and very powerful.

This is what we mean when we say that we’re changing the way people approach data. Welcome to the future of analysis.