Note: this part is part two of our series on doing your best in interviews. Part one: “How to Rock an Algorithms Interview”.

Here at Palantir algorithms are important, but code is our lifeblood. We live and die by the quality of the code we ship. It’s no surprise, then, that coding ability is what we stress the most in our interview process. A candidate can get by with mediocre algorithm skills (depending on the role), but no one can skimp on coding.

Suppose you’re confident in your ability to write great software. Your task in a coding interview (of which there will be several) is to show the interviewers that you in fact do have the programming chops — that you’re an experienced coder who knows how to write solid, production-quality code.

This is easier said than done. After all, coding in your favorite IDE from the comfort of $familiar_place is very different from coding on a whiteboard (on a problem you’re totally unfamiliar with) in a pressure-filled 45-minute interview. We realize that the interview environment is not the real world, and we adjust our expectations accordingly. Nonetheless, there are a number of things you can do to put your best foot forward during the interview.

First, though, we’d like to give you a sense for what we look for during a coding interview. Most important is the ability to write clean and correct code — it’s not enough just to be correct. A lot of people will be interacting with your code once you’re on the job, so it should be readable, maintainable, and extensible where appropriate. If your solution is clean and correct, and you produced it in a reasonable amount of time without a lot of help, you’re in good shape. But even if you stumble a bit, there are other ways to demonstrate your ability. As you work, we also watch for debugging ability, problem-solving and analytical skills, creativity, and an understanding of the ecosystem that surrounds production code.

With our evaluation criteria in mind, here are some suggestions we hope will help you perform at your very best.

Before you start coding

• Make sure you understand the problem. Don’t hesitate to ask questions. Specifically, if any of the problem requirements seem loosely defined or otherwise unclear, ask your interviewer to make things more concrete. There is no penalty for asking for clarifications, and you don’t want to miss a key requirement or proceed on unfounded assumptions.
• Work through simple examples. This can be useful both before you begin and after you’ve finished coding. Working through simple examples before coding can give you additional clarity on the nature of the problem — it may help you notice additional cases or patterns in the problem that you would otherwise have missed had you been thinking more abstractly.
• Make a plan. Be wary of jumping into code without thinking about your program’s high-level structure. You don’t have to work out every last detail (this can be difficult for more meaty problems), but you should give the matter sufficient thought. Without proper planning, you may be forced to waste your limited time reworking significant parts of your program.
• Choose a language. At Palantir, we don’t care what languages you know as long as you have a firm grasp on the fundamentals (decomposition, object-oriented design, etc.). That said, you need to be able to communicate with your interviewer, so choose something that both of you can understand. In general, it’s easier for us if you use Java or C++, but we’ll try to accommodate other languages. If all else fails, devise your own pseudo-code. Just make sure it’s precise (i.e. not hand-wavy) and internally consistent, and explain your choices as you go.

While you’re coding

• Think out loud. Explain your thought process to your interviewer as you code. This helps you more fully communicate your solution, and gives your interviewer an opportunity to correct misconceptions or otherwise provide high-level guidance.
• Break the problem down and define abstractions. One crucial skill we look for is the ability to handle complexity by breaking problems into manageable sub-problems. For anything non-trivial, you’ll want to avoid writing one giant, monolithic function. Feel free to define helper functions, helper classes, and other abstractions to reach a working solution. You can leverage design patterns or other programming idioms as well. Ideally, your solution will be well-factored and as a result easy to read, understand, and prove correct.
• Delay the implementation of your helper functions. (this serves a corollary to the previous point) Write out the signature, and make sure you understand the contract your helper will enforce, but don’t implement it right away. This serves a number of purposes: (1) it shows that you’re familiar with abstractions (by treating the method as an API); (2) it allows you to maintain momentum towards the overall solution; (3) it results in fewer context-switches for your brain (you can reason about each level of the call stack separately); and (4) your interviewer may grant you the implementation for free, if he or she considers it trivial.
• Don’t get caught up in trivialities. At Palantir we are much more interested in your general problem solving and coding abilities than your recall of library function names or obscure language syntax. If you can’t remember exactly how to do something in your chosen language, make something up and just explain to your interviewer that you would look up the specifics in the documentation. Likewise, if you utilize an abstraction or programming idiom which admits a trivial implementation, don’t be afraid to just write out the interface and omit the implementation so you can concentrate on more important aspects of the problem (e.g., “I’m going to use a circular buffer here with the following interface without writing out the full implementation”).

Once you have a solution

• Think about edge cases. Naturally, you should strive for a solution that’s correct in all observable aspects. Sometimes there will be a flaw in the core logic of your solution, but more often your only bugs will be in how you handle edge cases. (This is true of real-world engineering as well.) Make sure your solution works on all edge cases you can think of. One way you can search for edge-case bugs is to…
• Step through your code. One of the best ways to check your work is to simulate how your code executes against a sample input. Take one of your earlier examples and make sure your code produces the right result. Huge caveat here: when mentally simulating how your code behaves, your brain will be tempted to project what it wants to happen rather than what actually says happen. Fight this tendency by being as literal as possible. For example, if you’re calculating a string index with code like str.length()-suffix.length(), don’t just assume you know where that index will land; actually do the math and make sure the value is what you were hoping for.
• Explain the shortcuts you took. If you skipped things for reasons of expedience that you would otherwise do in a “real world” scenario, please let us know what you did and why. For example, “If I were writing this for production use, I would check an invariant here.” Since whiteboard coding is an artificial environment, this gives us a sense for how you’ll treat code once you’re actually on the job.

As an addendum, here are a few suggestions for books we like about the art of software construction:

Clean Code: A Handbook of Agile Software Craftsmanship – Robert C. Martin
Code Complete: A Practical Handbook of Software Construction – Steve McConnell
The Practice of Programming – Brian Kernighan, Rob Pike
Design Patterns: Elements of Reusable Object-Oriented Software – Erich Gamma, et al.
Effective Java – Joshua Bloch

One thing about being a developer on the Palantir Finance product that doesn’t get nearly enough publicity is the fact that we have our own programming language. I’m pretty excited about it so let me repeat, with emphasis: we have our own programming language. Yeah, it’s awesome. All those late hours you spent in the lab working on your final project in compilers: turns out they’re actually good for something other than getting into grad school.

Building this language ourselves — as opposed to, say, using an existing language that already just works — wasn’t an easy decision. In fact, it wasn’t even a single decision. We wracked our collective brain dozens of times trying to think of a better approach. But every which way we sliced it, the problems we needed to solve always pointed to building our own language. I still question this decision sometimes, but on the whole I’m very happy with how things have turned out.

The Palantir Finance programming language — Hedgehog as we know it — is an interpreted, statically typed, object-oriented language. With a syntax that’s based loosely on Java, it mixes roughly Java-style semantics and a few idiosyncrasies that make it a really interesting case study in language design. It’s built to be extremely efficient for batch operations on time series, which is the heavy lifting in financial analysis. It also allows you to dynamically add methods to a class from outside the class itself (conceptually similar to Ruby’s Mixins) — you define the function and its input type, and when you type the dot operator, your new method is auto-completed alongside all the “native” methods. Hedgehog also has a vast number of effectively global constants: all the stocks, bonds, and other financial instruments that are essential to the user experience, but that make for quite a design challenge.

I’m not a language guy myself, so instead of continuing to geek out over the core language features, I want to geek out about an emergent property that’s truly unique to the Hedgehog language. But first I’m going to back up and talk about something else that’s really important to us at Palantir: user experience. (I’ll get back to languages I promise.)

There’s a UX principle that says your interface should be “low threshold, high ceiling”. That is, it should be easy for the user to get started, but also able to do powerful things. This is actually a corollary of a more general principle: that your interface should strive for the optimal learning curve. My first CS professor explained this with a set of three diagrams, each representing one of the major OS families. I don’t remember exactly how he drew these diagrams at the time, but an updated version of them might look like this:

The x-axis of each curve represents “wizardry,” a measure of the user’s technical sophistication. The y-axis represents the power of the system — how much the user can accomplish at a given level of wizardry.

The best of the three curves, my prof argued, was the third curve. The first learning curve is great for providing incentives to learn. Each unit of effort spent to increase your wizardry yields an appropriate amount of reward or power. The drawback is that it’s hard for new users to do anything useful; its reward threshold is too high. The middle curve has a lower threshold and is better for novice users, but will frustrate an intermediate user because of the great plateau in the middle. (This might represent a place where the GUI isn’t powerful enough for the tasks you want to accomplish but scripting is still too difficult, leaving no way to express your commands) The third curve, however, is the best of both worlds: a low threshold and a smooth trajectory to the top.

Now let’s apply this back to our topic at hand, programming languages. Specifically, what does the learning curve look like for learning a first language? (Once you’ve learned one, of course, the rest come pretty easily.)

If your experience of learning to program was anything like mine, the first few projects in your first language were painful. You could sense the power further up the curve — it’s what convinced you to stick with CS — but simple tasks took a lot more effort than they should have, at the beginning.

Hedgehog on the other hand — our little homebrew that will someday have its own Wikipedia page — has the smoothest learning curve I’ve ever seen in a programming language. That’s the emergent property I wanted to talk about, because it’s a thing of beauty. You can get started with Hedgehog right away and accomplish quite a bit — without even knowing that you’re “programming” and the slope on the curve stays relatively constant throughout your trajectory.

We didn’t realize it at the time, but we were probably destined to create a low-threshold, high-ceiling language with a smooth learning curve, due to the nature of our user base. Financial analysts are impatient, and they still need to perform many kinds of complicated analysis. They definitely don’t have the time or inclination to spend a semester learning how to program. The solution to their problem is Hedgehog.

Allow me to illustrate with one of the earliest things a user might type into the expression bar:

And that’s it. The user types a ticker symbol and he gets a chart of IBM’s stock price. At no point did he have to wonder about variables or types or #includes. This experience is so frictionless he probably doesn’t even realize he’s writing code in a programming language. He just starts with what he knows, and the system gives him what he wants.

It starts to get interesting as you move further up the curve. Take this user input:

Of course that innocent dot between “IBM” and “volume” means a method invocation to anyone who’s familiar with C++ or Java. But to a new Palantir Finance user it simply means, “Let me access all the types of data associated with IBM.” Conceptually painless.

Or how about this one?

The volume/1000 expression is an anonymous method acting in the scope of a Stock object; it’s syntactic sugar for return this.volume()/1000;. But by allowing the user to strip away all the unnecessary syntax, we make learning the language that much easier.

I could go on tracing the curve here (I’ve only scratched the surface), but I hope I’ve made my point: we coax new users into writing code by making it look as much as possible like performing operations that they already intuitively understand. This is one of the benefits of creating a domain-specific language — we got the richness of the domain for free, and all the understanding that comes with it — and then we went above and beyond the simplification of a traditional DSL to really pare down the complexity of the language for novice users.

From simple beginnings like the ones I’ve shown here, it doesn’t take our users long at all to cross the threshold to more intermediate-level work, such as chaining function calls together or creating their own methods. As far as the high ceiling goes, we’re still working on it, but the language is currently capable of producing not only a quine, as one of our candidates showed us (yes, we ended up hiring him), but also code that can generate studies like the one below:

So Hedgehog has a low threshold and a smooth learning curve, and the ceiling is high enough that our users can do some really serious information processing with it — tasks that would make their other tools break down and cry. But there’s still a lot of interesting work for us to do, especially in pushing the language’s ceiling higher (developing better interactive debugging; working with large objects efficiently) — and as always, making it faster.

If you’d like to see the Hedgehog Programming Language in action, you can sign up for an account at Palantir JoyRide. the Palantir Finance public demo.

Cyber security is a hot topic, especially in national security circles. The world has witnessed a number of high-profile incidents in the past two years that have been notable for sharing three very important aspects:

• they were targeted attacks, carried out against specific institutions
• they were politically motivated, and, inconclusively, appear to be state-sponsored
• they used multiple-step, multi-vectors attacks and managed to evade existing security countermeasures

This deviates from the types of attacks that IT-centric approaches have sought to defend networks against. Traditional approaches neutralize the perceived threats against a network with a host of countermeasures: firewalls, malware scanners, automated network vulnerability scanning, patch policies, and intrusion detection systems. The network defenses can learn new tricks when the administrators update the signatures, or, for certain types of data, employ a Bayesian inference strategy (as has been employed to fight spam). This approach does a good job of protecting against untargeted attacks as well as weak targeted attacks.

Full network defense requires human analysts looking at anomalies at a level above the automated countermeasures. Check out the rest of this post to take a look at how human-driven, computer-aided analysis is a game changer in cyber security.

A classic doctrine: the immune system

If you’ve worked in network security, you’re undoubtedly familiar with most (if not all) of the countermeasure systems listed above. The question we don’t often ask is:

What is the defensive doctrine being employed by this security architecture?

Classic network security can be summed up as this philosophy:

Become unattractive as a target-of-opportunity to the legions of script kiddies and somewhat more sophisticated opportunists who search for network defenses they can easily breach.

The goal of the IT-based approach is to be a tougher nut to crack than the network next door. Attackers throw themselves against the defenses, find no exploitable vulnerabilities and move on to the next target-of-opportunity.

As the old joke goes: when a tiger attacks your safari group, you don’t have to run faster than the tiger, you just need to run faster than your friends. We might rewrite that today as: when the ‘l33t h4cker comes a’knocking in your network neighborhood, just make sure that you’re less of a n00b than the next guy and you’ll probably avoid getting pwned too hard.

And so we’re faced with this reality: today’s state-of-the-art network defense is a patchwork system of automated countermeasures designed to stop dumb, undirected, automated attacks. This architecture is not unique to cyber security — it has a close analog in biology.

The human immune system produces antibodies that recognize and defend against specific attacks; it learns over time through successful defense of the organism and, more recently, vaccinations. Millions of bacteria and viruses are foiled every day by immune systems. We can observe this same pattern in cyberspace: hijacked systems tirelessly scour the Internet’s address space, looking for hapless networks ripe for takeover. The Pentagon is probed something like 250,000 times a day.

It would be insanity to connect a network to the modern Internet without security countermeasures in place to defend against these sort of attacks. However, while they are necessary to the task of securing a network, they are certainly not sufficient.

Targeted attacks: slipping past the immune system

The Lifecycle of Hookworm

The countermeasures discussed thus far are essential but not infallible and can be bypassed by things like never-before-seen viruses or carefully crafted penetration attempts. In the biological domain a targeted attack might come in the form of HIV (evolved to slip past the immune defenses), a toxin (non-biological, nothing the immune system can do), or a parasite.

The original crafty adversary

A parasite can survive and thrive inside its host while evading or suppressing the normal immune response to invaders . They take up comfortable residence inside the body of their host, using it as source of food and protection; finally, they use the host as a place to reproduce and spread to other individuals in the host species. Parasites don’t generally kill or gravely harm their hosts (or at least they don’t do it quickly), as it’s in their own self-interest to have the host continue living.

Targeted parasite networks: GhostNet and the Shadow network

Cyber analog? You betcha: Vint Cerf was quoted just last week, “The hackers don’t want to destroy the network. They want to keep it running, so they can keep making money from it.”

The Citizen Lab, a University of Toronto-based non-profit that does in-depth, hands-on, technical research in the cyber security domain had this to say:

Crime and espionage form a dark underworld of cyberspace. Whereas crime is usually the first to seek out new opportunities and methods, espionage usually follows in its wake, borrowing techniques and tradecraft.

That’s in the foreword from their recent report, “Shadows in the Cloud: Investigating Cyber Espionage 2.0“. The report details their experiences tracking down the size, scope, and tradecraft behind a massive cyber-espionage botnet, dubbed GhostNet:

Tracking GhostNet: Investigating a Cyber Espionage Network [their first report on this botnet] was the product of a ten-month investigation and analysis focused on allegations of Chinese cyber espionage against the Tibetan community. The research entailed field-based investigations in India, Europe and North America working directly with affected Tibetan organizations, including the Private Office of the Dalai Lama, the Tibetan Government-in-Exile, and several Tibetan NGOs in Europe and North America. The fieldwork generated extensive data that allowed us to examine Tibetan information security practices, as well as capture evidence of malware that had penetrated Tibetan computer systems. We also engaged in extensive data analysis and technical investigation of web-based interfaces to command and control servers that were used by attackers to send instructions to, and receive data from compromised computers.

The report documented a wide ranging network of compromised computers, including at least 1,295 spread across 103 countries, 30 percent of which we identified and determined to be “high-value” targets, including ministries of foreign affairs, embassies, international organizations, news organizations, and a computer located at NATO headquarters.

These attacks used carefully forged email attacks, known as spearphishing, to entice their targets to unknowingly infect themselves with remote control software. The infections allowed the attackers to exfiltrate data from compromised machines and use them as springboards to attack other systems using similar targeted attacks. Sound familiar?

A New Doctrine: The Doctor

Without an immune system, we’d be dead within hours; our immune system is absolutely necessary but, again, not sufficient to keep us healthy. For those things that the immune system can’t take care of, we use doctors. Doctors are adaptive adversaries to disease: they can run tests, they can talk to the patient, they can apply insights learned from other patients or diseases. Most importantly, a doctor has a much more omniscient view of the patient than the immune system.

Network Security – a 10,000 ft. discipline

Applying this approach to the network enables security responses that can actually counter targeted attacks. A security officer (our network’s “doctor”) starts an investigation with some sort of anomalous event, a unexpected IP address in a log, an alert from intrusion detection system.

Remember that a runny nose or flagged packet is not an illness or a network compromise, it’s a symptom. Symptoms suggest causes, but are only clues. Taken in isolation, they don’t often offer conclusive information on the health of the patient. In fact, finding the root cause of a symptom (a diagnosis) requires the synthesis of multiple sources of data into a complete, coherent picture of the network or patient. This often includes things that you can’t see in the blood or packet stream, like understanding where the patient or user has travelled, what environmental factors might be present in their home, existing allergies, open wireless networks, insecure web apps, drug use, etc.

Node health vs. network health

A node gets an infection on your network? Re-image it, the symptoms go away. In the domain of human medicine, re-imaging of humans when they get sick has not yet gained FDA approval – something doctors have been uttering oaths about since way before the days of Hippocrates.

But it’s not the symptoms we’re after, it’s the root cause. Couple that with how easy it is to treat the symptoms via re-imaging, and security officers are more akin to public health officials, more concerned about the overall health of the network than the health of a single node. This broader concern manifests as an instant list of begged questions about any security anomaly on the network:

• How did this happen? Was it a machine (network exploit) or human vector (somebody clicked on something they shouldn’t have)?
• What is the extent of this infection? Is it limited to a single node? Why does this small moon appear to have a tractor beam locked on to our ship?
• Is this part of a larger attack? What is the true target of this attack? Is this a trap?
• Do the tracks lead out of or deeper into my network? Was this an inside job? Did I find an intermediary node in a multi-node penetration?
• Who is behind this attack and why do they want in? Can I match this modus operandi with any other known attacks on this or other networks?
• How do I prevent this sort of attack in the future? Do I need to deploy new countermeasures, re-architect parts of the network, and/or teach my people to be more careful?

The answer to any of these questions does not appear in a single log file on your network, no more than any single antibody can tell you that the H1N1 flu you’re now infected with came from the grocery clerk who got it from her boyfriend who, in turn, acquired it on his recent trip to Mexico.

The trees don’t know how big the forest is.

Cyber security doctors

A doctor examines a boy looking for hookworm.

The way to find the answers to these questions is to give a skilled, experienced analyst powerful tools to use against all the data about the attack on all of the systems on your network mashed up with relevant data about the messy meatspace that contains the computers, users, and attackers in question.

You need firewall logs, intrusion detection system logs, malware detection logs, badge logs to determine who had physical access to the network, travel records of where you expect your employees to be logging into the VPN from, and a dozen other sources of data that are unique to this network.

The data is not enough — they must to be accessible in a way that enable expedient analysis. In most shops, many of the aforementioned data sources exist, but accessing and cross-referencing them requires a high-level of technical fluency in the storage systems themselves, even for a user that has strong grasp of the story that the data are telling. Some combination of SQL, shell, grep, awk, sed, perl, and Mk I Eyeball are used to suss out answers from the data. It’s a slow, fragile, error-prone game, and the bar is high to even begin playing.

Whenever computers are recording information about the activities of other computers, the data gets big and it gets big fast. For example, grep is a very powerful and flexible tool, but its linear search through data starts to falter as the data size exceed about 10 GB on rotational media.

In order address and solve these sorts of problems, the world needs a platform with the following properties:

• Has access to all known information about a given incident
• Makes querying and exploring relationships conceptual and interactive
• Scale to handle large data sizes

It probably looks something like this:

Palantir is an intense place to work. There are people here around the clock (since developers set their own schedules) and folks and equipment arriving and leaving all the time. We’re a very focused bunch, trying to change the world as fast as we can by creating a whole new class of tools.

However, we’re not just people who build software; we’re sons and daughters, mothers and fathers and citizens of our community. As we headed into the holidays, Palantir employees decided to give something back: we signed up with a local organization call The Family Giving Tree, a now national charity that started as an MBA project out of San Jose State University.

The Family Giving Tree is unique in that it allows children to request the presents that they want. In this way, rather than putting money into a black box of a charity, you purchase the gift itself and donate that.

The people of Palantir Technologies purchased over 100 gifts, fulfilling the holiday wishes of the children that asked for them as well as cash donations that will buy gifts for at least 40 more.

The pile of presents collected by Palantir employees for The Family Giving Tree.

Happy Holidays, everyone! We’ll be back next year with more technical articles and information about Palantir.

On Oct. 9th, Palantir hosted our quarterly Government Conference in the DC area. The idea was to bring together customers of Palantir Government from across the defense and intelligence community to create a forum for them to:

• Talk candidly about their experiences using Palantir
• Discuss the many different domains they apply our technology against, everything from cyber defense to counter-terrorism to counter-proliferation
• Share experiences deploying our large distributed systems
• Learn about and see what new features and capabilities are in the pipeline for our next quarterly release

Most of the conference time is allocated to our government customers to present information on how they are using Palantir to provide deep mission impact. While this is only the second conference we have held using this open, customer-focused forum, nearly 200 people attended.

The speakers included:

• Lt. Col. Robert “Pic” Piccerillo (ret), from the Counter IED Operations Integration Center (COIC)
• David Arsenault, Assistant Department Head at MITRE
• Mike Jennings, an intelligence analyst from the FBI

In addition to presentations/demonstrations from our customers, there were several presentations of new functionality and demos by us—including demonstrations of our:

• Application platform, which allows customers to easily extend Palantir’s frontend by writing applications and helpers that embed in our platform framework
• New geospatial capabilities, including geosearch, geotagging, and other integrated workflows not seen elsewhere
• PalantirWeb—the new Palantir thin client/web frontend for expanded organizational integration

We also had a very special presentation from Jeff Carr, author of the IntelFusion blog. Jeff launched an open-source intelligence effort to analyze the actors and nature of the cyber war launched against Georgia that paralleled the Russian invasion called Project Grey Goose. Jeff presented some very compelling analytic tradecraft used in and some preliminary results from Project Grey Goose. The iteration 1 report comes out next week!

All in all, the conference went extremely well: it was gratifying for the Palantir team to see some of the innovative uses of the product. When your users are surprising and delighting you with the depth and quality of analysis they’re presenting back to you, you know you’re building and selling the right platform to truly change the way that people relate to data.

We’re witnessing the end of the data age and the first sparks of the age of analysis.

The following is my attempt at describing what we do at a high level without oversimplifying. I hope that after reading this a candidate will ‘get’ what we’re about, or at least understand enough not to apply tiny labels to our expansive vision.

The problem: implementing analysis

At Palantir we specialize in analysis.

Yes, that’s painfully abstract, and I’ll get to it in a second.

In real-world terms, we are building a software platform that enables people to take whatever data is relevant to them and understand it more easily and thoroughly than ever before, using concepts that they already understand. And we are applying this vision, at first, to solving problems in the finance sector and the government intelligence community.

The first important thing to note is that we don’t actually do the analysis ourselves. We don’t devise winning trading strategies and we don’t catch terrorists. We write software that enables other people to pull off these feats. These people, experts in their respective fields, are called analysts.

So what exactly do analysts do? What is analysis?

Analysis is everything necessary to extract insight from information.

Let’s break that down a bit.

Information is easy: It’s data. It lives in a relational database or as files indexed on a hard drive, and you can easily run queries against it. It comes in two forms, structured and unstructured. And there is a lot of it in the modern world – too much, actually, for current tools to make sense of.

Insight is trickier. Insight is something only a person can generate, and understanding this is critical for any organization that wants to do analysis right. Thus the challenge of data analysis is how to bring vast amounts of information into productive contact with human intelligence. In other words, the challenge is how to enable the analyst.

From the analyst’s perspective there are five essential features of an analysis platform:

1. First, and most important, the analyst should be in control. In other words, the primary way of interacting with an analysis tool should be human-driven queries. While automated approaches can complement a human-driven approach, there simply is no substitute for human intelligence. Unless you put a person behind the wheel, the system can never be flexible or creative enough to uncover truly original insight. Artificial Intelligence just isn’t there yet.
2. Ability to summarize large data sets. Some of this is what has traditionally been called data mining: the largely automated approach—using machine learning or other statistical techniques—of processing lots of data at once and extracting nuggets that capture something interesting about the data. Unlike Palantir, traditional approaches have focused almost exclusively on this aspect of analysis.
3. Ability to visualize large data sets. Here the analyst wants interesting and informative ways of viewing data graphically, to make it easier for him to digest. The analyst wants more than just a summary of the data; he wants a nuanced view of what’s going on inside these data sets: What’s the overall shape of the distribution? What are the outliers? What are important structures within the data?
4. Ability to iterate rapidly. This means enabling the analyst to ask a question, get the answer, and then quickly ask either a variant on the initial question or a follow-up question that depends on the answer to the initial question. This rapid, iterative process allows the analyst to quickly test out hypotheses and develop theories about what’s going on in the data, and by extension to discover what’s going on in the world.
5. Ability to collaborate with other analysts. Getting a handle on a terabyte of data, especially when it comprises multiple data types, is definitely more than a one-person job. Any organization that’s serious about understanding the world needs a team of analysts that can work together as more than the sum of its parts. This requires the ability for one analyst to effortlessly share the results of his analysis with his colleagues.

The Palantir approach

That’s what analysis looks like to the analyst, or rather what it should look like in an ideal world. (Current tools fall far short of this vision.) So what do we do at Palantir in order to make analysis this smooth and easy?

You could say that we help summarize large data sets, in the sense that we have to provide the analyst with a rich library of techniques and algorithms. You could also say that we do visualization, in the sense that we have to provide the analyst with a set of interesting and informative ways of visualizing their data. We do both of these things, and we have to be creative and solve hard problems in order to add value in these areas. But we do a lot more than that.

Probably the most central hard problem that we address in trying to enable the analyst is data modeling, the process of figuring out what data types are relevant to a domain, defining what they represent in the world, and deciding how to represent them in the system. At Palantir we make sure our data model (ontology) is both flexible and dynamic, and that it mirrors the concepts people naturally use when reasoning about the domain. This is no small challenge, but we’re already making it a reality. In finance our basic data types include financial instruments, dates, portfolios, indices, and strategies—the same things that financial researchers think about, talk about, and reason with. In the intelligence product our basic data types include people, places, and events (all with associated properties), which is exactly the way we all represent the world in our minds.

Data modeling, data summarization, and data visualization are the core disciplines for approaching large data sets. Human-driven queries, rapid iteration, and collaboration are multipliers, taking the power unlocked by the core disciplines to the next level. When these pieces are brought together in a coherent system, the result is in an analysis platform both very generic and very powerful.

This is what we mean when we say that we’re changing the way people approach data. Welcome to the future of analysis.