Palantir has entered the 2008 VAST Challenge. We present an in-depth look at one of our challenge solutions as the first public example of the Palantir platform in action.

Two years ago, the IEEE began an annual conference called VAST (Visual Analytics in Science and Technology). The VAST symposium focuses on the fundamental research contributions and real-world application of visual analytics. As a part of the conference, the VAST Challenge allows teams to compete on delivering analytic solutions against a synthetic real-world dataset.

Each year, the organizers build a very vast (pun intended) dataset from scratch. The data is entirely fictional but mirrors real-world use cases and scenarios. This year’s dataset is about a new religious movement that started on an imaginary Caribbean island (cleverly titled Isla del Sueño, or “Island of Dreams”) situated between Florida and Cuba. There are four subsets of the synthetic data: the Wikipedia page for the movement and its associated edit and discussion pages; landing and Coast Guard interdiction records for boats leaving Isla del Sueño for Florida/Mexico; cell phone records from the island; and RFID tracking data from people in a building that was attacked with an IED.

The types of questions asked in the problem sets are qualitative questions that require answers backed by data. These are the sorts of questions that don’t yield answers using a machine-learning/data-mining approach nor can an unassisted human get these answers by simple inspection of the data. They require some sort of human-computer symbiosis to solve.

To solve the VAST problems, we assembled an ad-hoc team of analysts — composed of a mix of engineers, in-house professional analysts, and one senior executive — and asked them to use the Palantir Government software to extract insights from the data.

The results speak for themselves: the complete set of Palantir’s VAST solutions are available here.

Read on for an in-depth look at how we deconstructed and solved one of the problems.

(more…)

About 10 months ago, we released set of nine screenshots from our applications. Time has passed and we have not stopped working; the look of the applications has evolved. Here are some updated screenshots:

 
 

(more…)

Raw data is an increasingly abundant and inexpensive commodity. Intelligently filtering, analyzing and visually understanding data is where the value is. Palantir invents technology and products that enables human analysts to harness the power of computers in an intuitive way to quickly and deeply analyze large amounts of data.

The value of data analysis as a career was recently emphasized by Hal Varian in the Freakonomics blog in The New York Times. Hal is an internationally known economist who is currently serving as Google’s Chief Economist while on leave from his three professorships at the University of California at Berkeley.

Q: Your job sounds extremely interesting. What jobs would you recommend to a young person with an interest, and maybe a bachelors degree, in economics?

A: If you are looking for a career where your services will be in high demand, you should find something where you provide a scarce, complementary service to something that is getting ubiquitous and cheap. So what’s getting ubiquitous and cheap? Data. And what is complementary to data? Analysis. So my recommendation is to take lots of courses about how to manipulate and analyze data: databases, machine learning, econometrics, statistics, visualization, and so on. [emphasis added]

Following the discovery that our offices were the birthplace of Java (or least the place where it had its childhood), I invited James Gosling to come visit. For those that don’t know who James Gosling is, he’s more-or-less the father of Java. Java started as a project of James Gosling’s in 1991; today, 17 years later, he’s still at Sun, in charge of guiding the Java platform into the future.

How does one invite such a luminary to come visit one’s offices? One guesses what his email address is and sends him an email out of the blue:

James,

My name is Ari Gordon-Schlosberg, an engineer at Palantir Technologies. I recently became interested in the storied history of our current facilities at 100 Hamilton Ave. in Palo Alto. As Java programmers, our engineering team is really excited to be working in the same place that gave the world Java.

You may not have heard of Palantir, but we’re working on some pretty interesting problems, using Java to build large-scale analysis applications that really push forward the state-of-the-art. We’ve won some accolades for our use of Swing by Romain Guy. If you felt like dropping by the next time you’re in the valley, we’d love to have you come by, see your old digs, and take a peek at what we’re working on.

Sincerely,

Ari Gordon-Schlosberg

To quote the Microsoft Program Manager’s book of proverbs: 90% of making things happen is sending email.

So James dropped by one Thursday for demos, lunch, and schmoozing with our engineers.

The first order of business was to demo our software to James. We got a bunch of the senior engineers together and showed him an abbreviated demo of both Palantir Government and Palantir Finance. We focused less on the problem-space aspects of the software and more on how we’re using Java to build the application. We went over how both of our apps are completely written in Java and that our GUIs are built with custom Swing components.

The most memorable part of the conversation went something like this:

LEAD DEV: So… what do you think of our applications?

GOSLING: It makes me want to weep.

LEAD DEV: Uh… ?

GOSLING: Yeah, we’ve been working on this infrastructure for years to be able to build applications like this and finally someone is doing it.

The rest of the visit was spent talking about Java, its history and its future. Topics ranged from why it’s hard to get dinosaurs like cable companies and mobile carriers to use modern technology to some of the complication in building an optimizing JIT compiler.

After lunch, I walked him to the elevator to see him off. We said our goodbyes and he stepped into the elevator, which was already occupied by the mailman making his rounds. As the doors closed, I hear the mailman say to James:

“Well, I haven’t seen you around here in a while.”

Palantir started in a very small office on Sand Hill Rd. For a time (while between offices), all work was done in a founder’s bedroom. In late 2005, the Palantir moved to medium-sized, nondescript suite on Page Mill Rd., just across the street from Hewlett-Packard in south Palo Alto. I joined a few months later, as the fifteenth employee. Since that time, we’ve sextupled in size and we’re currently pushing a triple-digit headcount!

We almost ran out of space last year, but managed to find a fantastic office space in downtown Palo Alto, at the intersection of Hamilton and Alma. At the time we acquired the space, we were told that it had been the home of Digital Equipment Corporation’s Western Research Lab. I mentioned this to an acquaintance of mine who mentioned, offhandedly, that this same set of offices was where Java was developed.

A few weeks ago, I got curious about this potentially dangling reference and decided to see if I could transmute it from rumor into fact. It sure would be neat if the fundamental technology that underlies our software was first developed in the very offices in which we work.

Some quick research confirmed the rumors and they’re all true! After the jump, the history of 100 Hamilton Ave.
(more…)

I often ask candidates if they’re familiar with what we do at Palantir. Most people think they are. “Oh, you’re that data viz. company,” or, worse, “You guys do data mining, right?” At least they’ve heard of us and at least they’re on the right track, but I cringe anyway. We aren’t just a “data visualization” company and we don’t do “data mining.” It’s almost impossible to convey the scope and complexity of what we do in a few short minutes—or to do so without taking the conversation to an eye-glazing level of abstraction.

The following is my attempt at describing what we do at a high level without oversimplifying. I hope that after reading this a candidate will ‘get’ what we’re about, or at least understand enough not to apply tiny labels to our expansive vision.

(more…)

We recently had a visit from some distinguished guests. Chris Campbell, a member of the Java 2D Team at Sun, came to see demonstrations of the Palantir products. We were very pleased and flattered by his positive reactions to the work that we’ve done.On the basis of that visit, we were added to the Java Desktop community site as a Swing Sighting Preview and merited a mention on Romain Guy’s blog: Another Pretty Java Application.We’re excited to present this series of screenshots as the first public unveiling of the Palantir applications. On a technical note: everything you see in these screenshots is from live, running applications. The applications are entirely written in Java and the GUIs are composed of custom Swing components.After the jump: bigger thumbnails with a description of each screenshot. You can click above or below to see the full resolution screenshots. (more…)

A couple of weeks ago, six of us from Palantir (five engineers and one director of business development) headed up to Seattle to host a series of unique events. We invited all of our friends, former colleagues, and schoolmates who reside in Seattle to come see what we were working on and hear about Palantir’s vision for changing the way that people relate to data. It was an exciting trip for us as it was the first public presentation of our software to a technical audience and showcased all the work that we’ve been doing.

The trip ended up being a lot of fun. We rented out facilities at the fantastic Hotel 1000, where we were served dinner and showed our guests a presentation on Palantir. After a question-and-answer session and dessert, we adjourned to Marcus’ Martini Heaven for drinks and further discussion.

People were impressed. For those of us on the trip, all the positive feedback was wonderful to hear. We’ve been working hard on these products for over a year, and it’s great validation to share your work with your peers and have it warmly received.

Hit the link for some pictures from the trip.
(more…)

Here at Palantir, Charles Cooper’s recent piece on CNET about J. C. R. Licklider has struck us as a very timely piece of journalism.

Licklider was an very influential man, with Cooper even crediting him for the existence of Computer Science as a modern-day field:

Until Licklider began his work at ARPA, there were no Ph.D. programs in computer science at American universities. That changed after ARPA began handing out grants to promising students, a practice that convinced MIT, Stanford, UC Berkeley and Carnegie Mellon to start their own graduate programs in computer science in 1965. Maybe that should go down as Licklider’s most lasting legacy.

In the piece, Cooper references this influential and well known work by Licklider: Man-Computer Symbiosis, by J. C. R. Licklider, published in IRE Transactions on Human Factors in Electronics, volume HFE-1, pages 4-11, March 1960.

That’s right, it was written almost 50 years ago. That said, it’s incredibly relevant today, perhaps more than ever.

Here’s the abstract:

Man-computer symbiosis is an expected development in cooperative interaction between men and electronic computers. It will involve very close coupling between the human and the electronic members of the partnership. The main aims are 1) to let computers facilitate formulative thinking as they now facilitate the solution of formulated problems, and 2) to enable men and computers to cooperate in making decisions and controlling complex situations without inflexible dependence on predetermined programs. In the anticipated symbiotic partnership, men will set the goals, formulate the hypotheses, determine the criteria, and perform the evaluations. Computing machines will do the routinizable work that must be done to prepare the way for insights and decisions in technical and scientific thinking. Preliminary analyses indicate that the symbiotic partnership will perform intellectual operations much more effectively than man alone can perform them. Prerequisites for the achievement of the effective, cooperative association include developments in computer time sharing, in memory components, in memory organization, in programming languages, and in input and output equipment.

This description is still a pretty accurate description of how most analysts (in any industry or field) go about their business:

Despite the fact that there is a voluminous literature on thinking and problem solving, including intensive case-history studies of the process of invention, I could find nothing comparable to a time-and-motion-study analysis of the mental work of a person engaged in a scientific or technical enterprise. In the spring and summer of 1957, therefore, I tried to keep track of what one moderately technical person actually did during the hours he regarded as devoted to work. Although I was aware of the inadequacy of the sampling, I served as my own subject.
…
About 85 per cent of my “thinking” time was spent getting into a position to think, to make a decision, to learn something I needed to know. Much more time went into finding or obtaining information than into digesting it. Hours went into the plotting of graphs, and other hours into instructing an assistant how to plot. When the graphs were finished, the relations were obvious at once, but the plotting had to be done in order to make them so.
…
Throughout the period I examined, in short, my “thinking” time was devoted mainly to activities that were essentially clerical or mechanical: searching, calculating, plotting, transforming, determining the logical or dynamic consequences of a set of assumptions or hypotheses, preparing the way for a decision or an insight. Moreover, my choices of what to attempt and what not to attempt were determined to an embarrassingly great extent by considerations of clerical feasibility, not intellectual capability.

This quote is an eerily accurate description of how trading strategies are formulated, back-tested, and implemented these days. As analogy, it’s also an accurate reflection of the modern use of information processing in the intelligence space.

To wit:

It is to bring computing machines effectively into processes of thinking that must go on in “real time,” time that moves too fast to permit using computers in conventional ways. Imagine trying, for example, to direct a battle with the aid of a computer on such a schedule as this. You formulate your problem today. Tomorrow you spend with a programmer. Next week the computer devotes 5 minutes to assembling your program and 47 seconds to calculating the answer to your problem. You get a sheet of paper 20 feet long, full of numbers that, instead of providing a final solution, only suggest a tactic that should be explored by simulation. Obviously, the battle would be over before the second step in its planning was begun. To think in interaction with a computer in the same way that you think with a colleague whose competence supplements your own will require much tighter coupling between man and machine than is suggested by the example and than is possible today.

So what how does this relate to what we do? In the finance world, much of what fund managers and analysts do in building strategies has to do with formulating trading models and then building spreadsheets that can back test or simulate the performance of those models.

Our finance tool obviates the need for this “clerical, mechanical” work, allowing strategists to spend more time making sense of the interconnections in the market and formulating nuanced trading strategies and less time doing model-building in Excel. We take the state-of-art a quantum leap forward in terms of financial analysis: rather than even just allowing analysts to quickly build models and back test trading strategies, we’ve built a tool that allows for a smooth flow from hypothesis to theory with the software doing all the heavy lifting, data wrangling, eye-candy-class presentation. New variables or market conditions can be incorporated on the fly without the need for a pause from high-level thinking to gather data or marshal it into the right format. Knowledge can be divined by asking questions relative to high-level concepts of things like dynamic market conditions and meta-conditions like the volatility-of-volatility.

The question has traditionally been, “How do I effectively model this financial space?” With Palantir, we’re transforming that question into the core question asked in the finance industry, namely, “How can I better understand the interactions at work in today’s markets?” So the focus moves to the human-level questions while the software takes care of the data level machinations.

In the intelligence space, the composite views of data that the government team creates save the analysts from having to painstakingly research and record correlations across multiple informational domains. Instead, the analyst can spend time divining the meaning behind the connections and correlations. Our take on perpetual analytics takes things a step further, alerting the analyst as relevant new information enters the system. And finally, we’re building workflows that allow analysts to quickly attach ‘handles’ to data to allow what has been traditionally unstructured data get seat at this table of computer-enhanced human analysis.

We’re speeding up the process of analysis by creating an analyst-computer symbiosis. No longer will people need to spend time doing menial data processing, the computers will do it for them, while the humans provide the spark of insight, semantics, and cognition that computers lack.

It’s conceptual analysis at the speed of thought.

This is why I’m excited to come to work every day: we’re building the software that embodies a broad vision of the future. This vision of human computer symbiosis dates from five decades ago but is also apparent in every interaction we see with computers on the big and small screens (no, not our monitors). From Star Trek to 24, people want to the computers to do the repetitive and time-consuming simple work but let them have final say on any complex decisions. As one of our customers told us when shown our application: this is the future.

Welcome to the Palantir TechBlog. Here’s a place for our engineers to sound off about all things technical. For now, it will be mostly related to general software engineering topics with a focus on development of enterprise software in Java. This will lead us to talk about SQL, Unix, Swing tricks, scaling Java, development practices and a host of other attendant topics related to what we do.

To be clear: we’re not a J2EE shop, we’re building enterprise class software from the ground up out of Java. So what does that mean? We’re building large scale data systems that have a server component and desktop client component. Which means that we sort of touch on everything end-to-end, from making Swing performant and pretty to building custom database code to model some of the very interesting things that we’re dealing with.

Please feel free to comment, we’d love to hear from you.