Here at Palantir, we use continuous integration as one of our development practices. Part of this includes running automated builds and tests. This practice is quite common, and is useful because it gives immediate feedback on (some of) the software’s correctness.

How it usually works

• Developers submit changes to the code base.
• The continuous build system detects a problem and the build “breaks” (or “goes red”).
• All developers who made recent changes are notified via email.
• Someone (usually the guilty developer) locates and fixes the problem.
• The continuous build system verifies the corrected problem and “goes green”.

Ideally, the delay between failure and fix should be as short as possible. But failure notifications are typically sent by email, which can easily be overlooked or ignored. Clearly we can do better.
Read the rest of this entry »

On Oct. 9th, Palantir hosted our quarterly Government Conference in the DC area. The idea was to bring together customers of Palantir Government from across the defense and intelligence community to create a forum for them to:

• Talk candidly about their experiences using Palantir
• Discuss the many different domains they apply our technology against, everything from cyber defense to counter-terrorism to counter-proliferation
• Share experiences deploying our large distributed systems
• Learn about and see what new features and capabilities are in the pipeline for our next quarterly release

Most of the conference time is allocated to our government customers to present information on how they are using Palantir to provide deep mission impact. While this is only the second conference we have held using this open, customer-focused forum, nearly 200 people attended.

The speakers included:

• Lt. Col. Robert “Pic” Piccerillo (ret), from the Counter IED Operations Integration Center (COIC)
• David Arsenault, Assistant Department Head at MITRE
• Mike Jennings, an intelligence analyst from the FBI

In addition to presentations/demonstrations from our customers, there were several presentations of new functionality and demos by us—including demonstrations of our:

• Application platform, which allows customers to easily extend Palantir’s frontend by writing applications and helpers that embed in our platform framework
• New geospatial capabilities, including geosearch, geotagging, and other integrated workflows not seen elsewhere
• PalantirWeb—the new Palantir thin client/web frontend for expanded organizational integration

We also had a very special presentation from Jeff Carr, author of the IntelFusion blog. Jeff launched an open-source intelligence effort to analyze the actors and nature of the cyber war launched against Georgia that paralleled the Russian invasion called Project Grey Goose. Jeff presented some very compelling analytic tradecraft used in and some preliminary results from Project Grey Goose. The iteration 1 report comes out next week!

All in all, the conference went extremely well: it was gratifying for the Palantir team to see some of the innovative uses of the product. When your users are surprising and delighting you with the depth and quality of analysis they’re presenting back to you, you know you’re building and selling the right platform to truly change the way that people relate to data.

We’re witnessing the end of the data age and the first sparks of the age of analysis.

At Palantir, we write software that gets deployed at each client, integrated across their sensitive data sets, and maintained and administered by that client’s in-house admins. Most deployed enterprise software is run on a single beefy box: consider wikis, blogging systems, bug tracking systems, or practically any client/server or web client software software used today. On the other hand, most enterprise software that runs as a distributed system is hosted: Salesforce.com, Google Apps, or any approach that sells software as a service. What’s fairly unusual about our software is that it’s deployed as a distributed system at each client.

Distributed systems are hard to build and hard to maintain. As long as that distributed system is built and maintained in-house, however, you have a number of advantages:

• The administrators are full-time product experts who are focused on the mission of keeping your system available and responsive.
• The development organization can build internal tools for the administrators that only have to be “good enough” and can step in if necessary.
• It’s easy to get feedback on how the system performs, because there are no sensitivity, privacy, or legal constraints.
• A single, large deployment allows you to optimize your hardware purchasing and amortize installation headaches across a large number of machines.

This is all great, of course, and if you can host and maintain your distributed system yourself, I’d highly recommend it. Sometimes, however, it’s just not possible. At Palantir, the client data we work with is so sensitive that even we cannot see it, except under very strictly controlled circumstances. It’s also so large that the bandwidth limitations of pushing it into a system hosted by us would be prohibitive.

So suppose that you have to deploy your distributed system in a customer datacenter with external parties maintaining the system. What do you need to consider? In this post, I’ll go into a number of key points that we have faced and addressed at Palantir.

Read the rest of this entry »