“You’re asking us to test our platform’s programming language? How am I supposed to do that?”

My head itches from trying to recall the bits and pieces of what I learned in high school about programming, specifically the semantics of a programming language. Sure, I did a bit of programming for homework assignments in college, but I was no CS major. This was a much different challenge for a QA engineer to test. Compared to an application, a programming language is completely open ended; there are no specifications to test, guidelines to follow, or limits to break.

The Hedgehog language had the basic set of tools laid out for me already: I could declare variables, create data structures, and use loops for iteration. As I was trying out individual usage examples, such as how to structure if statements or how to cast an object to a different type, I realized that this was no way to test something as powerful and flexible as an entire language. It would be like a doctor who claims that since each individual organ works fine, there are no problems with the entire system. This is insufficient: one needs to look at the system as a whole, including examining the interactions between each component. I decided I needed to create much larger and elaborate code samples in order to test the Hedgehog language in a larger scope.

Using the Hedgehog language, I had programmed several algorithms, solving puzzles that would output a number. This was getting bit boring, since once the output value was matched the expected number, there was nothing more to be done. I wanted to create something more dynamic, a toy I could play around and experiment with. And opportunity presented itself in the form of one of our newest tools: the spreadsheet application. With the capability of setting the value of each cell programmatically and then coloring them depending on their value… hmm what could I do with this?

Hedgehog is a powerful tool in coding functions and workflows that directly interact with our applications. Most of the time, the language is used to write expressions for an input value, create custom metrics that return values after a set of calculations, or even to set inputs, calculate, and save documents. Given the language’s ability to integrate with Spreadsheet, the capabilities of the Hedgehog language can literally be visually shown to the user, resulting in some stunning displays. Below are three examples I’ve coded in Palantir Finance’s own language: calculating and drawing the Mandelbrot fractal, simulating Conway’s Game of Life, and solving a Nonograms puzzle.
Read the rest of this entry »

We talk a lot about Human-Computer Symbiosis on this blog – it’s a systems design approach that guides us in our construction of our technology stacks. Given that, we’re always on the lookout for example of HCS systems built by other people.

Here’s an unlikely example: the layout of names in the memorial was made according to ‘meaningful adjacencies’ (as described by Jer Thorp in his blog post, All The Names: Algorithmic Design and the 9/11 Memorial):

The project was to design an algorithm for placement of names on the 9/11 memorial in New York City. In architect Michael Arad‘s vision for the memorial, the names were to be laid according to where people were and who they were with when they died – not alphabetical, nor placed in a grid. Inscribed in bronze parapets, almost three thousand names would stream seamlessly around the memorial pools. Underneath this river of names, though, an arrangement would provide a meaningful framework; one which allows the names of family and friends to exist together. Victims would be linked through what Arad terms ‘meaningful adjacencies’ – connections that would reflect friendships, family bonds, and acts of heroism. through these connections, the memorial becomes a permanent embodiment of not only the many individual victims, but also of the relationships that were part of their lives before those tragic events.

Read on for details on the approach they used and how it embodies HCS architecture (not to mention, a video of their tool in action).
Read the rest of this entry »

Late last year, we were honored to be invited to talk at Reflections|Projections, ACM@UIUC’s annual student-run computing conference. We decided to bring a talk about Horizon, our system for doing aggregate analysis and filtering across very large amounts of data. The video of the talk was posted a few weeks back on the conference website.

Horizon started as research project / technology demonstrator built as part of Palantir’s Hack Week – a periodic innovation sprint that our engineering team uses to build brand new ideas from whole cloth. It was then used by the Center For Public Integrity in their Who’s Behind The Subprime Meltdown report. We produced a short video on the subject, Beyond the Cloud: Project Horizon, released on our analysis blog. Subsequently, it was folded into our product offering, under the name Object Explorer.

In this hour-long talk, two of the engineers that built this technology tell the story of how Horizon came to be, how it works, and show a live demo of doing analysis on hundreds of millions of records in interactive time.

Visualizing SMS hotspots in days following the earthquake in Palantir.

Screenshot courtesy of Palantir Technologies

[Editor's Note: an edited version of this post first appeared on O'Reilly's Radar blog.]

The prologue was an earthquake of unexpected magnitude and location that left 250,000 dead.

As computer scientists and technologists, we’re used to dealing with large numbers in the abstract. Expressed in human terms, the mind-boggling numbers of 250,000 dead, 300,000 injured and over 1 million people left homeless are hard to comprehend.

Hit the link to read more about how effective data management and analysis is crucial to recovery efforts and see specific examples of data about the situation in Haiti modeled in Palantir Government.
Read the rest of this entry »

Software engineering is a craft that blends science and art. This fact is easy to overlook as the artistic aspects are often eclipsed by discussions of the science and technology behind what we do.

This is not one of those times: the art in software engineering is most evident when building compelling visual interfaces, something Palantir knows a thing or two about.

A demo reel is an industry term in the movie business — a short reel that acts as a portfolio when applying for jobs, a highlight reel of the author’s visual career. We’re not in the movie business, we’re in the software business. We do, however, use moving pictures to tell stories, stories backed by data — this is our demo reel: two-and-a-half minutes of data visualization and user interface eye-candy (It has pounding music — you may want to put on headphones or turn down your speakers.):

The movie will take a few seconds to load. It’s 800×600, so expanding to full-screen is suggested. We’ve done our best to create a streamable-yet-good-looking video. The compression artifacts are there, but shouldn’t be too distracting. In a real Palantir client, there are no compression artifacts and everything looks even better than it does here.

The Palantir family of products is much more that just pretty pictures; we have the underlying intelligence infrastructure to make those realtime animations possible and (more importantly) meaningful. That said, we sure do think they’re pretty.

By the way, if you’re interested in the progression of our interfaces, this not the first time we’ve posted eye candy: we posted a set of updated screenshots a little over a year ago; think of this as the next installment in the series.

And yes, it’s really all Java Swing.

In a previous post, Eric W. covered how we analyze polled system health information. Now we’ll look at pushed information, in the form of logging events.

Use Cases & Constraints

We decided on three kinds of questions we wanted to answer:

• What is the health of the deployment?
  • Example: What errors have occurred in the last 24 hours?
• Which parts of the platform are our users engaged with?
  • Example: How much time do users spend in each application?
• How is our server performing over time?
  • Example: What is the average wait on a search query?

The chief constraint was that we build our platform on Log4J. We already use Log4J all over the project, so converting the logging was out of the question. Besides, Log4J provides a guideline for the kind of metadata our events should support, and Log4J makes it easy to record events to a database.

That left us with two problems to solve: how to store structured data with a Log4j message, and how to analyze the collected data.

Analysis is the easy part: just use Palantir! After all, a sequence of logging events has a lot in common with a time series. The rest is explained below.

Read the rest of this entry »

We put up a post last year on the 2008 VAST Grand Challenge. Well, the IEEE VAST Challenge 2009 is over and the awards are in. We had another strong year, scoring two awards:

• Grand Challenge: Analyst’s Tool Choice (Of 48 submissions, only 3 Grand Challenge awards were given)
• Intuitive Traffic Visualization and Video Description of the Analysis Process

Some background on the event: three years ago, the IEEE began an annual conference called VAST (Visual Analytics in Science and Technology). The VAST symposium focuses on the fundamental research contributions and real-world application of visual analytics. As a part of the conference, the VAST Challenge allows teams to compete on delivering analytic solutions against a synthetic real-world dataset.

A selection of choice quotes from the judges:

• An award for “highly usable integrated exploration environment”, “efficient analytic exploration platform” or something along these lines would be appropriate.
• Survey Question: How much novelty do you see in this submission (data processing, visualization, interaction, hypothesis generation or evaluation, overall process, etc.)? Answer: More so than novelty was the extremely efficient solution approach to this challenge, much more so than other solutions.
• The submission shows two things very clearly: One, it shows the analytical process as being a multi-faceted, simultaneous processing of different information that is quite common among analysts. Two, it shows how multiple perspectives can be displayed on a single monitor, enabling the analyst to visualize what his mind is analyzing. Outstanding!

Our submission

And finally, our submission to the Grand Challenge. Here we have our overview video, with a link to the full video below:

For an in-depth look at the data and techniques used to make this a reality, check out our full submission in Finding a Mole: Cyber Counter Intelligence on the Palantir Analysis Blog.

I used to think I understood MVC. In undergraduate CS programs, MVC is taught as an off-the-shelf pattern, explained once and then ready for use in the real world. Wikipedia also makes it seem pretty simple:

Model–View–Controller (MVC) is an architectural pattern used in software engineering. Successful use of the pattern isolates business logic from user interface considerations, resulting in an application where it is easier to modify either the visual appearance of the application or the underlying business rules without affecting the other. In MVC, the model represents the information (the data) of the application; the view corresponds to elements of the user interface such as text, checkbox items, and so forth; and the controller manages the communication of data and the business rules used to manipulate the data to and from the model.

They go on to show the classic triangle diagram and how it’s baked into various GUI and web frameworks. There’s only one clause in the entire article that hints at something deeper: “Though MVC comes in different flavors…”

Different flavors indeed. In fact MVC is not just a pattern but a whole family of patterns: MVC, MVA, MVP, PAC, Model-Delegate…. It very quickly gets very hairy.

In this article I want to describe one of MVC’s lesser-known variants, the Model-View-Adapter (MVA) pattern, and talk about its advantages over traditional MVC in the context of a Java Swing application.

Read the rest of this entry »

As we mentioned in an earlier post, Palantir was recently invited to the IEEE’s VisWeek in Dayton Ohio, and was honored to be invited to participate in the VAST Interactive Challenge as part of VisWeek.

After winning an award for Interactive Visual Analytic Environment, Palantir was one of three teams selected to participate in the interactive session from 73 VAST Challenge entries. For the challenge, we were given a completely new set of data to analyze. We had 30 minutes to import 3 disparate datasets into Palantir, 30 minutes to train an analyst that had never used Palantir, and then 2 hours for the analyst to explore the data.

The data for the challenge came from three different sources, with a set of questions to answer for each set of data. There was an infectious outbreak, a Wikipedia edit war, and an abduction from a city park. Over the three challenges, there were over 100,000 datapoints to analyze. All of the data revolved around a fictitious town in Florida, Barracuda Springs, and was linked to the fictitious cult that was the center of the 2008 VAST Challenge. While two members of our team were importing the three datasets, the third team member was working with our analyst (each of the three teams was given a analyst from a nearby analytical organization). In 30 minutes, our analyst was able to learn how to conduct relational, temporal, geospatial, and statistical analysis in Palantir. After the 30 minutes of training, she was able to easily navigate the Palantir workspace, and solve all three challenges. Below is her work (hit the link to check it out).

Her conclusion was that Palantir was “viciously good software” and that she would be asking her boss if they could acquire Palantir for their work. Hit the link below to see screenshots and explanations for one of the challenge workflows.

We really enjoyed the VAST Challenge, and our experience at VisWeek. There were a lot of outstanding papers, posters, and speakers at VisWeek, and we were inspired by many fantastic visualizations that might soon make their way into Palantir’s Finance and Government Platforms. We are also looking forward to the 2009 VAST Challenge!
Read the rest of this entry »

I recently had the opportunity to attend a talk by Ben Shneiderman, a big name in HCI and professor at the University of Maryland. He showed off a bunch of really cool visualizations he’s invented over the years, aimed at advancing the field of data analysis. One of the visualizations, the Rank-by-Feature framework, looked immediately useful to our product. It’s an overview of multidimensional data that uses coloring in a matrix of correlations. I decided to add it into Palantir Finance as a proof-of-concept.

We have a scatter plot in Palantir Finance, but it’s not designed to compare your data across many variables (called metrics here). I extended it by adding a small triangular matrix control that we call the scatter plot Quick Select. The control gives you a visual overview of the data, and allows you to identify interesting metric pairs and then drill down into a scatter plot for any particular pair.

Assume I want to view a set of 10 metrics (shown above). With our regular scatter plot, you have to choose two of these metrics and plot them to see their correlation. If you want to know how every possible pair of metrics correlate, you have to manually perform an O(n²) input operation.

With Quick Select, you enter the metrics once and the triangular matrix is formed. Each square in the matrix represents a pair of metrics, and each square’s color is the correlation between the pairs. For example, green is a strong positive correlation, light green a less positive correlation, white is no correlation, and red is a strong anti-correlation.

In the highlighted square above, we’re comparing percent return and correlation with the S&P 500 over the past year. The red shows a strong negative correlation. This makes sense in light of recent market behavior: the S&P 500 has not done well over the past year, so companies that were correlated with it also performed poorly, while companies that moved in the opposite direction performed well.

Drilling down to a scatter plot is as simple as clicking on the corresponding square. Below are 2 of the 45 possible scatter plots defined by this set of 10 metrics. You can quickly jump back and forth between different scatter plots while retaining a nice, condensed overview of the data in the triangular matrix on the left.

This prototype took less than a day to write. And it was written from outside the system, using only the pluggability points Palantir Finance provides. Overall it’s a powerful visualization component, added as an extension to the Palantir Finance platform, and done in under a day. Pretty cool!

Many thanks to Prof. Shneiderman for the idea.