Visualizing SMS hotspots in days following the earthquake in Palantir.

Screenshot courtesy of Palantir Technologies

[Editor's Note: an edited version of this post first appeared on O'Reilly's Radar blog.]

The prologue was an earthquake of unexpected magnitude and location that left 250,000 dead.

As computer scientists and technologists, we’re used to dealing with large numbers in the abstract. Expressed in human terms, the mind-boggling numbers of 250,000 dead, 300,000 injured and over 1 million people left homeless are hard to comprehend.

Hit the link to read more about how effective data management and analysis is crucial to recovery efforts and see specific examples of data about the situation in Haiti modeled in Palantir Government.

Chapter One: Rescue

There was one glimmer of hope in this sea of tragedy: the world’s reaction. In the early hours and days after the quake, the focus was on pinpointing, triaging, and rescuing those in grave danger. Since those first harrowing hours, the world has made plain its willingness to help the people of Haiti. Supplies of money, food, medicine, fresh water, and volunteers have been pouring into Haiti and fundraising efforts are on-going around the world.

Technology also played an early, crucial role, with Mission 4636, InSTEDD and Ushahidi reacting lighting-fast to create a data collection system that enabled people in trouble to quickly communicate their urgent needs to rescuers and relief workers . If you haven’t already read it, Lukas Biewald’s piece, How crowdsourcing helped Haiti’s relief efforts, is a great look at those first, and most urgent efforts to collect data and synthesize information about the situation on the ground.

Chapters Two Through Many: Recovery

The extent of the devastation in Haiti.

Photo courtesy of Marko Kokic / ICRC / American Red Cross

Unfortunately, even partial recovery in Haiti will take years at the bare minimum. U.S. Vice President Joe Biden stated on 16 January that President Obama “does not view this as a humanitarian mission with a life cycle of a month. This will still be on our radar screen long after it’s off the crawler at CNN. This is going to be a long slog.”

Building the Deep, Big Picture

The recovery from a disaster of this magnitude presents some important tasks in the sphere of information technology: coordination of effort, triaging those most in need, and getting good data into the hands of decision makers and aid workers.

Here’s a partial list of aid, relief, and rescue organizations currently in Haiti, gleaned from Wikipedia:

• An Argentine military field hospital
• The Red Cross/Crescent, in various forms
• The US military
• Multiple UN agencies
• Remnants of the Haitian government
• The French navy
• Sri Lankan relief workers
• At least 2000 rescuers from 43 different groups (along with 161 search dogs)

A wealth of collaborators like this presents some unique challenges around information fusion: unlike business competitors or opposing sides of a war, the different groups want to share as much information as possible to achieve their common goal. A unified organization, like a single national military will have pre-existing methods to model and share their situational awareness. That is not the case in Haiti: this is a collection of groups coming together to form an ad-hoc relief force. Everything from differences in human languages, database schema, collection methodology, and problem domain make most of the datasets seemingly disjoint from the others.

However, each organization has a produced a fairly detailed picture of the parts of Haiti that they are interacting with. Each organization also wants to consume every other’s organization’s detailed knowledge of the situation. To act effectively, they need to integrate that knowledge into a common operating picture that accurately models the situation on the ground yesterday, today, and tomorrow.

Analyzing the Haiti situation using Palantir Government

Our reaction to the earthquake was to try to help in the best way we knew how. We set up a publicly available instance of our Palantir Government product, already loaded with relevant data, for use by aid workers and organizations working in Haiti. Using relevant, open-source data we’ve started modeling a picture of what’s going in Haiti.

Our first cut was to include the locations and names of collapsed buildings, Internally Displaced People (IDP) camps, and Misson 4636 SMS messages, among others. We also added in map layers that let us see what administrative zone any point on the map is located in.

Having mapped the data into this model, users have access to it through a suite of visualization, analysis, querying, and collaboration tools that allow them to get useful answers to practical questions. Here are some examples:

• Which administrative sectors have had the most SMS requests for food in the past 24 hours?
• What collapsed buildings are there that may contained hazardous materials that will require special cleanup?
• Are any IDP camps near enough to these hazmat sites to warrant special precautions or moving the residents?

We’ve created a video showing all the pieces put together into a seamless whole, using live data in our publicly available Haiti instance:

The Next Chapter: Flooding

A view of the water point at the Citee Renault camp in Port-au-Prince, Haitii

Photo courtesy of Joe Lowry / IFRC / American Red Cross

From the Red Cross website:

“We’re racing against the clock with hurricane season just around the corner,” said Jean Pierre Taschereau, a Red Cross disaster expert just back from Haiti. “Getting semi-permanent structures in place as well as trenches for sanitation latrines will be critically important.”

From “Quake-ravaged Haiti faces flooding”:

The UN wants to move 200,000 people out of overcrowded camps like this one. The Haitian government is trying to find land. It’s identified five sites outside of the Haitian capital, but those five sites are about 200 hectares and by the UN’s estimates 600 hectares will be needed to house the people it plans to move safely to have proper drainage when the rainy season finally arrives.

Haiti’s rainy season is notorious for causing flooding. Now, with the infrastructure of the country destroyed, flood season will be more dangerous than usual. Not only are the normal structures that protect people from the waters gone, but they’ve moved out of the ruins of Port-au-Prince to hastily constructed IDP camps, some of which are sitting in the flood plains of Haiti’s waterways.

The essential question facing relief workers: Which of the approximately 2500 IDP camps are most at risk from flooding?

In a place like the United States, an earthquake response and recovery team could engage the services and expertise of the US Geological Survey, which maintains the National Water Information System, a warehouse of detailed information about all things water in this country. No such luck in Haiti, where the closest thing to the USGS is the Centre National de l’Information Géo-Spatiale. A quick look at their website shows that they didn’t really make it through the earthquake. (In the video, we feature a picture of what’s left of their facility — it’s not pretty).

Since we’re starting from square one we put together data from the Army Geospatial Center, the UN, NOAA, Haiti-based NGOs, a number of academic papers, and even geo-tagged photos from Flickr. The time it took to integrate this data? About six hours. Time it took to do the analysis? About seven minutes. Amount of that work that is reusable? All of it.

Check out this video for a walk-through of the analysis:

The best way to improve this analysis will be to add more detailed information about flooding, gathered from the field. We’re looking into getting new conduits of information into the Haiti instance to make this a reality as the rains really pick up.

A Call To Action

If you’d like to help us, we’re accepting new data sources, analyses, and contact with relief organizations.

Volunteers, supplies, and goodwill are only the raw ingredients to recovery; it’s the efficient and timely application of those resources to Haiti’s most pressing problems that will change lives and make recovery a reality instead of just a good intention.

This is not one of those times: the art in software engineering is most evident when building compelling visual interfaces, something Palantir knows a thing or two about.

A demo reel is an industry term in the movie business — a short reel that acts as a portfolio when applying for jobs, a highlight reel of the author’s visual career. We’re not in the movie business, we’re in the software business. We do, however, use moving pictures to tell stories, stories backed by data — this is our demo reel: two-and-a-half minutes of data visualization and user interface eye-candy (It has pounding music — you may want to put on headphones or turn down your speakers.):

The movie will take a few seconds to load. It’s 800×600, so expanding to full-screen is suggested. We’ve done our best to create a streamable-yet-good-looking video. The compression artifacts are there, but shouldn’t be too distracting. In a real Palantir client, there are no compression artifacts and everything looks even better than it does here.

The Palantir family of products is much more that just pretty pictures; we have the underlying intelligence infrastructure to make those realtime animations possible and (more importantly) meaningful. That said, we sure do think they’re pretty.

By the way, if you’re interested in the progression of our interfaces, this not the first time we’ve posted eye candy: we posted a set of updated screenshots a little over a year ago; think of this as the next installment in the series.

And yes, it’s really all Java Swing.

Use Cases & Constraints

We decided on three kinds of questions we wanted to answer:

• What is the health of the deployment?
  • Example: What errors have occurred in the last 24 hours?
• Which parts of the platform are our users engaged with?
  • Example: How much time do users spend in each application?
• How is our server performing over time?
  • Example: What is the average wait on a search query?

The chief constraint was that we build our platform on Log4J. We already use Log4J all over the project, so converting the logging was out of the question. Besides, Log4J provides a guideline for the kind of metadata our events should support, and Log4J makes it easy to record events to a database.

That left us with two problems to solve: how to store structured data with a Log4j message, and how to analyze the collected data.

Analysis is the easy part: just use Palantir! After all, a sequence of logging events has a lot in common with a time series. The rest is explained below.

Recording Structured Data

Consider the problem of plotting usage by a user on a given day. The simplest approximation is to log an event every time an application is closed, and provide the time spent in the application with that event. Posting the information as an unstructured String–e.g. “Andrew spent 46 seconds in Chart”–makes it difficult to later extract the data for analysis. To solve this problem we developed the class RichLogEntry.

RichLogEntrys contain a human-readable message and tagged data in the form of a set of key/value pairs, such as {duration, 46}, {user, Andrew}. This adds to the up-front cost since log messages become more complex, but the benefit is that the analysis engine can easily and generically access data in events.

Furthermore, RichLogEntry plays nicely with existing Log4J infrastructure. Loggers in Log4J already accept an arbitrary Object to pass to Appenders, and Log4J’s default Appenders call toString() on the Objects provided. For RichLogEntry the toString() is simply the human-readable message. So a call to the logging framework with a RichLogEntry would look like (pseudo-Java):

logger.info( (“Andrew just spent 46 seconds in Chart”,
{“duration” : 46.0, “application” : “Chart”, “user” : “Andrew”}) );


For most Appenders this would produce the human-readable String, but our custom Appender knows how to store the tagged data for later analysis.

Example: Application Usage

We implemented the above (i.e. log a “duration” message each time a Palantir application loses focus), and hooked in the data with a Palantir Data Provider plugin.

In the image below, we’re using our Explorer application to analyze the logging data. Our filter framework combines filtering and visualization into a single application. The image contains three filters from top to bottom, each containing a blue title bar. The results of each filter are fed into the filter below it:

• The top filter separates messages by application and displays statistics for each. We’ve selected the Explorer application, so its 144 messages will be fed into the next filter.
• The middle filter has a histogram of the number of seconds each Explorer instance was active (in log scale). Each “bucket” represents a range of durations, and its height is the number of messages with that duration. It looks like I usually spend around 10 seconds in Explorer before switching windows. We’re selecting the gray part of the histogram to avoid skewing our results for the times I’ve gone AFK with Palantir running.
• The bottom filter counts the number of log events over time.

In Palantir Finance, filters such as these can be saved and used anywhere in the platform. Let’s do that, and compare my usage to my coworker Eric L’s. Creating a new set of filters for Eric is easy–I just modify a single filter above to specify him instead of me (the filter isn’t shown for simplicity’s sake), and then save a new copy. Our Chart application is a good place to view the two series side by side:

Of course, I’m the harder worker!

Conclusion

Our logging framework is complete, and we’ve found many new use cases. We use the framework to:

• analyze performance metrics across builds of both Palantir products
• automatically compile usage reports on deployed installations
• import and explore exotic event data sets by running the events through Log4J

Building the Log4J analysis framework was valuable, fun and easy; and it demonstrates the flexibility of Palantir Finance for working with arbitrary data sets.

We put up a post last year on the 2008 VAST Grand Challenge. Well, the IEEE VAST Challenge 2009 is over and the awards are in. We had another strong year, scoring two awards:

• Grand Challenge: Analyst’s Tool Choice (Of 48 submissions, only 3 Grand Challenge awards were given)
• Intuitive Traffic Visualization and Video Description of the Analysis Process

Some background on the event: three years ago, the IEEE began an annual conference called VAST (Visual Analytics in Science and Technology). The VAST symposium focuses on the fundamental research contributions and real-world application of visual analytics. As a part of the conference, the VAST Challenge allows teams to compete on delivering analytic solutions against a synthetic real-world dataset.

A selection of choice quotes from the judges:

• An award for “highly usable integrated exploration environment”, “efficient analytic exploration platform” or something along these lines would be appropriate.
• Survey Question: How much novelty do you see in this submission (data processing, visualization, interaction, hypothesis generation or evaluation, overall process, etc.)? Answer: More so than novelty was the extremely efficient solution approach to this challenge, much more so than other solutions.
• The submission shows two things very clearly: One, it shows the analytical process as being a multi-faceted, simultaneous processing of different information that is quite common among analysts. Two, it shows how multiple perspectives can be displayed on a single monitor, enabling the analyst to visualize what his mind is analyzing. Outstanding!

Our submission

And finally, our submission to the Grand Challenge. Here we have our overview video, with a link to the full video below:

For an in-depth look at the data and techniques used to make this a reality, check out our full submission in Finding a Mole: Cyber Counter Intelligence on the Palantir Analysis Blog.

Model–View–Controller (MVC) is an architectural pattern used in software engineering. Successful use of the pattern isolates business logic from user interface considerations, resulting in an application where it is easier to modify either the visual appearance of the application or the underlying business rules without affecting the other. In MVC, the model represents the information (the data) of the application; the view corresponds to elements of the user interface such as text, checkbox items, and so forth; and the controller manages the communication of data and the business rules used to manipulate the data to and from the model.

They go on to show the classic triangle diagram and how it’s baked into various GUI and web frameworks. There’s only one clause in the entire article that hints at something deeper: “Though MVC comes in different flavors…”

Different flavors indeed. In fact MVC is not just a pattern but a whole family of patterns: MVC, MVA, MVP, PAC, Model-Delegate…. It very quickly gets very hairy.

In this article I want to describe one of MVC’s lesser-known variants, the Model-View-Adapter (MVA) pattern, and talk about its advantages over traditional MVC in the context of a Java Swing application.

Architecture

The best place to start is with an architecture diagram. While vanilla MVC is a triangle:

MVA puts the Adapter in a position to strictly mediate between Model and View:

Here a solid line represents a direct relationship while a dashed line represents an indirect relationship via the Observer pattern. Put another way, the Adapter holds a pointer both to the Model and to the View and directly calls methods on both. At the same time, it attaches itself as a listener both to the Model and to the View in order to receive events. It receives property change events from the Model and action events (checkbox ticked, text entered, etc.) from the View, and then routes appropriate changes to the other side. The Adapter is entirely responsible for keeping the Model and the View in sync; the Model and View are both relatively dumb structures, knowing nothing about the other.

The advantages to organizing code this way are:

• All “moving parts” are centralized in one place, the Adapter. No worrying about where to add a listener; no hunting around to find isolated listeners.
• Separation of concerns between the View and the Adapter. The View is responsible for layout and visual presentation while the Adapter is responsible for synchronization and the dynamic aspects of the user interface.
• Better decoupling between Models and Views. Specifically, the View doesn’t need to know anything about the Model.

Additionally, while it will never be possible to fully componentize any variant of the MVC pattern, MVA is more amenable to componentization and thus more of its implementation can be centralized (in a single class) and reused. Once componentized, we can augment the basic functionality with things like:

• Automatic registration and unregistration of listeners when the View enters and exits the Swing component hierarchy, thereby preventing certain kinds of memory leaks.
• Automatic unregistration of listeners when the program shuts down. This can help free up resources like realtime subscriptions.
• Method for swapping a new Model object in for an old Model object.
• Ability to execute a task without listeners attached, to help prevent event-action-event loops.

The downside to using MVA over MVC is that the Adapter tends to take on a lot of the responsibility and can get quite complicated. But in my experience that can be mitigated by having good conventions about which pieces (M, V, A) are allowed to communicate with which other pieces and at what times. Enforcing predictable control flow goes a long way toward managing complexity.

Read on for a code-level description of our implementation of the MVA pattern.

Palantir MVA Implementation

Our half-componentization of MVA resides in a single abstract class named Adapter:

public abstract class Adapter<ViewType extends Component, ModelType> {
// constructor
protected Adapter(ViewType view, ModelType model); { ... }

/**
* Attach listeners to the View's subcomponents (checkboxes etc.).
* Listeners should be stored as member variables in the Adapter
* subclass.
*/
protected abstract void registerViewListeners();

/**
* Detach the same listeners (member variables) that were
* attached in registerViewListeners().
*/
protected abstract void unregisterViewListeners();

/**
* Attach listener(s) to the Model.
*/
protected abstract void registerModelListeners();

/**
* Detach the same listeners (member variables) that were
* attached in registerModelListeners().
*/
protected abstract void unregisterModelListeners();

/**
* Bring the View fully in synch with the Model.  Typically
* this involves querying state from the Model and
* reconfiguring subcomponents of the View accordingly.
*/
protected abstract void fullSynchronize();

protected ModelType getModel() { ... }
protected ViewType getView() { ... }

// other methods elided
}

New View components that want to stay synchronized with a Model must instantiate a subclass of Adapter and implement the abstract methods. The Adapter parent class (itself an example of the Template Method design pattern) will then call into the appropriate abstract methods at the appropriate times. For example, after the View is constructed, as soon as it’s displayed in the Swing component hierarchy the Adapter parent class will automatically call fullSynchronize() (whose implementation should bring the View in line with the Model) and then registerViewListeners() and registerModelListeners(), so the Adapter is poised to react to events. Likewise, when the View is removed from the component hierarchy (when its containing frame is closed, say), both unregisterViewListeners() and unregisterModelListeners() will be called. This can help ensure that no memory will be leaked when a long-life-cycle object (like a system-wide singleton) retains a pointer to a short-life-cycle object (the View) via the Observer pattern.

Dealing With Listener Loops

One problem that confronts UI developers is the problem of “listener loops”: infinite loops that result when the View fires an event, the Adapter (or Controller) responds to it by setting some property on the Model, and an event is propagated from the Model back to the View, starting the whole cycle over again.

One way to combat this is to make sure your Model only fires events when the value that’s being set on the Model is different from the value currently stored in the Model. (This will cut off the infinite loop after one and a half cycles.) It’s a good practice but often isn’t enough, especially when your system is multithreaded and events start to queue up. You can sometimes get into situations where an M-V-C triplet will thrash forever between two different values for one of the Model’s properties.

Our solution to this problem is a protected method (on our Adapter base class) called runWithoutViewListeners:

/**
* Guarantees that the job r will be run:
*    - on the Swing thread
*    - with Model listeners attached
*    - with View listeners DEtached
*/
public final void runWithoutViewListeners(final Runnable r) { ... }

The implementation of this method checks to make sure the view listeners are attached when it’s called, detaches them via a call to unregisterViewListeners(), invokes the Runnable, then reattaches the view listeners via a call to registerViewListeners(). The code inside the Runnable can then make whatever changes it wants to the View without perturbing the Model downstream. Listener loop averted!

More To Come

I hope that’s given you some sense of the territory out there in the wide world of MVC-variants. In a week or two, Derek will show off some of the work he’s done on the M piece of the MVA triad related to “event bubbling.” Stay tuned!

As we mentioned in an earlier post, Palantir was recently invited to the IEEE’s VisWeek in Dayton Ohio, and was honored to be invited to participate in the VAST Interactive Challenge as part of VisWeek.

After winning an award for Interactive Visual Analytic Environment, Palantir was one of three teams selected to participate in the interactive session from 73 VAST Challenge entries. For the challenge, we were given a completely new set of data to analyze. We had 30 minutes to import 3 disparate datasets into Palantir, 30 minutes to train an analyst that had never used Palantir, and then 2 hours for the analyst to explore the data.

The data for the challenge came from three different sources, with a set of questions to answer for each set of data. There was an infectious outbreak, a Wikipedia edit war, and an abduction from a city park. Over the three challenges, there were over 100,000 datapoints to analyze. All of the data revolved around a fictitious town in Florida, Barracuda Springs, and was linked to the fictitious cult that was the center of the 2008 VAST Challenge. While two members of our team were importing the three datasets, the third team member was working with our analyst (each of the three teams was given a analyst from a nearby analytical organization). In 30 minutes, our analyst was able to learn how to conduct relational, temporal, geospatial, and statistical analysis in Palantir. After the 30 minutes of training, she was able to easily navigate the Palantir workspace, and solve all three challenges. Below is her work (hit the link to check it out).

Her conclusion was that Palantir was “viciously good software” and that she would be asking her boss if they could acquire Palantir for their work. Hit the link below to see screenshots and explanations for one of the challenge workflows.

We really enjoyed the VAST Challenge, and our experience at VisWeek. There were a lot of outstanding papers, posters, and speakers at VisWeek, and we were inspired by many fantastic visualizations that might soon make their way into Palantir’s Finance and Government Platforms. We are also looking forward to the 2009 VAST Challenge!

Challenge: Infectious Outbreak

Our analyst started with the first challenge, analyzing an infectious outbreak in Barracuda Springs. The notional source of this data was reported school absences, so we see gaps in the timeline for weekends.

The Outbreak

She started by searching for all infected houses, and then plotting them geospatially. She immediately identified that the infection was a 5 week outbreak, peaking in week three, and then tailing off. With Palantir’s temporal filters, she quickly identified the first house to have the disease was 208, on Thursday May 5th:

Week 1 – starting slow

Increasing our temporal filter to the first week, we saw that the infection was in the north of Barracuda Springs, isolated in two neighborhoods (points on the map that correspond to selected time range in the timeline (bottom) are highlighted in yellow):

Week 2 – burning bright in north

The second week saw much of the north infected, but not yet into the south:

Week 3 – everyone’s got it

The third week shows the infection peaking, and moving to the rest of community. The analyst also noted that the neighborhoods from the first week appear to be clearing up:

Week 4 – northside burnout

By week 4, infections are exclusively in the south of Barracuda Springs, and in week 5 the infection clears up:

Hidden Pattern: refusing treatment

Also of interest was the fact that there are certain geographic areas where the infection lasts for 10 days. This was an Easter egg put into the data — the areas of 10 day infections were neighborhoods in the city where the fictitious cult had settled, and they were refusing to take treatment. Using the histogram, the data leaps out you. We were the only group to discover this:

We have a scatter plot in Palantir Finance, but it’s not designed to compare your data across many variables (called metrics here). I extended it by adding a small triangular matrix control that we call the scatter plot Quick Select. The control gives you a visual overview of the data, and allows you to identify interesting metric pairs and then drill down into a scatter plot for any particular pair.

Assume I want to view a set of 10 metrics (shown above). With our regular scatter plot, you have to choose two of these metrics and plot them to see their correlation. If you want to know how every possible pair of metrics correlate, you have to manually perform an O(n²) input operation.

With Quick Select, you enter the metrics once and the triangular matrix is formed. Each square in the matrix represents a pair of metrics, and each square’s color is the correlation between the pairs. For example, green is a strong positive correlation, light green a less positive correlation, white is no correlation, and red is a strong anti-correlation.

In the highlighted square above, we’re comparing percent return and correlation with the S&P 500 over the past year. The red shows a strong negative correlation. This makes sense in light of recent market behavior: the S&P 500 has not done well over the past year, so companies that were correlated with it also performed poorly, while companies that moved in the opposite direction performed well.

Drilling down to a scatter plot is as simple as clicking on the corresponding square. Below are 2 of the 45 possible scatter plots defined by this set of 10 metrics. You can quickly jump back and forth between different scatter plots while retaining a nice, condensed overview of the data in the triangular matrix on the left.

This prototype took less than a day to write. And it was written from outside the system, using only the pluggability points Palantir Finance provides. Overall it’s a powerful visualization component, added as an extension to the Palantir Finance platform, and done in under a day. Pretty cool!

Many thanks to Prof. Shneiderman for the idea.

One of the things our customers love to do is print our beautiful object graphs and tape them to the wall for discussion. What they hate to do is print 30 pages, line them up, and tape them to a poster one at a time. So we bought a plotter, and I started plotting.

I needed to print directly to a Java Graphics object. Unfortunately, the available information on large output printing from Java is thin at best. While there are lots of ways to successfully place ink on paper, I was only able to find one that reliably lets the application pick odd paper sizes that plotters use, like 24×19.7 inches. (The term “plotter” used to mean something with pens for printing blueprints and such. Now it just means a large format printer, commonly printers that can use roll paper as a source.)

One of the first things you’ll learn when you start working with printing in Java is that a language intended to be all things to all people (i.e., cross-platform) is utterly lousy at tasks highly specific to a given environment, such as printing. It will not surprise you to hear that native print services on Windows are pretty different from those available on a Mac, which themselves are pretty different from the CUPS system common to Unix systems.

So, by and large, you are reduced to the least common denominator of printing. Part and parcel of this least common denominator is agreeing on what constitutes a piece of paper and sticking to it. This is fine for people thinking, “My paper is 8.5 inches wide by 11 inches tall.” It poses a bit of a problem for people with plotters who are thinking, “My paper is 24 inches wide by as many damned inches tall as I need.” Even relatively powerful programs like PhotoShop or GIMP don’t seem to support plotters well. I believe Photoshop works by specifying the exact paper size you want to use, but any technique in which the easiest solution for the user is to pull out a calculator does not meet with my approval.

Advice in a nutshell

A tutorial by MartyHall provides some valuable insights into printing which I applied to my printing component. While it didn’t address my specific issues, props to a well-done starting point.

I’m starting with the assumption that you’ve got yourself a class that implements the Printable interface so that once the Java printing subsystem is coerced into accepting the appropriate paper size, Printable.print(Graphics, PageFormat, int) will be called correctly and your job will print properly. Since my usage was screen-visible as well as printable, I simplified and asserted that one screen pixel is equivalent to one point (1/72 inch), which is how Java prepares the scaling on the Graphics object passed to Printable.print(…), thus making printouts look about the size of onscreen displays. Then, knowing that the printout needed, for example, 1200 vertical pixels/points, I can automatically calculate that I want to use a paper size 18″ tall or so (1200 / 72 dpi = 16.67″ + margins).

Here’s a summary of the critical steps:

1. Do the math yourself to get the paper size required in points (1/72 of an inch)
2. Implement the Pageable interface to report page count (usually one) and the PageFormat using the computed paper size.
3. Create a PrinterJob configured with the correct PrintService
4. Set your Pageable on the PrinterJob instance
5. Optionally display print setup dialog (PrinterJob.printDialog()) recognizing that changes to orientation or paper size will be ignored.
6. call PrinterJob.print()

Generally, don’t use the standard dialogs

This might seem contrary to desired behavior, but I found that giving the user access to the usual native or cross-platform print dialogs can be a bit sticky because those are places where the user can alter the selected paper; in order to properly print to roll paper sources, we must override any paper settings the dialogs might provide. Ultimately this can be confusing for the user, but that’s an unfortunate side effect of supporting plotters.

Don’t take any advice too seriously

We still choose to display the dialog because complex printers like plotters have many options that the users may need to alter and would otherwise have no way to modify.

StickFigure: looking at plotting in Palantir

This work didn’t happen in a vacuum: we have a need to print large graphs for display. Being able to print them in a large format is one of the simplest and most beloved methods of collaboration.

To give an example of why plotter printing is interesting, we created a stick figure on the graph in Palantir. After creating this monstrosity, we went go print it and examined the layout. The first layout was with letter-size paper, and just by looking at the head, we could see that it was going to be a lot of pages:

Zoomed-in view of the head of the stick figure. (click for full image)

Zooming out to view the entire graph, it’s clear that it’s going to be 35 pages!

Full pagination of the stick figure. (click for full image)

Even switching to the largest standard paper size only gets us to two sheets:

Paginated out to largest standard paper size, still two pages (click for full image)

When we switch over to using the plotter layout, we finally get to the one sheet we were looking for:

Paginated onto a plotter roll. (click for full image)

And finally, Mr. StickFigure comes to life:

Carl juggles with his creation

Palantir has entered the 2008 VAST Challenge. We present an in-depth look at one of our challenge solutions as the first public example of the Palantir platform in action.

Two years ago, the IEEE began an annual conference called VAST (Visual Analytics in Science and Technology). The VAST symposium focuses on the fundamental research contributions and real-world application of visual analytics. As a part of the conference, the VAST Challenge allows teams to compete on delivering analytic solutions against a synthetic real-world dataset.

Each year, the organizers build a very vast (pun intended) dataset from scratch. The data is entirely fictional but mirrors real-world use cases and scenarios. This year’s dataset is about a new religious movement that started on an imaginary Caribbean island (cleverly titled Isla del Sueño, or “Island of Dreams”) situated between Florida and Cuba. There are four subsets of the synthetic data: the Wikipedia page for the movement and its associated edit and discussion pages; landing and Coast Guard interdiction records for boats leaving Isla del Sueño for Florida/Mexico; cell phone records from the island; and RFID tracking data from people in a building that was attacked with an IED.

The types of questions asked in the problem sets are qualitative questions that require answers backed by data. These are the sorts of questions that don’t yield answers using a machine-learning/data-mining approach nor can an unassisted human get these answers by simple inspection of the data. They require some sort of human-computer symbiosis to solve.

To solve the VAST problems, we assembled an ad-hoc team of analysts — composed of a mix of engineers, in-house professional analysts, and one senior executive — and asked them to use the Palantir Government software to extract insights from the data.

The results speak for themselves: the complete set of Palantir’s VAST solutions are available here.

Read on for an in-depth look at how we deconstructed and solved one of the problems.

Analyzing Cell Phone Networks or “Where’s Waldo Ferdinando?”

Mini-Challenge 3 asks a couple of questions about the command and control structure of the religious movement by examining cell phone data from the island:

• What is the Catalano/Vidro social network, as reflected in the cell phone call data, at the end of the time period?
• Characterize the changes in the Catalano/Vidro social structure over the ten day period.

We started out with a CSV containing 9,385 rows that look like this, logging ten-days of calls:

A tabular format like this is about as far from “visual” or “analytic” as one can get; not even a social-network-analysis-savant could glean something interesting from the data in that form. In addition to the spreadsheet, we were given the approximate geographic locations of each cell tower and the following intelligence (from the contest’s synthetic dataset):

“We have medium confidence that Ferdinando Catalano is identifier 200. Close relatives and associates he would be calling would include David Vidro, Juan Vidro, Jorge Vidro, and Estaban Catalano. We believe Ferdinando would call brother Estaban most frequently. We also believe that David Vidro coordinates high-level Paraiso activities and communications. ”

Using nothing but that spreadsheet and the power of Palantir, we were able to uncover secretive attempts by member of the Paraiso movement’s highest echelon to conceal their cell phone traces by switching devices.

Most importantly, the analytic team did this without needing to be experts in the technical aspects of social network analysis, database queries, or Java. Instead, Palantir allows users — people who have a deep understanding of how the world works but are not technically savvy — to get answers backed by data. So answers to high-level questions such as, “Who were these people talking to during those last three days of network silence?” are actually easy to extract from the dataset. We realize that this sounds awfully hand-wavy, abstract, and vague, but the videos and explanations show exactly how it’s done.

Here’s the two-minute long Mini Challenge 3 Video and full explanation. The ten-minute long Palantir VAST 2008 Grand Challenge video gives a brief overview of Palantir and then presents all four solutions.

Palantir Finance, our product that has not yet been released, brings a similar level of power, flexibility, and high-level analytics to the realm of finance. Stay tuned for a sneak peek of those capabilities in the coming months.

If you found that video as much fun as we do, check out the other demonstrations of our ability to make data come to life. The “Boat Video” about migration data does an especially good job of showing off Palantir’s relational analysis abilities and its open-platform geospatial integration. Each of the write-ups includes additional screenshots that aren’t in the videos.

Welcome to the future of analysis!

 
 

Time Filtering

The graph is linked to the histogram which allows view filtering based on time ranges.

Financial Dashboard

The dashboard allows a financial analyst to follow a configurable set of widgets delivering real-time financial data.

Instrument Groups

The Instrument Groups tool lets analysts build up dynamic groups of financial instruments using expressive queries, filters, and calculations

Flows

More impressive in an animation than still image, the flows tool allow for the intuitive visualization of resources flowing on between entities. The configurable threshold enables the filtering of the ‘noise’ of smaller payments.