Late last year, we were honored to be invited to talk at Reflections|Projections, ACM@UIUC’s annual student-run computing conference. We decided to bring a talk about Horizon, our system for doing aggregate analysis and filtering across very large amounts of data. The video of the talk was posted a few weeks back on the conference website.

Horizon started as research project / technology demonstrator built as part of Palantir’s Hack Week – a periodic innovation sprint that our engineering team uses to build brand new ideas from whole cloth. It was then used by the Center For Public Integrity in their Who’s Behind The Subprime Meltdown report. We produced a short video on the subject, Beyond the Cloud: Project Horizon, released on our analysis blog. Subsequently, it was folded into our product offering, under the name Object Explorer.

In this hour-long talk, two of the engineers that built this technology tell the story of how Horizon came to be, how it works, and show a live demo of doing analysis on hundreds of millions of records in interactive time.

Here at Palantir we use test-driven development (or TDD for short). Integrated tools like Eclipse and JUnit simplify writing and running unit tests. However, once you need to test a broader swath of functionality, it’s time to write functional, integration, and system tests. While technically not ‘unit testing’, the testing framework that JUnit provides is basically the same infrastructure that you want to leverage for writing these more involved types of testing.

When you’re developing enterprise software, functional testing often means getting your clients to talk to your servers. For the main Palantir Government product, we integrate the process of bringing the server up and down with the Ant scripts that run our automated unit tests: our testing tasks bring up the server, run the test suite, and then kill the server. This works great and produces nice results.

When I started working on our authentication server, the pattern that we had used before didn’t work for me. While the Palantir Government tests ran with a single, static configuration file, I needed to run the authentication server with multiple configurations in the course of running through the all the different functional tests. I determined that I needed a way to programmatically bring the server up and down for testing. In JUnit parlance, I needed a way to programmatically launch the server component as part of my setup() function for my unit tests and stop it in my teardown().

With my itch-to-scratch firmly in hand (or some other mixed metaphor), I set out to figure out how to invoke new Java processes from inside a unit test. The solution I came up with (with source code and examples) after the jump.

The Six Ingredients

So there are six ingredients that go into spawning a new VM:

• The classpath to use for the new VM
• The name of the class to run
• The directory to be used as the current directory for the process
• The command line arguments to pass to the process
• The set of Java system properties to use for this process
• The environment to pass to the process

Let’s look at each item individually.

Classpath

The classpath will tell the spawned VM where to load classes from. In JavaInvoke, we use the existing classpath (from the spawning VM) as a starting point and then prepend any new entries to allow overriding the classpath for the spawned VM.

This takes a lot of the tedium out of having to figuring out what to put in the classpath. Most likely, you want something similar to what you already have, if not completely identical.

We get the classpath from System.getProperty("java.class.path") and can add new entries by prepending the new entry, using the value of File.pathSeparatorChar as the entry delimiter. Using File.pathSeparatorChar makes the code cross-platform friendly (since the path separator is ‘;’ on Windows and ‘:’ on Unix (Linux, Solaris, OS/X, etc.).

Caveat: if you change the working directory and your original classpath was constructed using relative paths, you’ll probably have trouble getting anything to run (since your classpath will no longer point to right locations).

Class name

Pretty simple: what do you want to run in the spawned VM? The class must have a static void main(String args[]) defined, and it must be available for loading via the classpath.

Working Directory

If it should be different from the current working directory (CWD) of the running process, then set it and JavaInvoke will change it in the environment.

Command line arguments

If the process needs any command line arguments, including VM options, specify them in a string array. Note that not all of these arguments will necessarily make it to your main method, since the VM executable will parse it first and remove the VM arguments, passing through the program arguments.

Java System Properties

System properties can be used to control many aspects of how a VM runs. You can set them programmatically in your code or you can set set them on the command line by passing -Dkey=value. Our JavaInvoke implementation will take a Map of properties as a convenience argument; all it does is rewrite the map into the command line.

Process environment

This is an operating-system level construct. This is the set of environment variables, also in a Map that you would like merged with the current environment. This would be the place that you set things like LD_LIBRARY_PATH on Unix.

Dealing with input and output

So you might ask the question, “where does the output from the process go?” Or more troubling, “How do I send the process some input?” The Java Process object has methods to deal with this, allowing you to get streams that give you access to the input, output, and error streams of spawned process. That API is straight-forward to deal with, just like any other use of the java.io streams.

However, we want to make the typical case really easy: pulling the output from the spawned process back to the parent that spawned it. To that end, we add into the mix a class called OutputPiper. It fires up a thread that pulls all input from the spawned process, tags it with an identifier, and then outputs to the spawner’s stdout/stderr.

OutputPiper

(as extracted from ProcessSpawner.java)

	public static class OutputPiper extends Thread  {
		InputStream in;
		PrintStream out;
		String tag = null;

		public OutputPiper(String tag, InputStream in,PrintStream out) {
			this.in = in;
			this.out = out;
			this.tag = tag;
			// make sure that we don't keep the VM alive
			this.setDaemon(true);
			this.setName("OutputPiper-" + tag);
			out.println("Starting output piper for tag: " + tag);
			this.start();
		}

		@Override
		public void run() {
			try {
				BufferedReader reader = new BufferedReader(new InputStreamReader(in));
				String line = null;
				do {
					line = reader.readLine();
					if(line != null) {
						out.println(tag + ": " + line);
					}
				}while(line != null);
			}
			catch (Exception e) {
				//
			}
			out.println("Output piper exiting for tag: " + tag);
		}

		public static OutputPiper createOutputPiper(String tag, InputStream in, PrintStream out) {
			OutputPiper rc = new OutputPiper(tag, in,out);
			return rc;
		}
	}

Outpiper extends Thread so that all the output will arrive back to the controlling process in a timely manner. For each given process, we spawn off two OutputPipers, one for stdout and one for stderr, corresponding to the Process.getInputStream() and the Process.getErrorStream().

ProcessSpawner & JavaInvoke

There are two key classes in the example:

• ProcessSpawner.java – Essentially a wrapper around ProcessBuilder, a generic process spawner that makes it simple to invoke processes that that use OutputPipers to forward their output back to their parent. This class allows you to specify the working directory, process environment, and command line for the process to be invoked.
• JavaInvoke.java – a specialized subclass of ProcessSpawner, this class makes spawning new VMs a piece of cake, doing the necessary translation for Java system properties, setting the proper classpath environment variable with potential overrides, and fills in the fully qualified class name to run.

The Example & Source Code

I’ve put together a running example that implements a trivial client and server in JUnit test. The setup() method spawns the server and then the tests run the client code against the server, tearing it down after each test. It’s available in the PalantirVMSpawnerExample.zip zip file. Unzip it, run the run.sh or run.bat script as appropriate. It should generate output that looks like this:

-----------------------------------------------------
Starting test testAck
INFO [main] JavaInvoke - CLASSPATH=./lib/devblog-vmspawner.jar
INFO [main] ProcessSpawner - Build process spawner for the following command line:
INFO [main] ProcessSpawner - /home/pteng/java/i586/jdk1.5.0_14/jre/bin/java com.palantir.blog.processspawner.Server
Starting output piper for tag: server-stdout
Starting output piper for tag: server-stderr
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: [Socket Handler2]: Got message: some message
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: [Socket Handler3]: Got message: SHUTDOWN
Output piper exiting for tag: server-stdout
Output piper exiting for tag: server-stderr
Finished test testAck
-----------------------------------------------------
-----------------------------------------------------
Starting test testShutdown
INFO [main] JavaInvoke - CLASSPATH=./lib/devblog-vmspawner.jar
INFO [main] ProcessSpawner - Build process spawner for the following command line:
INFO [main] ProcessSpawner - /home/pteng/java/i586/jdk1.5.0_14/jre/bin/java com.palantir.blog.processspawner.Server
Starting output piper for tag: server-stdout
Starting output piper for tag: server-stderr
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: [Socket Handler3]: Got message: SHUTDOWN
Output piper exiting for tag: server-stdout
Output piper exiting for tag: server-stderr
Took 3 ms to send shutdown.
Took 335 ms for process to die.
Finished test testShutdown
-----------------------------------------------------
SUCCESS: all 2 tests passed

The source is included in the zip file, but if you wanted to look at it or link to it on the web, here are the classes involved:

• Client.java
• Example.java
• JavaInvoke.java
• ProcessSpawner.java
• Server.java
• ServerSpawningTest.java

And as an added bonus, there’s an Ant build.xml that will let you tweak and rebuild the demo yourself.

Comments and questions welcome. Enjoy.

“Freaking lag!” It had started to become a common refrain around the developer pit. Listed as a project on a candidate’s resume, MultiSnake was a game that we had started to play during our coding breaks. The game was really quite fun — it was easy to play, games were short, and its multi-player nature fostered great competition. The only real drawback was that we seemed to experience network lag. There was nothing more infuriating than having your long snake die by running straight into a completely avoidable wall because the game lagged and didn’t respond to your keyboard commands in time. During one of our particularly lag-heavy games, someone yelled out a gripe that would change our MultiSnaking days for good: “Man, we could totally write this game ourselves, in our app.”

The gripe stuck around, and one day someone finally called out the person making the claim. “Do you seriously think we could write this ourselves?”

“Sure, why not? We have all of the architecture that we need to make this work. We could do it in four hours.”

“I bet you we couldn’t.”

The rest of the story and a video of MultiSnake in action follows.

The Challenge

A challenge was born. The task sounded fun, and it also provided us with a great chance to test the extensibility of our platform. Our most recent milestone had focused on solidifying the public APIs of our platform, and this challenge seemed like a way to test its pluggability. We also thought that it would be a great showcase to demonstrate how easily one could add capabilities to our platform — if we could write a multi-player network game using only the same public APIs available to our clients, it would be a strong signal that our framework was solid.

The Rules

Once we decided that we were going to take on the challenge, we decided that we would do it on Sunday from 8:00PM to midnight (our normal peak productivity hours) and laid out the following rules:

• There would be a strict four-hour time limit for all planning, design and coding.
• The game had to be implemented using only our public APIs — no touching core code.
• The game had to support all of the features provided by the online game and had to be lag-free (after all, that was the whole point of writing our own!).

The Race

Sunday night came along, and it was off to the races! We projected a countdown timer onto a whiteboard in the middle of the developer space, blasted some techno music and got to work! Five developers decided to participate, and it was a pretty collaborative effort in which most of the participants ended up contributing in their standard roles — the frontend people did the game graphics and UI, the backenders worked on the server and game logic, and our data folks dealt with creating a game board provider.

The pluggability of our platform and the fact that it already had support for multiple users and sending out realtime messages from the server to clients made most of the work go pretty smoothly. Besides Eclipse crashing on one of our machines, development was pretty seamless and fast-paced, with team members yelling out at each other across the room to communicate. Within 45 minutes, most of us had finished our first iteration of code, and at the one hour mark we verified that we could successfully get the server and client communicating with each other and draw some game state. By the time we were two hours in, we had most of the core game features implemented, and once we hit the three hour mark we had a fully functional snake game that allowed us to play against each other. We spent the last hour doing some UI polish and ironing out a few bugs, and by the time midnight rolled around we had a fully functional MultiSnake implementation in our platform that was written using only our public APIs. We were even able to get in a few extra features such as a visible timer to count down to the end of the game, a circle to show where a snake respawned and a name that followed the snakes vertically to identify them. We celebrated by eating a rum cake that a coworker had brought in earlier and playing multiple rounds of lag-free MultiSnake against each other. Success!

The Results

MultiSnake lives on as an add-on to our platform. Even after the challenge was completed, devs have been playing around with the code on slow weekends, adding extra maps, new features such as wormholes, and overhauling the graphics. The final product is a pretty impressive and fun-to-play game that we now often demo as an example of the versatility and power of the APIs for our platform.

At Palantir, we work in Silicon Valley, read High Scalability, and think of web companies like Facebook and Google as our peers. Most of the time, this is exactly the right recipe for bringing disruptive innovation into the intelligence community. Sometimes, though, it’s misleading – when discussing a design decision, it’s received knowledge that “Disk is cheap.” or “CPU is cheap”. For a web company with a deployment in a commercial data center (or its own data center), this received knowledge is correct. But for a company that ships distributed systems instead of hosting them, and for whom the deployment environment is the kind of locked-down server room in which classified data can reside, these assumptions couldn’t be more false.

At Palantir, we are almost never able to host our customers’ data – typically, as the data is very sensitive, we are not even allowed to see it! Our customers’ highly sensitive data has to reside in a Secure Compartmented Information Facility or SCIF – a building which has been built to be resistant to attempts to access the information within, whether through active or passive measures. The network inside a SCIF is physically separated – “airgapped” – from the public Internet to prevent information leakage. As the entire rationale for such facilities is to prevent information leakage, moving information into or out of one is a tightly regulated process, almost always requiring a human to be in the loop.

Bandwidth is narrow

Bandwidth in and out of a data center is cheap. Bandwidth in and out of a SCIF is not – and this manifests in surprising ways. First off, what does it take to get data into a SCIF? First, the data has to be downloaded from wherever it’s hosted and burned to a CD. Then, someone has to carry it into the SCIF and find a security officer to approve adding it to the network. Finding the security officer can take anywhere from 10 minutes to an entire day. Once you’ve found the security officer, he has to run a virus scan on the CD, which can run at a rate of roughly 20 minutes per 100MB.

If you look at the entire process, you can model our connection into the SCIF as averaging about an 8 hour latency and 640 Kbps bandwidth. That’s about the bandwidth of a slow DSL line and the latency of a radio connection to Pluto. (Actually, it’s somewhat slower.) There’s also a big non-linearity at 700MB, which is the amount of data that fits on a single CD. For instance, this non-linearity is the big reason why we prefer to send patches to our customers rather than full distributions, which are slightly less than a gigabyte including dependencies – and thus why it’s worth it to us to build a system for automating patch application rather than simply replacing jar files by hand.

Disks are expensive

Similarly, if you are running a data warehouse, disk is cheap. You can buy a 1 TB, 7200 RPM disk for about $100, which is perfect for the kind of large, serial reads or writes that a data warehousing workflow requires. However, Palantir uses disk for our database and our search engine, both of which have an OLTP-style usage pattern. As opposed to a data warehouse access pattern, which emphasizes full table scans, OLTP emphasizes random access and therefore requires fast disk. To get 1TB at 15k RPMs costs about $1000, and requires a disk array rather than a single disk. In order to keep the disk fast, you also want to leave it only about 20% full, which overall makes fast disk about 50 times more expensive than slow disk. Most importantly, however, installing a disk array requires trained personnel, a special approval process, and reconfiguring the system to use the new disks, which is a fairly complicated and error-prone process.

CPUs are hot

Finally, in a commercial data center, CPU is the cheapest resource of all. In a secure server room, however, it can be quite expensive. Each CPU or additional box requires more power and cooling. If the room is nearly full, adding that extra box may require building out an entirely new server room, which can cost months and hundreds of thousands of dollars just for an office building. Building a server room in a SCIF is much more expensive and prohibitively time-consuming.

RAM to the rescue

On the other hand, some things in a SCIF are comparatively cheap. We never use boxes with less than 32GB of memory, and, in fact, lots of sites use 128GB of memory. RAM requires negligible power and cooling, and compared to disk, it’s relatively simple to install. It’s also easy to reconfigure the setup to use the additional memory.

The upshot

The design guidelines that follow from this are simple: build a system that is as autonomous as possible and scales down as well as it scales out.

All these statistics are compiled from our day-to-day experiences in the office environment of a SCIF. Deploying to soldiers in the field makes the issues involved in deploying to a SCIF seem minor. Of course, that’s what makes what we do fun.