This is a response to Ari’s awesome post on human-computer symbiosis. Ari and I were chatting about the equation he developed and I was wondering if there were some further refinements that are possible… let’s take a look:

We are attempting to understand the total analytic capability for a given task a of a human-computer team. Analytic capability in this case probably means:

(1)

Where A is the answer to the analytic problem in question and t_A is the time needed to arrive at the answer based on the inputs available. In the case of chess, A could be the optimum next move given all previous information and t_A would be how long it takes to decide on this move.

Read on for a look at how this generalizes in human-computer symbiotic systems.

In the case of the human-computer team, we know that a is going to be a function of both the human’s analytical capability h and the computer’s analytical capability c (where both h and c have units of answers/time). In the limit case we know that:

(2)

(3)

Or in plain English, if there is no human present, the total analytic capability is simply the analytic capability of the computer. So the naïve solution would be that:

(4)

(4) clearly meets the limiting cases described in (2) and (3). Kasparov noticed a mixing function where the ability of the human and computer to work together becomes the dominant term — we might call this the mixing capability for the given task or m. Including this phenomenon, the total analytic capability (4) would be re-defined as:

(5)

where m has the property that:

(6)

Thus maintaining the limits expressed in (2) and (3) and adhering to the observation that if there is no human or computer component then there will be no mixing advantage. A naïve solution to this constraint would be simple linear mixing:

(7)

where M (units of time per answer) is the mixing efficiency and will be primarily based on the type of task being solved — some analytical tasks lend themselves to a combined process more than others (for example, multiplying 20 digit numbers does not really benefit from the intuition of a human so the ability of a human and computer to perform this task is merely their additive ability).

What Kasparov noticed is that the mixing was primarily based on the quality of the process rather than the analytical power of either the human or computer separately. This seems to imply that we must somehow account for the fact that the quality of the human-computer interface is responsible for the quality of the mixing. This can be modeled as a unitless friction of interaction f_i that impedes the ability of the human and computer to work together.

Equation (7) can thus be re-written as:

(8)

In this case, the maximum value for the mixing capability is realized when the friction of interaction goes to zero. This mixing capability is the same as the equation Ari developed (less the coefficient which is necessary to maintain consistent units throughout).

We can now re-write our analytic capability in (5) as:

(9)

Below, see a plot of this function over a range of values for h, c and f_i:

As can clearly be seen from this functional plot (note the vertical scale), the effect of interface friction dominates over the other terms whenever both the human and computer can make important contributions to the task at hand. The conclusion can be drawn that the most effective way to solve analytical problems is to minimize the friction of the human-computer interface; or to put it another way: optimal analytical systems are those that are built specifically to maximize the ability of the human to leverage the ability of the computer.

I am certain there is still the possibility for further refinement, for example:

(10)

As we build our platforms and applications following a human-computer symbiosis approach, we keep an ear to the ground for interesting examples that illuminate new techniques or validate our approach in some empirical way.

One of the areas that we’re interested is in the overall friction of analysis systems. The systems that we build are built on commodity hardware — we’re not building faster computers and yet we can deliver orders-of-magnitude better performance on analysis tasks than existing solutions. How do we do this? By building software in such a way that it reduces the friction experienced at the boundaries between the computing power, the analyst, and the source data.

Chess as analysis laboratory

Chess is, at its heart, a predictive venture. The player attempts to anticipate their opponent’s moves, planning their own moves accordingly, with the straightforward goal of finding a sequence of piece moves that force checkmate.

This game is, in its ideal form, analysis. (The moves made are the logical extension of the analysis.) The data are clean, the problem is well-defined and everyone plays by the same rules. There are even well-defined metrics for ranking chess players by skill — a better chess player is a better chess-game analyst.

In the realm of evaluation of analysis systems, this is as about as good as it gets in terms of designing controlled experiments to study the relative strengths of different analysis systems.

Garry Kasparov, widely considered to be the greatest chess player of all time, recently wrote a review of Diego Rasskin Gutman’s book, Chess Metaphors: Artificial Intelligence and the Human Mind.

The review is excellent and covers a lot of ground. However, one particular anecdote stood out as a very interesting example of human-computer symbiosis (emphasis added):

In 2005, the online chess-playing site Playchess.com hosted what it called a “freestyle” chess tournament in which anyone could compete in teams with other players or computers. Normally, “anti-cheating” algorithms are employed by online sites to prevent, or at least discourage, players from cheating with computer assistance. (I wonder if these detection algorithms, which employ diagnostic analysis of moves and calculate probabilities, are any less “intelligent” than the playing programs they detect.)

Lured by the substantial prize money, several groups of strong grandmasters working with several computers at the same time entered the competition. At first, the results seemed predictable. The teams of human plus machine dominated even the strongest computers. The chess machine Hydra, which is a chess-specific supercomputer like Deep Blue, was no match for a strong human player using a relatively weak laptop. Human strategic guidance combined with the tactical acuity of a computer was overwhelming.

The surprise came at the conclusion of the event. The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.

After the jump, we look at this finding in a more generalized way and map it onto the Palantir approach.

The cyborg Grandmaster: a fearsome opponent

The tournament Kasparov recalls was a showcase of chess talent, human-computer symbiosis, and raw computing power. Among those entered in the tournament were a purpose-made chess machine (similar to Deep Blue) named Hydra and a team of Grandmasters assisted by computer programs.

One losing participant had this to say about the computer-aided Grandmasters:

Secondly, I have learned that a Grandmaster armed with a chess engine is a killer combination against a plain Engine. Engines see everything via brute force, Grandmasters use their intuition and are able to see “obvious” moves at once. So the two of them together are a mighty force.

This is just as Licklider predicted 50 years ago — quoting Man-Computer Symbiosis (if I could put it better, I would):

Men will set the goals and supply the motivations, of course, at least in the early years. They will formulate hypotheses. They will ask questions… In general, they will make approximate and fallible, but leading, contributions, and they will define criteria and serve as evaluators, judging the contributions of the equipment and guiding the general line of thought.

…

In addition, the computer will serve as a statistical-inference, decision-theory, or game-theory machine to make elementary evaluations of suggested courses of action whenever there is enough basis to support a formal statistical analysis. Finally, it will do as much diagnosis, pattern-matching, and relevance-recognizing as it profitably can, but it will accept a clearly secondary status in those areas.

So in classic intelligence amplification fashion, having computer programs that can quickly evaluate a move’s likelihood of success can amplify the power of the Grandmaster.

While empirically true, it does beg the question: how much does it amplify the power of the Grandmaster?

One approximation might be product as a simple linear amplification. Let’s imagine a function, a(h,c), in which the analytic power (a) is the product of power of the human (h) and the computing power of the chess engine being used (c). This gives us the equation:

One term to dominate them all: friction-of-interface

Does this simple approximation hold up? It does not. The team that won the PAL/CSS Freestyle Tournament in 2005 was composed of two amateur chess players that were able to best a computer-assisted Grandmaster.

How did they accomplish this feat? It was not through superior compute power. Instead, they did so by more effectively feeding insights to their three chess engines. They played so well that a large number of people actually assumed that it was actually Kasparov himself playing:

Many speculated that it might be Garry Kasparov, who was the initiator of this kind of computer assisted chess matches. When we asked him Kasparov confirmed that was not the case. But he reminded us that it doesn’t really matter. The guiding principle of Freestyle Chess: anything is allowed. “Even if they were assisted by the devil, that would probably be covered by the rules,” he joked. “Only the moves they played count.”

What does this mean for our simple equation? Well, it looks it’s missing a term, one we’ll call f, that describes the efficiency or friction of the interface between human and computer.

Quoting Kasparov again:

Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.

The implication being that the equation actually looks like this:

So as the friction of the interface goes to zero, the full amplification of the chess engine is brought to bear. A quick gut-check in the opposite direction agrees: one can imagine the world’s most powerful chess engine with the world’s worst interface; spending the time it would take to express commands to this theoretically awful program would actually be worse than playing without it.

Palantir: a low-friction interface to data

As analysis problems go, chess resembles a spherical cow in a vacuum. Analysis problems in the real world are orders of magnitude messier.

Let’s reframe the terms of our equation above into a more general approach to analysis:

• H – this is power of the analyst. In chess, the value of this terms varies widely between players; in designing real-world data analysis systems, this is more or less a constant (which is why h above becomes H below). Of course there are differing levels of expertise, training, and raw ability amongst the user population, but when we design systems, it’s with the average case in mind.
• c – computing power. How fast are the machines? How well do they scale? How efficiently do they perform the data tasks at hand? Palantir spends significant engineering effort on optimizing the c term, but most of the growth in this term comes from the layers we depend on, built by companies like Intel, Sun, Oracle, etc.
• f – friction. How easy is it to bring c to bear on the problem? Note that when we talk about friction of interface, this is not exclusively referring to user interface. More generally, friction can be present at any interface between two systems: data-software, software-software, human-software, etc. The f that we consider in this simple model is sum total system friction.

So our final formulation is just in terms of c and f (holding H as a constant):

When we discuss friction in real-world analysis systems, the friction actually exists at multiple levels:

1. Creating an analysis model that will enable answering the questions that need to be explored
2. Integrating the data into a single coherent view of the problem
3. Enabling analysis tools to efficiently query and load the data
4. Exposing APIs that allow developers to develop custom solutions quickly and efficiently for modeling and analysis tasks not covered by general tools
5. User interface that makes the tools easy, enjoyable, and quick to use

Minimizing f: Haiti Flooding Predictions

If this is starting to sound very similar to Palantir’s marketing information, this is no accident. While some of our backend engineers are concerned with things like scaling and speed-of-querying, the overall innovation that we’re bringing to the field is not simply about faster data processing systems (even if they are) but reducing the friction at every interface inside a complex human-computer symbiotic system.

You want an example that ties it all together? It starts with a simple question: which of the many displaced-person camps in Haiti are most at risk for flooding as the rainy season approaches? Easy to ask, but not so simple to answer.

The original introduction to this video:

As we enter the beginning of the rainy season in Haiti, one of the biggest problems facing relief organizations today is the spectre of flooding and mudslides destroying Internally Displaced Persons (IDP) Camps. In this video, we integrate data from many sources to determine high risk aid locations.

The data integration for this video took about six hours, using sources of data that had never before been fused. The analysis itself takes a few minutes and quickly comes to an actionable answer to the original question.

Here at Palantir, a lot of our automatic tests are full-chain tests. A backend server is fired up, client code runs against it, and everything runs much like a production environment. This makes intuitive sense because it’s a faithful approximation of how the system will run in the field.

However, there are some disadvantages to this:

• Full-pass tests don’t always localize the problem. Tests on a client class might fail even if it was the service that behaved incorrectly.
• These full-pass tests are relatively slow. Client code is running against an actual remote service. If a client is being tested, the server code still has to do work — sometimes a lot of work — even if that isn’t the focus of the test.
• The constraints of the test are loose. Full-chain tests can mostly only see whether the operation finished correctly. It’s much harder to figure out whether the operation was done efficiently and without making unnecessary service calls.
• They’re very little setup flexibility. If you want an RPC to return a specific value, you have little choice but to have your test get the service into a state where it can return that value. This is easy in some cases, but prohibitively difficult in others.
• Client tests are forced to share any non-determinism leaked from the service. For example, under real conditions, a request to call A might respond before call B, and sometimes the other way around. This can result in flaky tests or tests that don’t always simulate the conditions you want to exercise.

What’s to be done? Fortunately, there’s an option that handles these cases elegantly. We also test with jMock, a library that dynamically generates mock objects from arbitrary interfaces. These mock objects can be configured to check that particular methods are called with particular inputs a particular number of times, and then give prescribed responses.

Hit the link to see a concrete example of jMock in action.

jMock in action

Let’s say I want to test my object viewer page in Palantir Web, but I don’t want to fire up a dispatch server at all. First, I create my mock service object.

Mockery context = new Mockery();
final PalantirService service = context.mock(PalantirService.class);

Then, I set the expectations of my mock object. In this case, I want to tell my mock object to expect a call to PalantirService.getObject() and PalantirService.getDataSources(). getObject() will return a specific object. Any call made to the service apart from these will make the test fail.

context.checking(new Expectations() {{
        oneOf(service).getObject(realm.getId(), myObject.getId());
        will(returnValue(myObject));
        oneOf(service).getDataSources(myObject.getDataSources());
}});

Now, I create the object I want to test and inject the service.

ObjectViewController controller = new ObjectViewController();
controller.setService(service);

And then we fire away.

ModelMap model = new ModelMap();
controller.doGet(myObject.getId(), model);

Now that the controller (the class we’re exercising) has gone off and populated the model, we check to see that the model is populated correctly. Just like we would in any other test.

assertEquals(myObject.getName(), model.get("objectName"));
assertEquals(myObject, model.get("object"));

But in addition, we also assert that the expectations specified above were satisfied.

context.assertIsSatisfied();

Not only can we be sure that the right calls were made with the right parameters, but we can also be sure that no calls besides the expected calls were made. So the next time you want more speed or control over your tests, take a look at jMock or another framework like it. It’s a powerful tool in the effort to test your best!

A Palantir cluster seamlessly integrates many pieces of proven technology. One of them is our customized version of the venerable Java search engine, Lucene. Search engine technology tends to be optimized for the common use case of indexing web documents (or similar information architectures) where you have a few search terms in each query and many, many documents as results. We want to leverage the inverted index capabilities of Lucene, but our data access patterns are a bit different than the typical use case: we need things like pervasive range-querying, different types of relevance, and dynamic views of the data based on security constraints. So in building our data platform, we’ve run into some interesting challenges that are pretty unique in the information retrieval realm, specifically:

1. Raising memory efficiency
2. Real-time indexing
3. Preventing information leaks across access boundaries in an efficient manner

I’ll cover (1) in this post and (2) and (3) in a later post, due out in about two weeks.

Hit the link and we’ll delve into this topic.

Raising memory efficiency

We’ve addressed the issue of resource constraints, generally, in our earlier post: Bandwidth isn’t cheap. Disk isn’t cheap. CPU isn’t cheap. In that post, we posited “RAM to the rescue”:

On the other hand, some things in a SCIF are comparatively cheap. We never use boxes with less than 32GB of memory, and, in fact, lots of sites use 128GB of memory. RAM requires negligible power and cooling, and compared to disk, it’s relatively simple to install. It’s also easy to reconfigure the setup to use the additional memory.

While this is true, no matter how much RAM you buy, your users will find a way to use it all — search is no exception. In many of our environments, the search processes share hardware with other processes in the Palantir cluster, so while the OS may have 128 GB of RAM available, the search process’s VM has substantially less available to it. Compare this to a cluster of dedicated search nodes, where each node will have indexes sized to fit specifically into the memory available.

The upshot is that we needed to modify parts of Lucene to deal with tighter memory constraints than it was designed for.

Priority queue results accumulation

Most systems that implement search include some notion of paging through the results. We use a multi-level paging system, with the search server maintaining a server-side page for each query and serving smaller client-facing pages from.

Vanilla Lucene uses the following algorithm for accumulating search results:

1. Load all matching results.
2. Sort by some relevance metric(s).
3. Return the top n results.

The results are cached as a server-side page in case the client wants to load more than the first n results. You can see where this could run into trouble: if the total number of matching documents is high, that’s a lot of wasted RAM while we winnow it down to the size of the server page. So we use the following algorithm:

1. Construct a priority queue of constrained size with priority computed using the chosen relevance metric
2. Stream through the results, inserting into the queue
3. Return the set of results in the priority queue

Now we never need more RAM than the size of a server-side page to serve results. The downside is that if the client wants more than one server-side page, we have to run the search — in its entirety — twice (ouch). To avoid the first set of results, we adjust the priority queue to kick out all results that were in the first page based on relevance metric.

Using bitsets to optimize range queries

A range query can return a result set of very high cardinality – a range is a very compact way of describing a large set of matching terms (even if they are discrete values, like dates). One way to think about a range query of, say, 10 <= age <= 15, is that it expands to age = 10 OR age = 11 OR age = 12 OR age = 13 OR age = 14 OR age = 15. Rather than treat range queries in any special way, Lucene just does this expansion of the range and runs the query like a normal query.

Internally, Lucene stores a list of metadata nodes, ordered by document id, of each document that matches a given term. The algorithm goes something like this:

1. Open the document id lists for all matching terms
2. Walk the list pointers for each potential match such that you accumulate all the metadata for a given document.
3. Pass all this metadata up to the query processor which decides:
   1. Does this document match the overall query? (remember that terms can be inverted)
   2. Use term frequency taken from the metadata to calculate the relevance.

This structure and attendant algorithm has some nice properties:

• All documents are processed in a set order.
• Everything is known about a document all at once.
• It terminates in a single linear scan.

… and has one very nasty property:

• All of the term value buckets that match the range must be open simultaneously.

This is not a big deal for most English language queries. However, for large ranges and the like, there can be thousands or even millions of terms.

The semantics of range queries have an interesting feature: a document that matches the range twice is not more relevant than one that matches once. (Contrast this with a simple term query: multiple matches do indicate higher relevance). Being able to discard the accounting of how many time we match the range leads to a huge win:

1. We only need a single bit to represent a match
2. We can process a single term value bucket at a time instead of holding all buckets open in memory.

Our search engine accumulates range queries into bitset objects, allowing for a very compact representation of results. We need much less memory than we did before since we only load one term value bucket at a time. And the algorithm is simpler: no more walking pointers or O(n) check before figuring out which pointer moves next.

The next episode

Tune in for Palantir: search with a twist (part two) in a few weeks. I’ll cover the following topics:

• Real-time indexing
• Preventing information leaks across access boundaries in an efficient manner. (see Jason’s Multi-Level Security post over on the Palantir Government Analysis Blog for a high-level look at why these feature are important. and check out Bob McGrew’s “Access Control Model” White Video for in-depth look at how we apply security to our object model.)

Here at Palantir we use test-driven development (or TDD for short). Integrated tools like Eclipse and JUnit simplify writing and running unit tests. However, once you need to test a broader swath of functionality, it’s time to write functional, integration, and system tests. While technically not ‘unit testing’, the testing framework that JUnit provides is basically the same infrastructure that you want to leverage for writing these more involved types of testing.

When you’re developing enterprise software, functional testing often means getting your clients to talk to your servers. For the main Palantir Government product, we integrate the process of bringing the server up and down with the Ant scripts that run our automated unit tests: our testing tasks bring up the server, run the test suite, and then kill the server. This works great and produces nice results.

When I started working on our authentication server, the pattern that we had used before didn’t work for me. While the Palantir Government tests ran with a single, static configuration file, I needed to run the authentication server with multiple configurations in the course of running through the all the different functional tests. I determined that I needed a way to programmatically bring the server up and down for testing. In JUnit parlance, I needed a way to programmatically launch the server component as part of my setup() function for my unit tests and stop it in my teardown().

With my itch-to-scratch firmly in hand (or some other mixed metaphor), I set out to figure out how to invoke new Java processes from inside a unit test. The solution I came up with (with source code and examples) after the jump.

The Six Ingredients

So there are six ingredients that go into spawning a new VM:

• The classpath to use for the new VM
• The name of the class to run
• The directory to be used as the current directory for the process
• The command line arguments to pass to the process
• The set of Java system properties to use for this process
• The environment to pass to the process

Let’s look at each item individually.

Classpath

The classpath will tell the spawned VM where to load classes from. In JavaInvoke, we use the existing classpath (from the spawning VM) as a starting point and then prepend any new entries to allow overriding the classpath for the spawned VM.

This takes a lot of the tedium out of having to figuring out what to put in the classpath. Most likely, you want something similar to what you already have, if not completely identical.

We get the classpath from System.getProperty("java.class.path") and can add new entries by prepending the new entry, using the value of File.pathSeparatorChar as the entry delimiter. Using File.pathSeparatorChar makes the code cross-platform friendly (since the path separator is ‘;’ on Windows and ‘:’ on Unix (Linux, Solaris, OS/X, etc.).

Caveat: if you change the working directory and your original classpath was constructed using relative paths, you’ll probably have trouble getting anything to run (since your classpath will no longer point to right locations).

Class name

Pretty simple: what do you want to run in the spawned VM? The class must have a static void main(String args[]) defined, and it must be available for loading via the classpath.

Working Directory

If it should be different from the current working directory (CWD) of the running process, then set it and JavaInvoke will change it in the environment.

Command line arguments

If the process needs any command line arguments, including VM options, specify them in a string array. Note that not all of these arguments will necessarily make it to your main method, since the VM executable will parse it first and remove the VM arguments, passing through the program arguments.

Java System Properties

System properties can be used to control many aspects of how a VM runs. You can set them programmatically in your code or you can set set them on the command line by passing -Dkey=value. Our JavaInvoke implementation will take a Map of properties as a convenience argument; all it does is rewrite the map into the command line.

Process environment

This is an operating-system level construct. This is the set of environment variables, also in a Map that you would like merged with the current environment. This would be the place that you set things like LD_LIBRARY_PATH on Unix.

Dealing with input and output

So you might ask the question, “where does the output from the process go?” Or more troubling, “How do I send the process some input?” The Java Process object has methods to deal with this, allowing you to get streams that give you access to the input, output, and error streams of spawned process. That API is straight-forward to deal with, just like any other use of the java.io streams.

However, we want to make the typical case really easy: pulling the output from the spawned process back to the parent that spawned it. To that end, we add into the mix a class called OutputPiper. It fires up a thread that pulls all input from the spawned process, tags it with an identifier, and then outputs to the spawner’s stdout/stderr.

OutputPiper

(as extracted from ProcessSpawner.java)

	public static class OutputPiper extends Thread  {
		InputStream in;
		PrintStream out;
		String tag = null;

		public OutputPiper(String tag, InputStream in,PrintStream out) {
			this.in = in;
			this.out = out;
			this.tag = tag;
			// make sure that we don't keep the VM alive
			this.setDaemon(true);
			this.setName("OutputPiper-" + tag);
			out.println("Starting output piper for tag: " + tag);
			this.start();
		}

		@Override
		public void run() {
			try {
				BufferedReader reader = new BufferedReader(new InputStreamReader(in));
				String line = null;
				do {
					line = reader.readLine();
					if(line != null) {
						out.println(tag + ": " + line);
					}
				}while(line != null);
			}
			catch (Exception e) {
				//
			}
			out.println("Output piper exiting for tag: " + tag);
		}

		public static OutputPiper createOutputPiper(String tag, InputStream in, PrintStream out) {
			OutputPiper rc = new OutputPiper(tag, in,out);
			return rc;
		}
	}

Outpiper extends Thread so that all the output will arrive back to the controlling process in a timely manner. For each given process, we spawn off two OutputPipers, one for stdout and one for stderr, corresponding to the Process.getInputStream() and the Process.getErrorStream().

ProcessSpawner & JavaInvoke

There are two key classes in the example:

• ProcessSpawner.java – Essentially a wrapper around ProcessBuilder, a generic process spawner that makes it simple to invoke processes that that use OutputPipers to forward their output back to their parent. This class allows you to specify the working directory, process environment, and command line for the process to be invoked.
• JavaInvoke.java – a specialized subclass of ProcessSpawner, this class makes spawning new VMs a piece of cake, doing the necessary translation for Java system properties, setting the proper classpath environment variable with potential overrides, and fills in the fully qualified class name to run.

The Example & Source Code

I’ve put together a running example that implements a trivial client and server in JUnit test. The setup() method spawns the server and then the tests run the client code against the server, tearing it down after each test. It’s available in the PalantirVMSpawnerExample.zip zip file. Unzip it, run the run.sh or run.bat script as appropriate. It should generate output that looks like this:

-----------------------------------------------------
Starting test testAck
INFO [main] JavaInvoke - CLASSPATH=./lib/devblog-vmspawner.jar
INFO [main] ProcessSpawner - Build process spawner for the following command line:
INFO [main] ProcessSpawner - /home/pteng/java/i586/jdk1.5.0_14/jre/bin/java com.palantir.blog.processspawner.Server
Starting output piper for tag: server-stdout
Starting output piper for tag: server-stderr
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: [Socket Handler2]: Got message: some message
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: [Socket Handler3]: Got message: SHUTDOWN
Output piper exiting for tag: server-stdout
Output piper exiting for tag: server-stderr
Finished test testAck
-----------------------------------------------------
-----------------------------------------------------
Starting test testShutdown
INFO [main] JavaInvoke - CLASSPATH=./lib/devblog-vmspawner.jar
INFO [main] ProcessSpawner - Build process spawner for the following command line:
INFO [main] ProcessSpawner - /home/pteng/java/i586/jdk1.5.0_14/jre/bin/java com.palantir.blog.processspawner.Server
Starting output piper for tag: server-stdout
Starting output piper for tag: server-stderr
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: [Socket Handler3]: Got message: SHUTDOWN
Output piper exiting for tag: server-stdout
Output piper exiting for tag: server-stderr
Took 3 ms to send shutdown.
Took 335 ms for process to die.
Finished test testShutdown
-----------------------------------------------------
SUCCESS: all 2 tests passed

The source is included in the zip file, but if you wanted to look at it or link to it on the web, here are the classes involved:

• Client.java
• Example.java
• JavaInvoke.java
• ProcessSpawner.java
• Server.java
• ServerSpawningTest.java

And as an added bonus, there’s an Ant build.xml that will let you tweak and rebuild the demo yourself.

Comments and questions welcome. Enjoy.

The solution was to select a single event firing system. We wanted something that was easy-to-use yet powerful enough to express all the changes that might be made to a data model. Java’s Property Change Support (PCS) was a good fit because it can support arbitrary events in a very lightweight fashion.

Read on for details of our implementation…

Property Change Support

Java’s PropertyChangeSupport class (PCS) basically allows an object to easily fire events consisting of 4 pieces of information:

• source object – the thing that fired the event
• property name – allows the listener to tell events for different things apart
• old value – the old value for the property
• new value – the new value for the property

PCS handles all the bookkeeping for adding and removing listeners and firing events. It is very useful for creating listenable models, but we wanted to make it just a little bit easier by having an abstract class that exposed the add/remove listener calls and took care of initializing PCS:

public abstract class AbstractListenableModel implements Serializable {

    private static final long serialVersionUID = 1L;

    private transient PropertyChangeSupport pcs;

    protected AbstractListenableModel() {
        this.init();
    }

    /**
    * Adds a property change listener to the model.
    */
    public final void addPropertyChangeListener(PropertyChangeListener listener) {
        this.pcs.addPropertyChangeListener(listener);
    }

    /**
    * Removes a property change listener from the model.
    */
    public final void removePropertyChangeListener(PropertyChangeListener listener) {
        this.pcs.removePropertyChangeListener(listener);
    }

    /**
    * Fires a property change event to listeners of the model.
    */
    protected final void firePropertyChange(String propertyName, Object oldValue, Object newValue) {
        this.pcs.firePropertyChange(propertyName, oldValue, newValue);
    }

    /**
    * Initializes transient fields during deserialization.
    */
    protected Object readResolve() {
        this.init();
        return this;
    }

    /**
    * Initializes transient fields.
    */
    private void init() {
        this.pcs = new PropertyChangeSupport(this);
    }
}

AbstractListenableModel is basically just a simple wrapper for exposing the functionality of PCS. By extending this abstract class, it’s very easy to create a listenable model:

public final class MyModel extends AbstractListenableModel {

    public static final String PROP_FOO = MyClass.class.getName() + ".Foo";

    private int foo;

    public int getFoo() {
        return this.foo;
    }

    public void setFoo(int foo) {
        //
        // The semantics of the following line are a little hard to unpack,
        // but it does exactly what it needs to do, and the tradeoff
        // for conciseness over immediate readability is worth it for
        // large models with lots of properties.
        //
        // First, the JVM starts to create a stack frame for the call
        // into firePropertyChange().  It begins binding parameter values
        // from the left to the right.  The pointer to the String contained
        // in PROP_FOO is passed in first, then the current value of
        // this.foo is passed in, then the expression
        //        this.foo = foo
        // is evaluated (setting this.foo to the new value of foo), which
        // returns the new value of foo.  All the parameters are then
        // passed down into firePropertyChange(), which checks whether
        // the oldValue is equal to the newValue.  If they're not equal,
        // it fires the event.  If they are equal, it ignores the event.
        //
        this.firePropertyChange(PROP_FOO, this.foo, this.foo = foo);
    }
}

In this example, MyModel contains a single property called foo. When the value of foo is changed, a property change event will be fired to listeners of the model.

You may notice that the value of PROP_FOO is prefixed by the name of the class. This ensure that naming collisions do not occur for scenarios in which the same listener is used to listen to multiple models which happen to use the same property name. This scenario becomes much more likely in the case of event bubbling, which I’ll talk about next.

Event Bubbling

Imagine a scenario in which we have a nested model:

Normally, if a listener needs to receive events from both models A and B, it will need to add itself as a listener to each individual model. While this solution would work, it’s a little cumbersome, especially when model B can get swapped out for model B’—the listener then has to keep itself synched to the internal state of model A. It would be nice if model A could just automatically forward all the events from model B (or B’) via its PCS support so that a listener only needs to attach itself to one model instead of multiple models. With a bit more code in AbstractListenableModel, this is possible:

public abstract class AbstractListenableModel implements Serializable {

    private transient PropertyChangeListener childModelListener;

    ...

    /**
    * Registers a child model to this model.
    */
    protected void registerChildModel(ListenableModel childModel, String propertyName) {
        childModel.addPropertyChangeListener(this.childModelListener);
    }

    /**
    * Initializes transient fields.
    */
    private void init() {
        ...
        this.childModelListener = new ChildModelListener();
    }

    /**
    * Listener for property change events fired from child models.
    */
    private final class ChildModelListener implements PropertyChangeListener {
        public void propertyChange(PropertyChangeEvent event) {
            // This is where the bubbling happens
            pcs.firePropertyChange(event);
        }
    }
}

Now, whenever model B fires a property change event, this event will also be fired by model A. This makes it much easier for the listener to listen to events arbitrarily deep in the model hierarchy, because each event fired by a child model gets re-fired (bubbled) by all its ancestors. All you have to do is attach a listener to the root model and you’ll automatically receive events from all models in the hierarchy.

Note that the registerChildModel method above takes an unused propertyName argument. In the full implementation of this class, events with the provided property name are monitored. When an event with the provided property name is fired, childModelListener is detached from the old child model and attached to any new child model. This ensures that the listenable model is always listening to the current child models.

Events for Collections

Any model event support would not be complete without some consideration of how to handle collections such as sets and lists. To solve this scenario, we created specialized collection classes called ListenableModelSet and ListenableModelList. These collections hold AbstractListenableModels as their elements and fire events whenever their contents change. Since the changes to collections can vary widely, the solution we came up for communicating collection changes with full fidelity is basically to fire events with a copy of the old set as the old value and the new set as the new value. Listeners can then diff the old and new values to determine exactly what changed if necessary. Additionally, each ListenableModelSet or List adds a ChildModelListener to all of its children (themselves AbstractListenableModels), thereby ensuring that events are bubbled from all models in the collection.

Conclusion

Just as we saw with the Adapter piece of the MVA triad, when we componentize the Model piece there are huge gains to be had. Once we started using a base Model class and a consistent eventing infrastructure (PropertyChangeSupport), we could add features that made coding across our entire application a lot more pleasant.

It’s always fun to release a new piece of jargon into the wild. I’ve run into a number of bugs in our codebase that caused by an anti-pattern I’d like to dub The Pokémon Problem.

Much like the game of Whac-a-Mole, this is a class of bugs where fixing every occurrence does not prevent the bug from returning in new code: it is easy for code delta to result in an instance of the bug being re-introduced into the code base. Even if you “catch ‘em all“, nothing prevents someone else from introducing new Pokémon bugs later.

Not only is this bug easy to re-introduce, but it sometimes can be hard to find all currently existing instances of this pattern. Although tools like Eclipse make it easier to track down all the places that code is called, sometimes you’re looking for things that happen in a certain sequence (which tools like Eclipse don’t do a good job of searching for) and dynamic invocation mechanisms like Java Reflection can sometimes make it impossible to be exhaustive. This type of bug is also resistant to automated refactoring: changing the protocol of dealing with this corner of your code will require you to track down all places it was touched and manually refactor them. It generally signals a failure to use sufficient separation of concerns.

In general, this anti-pattern is a result of APIs that require the caller to be responsible for state management of resources that the API owns. This can include things like an object that requires the caller to have run an initialization method before calling any other method on the object. These bugs get even more insidious when a failure to do things in the right order does not cause a hard failure (like throwing an exception) but instead creates some sort of subtle corruption that may not be noticed or cause subsequent calls to fail unexpectedly.

Read on for some strategies on dealing with the Pokémon problem.

Solving the Pokémon Problem

How do we solve the Pokémon problem?  Sometimes it is as easy as writing a unit test to verify that no new Pokémon exist. More often than not, though, you’ll have to use better encapsulation to solve the problem. If the Pokemon problem is the anti-pattern, encapsulation is its opposite.

For an example, I’ll turn to a real problem I ran into in the Palantir Government codebase: Let’s say I have class Parser and it has a method getAttribute(). I want to add the ability to have shortened (obfuscated) tag names to the Parser class.

The wrong approach

One approach I can take:

1. Create a class called AttributeHandler that wraps calls to Parser.getAttribute(). It handles detection of the short or full XML dialect and then returns the appropriate attribute value.
2. Stick an instance of this AttrbributeHandler class in a member variable of Parser and make sure to call AttributeHandler.getAttribute() instead of Parser.getAttribute() when processing attributes.
3. Put comment on Parser.getAttribute() mentioning that it shouldn’t be called directly anymore.

I just introduced a Pokémon problem.  The next person that edits this code may not read my comment and know that calling Parser.getAttribute() is a bug.  They may not test the short tag case and find the bug. How do I fix this Pokémon problem? 

The right approach

Approaching this encapsulation problem has a number of different approaches one could take. Here’s what I chose:

1. Create a class called AttributeHandler that wraps calls to Parser.getAttribute(). It handles detection of the short or full XML dialect and then returns the appropriate attribute value.
2. Stick an instance of this AttrbributeHandler class in a member variable of Parser.
3. Create a wrapper class around Parser that delegates the getAttribute() call to the AttributeHandler member when processing attributes.

(You might ask why I used a delegate instead of sub-classing the parser and overriding the getAttribute() method: it didn’t fit for architectural reasons that aren’t relevant here). Now the developer no longer has access to the Parser.getAttribute() method that would cause the undesired behavior.

In-depth Example: Resource Management

Another common place you might run into the Pokémon problem is code that needs to clean up resources.  Some coders are lazy and don’t want to always write their try/finally/close blocks properly. This can lead to resource leaks; for things like file handles, this usually isn’t a large issue, but for scarce resources like database connections, it’s critical that things get cleaned and returned to the pool. For locks, it’s absolutely essential to avoid deadlocks.

The wrong way

Here’s a simple example of bad resource management. In this (admittedly contrived) scenario, we have class that’s managing a resource for us, in this case it’s a status file. Different parts of the app need to read the status file where, presumably, something is storing its status.

Here’s the naive implementation of the status manager class:

public class PokemonStatusFileManager {

	final File statusFile;

	public PokemonStatusFileManager(File statusFilePath) {
		this.statusFile = statusFilePath;
	}

	public InputStream getStatusFileInputStream() throws FileNotFoundException {
		return new FileInputStream(this.statusFile);
	}
}

Based on this implementation, here’s some code that would use it. Note that first method, parseStatusFileIncorrectly() does no error checking and may not close the InputStream properly if an exception is thrown. The second method does proper resource handling, but it’s kind of ugly to read.

public class PokemonParseStatusFile {

	/**
	 * This is not proper resource management.
	 * @param manager
	 * @throws IOException
	 */
	public static void parseStatusFileIncorrectly(PokemonStatusFileManager manager)
		throws IOException {

		InputStream statusFile = manager.getStatusFileInputStream();
		readFrom(statusFile);
		statusFile.close();

	}

	/**
	 * This is proper resource management, but it's tedious to have to write.
	 * Some coders are too lazy to always do this the right way.
	 * @param manager
	 * @throws IOException
	 */
	public static void parseStatusFileCorrectly(PokemonStatusFileManager manager)
		throws IOException {

		InputStream statusFile = null;
		try {
			statusFile = manager.getStatusFileInputStream();
			readFrom(statusFile);
		} finally {
			// carefully close the resource
			if(statusFile != null) {
				try {
					statusFile.close();
				} catch(Exception e) {
					System.err.println("Error closing statusFile!");
					e.printStackTrace(System.err);
				}
			}
		}
	}

	static void readFrom(InputStream statusFile) throws IOException {
		// do the reading here...
	}
}

So this produces a classic Pokémon problem: everyone who interacts with the StatusFileManager has to do proper resource handling. Now imagine that this is an important lock instead of just a file handle: this Pokémon problem could cause deadlock (which would be bad).

So how do we fix this? The Visitor Pattern.

Solving the Pokémon problem with the visitor pattern

The Visitor Pattern is a potent weapon in fighting the Pokémon problem: it allows you to fully encapsulate access to a resource by injecting into the places it’s needed but preserving overall control of the resource in the code that “owns” the resource. Classically used for controlling things like iteration order, here the visitor pattern is applied as a form of dependency injection to enable lifecycle management.

The visitor pattern as applied to resource management is fairly straightforward:

1. Define an interface, ResourceVisitor with one method, visit(Resource r) (where Resource is the type of resource we’re managing.
2. Define the resource manager with a method that takes a ResourceVisitor as a parameter. Manage the lifecycle of the resource and call ResourceVisitor.visit(r) when appropriate, handling all initialization, error-handling and cleanup.

Here’s our re-spin of the status file manager class. You’ll notice that we’ve moved the resource handling code from the correct example above and encapsulated it in the visitor pattern:

public class UnPokemonStatusFileManager {

	/**
	 * Visitor interface implemented by callers wishing to
	 * interact with the status file.
	 */
	public static interface StatusFileVisitor {
		public void visitStatusFile(InputStream statusFile) throws IOException;
	}

	final File statusFile;

	public UnPokemonStatusFileManager(File statusFilePath) {
		this.statusFile = statusFilePath;
	}

	/**
	 * Note that this method is now private.
	 */
	private InputStream getStatusFileInputStream() throws FileNotFoundException {
		return new FileInputStream(this.statusFile);
	}

	/**
	 * Here's the method that takes the visitor and
	 * fully encapsulates the lifecycle of the status file.
	 */
	public void parseStatusFile(StatusFileVisitor parser)
	throws IOException {
		InputStream statusFile = this.getStatusFileInputStream();
		try {
			parser.visitStatusFile(statusFile);
		} finally {
			// carefully close the resource
			try {
				statusFile.close();
			} catch(Exception e) {
				System.err.println("Error closing statusFile!");
				e.printStackTrace(System.err);
			}
		}
	}
}

Now that we’ve encapsulated the complexity of the resource handling the UnPokemonStatusFileManager class, code that needs to access the status file can be written in a highly correct manner without much work.

public class UnPokemonParseStatusFile {

	public static void parseStatusFile(UnPokemonStatusFileManager statusFileManager)
		throws IOException {

		// generate anonymous visitor to do the processing
		StatusFileVisitor visitor = new StatusFileVisitor() {
			public void visitStatusFile(InputStream statusFile) throws IOException {
				readFrom(statusFile);
			}
		};
		statusFileManager.parseStatusFile(visitor);
	}

	static void readFrom(InputStream statusFile) throws IOException {
		// do the reading here...
	}
}

By encapsulating the full lifecycle of the status file in the visitor pattern, I’ve ensured that it’s always accessed properly. If I change the place where we store status to be a database rather than a file, nothing needs to change except this one class; the calling code remains the same.

We now have easy re-factoring, no resource leaks, and have simplified calling code. And finally: there are no new bugs to be introduced by callers that aren’t sure how to use our resource. Looks like we caught ‘em all!

Wrapping It All Up

So there it is, your new piece of jargon: The Pokémon Problem anti-pattern. You heard it here first! Please post any other great examples to the comments section on this post.

Distributed systems are complex. Getting them right is hard, and when things don’t go right, it can be difficult to understand what went wrong. In an environment like ours, a good monitoring system isn’t just nice to have; it’s a critical component necessary for understanding behavior and diagnosing problems.

We had three primary goals for the initial monitoring system: graphing of time-series data, alerting on event triggers, and notifications to users. Furthermore, as a product company, we had a design goal of a simple, intuitive (yet powerful and flexible) solution.

Before starting, we did a quick survey of existing open-source packages. Unfortunately, nothing we found quite fit our needs, given our specific requirements of security, protocol, licensing, and integrability into our product. Given that, we made the decision to forge ahead and build our own; we try not to re-invent the wheel but it seemed to make sense here.

For an in-depth look at the architecture of the Monitoring Server and components we used to build it, read on…

Architecture

At the highest level, a two-tiered architecture made the most sense. The back-end, standalone server component would be responsible for collecting, processing, and exposing data through an API. The front-end component would be web-based portlets integrated into our existing management interface.

The server architecture was designed to allow generic components to work together, with everything connected up via Spring. While we started with JMX as our collection method for monitoring data, the architecture sees this as just one pluggable component, with multiple data backends supported. A Spring webservices API allows the front-end portlets to query and manipulate the components at each level.

For our first shipping release, we’ve only shipped the JMX backend, and so this is what production architecture looks like for now:

Components

Any time you choose build instead of buy, there’s a lot of work to be done to get the full set of functionality you need. Fortunately, the Java platform has an extremely rich set of freely available projects and libraries, and we leveraged many of them for the back-end:

• JMX: the core of our system, the Java Management Extensions is a standard for managing and monitoring applications. We use JMX to instrument and monitor our own servers, and because it’s an adopted standard, we gain access to MBeans exposed by third-party components as well.
• rrd4j: round-robin databases (RRDs) are an excellent storage format for time-series data, and RRD4J is a pure Java implementation of the legendary RRDTool. The round-robin format allows for a fixed size file, since older data is overwritten as newer data arrives. The multi-resolution aspect of the files provides long historical views without a space premium. For example, an RRD can contain a high resolution series for recent information and a low resolution series for long-term data.
• HSQLDB: a lightweight, native Java, SQL database that can be run in-process. We use HSQLDB to store all non–time-series information, such as metadata about metrics we’re monitoring.
• Quartz: an open source job scheduling system, we use Quartz primarily for scheduling Alerts. Alerts run periodically to check for a condition, and notify if triggered. Each Alert’s wait period is specified by the user, and fortunately, with Quartz it’s easy to schedule many Alerts at different frequencies.
• Groovy: self-described as “an agile dynamic language for the Java Platform,” Goovy is integrated into our alerting system. Alerts can contain Groovy scriptlets, which give us the expressiveness to create Alerts such as “alert if a metric’s average value over the past 5 minutes is greater than X,” or “alert if the variation of a set of metrics’ values across all servers of type Y is greater than Z.”
• JavaMail: a full-featured email framework. Supports SSL/TLS secure connection protocols, which our clients require.
• JAXB: a simple-to-use Java to XML API, JAXB allows us to convert XML into Java objects (and vice-versa). We use JAXB for parsing configuration files and persisting objects into HSQLDB.
• Spring: a framework for developing enterprise Java applications, Spring is the foundation for our monitoring server.

  Having never used a component framework before, using Spring’s Inversion of Control and Dependency Injection paradigms to build an application turned out to be a pleasant and educational experience. While it enforced discipline in using interfaces, it rewarded us with the ability to easily swap implementations of a component. For example, switching to a HSQLDB-based data store required only a single-line edit, and everything just worked. Seriously.

  We also leveraged Spring early in our development process: we pair-coded interfaces, created stub objects, and then wired everything up in Spring. Once our skeleton was in place, we independently worked on component implementations and swapped them in as they were completed. Later in the cycle, we used Spring in our unit tests to compose our application differently for specific tests, isolating important functionality and using dummy components for non-relevant areas.

User Interface

By moving the user interface into the portlets, we were able to re-skin the fairly ugly native graphing capability that rrd4j provides with a more generic solution that looks good. For comparison, here’s an MRTG style graph produced by rrd4j:

And here’s some graphs from our Monitoring Server (note the portlet UI components for controlling display of the graphs):

While the difference is not that stark, our graphs are much easier on the eyes.

Monitoring Server: present and future

We recently released the monitoring system, and it’s already providing insights into our product’s behavior. We have more features planned: eventing, which will help us track system events such as a server restart or job completion; generating new time-series data from existing data (for example, a series of the rolling standard deviation of a metric, or the number of failure events in the past 24 hours), and Groovy scripting directly against the monitoring server. The last feature is particularly helpful when our engineering team can’t physically access a system due to security restrictions.

From an analysis perspective, we can now start to better understand our system’s behavior, which will help us identify problems before they occur and help steer our development energy going forward. Even the world’s best data analysis software needs a little analysis itself sometimes.

At Palantir, we write software that gets deployed at each client, integrated across their sensitive data sets, and maintained and administered by that client’s in-house admins. Most deployed enterprise software is run on a single beefy box: consider wikis, blogging systems, bug tracking systems, or practically any client/server or web client software software used today. On the other hand, most enterprise software that runs as a distributed system is hosted: Salesforce.com, Google Apps, or any approach that sells software as a service. What’s fairly unusual about our software is that it’s deployed as a distributed system at each client.

Distributed systems are hard to build and hard to maintain. As long as that distributed system is built and maintained in-house, however, you have a number of advantages:

• The administrators are full-time product experts who are focused on the mission of keeping your system available and responsive.
• The development organization can build internal tools for the administrators that only have to be “good enough” and can step in if necessary.
• It’s easy to get feedback on how the system performs, because there are no sensitivity, privacy, or legal constraints.
• A single, large deployment allows you to optimize your hardware purchasing and amortize installation headaches across a large number of machines.

This is all great, of course, and if you can host and maintain your distributed system yourself, I’d highly recommend it. Sometimes, however, it’s just not possible. At Palantir, the client data we work with is so sensitive that even we cannot see it, except under very strictly controlled circumstances. It’s also so large that the bandwidth limitations of pushing it into a system hosted by us would be prohibitive.

So suppose that you have to deploy your distributed system in a customer datacenter with external parties maintaining the system. What do you need to consider? In this post, I’ll go into a number of key points that we have faced and addressed at Palantir.

Understand Your Administrators

Assume that your administrators are part-time, not product experts, and constantly distracted by their other responsibilities. They aren’t even experts in the technologies your system is based on: for example, they don’t really know much about databases and they are more comfortable with Windows than with Linux. Even if these assumptions aren’t all true in any particular case, there will be administrators who meet each of these assumptions.

Design For Manageablility

This means building powerful management tools for your system that are web-based and also scriptable. Remember that your administrators are part-time, so usability is important: by the time your administrator touches the Foobar Configuration Widget the second time, he’s forgotten everything he learned a month ago when he did it the first time. You also want to build management tools that go all the way down the stack: using low-level tools for occasional jobs leads to mistakes, because those low-level tools tend to be far more powerful than necessary for your system.

Design In Monitoring And Notification

Visibility is one of the biggest reasons people want control – but unnecessary control leads to mistakes. Your administrator shouldn’t have to go to the command line to run top just to figure out whether the system is overloaded. Each metric that is being monitored needs to have historical data so that your system can distinguish baseline behavior from anomalies. Each metric displayed to an administrator needs to be displayed with context, whether that’s the mean and standard deviation of the metric, or whether it’s similar metrics on other servers. Anomalous behavior should trigger human action through a notification.

Notifications also need to be carefully designed (as well as extensible by the administrator). Carefully distinguish actionable items from non-actionable items, and try to reduce ambiguity as to what action is required. It’s similar to error logging: if you let standard system events pollute your error logs, the administrator will soon stop paying attention to them. Although you may not be able to send monitoring information directly back to the development team, you may also want to prepare reports of what’s gone wrong to collect every so often; just make sure that these reports are human-readable so that they can be vetted to make sure they don’t leak any sensitive information.

Design For Autonomy

Where possible, design the system to handle error conditions that can be systemically fixed. The best kind of failure is one that requires no admin intervention. If you can automatically extend your tablespace when it runs out of allocated space, do it. But be sure to give sufficient warning if a long lead-time action is going to be required (like ordering and installing an additional disk). You won’t be able to figure out everything that can go wrong ahead of time, but you can iterate to drive down the number of events for which human intervention is required.

In future posts, we plan to drill down on each of these challenges and look at what approaches and technologies worked for us.