This is a response to Ari’s awesome post on human-computer symbiosis. Ari and I were chatting about the equation he developed and I was wondering if there were some further refinements that are possible… let’s take a look:

We are attempting to understand the total analytic capability for a given task a of a human-computer team. Analytic capability in this case probably means:

(1)

Where A is the answer to the analytic problem in question and t_A is the time needed to arrive at the answer based on the inputs available. In the case of chess, A could be the optimum next move given all previous information and t_A would be how long it takes to decide on this move.

Read on for a look at how this generalizes in human-computer symbiotic systems.

In the case of the human-computer team, we know that a is going to be a function of both the human’s analytical capability h and the computer’s analytical capability c (where both h and c have units of answers/time). In the limit case we know that:

(2)

(3)

Or in plain English, if there is no human present, the total analytic capability is simply the analytic capability of the computer. So the naïve solution would be that:

(4)

(4) clearly meets the limiting cases described in (2) and (3). Kasparov noticed a mixing function where the ability of the human and computer to work together becomes the dominant term — we might call this the mixing capability for the given task or m. Including this phenomenon, the total analytic capability (4) would be re-defined as:

(5)

where m has the property that:

(6)

Thus maintaining the limits expressed in (2) and (3) and adhering to the observation that if there is no human or computer component then there will be no mixing advantage. A naïve solution to this constraint would be simple linear mixing:

(7)

where M (units of time per answer) is the mixing efficiency and will be primarily based on the type of task being solved — some analytical tasks lend themselves to a combined process more than others (for example, multiplying 20 digit numbers does not really benefit from the intuition of a human so the ability of a human and computer to perform this task is merely their additive ability).

What Kasparov noticed is that the mixing was primarily based on the quality of the process rather than the analytical power of either the human or computer separately. This seems to imply that we must somehow account for the fact that the quality of the human-computer interface is responsible for the quality of the mixing. This can be modeled as a unitless friction of interaction f_i that impedes the ability of the human and computer to work together.

Equation (7) can thus be re-written as:

(8)

In this case, the maximum value for the mixing capability is realized when the friction of interaction goes to zero. This mixing capability is the same as the equation Ari developed (less the coefficient which is necessary to maintain consistent units throughout).

We can now re-write our analytic capability in (5) as:

(9)

Below, see a plot of this function over a range of values for h, c and f_i:

As can clearly be seen from this functional plot (note the vertical scale), the effect of interface friction dominates over the other terms whenever both the human and computer can make important contributions to the task at hand. The conclusion can be drawn that the most effective way to solve analytical problems is to minimize the friction of the human-computer interface; or to put it another way: optimal analytical systems are those that are built specifically to maximize the ability of the human to leverage the ability of the computer.

I am certain there is still the possibility for further refinement, for example:

(10)

Here at Palantir, a lot of our automatic tests are full-chain tests. A backend server is fired up, client code runs against it, and everything runs much like a production environment. This makes intuitive sense because it’s a faithful approximation of how the system will run in the field.

However, there are some disadvantages to this:

• Full-pass tests don’t always localize the problem. Tests on a client class might fail even if it was the service that behaved incorrectly.
• These full-pass tests are relatively slow. Client code is running against an actual remote service. If a client is being tested, the server code still has to do work — sometimes a lot of work — even if that isn’t the focus of the test.
• The constraints of the test are loose. Full-chain tests can mostly only see whether the operation finished correctly. It’s much harder to figure out whether the operation was done efficiently and without making unnecessary service calls.
• They’re very little setup flexibility. If you want an RPC to return a specific value, you have little choice but to have your test get the service into a state where it can return that value. This is easy in some cases, but prohibitively difficult in others.
• Client tests are forced to share any non-determinism leaked from the service. For example, under real conditions, a request to call A might respond before call B, and sometimes the other way around. This can result in flaky tests or tests that don’t always simulate the conditions you want to exercise.

What’s to be done? Fortunately, there’s an option that handles these cases elegantly. We also test with jMock, a library that dynamically generates mock objects from arbitrary interfaces. These mock objects can be configured to check that particular methods are called with particular inputs a particular number of times, and then give prescribed responses.

Hit the link to see a concrete example of jMock in action.

jMock in action

Let’s say I want to test my object viewer page in Palantir Web, but I don’t want to fire up a dispatch server at all. First, I create my mock service object.

Mockery context = new Mockery();
final PalantirService service = context.mock(PalantirService.class);

Then, I set the expectations of my mock object. In this case, I want to tell my mock object to expect a call to PalantirService.getObject() and PalantirService.getDataSources(). getObject() will return a specific object. Any call made to the service apart from these will make the test fail.

context.checking(new Expectations() {{
        oneOf(service).getObject(realm.getId(), myObject.getId());
        will(returnValue(myObject));
        oneOf(service).getDataSources(myObject.getDataSources());
}});

Now, I create the object I want to test and inject the service.

ObjectViewController controller = new ObjectViewController();
controller.setService(service);

And then we fire away.

ModelMap model = new ModelMap();
controller.doGet(myObject.getId(), model);

Now that the controller (the class we’re exercising) has gone off and populated the model, we check to see that the model is populated correctly. Just like we would in any other test.

assertEquals(myObject.getName(), model.get("objectName"));
assertEquals(myObject, model.get("object"));

But in addition, we also assert that the expectations specified above were satisfied.

context.assertIsSatisfied();

Not only can we be sure that the right calls were made with the right parameters, but we can also be sure that no calls besides the expected calls were made. So the next time you want more speed or control over your tests, take a look at jMock or another framework like it. It’s a powerful tool in the effort to test your best!

[A number of weeks ago, we published a post on the search technology used by Palantir. That post covered raising the memory efficiency of a couple of operations. This is part two of that series.]

The most familiar use of search engines is to index documents made available on the Internet via the hypertext transfer protocol. Forgotten names like AltaVista, names not-yet-really-learned like Bing, and, of course, Google come to mind.

This one, massive use case has a couple of properties that I’d like to highlight:

• Asynchronous indexing and querying – web search engines tend to use crawlers and indexers to build up an index of the web. After each crawl is finished, the new index is brought online for use by the query engine.
• Lack of access controls – all the data in the index is available to any query. In fact, most queries are (from the standpoint of the index) completely anonymous.

Palantir: not a web search engine

Search technology is just one part of what makes up a Palantir system. For us, it’s a way to quickly retrieve Palantir objects in a Palantir system, it’s not the whole of the application.

I’d like to highlight a couple of differences from the web search engine case. A Palantir system needs the following properties:

• Realtime indexing and querying – we need information to be available immediately as it changes in the system.
• Leak-proof access controls – we need the search engine to help us make sure that we don’t have information leaking across access control boundaries.

Hit the link to read more about these topics.

Realtime indexing

The Palantir platforms implement realtime indexing: as soon as an analyst changes an object in the system, it needs to be available to query. This could be a change to data in the object or a change to the security tags on the object.

From a programming perspective, this is pretty straightforward: a Palantir transaction will not commit until the search engine is finished indexing the new data.

From a search engine operational perspective, this induces some challenges. Asynchronous indexing allows the search engines to bring online a highly optimized static form of the index. Contrast that with realtime indexing, where every cycle spent optimizing the index is removing cycles from serving other queries and there is likely a human waiting for the optimizing process to finish.

When using the static index, a query only accesses one, optimized index file which then points to the documents containing the results. However, as changes and additions are indexed into the system, there is a lot of overhead to merging them into the master index.

Instead of merging and optimizing on every change, Lucene can keep around a number of smaller indexes that hold all the fresh entries. These are fixed-size append-only segments that are much cheaper to write to than the optimized and merged form of the index. So basically, these ‘dynamic’ indexes are linear lists of single-document indexes. When the search engine goes to run a query it has to follow this simple (yet expensive) algorithm:

• Query the static, merged index, accumulating results. (this part is reasonably fast)
• For each of the dynamic indexes:
  • Open the file, incurring IO overhead.
  • Query each single-document index and look for additional records or newer records that supersede one of the existing found results.

You can see how the overhead of this can quickly get pretty large as the number of dynamic indexes grows: it grows linearly with number of new indexed records. Compare that with the optimized index, which should be close to constant time for any given query.

To get around this, the indexer will only allow a certain number of these dynamic indexes to accumulate before it kicks off a background merge job. During the merge job, we take a noticeable performance hit, but by batching up the merge run we amortize the overhead away for an overall performance win. This hybrid mode didn’t require us to write any new code, but just to tune Lucene to give us the performance profile we wanted.

Preventing Information Leaks

The Palantir data platform has a fairly sophisticated security model baked in (see The White Videos for a more in-depth look at the security model). One of the features that we have implemented is the ability to show a narrower view of an object based the user’s permissions: the user only sees the slice of the data that they have been granted access to. Part of the complexity in implementing this was that we can’t even hint that the other, hidden data exists at all.

Search engines ranks their results by relevance, showing the matches to the query that it believes to be most relevant first. One common way to make these relevance calculations is by comparing the length of the search term or phrase to the length of the term that it matched. Consider the search term ‘king’: it will match the following phrases:

• “I’m the king of the world!”
• “King salmon are often found in the Pacific Northwest and are also known as Chinook salmon.”
• “Yes, my king.”

Using a length-computed relevance, the phrase, “Yes, my king.” is the most relevant.

Getting back to the Palantir object model: for each distinct set of permissions that an object has, we compute a different object label based on the properties that are visible to that particular slice. These multiple titles all go into the search engine. If we were to compute relevance based on the length of the phrase that matched, and the shortest match on the object is shorter than the match that is actually visible to us, we could return the object with a higher-than-obvious relevance. If we were to do that, we’d be leaking information, namely that there’s data on this object that the user making the query is not privy to. (Note that filtering of objects that aren’t at all visible to the user is done in a higher layer after the results have been accumulated and ranked by the search engine.)

Given this problem, there are two approaches one can take:

1. Store all the information needed to decide which labels are visible to the user running the query and then use only the visible labels when calculating the relevance of a match. Note that is a pretty expensive operation.
2. Don’t use the length of match to compute relevance. Note that skipping a relevance calculation is, obviously, a very cheap thing do.

Which do we do? Both.

When matching against object labels, the length metric actually lets us discern between better and worse matches. So in that case, we incur the cost of this calculation in order to return higher quality results.

However, when matching against things like document bodies, the ratio of the size of the match to the size of the search term starts to have less meaning but still has the possibility of leaking information in the query results. For fields like this, we turn off the relevance calculations based on length of match. The upshot is the we don’t have to store the permissions information in the index nor incur the cost of the permissions/views calculation for these fields.

A heartfelt thank you

To be clear, this post highlights the ways in which our search code diverges from the main Lucene code base. We’re huge fans of Lucene and have great respect for the developers that built and maintain what is probably the world’s greatest open-source search engine.

A Palantir cluster seamlessly integrates many pieces of proven technology. One of them is our customized version of the venerable Java search engine, Lucene. Search engine technology tends to be optimized for the common use case of indexing web documents (or similar information architectures) where you have a few search terms in each query and many, many documents as results. We want to leverage the inverted index capabilities of Lucene, but our data access patterns are a bit different than the typical use case: we need things like pervasive range-querying, different types of relevance, and dynamic views of the data based on security constraints. So in building our data platform, we’ve run into some interesting challenges that are pretty unique in the information retrieval realm, specifically:

1. Raising memory efficiency
2. Real-time indexing
3. Preventing information leaks across access boundaries in an efficient manner

I’ll cover (1) in this post and (2) and (3) in a later post, due out in about two weeks.

Hit the link and we’ll delve into this topic.

Raising memory efficiency

We’ve addressed the issue of resource constraints, generally, in our earlier post: Bandwidth isn’t cheap. Disk isn’t cheap. CPU isn’t cheap. In that post, we posited “RAM to the rescue”:

On the other hand, some things in a SCIF are comparatively cheap. We never use boxes with less than 32GB of memory, and, in fact, lots of sites use 128GB of memory. RAM requires negligible power and cooling, and compared to disk, it’s relatively simple to install. It’s also easy to reconfigure the setup to use the additional memory.

While this is true, no matter how much RAM you buy, your users will find a way to use it all — search is no exception. In many of our environments, the search processes share hardware with other processes in the Palantir cluster, so while the OS may have 128 GB of RAM available, the search process’s VM has substantially less available to it. Compare this to a cluster of dedicated search nodes, where each node will have indexes sized to fit specifically into the memory available.

The upshot is that we needed to modify parts of Lucene to deal with tighter memory constraints than it was designed for.

Priority queue results accumulation

Most systems that implement search include some notion of paging through the results. We use a multi-level paging system, with the search server maintaining a server-side page for each query and serving smaller client-facing pages from.

Vanilla Lucene uses the following algorithm for accumulating search results:

1. Load all matching results.
2. Sort by some relevance metric(s).
3. Return the top n results.

The results are cached as a server-side page in case the client wants to load more than the first n results. You can see where this could run into trouble: if the total number of matching documents is high, that’s a lot of wasted RAM while we winnow it down to the size of the server page. So we use the following algorithm:

1. Construct a priority queue of constrained size with priority computed using the chosen relevance metric
2. Stream through the results, inserting into the queue
3. Return the set of results in the priority queue

Now we never need more RAM than the size of a server-side page to serve results. The downside is that if the client wants more than one server-side page, we have to run the search — in its entirety — twice (ouch). To avoid the first set of results, we adjust the priority queue to kick out all results that were in the first page based on relevance metric.

Using bitsets to optimize range queries

A range query can return a result set of very high cardinality – a range is a very compact way of describing a large set of matching terms (even if they are discrete values, like dates). One way to think about a range query of, say, 10 <= age <= 15, is that it expands to age = 10 OR age = 11 OR age = 12 OR age = 13 OR age = 14 OR age = 15. Rather than treat range queries in any special way, Lucene just does this expansion of the range and runs the query like a normal query.

Internally, Lucene stores a list of metadata nodes, ordered by document id, of each document that matches a given term. The algorithm goes something like this:

1. Open the document id lists for all matching terms
2. Walk the list pointers for each potential match such that you accumulate all the metadata for a given document.
3. Pass all this metadata up to the query processor which decides:
   1. Does this document match the overall query? (remember that terms can be inverted)
   2. Use term frequency taken from the metadata to calculate the relevance.

This structure and attendant algorithm has some nice properties:

• All documents are processed in a set order.
• Everything is known about a document all at once.
• It terminates in a single linear scan.

… and has one very nasty property:

• All of the term value buckets that match the range must be open simultaneously.

This is not a big deal for most English language queries. However, for large ranges and the like, there can be thousands or even millions of terms.

The semantics of range queries have an interesting feature: a document that matches the range twice is not more relevant than one that matches once. (Contrast this with a simple term query: multiple matches do indicate higher relevance). Being able to discard the accounting of how many time we match the range leads to a huge win:

1. We only need a single bit to represent a match
2. We can process a single term value bucket at a time instead of holding all buckets open in memory.

Our search engine accumulates range queries into bitset objects, allowing for a very compact representation of results. We need much less memory than we did before since we only load one term value bucket at a time. And the algorithm is simpler: no more walking pointers or O(n) check before figuring out which pointer moves next.

The next episode

Tune in for Palantir: search with a twist (part two) in a few weeks. I’ll cover the following topics:

• Real-time indexing
• Preventing information leaks across access boundaries in an efficient manner. (see Jason’s Multi-Level Security post over on the Palantir Government Analysis Blog for a high-level look at why these feature are important. and check out Bob McGrew’s “Access Control Model” White Video for in-depth look at how we apply security to our object model.)

Here at Palantir we use test-driven development (or TDD for short). Integrated tools like Eclipse and JUnit simplify writing and running unit tests. However, once you need to test a broader swath of functionality, it’s time to write functional, integration, and system tests. While technically not ‘unit testing’, the testing framework that JUnit provides is basically the same infrastructure that you want to leverage for writing these more involved types of testing.

When you’re developing enterprise software, functional testing often means getting your clients to talk to your servers. For the main Palantir Government product, we integrate the process of bringing the server up and down with the Ant scripts that run our automated unit tests: our testing tasks bring up the server, run the test suite, and then kill the server. This works great and produces nice results.

When I started working on our authentication server, the pattern that we had used before didn’t work for me. While the Palantir Government tests ran with a single, static configuration file, I needed to run the authentication server with multiple configurations in the course of running through the all the different functional tests. I determined that I needed a way to programmatically bring the server up and down for testing. In JUnit parlance, I needed a way to programmatically launch the server component as part of my setup() function for my unit tests and stop it in my teardown().

With my itch-to-scratch firmly in hand (or some other mixed metaphor), I set out to figure out how to invoke new Java processes from inside a unit test. The solution I came up with (with source code and examples) after the jump.

The Six Ingredients

So there are six ingredients that go into spawning a new VM:

• The classpath to use for the new VM
• The name of the class to run
• The directory to be used as the current directory for the process
• The command line arguments to pass to the process
• The set of Java system properties to use for this process
• The environment to pass to the process

Let’s look at each item individually.

Classpath

The classpath will tell the spawned VM where to load classes from. In JavaInvoke, we use the existing classpath (from the spawning VM) as a starting point and then prepend any new entries to allow overriding the classpath for the spawned VM.

This takes a lot of the tedium out of having to figuring out what to put in the classpath. Most likely, you want something similar to what you already have, if not completely identical.

We get the classpath from System.getProperty("java.class.path") and can add new entries by prepending the new entry, using the value of File.pathSeparatorChar as the entry delimiter. Using File.pathSeparatorChar makes the code cross-platform friendly (since the path separator is ‘;’ on Windows and ‘:’ on Unix (Linux, Solaris, OS/X, etc.).

Caveat: if you change the working directory and your original classpath was constructed using relative paths, you’ll probably have trouble getting anything to run (since your classpath will no longer point to right locations).

Class name

Pretty simple: what do you want to run in the spawned VM? The class must have a static void main(String args[]) defined, and it must be available for loading via the classpath.

Working Directory

If it should be different from the current working directory (CWD) of the running process, then set it and JavaInvoke will change it in the environment.

Command line arguments

If the process needs any command line arguments, including VM options, specify them in a string array. Note that not all of these arguments will necessarily make it to your main method, since the VM executable will parse it first and remove the VM arguments, passing through the program arguments.

Java System Properties

System properties can be used to control many aspects of how a VM runs. You can set them programmatically in your code or you can set set them on the command line by passing -Dkey=value. Our JavaInvoke implementation will take a Map of properties as a convenience argument; all it does is rewrite the map into the command line.

Process environment

This is an operating-system level construct. This is the set of environment variables, also in a Map that you would like merged with the current environment. This would be the place that you set things like LD_LIBRARY_PATH on Unix.

Dealing with input and output

So you might ask the question, “where does the output from the process go?” Or more troubling, “How do I send the process some input?” The Java Process object has methods to deal with this, allowing you to get streams that give you access to the input, output, and error streams of spawned process. That API is straight-forward to deal with, just like any other use of the java.io streams.

However, we want to make the typical case really easy: pulling the output from the spawned process back to the parent that spawned it. To that end, we add into the mix a class called OutputPiper. It fires up a thread that pulls all input from the spawned process, tags it with an identifier, and then outputs to the spawner’s stdout/stderr.

OutputPiper

(as extracted from ProcessSpawner.java)

	public static class OutputPiper extends Thread  {
		InputStream in;
		PrintStream out;
		String tag = null;

		public OutputPiper(String tag, InputStream in,PrintStream out) {
			this.in = in;
			this.out = out;
			this.tag = tag;
			// make sure that we don't keep the VM alive
			this.setDaemon(true);
			this.setName("OutputPiper-" + tag);
			out.println("Starting output piper for tag: " + tag);
			this.start();
		}

		@Override
		public void run() {
			try {
				BufferedReader reader = new BufferedReader(new InputStreamReader(in));
				String line = null;
				do {
					line = reader.readLine();
					if(line != null) {
						out.println(tag + ": " + line);
					}
				}while(line != null);
			}
			catch (Exception e) {
				//
			}
			out.println("Output piper exiting for tag: " + tag);
		}

		public static OutputPiper createOutputPiper(String tag, InputStream in, PrintStream out) {
			OutputPiper rc = new OutputPiper(tag, in,out);
			return rc;
		}
	}

Outpiper extends Thread so that all the output will arrive back to the controlling process in a timely manner. For each given process, we spawn off two OutputPipers, one for stdout and one for stderr, corresponding to the Process.getInputStream() and the Process.getErrorStream().

ProcessSpawner & JavaInvoke

There are two key classes in the example:

• ProcessSpawner.java – Essentially a wrapper around ProcessBuilder, a generic process spawner that makes it simple to invoke processes that that use OutputPipers to forward their output back to their parent. This class allows you to specify the working directory, process environment, and command line for the process to be invoked.
• JavaInvoke.java – a specialized subclass of ProcessSpawner, this class makes spawning new VMs a piece of cake, doing the necessary translation for Java system properties, setting the proper classpath environment variable with potential overrides, and fills in the fully qualified class name to run.

The Example & Source Code

I’ve put together a running example that implements a trivial client and server in JUnit test. The setup() method spawns the server and then the tests run the client code against the server, tearing it down after each test. It’s available in the PalantirVMSpawnerExample.zip zip file. Unzip it, run the run.sh or run.bat script as appropriate. It should generate output that looks like this:

-----------------------------------------------------
Starting test testAck
INFO [main] JavaInvoke - CLASSPATH=./lib/devblog-vmspawner.jar
INFO [main] ProcessSpawner - Build process spawner for the following command line:
INFO [main] ProcessSpawner - /home/pteng/java/i586/jdk1.5.0_14/jre/bin/java com.palantir.blog.processspawner.Server
Starting output piper for tag: server-stdout
Starting output piper for tag: server-stderr
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: [Socket Handler2]: Got message: some message
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: [Socket Handler3]: Got message: SHUTDOWN
Output piper exiting for tag: server-stdout
Output piper exiting for tag: server-stderr
Finished test testAck
-----------------------------------------------------
-----------------------------------------------------
Starting test testShutdown
INFO [main] JavaInvoke - CLASSPATH=./lib/devblog-vmspawner.jar
INFO [main] ProcessSpawner - Build process spawner for the following command line:
INFO [main] ProcessSpawner - /home/pteng/java/i586/jdk1.5.0_14/jre/bin/java com.palantir.blog.processspawner.Server
Starting output piper for tag: server-stdout
Starting output piper for tag: server-stderr
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: [Socket Handler3]: Got message: SHUTDOWN
Output piper exiting for tag: server-stdout
Output piper exiting for tag: server-stderr
Took 3 ms to send shutdown.
Took 335 ms for process to die.
Finished test testShutdown
-----------------------------------------------------
SUCCESS: all 2 tests passed

The source is included in the zip file, but if you wanted to look at it or link to it on the web, here are the classes involved:

• Client.java
• Example.java
• JavaInvoke.java
• ProcessSpawner.java
• Server.java
• ServerSpawningTest.java

And as an added bonus, there’s an Ant build.xml that will let you tweak and rebuild the demo yourself.

Comments and questions welcome. Enjoy.

“Freaking lag!” It had started to become a common refrain around the developer pit. Listed as a project on a candidate’s resume, MultiSnake was a game that we had started to play during our coding breaks. The game was really quite fun — it was easy to play, games were short, and its multi-player nature fostered great competition. The only real drawback was that we seemed to experience network lag. There was nothing more infuriating than having your long snake die by running straight into a completely avoidable wall because the game lagged and didn’t respond to your keyboard commands in time. During one of our particularly lag-heavy games, someone yelled out a gripe that would change our MultiSnaking days for good: “Man, we could totally write this game ourselves, in our app.”

The gripe stuck around, and one day someone finally called out the person making the claim. “Do you seriously think we could write this ourselves?”

“Sure, why not? We have all of the architecture that we need to make this work. We could do it in four hours.”

“I bet you we couldn’t.”

The rest of the story and a video of MultiSnake in action follows.

The Challenge

A challenge was born. The task sounded fun, and it also provided us with a great chance to test the extensibility of our platform. Our most recent milestone had focused on solidifying the public APIs of our platform, and this challenge seemed like a way to test its pluggability. We also thought that it would be a great showcase to demonstrate how easily one could add capabilities to our platform — if we could write a multi-player network game using only the same public APIs available to our clients, it would be a strong signal that our framework was solid.

The Rules

Once we decided that we were going to take on the challenge, we decided that we would do it on Sunday from 8:00PM to midnight (our normal peak productivity hours) and laid out the following rules:

• There would be a strict four-hour time limit for all planning, design and coding.
• The game had to be implemented using only our public APIs — no touching core code.
• The game had to support all of the features provided by the online game and had to be lag-free (after all, that was the whole point of writing our own!).

The Race

Sunday night came along, and it was off to the races! We projected a countdown timer onto a whiteboard in the middle of the developer space, blasted some techno music and got to work! Five developers decided to participate, and it was a pretty collaborative effort in which most of the participants ended up contributing in their standard roles — the frontend people did the game graphics and UI, the backenders worked on the server and game logic, and our data folks dealt with creating a game board provider.

The pluggability of our platform and the fact that it already had support for multiple users and sending out realtime messages from the server to clients made most of the work go pretty smoothly. Besides Eclipse crashing on one of our machines, development was pretty seamless and fast-paced, with team members yelling out at each other across the room to communicate. Within 45 minutes, most of us had finished our first iteration of code, and at the one hour mark we verified that we could successfully get the server and client communicating with each other and draw some game state. By the time we were two hours in, we had most of the core game features implemented, and once we hit the three hour mark we had a fully functional snake game that allowed us to play against each other. We spent the last hour doing some UI polish and ironing out a few bugs, and by the time midnight rolled around we had a fully functional MultiSnake implementation in our platform that was written using only our public APIs. We were even able to get in a few extra features such as a visible timer to count down to the end of the game, a circle to show where a snake respawned and a name that followed the snakes vertically to identify them. We celebrated by eating a rum cake that a coworker had brought in earlier and playing multiple rounds of lag-free MultiSnake against each other. Success!

The Results

MultiSnake lives on as an add-on to our platform. Even after the challenge was completed, devs have been playing around with the code on slow weekends, adding extra maps, new features such as wormholes, and overhauling the graphics. The final product is a pretty impressive and fun-to-play game that we now often demo as an example of the versatility and power of the APIs for our platform.

The solution was to select a single event firing system. We wanted something that was easy-to-use yet powerful enough to express all the changes that might be made to a data model. Java’s Property Change Support (PCS) was a good fit because it can support arbitrary events in a very lightweight fashion.

Read on for details of our implementation…

Property Change Support

Java’s PropertyChangeSupport class (PCS) basically allows an object to easily fire events consisting of 4 pieces of information:

• source object – the thing that fired the event
• property name – allows the listener to tell events for different things apart
• old value – the old value for the property
• new value – the new value for the property

PCS handles all the bookkeeping for adding and removing listeners and firing events. It is very useful for creating listenable models, but we wanted to make it just a little bit easier by having an abstract class that exposed the add/remove listener calls and took care of initializing PCS:

public abstract class AbstractListenableModel implements Serializable {

    private static final long serialVersionUID = 1L;

    private transient PropertyChangeSupport pcs;

    protected AbstractListenableModel() {
        this.init();
    }

    /**
    * Adds a property change listener to the model.
    */
    public final void addPropertyChangeListener(PropertyChangeListener listener) {
        this.pcs.addPropertyChangeListener(listener);
    }

    /**
    * Removes a property change listener from the model.
    */
    public final void removePropertyChangeListener(PropertyChangeListener listener) {
        this.pcs.removePropertyChangeListener(listener);
    }

    /**
    * Fires a property change event to listeners of the model.
    */
    protected final void firePropertyChange(String propertyName, Object oldValue, Object newValue) {
        this.pcs.firePropertyChange(propertyName, oldValue, newValue);
    }

    /**
    * Initializes transient fields during deserialization.
    */
    protected Object readResolve() {
        this.init();
        return this;
    }

    /**
    * Initializes transient fields.
    */
    private void init() {
        this.pcs = new PropertyChangeSupport(this);
    }
}

AbstractListenableModel is basically just a simple wrapper for exposing the functionality of PCS. By extending this abstract class, it’s very easy to create a listenable model:

public final class MyModel extends AbstractListenableModel {

    public static final String PROP_FOO = MyClass.class.getName() + ".Foo";

    private int foo;

    public int getFoo() {
        return this.foo;
    }

    public void setFoo(int foo) {
        //
        // The semantics of the following line are a little hard to unpack,
        // but it does exactly what it needs to do, and the tradeoff
        // for conciseness over immediate readability is worth it for
        // large models with lots of properties.
        //
        // First, the JVM starts to create a stack frame for the call
        // into firePropertyChange().  It begins binding parameter values
        // from the left to the right.  The pointer to the String contained
        // in PROP_FOO is passed in first, then the current value of
        // this.foo is passed in, then the expression
        //        this.foo = foo
        // is evaluated (setting this.foo to the new value of foo), which
        // returns the new value of foo.  All the parameters are then
        // passed down into firePropertyChange(), which checks whether
        // the oldValue is equal to the newValue.  If they're not equal,
        // it fires the event.  If they are equal, it ignores the event.
        //
        this.firePropertyChange(PROP_FOO, this.foo, this.foo = foo);
    }
}

In this example, MyModel contains a single property called foo. When the value of foo is changed, a property change event will be fired to listeners of the model.

You may notice that the value of PROP_FOO is prefixed by the name of the class. This ensure that naming collisions do not occur for scenarios in which the same listener is used to listen to multiple models which happen to use the same property name. This scenario becomes much more likely in the case of event bubbling, which I’ll talk about next.

Event Bubbling

Imagine a scenario in which we have a nested model:

Normally, if a listener needs to receive events from both models A and B, it will need to add itself as a listener to each individual model. While this solution would work, it’s a little cumbersome, especially when model B can get swapped out for model B’—the listener then has to keep itself synched to the internal state of model A. It would be nice if model A could just automatically forward all the events from model B (or B’) via its PCS support so that a listener only needs to attach itself to one model instead of multiple models. With a bit more code in AbstractListenableModel, this is possible:

public abstract class AbstractListenableModel implements Serializable {

    private transient PropertyChangeListener childModelListener;

    ...

    /**
    * Registers a child model to this model.
    */
    protected void registerChildModel(ListenableModel childModel, String propertyName) {
        childModel.addPropertyChangeListener(this.childModelListener);
    }

    /**
    * Initializes transient fields.
    */
    private void init() {
        ...
        this.childModelListener = new ChildModelListener();
    }

    /**
    * Listener for property change events fired from child models.
    */
    private final class ChildModelListener implements PropertyChangeListener {
        public void propertyChange(PropertyChangeEvent event) {
            // This is where the bubbling happens
            pcs.firePropertyChange(event);
        }
    }
}

Now, whenever model B fires a property change event, this event will also be fired by model A. This makes it much easier for the listener to listen to events arbitrarily deep in the model hierarchy, because each event fired by a child model gets re-fired (bubbled) by all its ancestors. All you have to do is attach a listener to the root model and you’ll automatically receive events from all models in the hierarchy.

Note that the registerChildModel method above takes an unused propertyName argument. In the full implementation of this class, events with the provided property name are monitored. When an event with the provided property name is fired, childModelListener is detached from the old child model and attached to any new child model. This ensures that the listenable model is always listening to the current child models.

Events for Collections

Any model event support would not be complete without some consideration of how to handle collections such as sets and lists. To solve this scenario, we created specialized collection classes called ListenableModelSet and ListenableModelList. These collections hold AbstractListenableModels as their elements and fire events whenever their contents change. Since the changes to collections can vary widely, the solution we came up for communicating collection changes with full fidelity is basically to fire events with a copy of the old set as the old value and the new set as the new value. Listeners can then diff the old and new values to determine exactly what changed if necessary. Additionally, each ListenableModelSet or List adds a ChildModelListener to all of its children (themselves AbstractListenableModels), thereby ensuring that events are bubbled from all models in the collection.

Conclusion

Just as we saw with the Adapter piece of the MVA triad, when we componentize the Model piece there are huge gains to be had. Once we started using a base Model class and a consistent eventing infrastructure (PropertyChangeSupport), we could add features that made coding across our entire application a lot more pleasant.

Model–View–Controller (MVC) is an architectural pattern used in software engineering. Successful use of the pattern isolates business logic from user interface considerations, resulting in an application where it is easier to modify either the visual appearance of the application or the underlying business rules without affecting the other. In MVC, the model represents the information (the data) of the application; the view corresponds to elements of the user interface such as text, checkbox items, and so forth; and the controller manages the communication of data and the business rules used to manipulate the data to and from the model.

They go on to show the classic triangle diagram and how it’s baked into various GUI and web frameworks. There’s only one clause in the entire article that hints at something deeper: “Though MVC comes in different flavors…”

Different flavors indeed. In fact MVC is not just a pattern but a whole family of patterns: MVC, MVA, MVP, PAC, Model-Delegate…. It very quickly gets very hairy.

In this article I want to describe one of MVC’s lesser-known variants, the Model-View-Adapter (MVA) pattern, and talk about its advantages over traditional MVC in the context of a Java Swing application.

Architecture

The best place to start is with an architecture diagram. While vanilla MVC is a triangle:

MVA puts the Adapter in a position to strictly mediate between Model and View:

Here a solid line represents a direct relationship while a dashed line represents an indirect relationship via the Observer pattern. Put another way, the Adapter holds a pointer both to the Model and to the View and directly calls methods on both. At the same time, it attaches itself as a listener both to the Model and to the View in order to receive events. It receives property change events from the Model and action events (checkbox ticked, text entered, etc.) from the View, and then routes appropriate changes to the other side. The Adapter is entirely responsible for keeping the Model and the View in sync; the Model and View are both relatively dumb structures, knowing nothing about the other.

The advantages to organizing code this way are:

• All “moving parts” are centralized in one place, the Adapter. No worrying about where to add a listener; no hunting around to find isolated listeners.
• Separation of concerns between the View and the Adapter. The View is responsible for layout and visual presentation while the Adapter is responsible for synchronization and the dynamic aspects of the user interface.
• Better decoupling between Models and Views. Specifically, the View doesn’t need to know anything about the Model.

Additionally, while it will never be possible to fully componentize any variant of the MVC pattern, MVA is more amenable to componentization and thus more of its implementation can be centralized (in a single class) and reused. Once componentized, we can augment the basic functionality with things like:

• Automatic registration and unregistration of listeners when the View enters and exits the Swing component hierarchy, thereby preventing certain kinds of memory leaks.
• Automatic unregistration of listeners when the program shuts down. This can help free up resources like realtime subscriptions.
• Method for swapping a new Model object in for an old Model object.
• Ability to execute a task without listeners attached, to help prevent event-action-event loops.

The downside to using MVA over MVC is that the Adapter tends to take on a lot of the responsibility and can get quite complicated. But in my experience that can be mitigated by having good conventions about which pieces (M, V, A) are allowed to communicate with which other pieces and at what times. Enforcing predictable control flow goes a long way toward managing complexity.

Read on for a code-level description of our implementation of the MVA pattern.

Palantir MVA Implementation

Our half-componentization of MVA resides in a single abstract class named Adapter:

public abstract class Adapter<ViewType extends Component, ModelType> {
// constructor
protected Adapter(ViewType view, ModelType model); { ... }

/**
* Attach listeners to the View's subcomponents (checkboxes etc.).
* Listeners should be stored as member variables in the Adapter
* subclass.
*/
protected abstract void registerViewListeners();

/**
* Detach the same listeners (member variables) that were
* attached in registerViewListeners().
*/
protected abstract void unregisterViewListeners();

/**
* Attach listener(s) to the Model.
*/
protected abstract void registerModelListeners();

/**
* Detach the same listeners (member variables) that were
* attached in registerModelListeners().
*/
protected abstract void unregisterModelListeners();

/**
* Bring the View fully in synch with the Model.  Typically
* this involves querying state from the Model and
* reconfiguring subcomponents of the View accordingly.
*/
protected abstract void fullSynchronize();

protected ModelType getModel() { ... }
protected ViewType getView() { ... }

// other methods elided
}

New View components that want to stay synchronized with a Model must instantiate a subclass of Adapter and implement the abstract methods. The Adapter parent class (itself an example of the Template Method design pattern) will then call into the appropriate abstract methods at the appropriate times. For example, after the View is constructed, as soon as it’s displayed in the Swing component hierarchy the Adapter parent class will automatically call fullSynchronize() (whose implementation should bring the View in line with the Model) and then registerViewListeners() and registerModelListeners(), so the Adapter is poised to react to events. Likewise, when the View is removed from the component hierarchy (when its containing frame is closed, say), both unregisterViewListeners() and unregisterModelListeners() will be called. This can help ensure that no memory will be leaked when a long-life-cycle object (like a system-wide singleton) retains a pointer to a short-life-cycle object (the View) via the Observer pattern.

Dealing With Listener Loops

One problem that confronts UI developers is the problem of “listener loops”: infinite loops that result when the View fires an event, the Adapter (or Controller) responds to it by setting some property on the Model, and an event is propagated from the Model back to the View, starting the whole cycle over again.

One way to combat this is to make sure your Model only fires events when the value that’s being set on the Model is different from the value currently stored in the Model. (This will cut off the infinite loop after one and a half cycles.) It’s a good practice but often isn’t enough, especially when your system is multithreaded and events start to queue up. You can sometimes get into situations where an M-V-C triplet will thrash forever between two different values for one of the Model’s properties.

Our solution to this problem is a protected method (on our Adapter base class) called runWithoutViewListeners:

/**
* Guarantees that the job r will be run:
*    - on the Swing thread
*    - with Model listeners attached
*    - with View listeners DEtached
*/
public final void runWithoutViewListeners(final Runnable r) { ... }

The implementation of this method checks to make sure the view listeners are attached when it’s called, detaches them via a call to unregisterViewListeners(), invokes the Runnable, then reattaches the view listeners via a call to registerViewListeners(). The code inside the Runnable can then make whatever changes it wants to the View without perturbing the Model downstream. Listener loop averted!

More To Come

I hope that’s given you some sense of the territory out there in the wide world of MVC-variants. In a week or two, Derek will show off some of the work he’s done on the M piece of the MVA triad related to “event bubbling.” Stay tuned!

It’s always fun to release a new piece of jargon into the wild. I’ve run into a number of bugs in our codebase that caused by an anti-pattern I’d like to dub The Pokémon Problem.

Much like the game of Whac-a-Mole, this is a class of bugs where fixing every occurrence does not prevent the bug from returning in new code: it is easy for code delta to result in an instance of the bug being re-introduced into the code base. Even if you “catch ‘em all“, nothing prevents someone else from introducing new Pokémon bugs later.

Not only is this bug easy to re-introduce, but it sometimes can be hard to find all currently existing instances of this pattern. Although tools like Eclipse make it easier to track down all the places that code is called, sometimes you’re looking for things that happen in a certain sequence (which tools like Eclipse don’t do a good job of searching for) and dynamic invocation mechanisms like Java Reflection can sometimes make it impossible to be exhaustive. This type of bug is also resistant to automated refactoring: changing the protocol of dealing with this corner of your code will require you to track down all places it was touched and manually refactor them. It generally signals a failure to use sufficient separation of concerns.

In general, this anti-pattern is a result of APIs that require the caller to be responsible for state management of resources that the API owns. This can include things like an object that requires the caller to have run an initialization method before calling any other method on the object. These bugs get even more insidious when a failure to do things in the right order does not cause a hard failure (like throwing an exception) but instead creates some sort of subtle corruption that may not be noticed or cause subsequent calls to fail unexpectedly.

Read on for some strategies on dealing with the Pokémon problem.

Solving the Pokémon Problem

How do we solve the Pokémon problem?  Sometimes it is as easy as writing a unit test to verify that no new Pokémon exist. More often than not, though, you’ll have to use better encapsulation to solve the problem. If the Pokemon problem is the anti-pattern, encapsulation is its opposite.

For an example, I’ll turn to a real problem I ran into in the Palantir Government codebase: Let’s say I have class Parser and it has a method getAttribute(). I want to add the ability to have shortened (obfuscated) tag names to the Parser class.

The wrong approach

One approach I can take:

1. Create a class called AttributeHandler that wraps calls to Parser.getAttribute(). It handles detection of the short or full XML dialect and then returns the appropriate attribute value.
2. Stick an instance of this AttrbributeHandler class in a member variable of Parser and make sure to call AttributeHandler.getAttribute() instead of Parser.getAttribute() when processing attributes.
3. Put comment on Parser.getAttribute() mentioning that it shouldn’t be called directly anymore.

I just introduced a Pokémon problem.  The next person that edits this code may not read my comment and know that calling Parser.getAttribute() is a bug.  They may not test the short tag case and find the bug. How do I fix this Pokémon problem? 

The right approach

Approaching this encapsulation problem has a number of different approaches one could take. Here’s what I chose:

1. Create a class called AttributeHandler that wraps calls to Parser.getAttribute(). It handles detection of the short or full XML dialect and then returns the appropriate attribute value.
2. Stick an instance of this AttrbributeHandler class in a member variable of Parser.
3. Create a wrapper class around Parser that delegates the getAttribute() call to the AttributeHandler member when processing attributes.

(You might ask why I used a delegate instead of sub-classing the parser and overriding the getAttribute() method: it didn’t fit for architectural reasons that aren’t relevant here). Now the developer no longer has access to the Parser.getAttribute() method that would cause the undesired behavior.

In-depth Example: Resource Management

Another common place you might run into the Pokémon problem is code that needs to clean up resources.  Some coders are lazy and don’t want to always write their try/finally/close blocks properly. This can lead to resource leaks; for things like file handles, this usually isn’t a large issue, but for scarce resources like database connections, it’s critical that things get cleaned and returned to the pool. For locks, it’s absolutely essential to avoid deadlocks.

The wrong way

Here’s a simple example of bad resource management. In this (admittedly contrived) scenario, we have class that’s managing a resource for us, in this case it’s a status file. Different parts of the app need to read the status file where, presumably, something is storing its status.

Here’s the naive implementation of the status manager class:

public class PokemonStatusFileManager {

	final File statusFile;

	public PokemonStatusFileManager(File statusFilePath) {
		this.statusFile = statusFilePath;
	}

	public InputStream getStatusFileInputStream() throws FileNotFoundException {
		return new FileInputStream(this.statusFile);
	}
}

Based on this implementation, here’s some code that would use it. Note that first method, parseStatusFileIncorrectly() does no error checking and may not close the InputStream properly if an exception is thrown. The second method does proper resource handling, but it’s kind of ugly to read.

public class PokemonParseStatusFile {

	/**
	 * This is not proper resource management.
	 * @param manager
	 * @throws IOException
	 */
	public static void parseStatusFileIncorrectly(PokemonStatusFileManager manager)
		throws IOException {

		InputStream statusFile = manager.getStatusFileInputStream();
		readFrom(statusFile);
		statusFile.close();

	}

	/**
	 * This is proper resource management, but it's tedious to have to write.
	 * Some coders are too lazy to always do this the right way.
	 * @param manager
	 * @throws IOException
	 */
	public static void parseStatusFileCorrectly(PokemonStatusFileManager manager)
		throws IOException {

		InputStream statusFile = null;
		try {
			statusFile = manager.getStatusFileInputStream();
			readFrom(statusFile);
		} finally {
			// carefully close the resource
			if(statusFile != null) {
				try {
					statusFile.close();
				} catch(Exception e) {
					System.err.println("Error closing statusFile!");
					e.printStackTrace(System.err);
				}
			}
		}
	}

	static void readFrom(InputStream statusFile) throws IOException {
		// do the reading here...
	}
}

So this produces a classic Pokémon problem: everyone who interacts with the StatusFileManager has to do proper resource handling. Now imagine that this is an important lock instead of just a file handle: this Pokémon problem could cause deadlock (which would be bad).

So how do we fix this? The Visitor Pattern.

Solving the Pokémon problem with the visitor pattern

The Visitor Pattern is a potent weapon in fighting the Pokémon problem: it allows you to fully encapsulate access to a resource by injecting into the places it’s needed but preserving overall control of the resource in the code that “owns” the resource. Classically used for controlling things like iteration order, here the visitor pattern is applied as a form of dependency injection to enable lifecycle management.

The visitor pattern as applied to resource management is fairly straightforward:

1. Define an interface, ResourceVisitor with one method, visit(Resource r) (where Resource is the type of resource we’re managing.
2. Define the resource manager with a method that takes a ResourceVisitor as a parameter. Manage the lifecycle of the resource and call ResourceVisitor.visit(r) when appropriate, handling all initialization, error-handling and cleanup.

Here’s our re-spin of the status file manager class. You’ll notice that we’ve moved the resource handling code from the correct example above and encapsulated it in the visitor pattern:

public class UnPokemonStatusFileManager {

	/**
	 * Visitor interface implemented by callers wishing to
	 * interact with the status file.
	 */
	public static interface StatusFileVisitor {
		public void visitStatusFile(InputStream statusFile) throws IOException;
	}

	final File statusFile;

	public UnPokemonStatusFileManager(File statusFilePath) {
		this.statusFile = statusFilePath;
	}

	/**
	 * Note that this method is now private.
	 */
	private InputStream getStatusFileInputStream() throws FileNotFoundException {
		return new FileInputStream(this.statusFile);
	}

	/**
	 * Here's the method that takes the visitor and
	 * fully encapsulates the lifecycle of the status file.
	 */
	public void parseStatusFile(StatusFileVisitor parser)
	throws IOException {
		InputStream statusFile = this.getStatusFileInputStream();
		try {
			parser.visitStatusFile(statusFile);
		} finally {
			// carefully close the resource
			try {
				statusFile.close();
			} catch(Exception e) {
				System.err.println("Error closing statusFile!");
				e.printStackTrace(System.err);
			}
		}
	}
}

Now that we’ve encapsulated the complexity of the resource handling the UnPokemonStatusFileManager class, code that needs to access the status file can be written in a highly correct manner without much work.

public class UnPokemonParseStatusFile {

	public static void parseStatusFile(UnPokemonStatusFileManager statusFileManager)
		throws IOException {

		// generate anonymous visitor to do the processing
		StatusFileVisitor visitor = new StatusFileVisitor() {
			public void visitStatusFile(InputStream statusFile) throws IOException {
				readFrom(statusFile);
			}
		};
		statusFileManager.parseStatusFile(visitor);
	}

	static void readFrom(InputStream statusFile) throws IOException {
		// do the reading here...
	}
}

By encapsulating the full lifecycle of the status file in the visitor pattern, I’ve ensured that it’s always accessed properly. If I change the place where we store status to be a database rather than a file, nothing needs to change except this one class; the calling code remains the same.

We now have easy re-factoring, no resource leaks, and have simplified calling code. And finally: there are no new bugs to be introduced by callers that aren’t sure how to use our resource. Looks like we caught ‘em all!

Wrapping It All Up

So there it is, your new piece of jargon: The Pokémon Problem anti-pattern. You heard it here first! Please post any other great examples to the comments section on this post.

One of the big challenges in Palantir Finance comes when integrating data from multiple data providers. When the server is launched, it needs to create a coherent model of the financial world based on data coming from potentially dozens of data providers. Each data provider defines a set of “models” that it supports. These models can be things like equities, currencies, futures, options, or even new types that the providers themselves define.

The major challenge occurs when multiple providers define models that represent the same real-world entity. Provider A might know about Google, have basic open/high/low/close data for the stock, and know its ticker, country, and ISIN. Provider B might also provide a Google model, have balance sheet data, and know its country, exchange, and ISIN. We want to expose only one Google model to the user, however, and so we need a means of resolving the two Googles together – recognizing that they’re the same instrument – and adding just one equity to the system that encompasses both.

Resolution logic can be fairly complicated. For equities, for example, there are several different ways in which resolution can take place. If two equities have identical ISINs, we can be pretty confident they match, since those identifiers are declared as globally unique. If two equities have the same ticker and the same country of exchange, we might also consider that a match, though perhaps of weaker quality. Two models resolve to each other if any form of resolution considers them equal (with errors being thrown if other forms of resolution contradict the form that considers them equal…i.e. provider A and provider B agree on an instrument’s ISIN but disagree on its ticker).

Read on for the details of how we solve this seemingly n² problem with a linear solution.

Given N models across providers of a given asset class (say, equities), there are N² potential checks that I need to do to properly “resolve” all models, since any model can resolve to any other model in the system (and I potentially do want to attempt to resolve a model from provider A to other models from provider A to do error checking, since I may consider it invalid for a provider to provide the same model twice). Obviously we would like to do better than this, and we can, assuming that most models do not resolve to each other.

Envision the set of all (model, provider) pairs as the set of nodes on a graph. Two models from different providers that resolve to each other can be represented by an edge between two nodes in the graph. If the number of providers k is small relative to N, the number of resolution forms for a given asset class is small, and our data is valid, we can come up with an algorithm that solves our problem in N time as follows:

1. For every form of resolution, ask the data providers for all the data necessary for resolution to take place. For ticker/country resolution, with our data provider interfaces, this gives us a map from every (model, provider) pair to its ticker and country.
2. We can then invert this map, giving us a map from (ticker, country) pairs to a set of (model, provider) pairs. Note that the values in the inverted map do have to be sets, since there can be multiple (model, provider) pairs with the same ticker and country (indeed, this is expected if ANY models can be resolved between providers).
3. Then, for every model, for each resolution form, we can look up the relevant properties for that model, and then look up in the inverse map any models that are equivalent to it. This tells us what edges to add to our (model, provider) graph.

We’re essentially building up an in-memory, inverted index of the relevant data each model is giving us. The amortized O(1) lookups that the hashtable-backed maps provide allows us to trade the O(N²) complexity for something more like O(N).

N checks, rather than N² (assuming that k is trivial compared to N).

Once we’ve done this for every model each connected component of our graph should correspond to one model to be added into our final system. Since the connected components of a graph can be computed in time linear to the number of nodes, we can compute all the final models in linear time. And what is nice is that the maps give us the ability to quickly post-process our data to look for errors. If any two models in a given connected component come from the same provider, this is an error (either the provider has incorrect data, or it is modeling the data improperly). If two models from two different providers resolve, but have conflicting data for a given resolution form, this is also an error. Note that since providers do not have to provide data for every resolution form, it is possible that k models from different providers that resolve together do not form a k-clique on the graph.

Writing data providers is not always easy. There are many data sources out there that are messy, and properly modeling real world data in code can be quite challenging. That’s why it is important to come up with sound, efficient resolution logic that fails noisily, and tells the engineer building the provider when they are and are not playing nicely with the rest of the system.