We’re big fans of open source. Libraries from Apache, Google, and various projects hosted on SourceForge.net make up a significant fraction of the third-party code we use to build our products.

We’re proud to be making our first set of open source releases with these two projects: Cinch and Sysmon.

We think it’s the right thing to do, to add our voice to the chorus of developers making software available to freely use, modify, and distribute. These two projects represent our first dip into the open source water – we’re just getting started. As time and other interests allow, we’ll be making other projects available to the dev community.

We’ve chosen the Apache License, Version 2.0 to make our contributions as free from encumberance as possible – our hope is that many people will find them useful and build on top of them just as we have with our own software.

The Projects

Cinch – Cinch makes MVC in Swing easy

Cinch is a Java library for simplifying certain types of GUI code. When developing Swing applications it’s easy to fall into the trap of not separating out Models and Controllers. It’s all too easy to just store the state of that boolean in the checkbox itself, or that String in the JTextField. The design goal behind Cinch was to make it easier to apply MVC than to not by reducing much of the typical Swing friction and boilerplate. Cinch uses Java annotations to reflectively wire up Models, Views, and Controllers.

Already in heavy use inside the Palantir Government product, Cinch changes GUI development in Java to be similar to iOS and OS X’s Cocoa, where annotations are used to bind controls to fields.

Sysmon – A lightweight platform monitoring tool for Java VMs

Sysmon is a lightweight platform monitoring tool. It was designed to gather performance data (CPU, disks, network, etc.) from the host running the Java VM. This data is gathered, packaged, and published via Java Management Extensions (JMX) for access using the JMX APIs and standard tools (such as jconsole). Sysmon can be run as a standalone daemon or as a library to add platform monitoring to any application.

Originally built as component in our Palantir cluster monitoring server, this project should be helpful in scenarios where you need to get data off a host platform and into a VM.

Let us know how we’re doing

We’d love to hear from you on how we’re doing. Aside from the normal outlets to communicate about the projects themselves (see the mailing lists and issue trackers for each project), please feel free to email me directly, Ari Gesher, as the curator of these projects.

Note: this part is part two of our series on doing your best in interviews. Part one: “How to Rock an Algorithms Interview”.

Here at Palantir algorithms are important, but code is our lifeblood. We live and die by the quality of the code we ship. It’s no surprise, then, that coding ability is what we stress the most in our interview process. A candidate can get by with mediocre algorithm skills (depending on the role), but no one can skimp on coding.

Suppose you’re confident in your ability to write great software. Your task in a coding interview (of which there will be several) is to show the interviewers that you in fact do have the programming chops — that you’re an experienced coder who knows how to write solid, production-quality code.

This is easier said than done. After all, coding in your favorite IDE from the comfort of $familiar_place is very different from coding on a whiteboard (on a problem you’re totally unfamiliar with) in a pressure-filled 45-minute interview. We realize that the interview environment is not the real world, and we adjust our expectations accordingly. Nonetheless, there are a number of things you can do to put your best foot forward during the interview.

First, though, we’d like to give you a sense for what we look for during a coding interview. Most important is the ability to write clean and correct code — it’s not enough just to be correct. A lot of people will be interacting with your code once you’re on the job, so it should be readable, maintainable, and extensible where appropriate. If your solution is clean and correct, and you produced it in a reasonable amount of time without a lot of help, you’re in good shape. But even if you stumble a bit, there are other ways to demonstrate your ability. As you work, we also watch for debugging ability, problem-solving and analytical skills, creativity, and an understanding of the ecosystem that surrounds production code.

With our evaluation criteria in mind, here are some suggestions we hope will help you perform at your very best.

Before you start coding

• Make sure you understand the problem. Don’t hesitate to ask questions. Specifically, if any of the problem requirements seem loosely defined or otherwise unclear, ask your interviewer to make things more concrete. There is no penalty for asking for clarifications, and you don’t want to miss a key requirement or proceed on unfounded assumptions.
• Work through simple examples. This can be useful both before you begin and after you’ve finished coding. Working through simple examples before coding can give you additional clarity on the nature of the problem — it may help you notice additional cases or patterns in the problem that you would otherwise have missed had you been thinking more abstractly.
• Make a plan. Be wary of jumping into code without thinking about your program’s high-level structure. You don’t have to work out every last detail (this can be difficult for more meaty problems), but you should give the matter sufficient thought. Without proper planning, you may be forced to waste your limited time reworking significant parts of your program.
• Choose a language. At Palantir, we don’t care what languages you know as long as you have a firm grasp on the fundamentals (decomposition, object-oriented design, etc.). That said, you need to be able to communicate with your interviewer, so choose something that both of you can understand. In general, it’s easier for us if you use Java or C++, but we’ll try to accommodate other languages. If all else fails, devise your own pseudo-code. Just make sure it’s precise (i.e. not hand-wavy) and internally consistent, and explain your choices as you go.

While you’re coding

• Think out loud. Explain your thought process to your interviewer as you code. This helps you more fully communicate your solution, and gives your interviewer an opportunity to correct misconceptions or otherwise provide high-level guidance.
• Break the problem down and define abstractions. One crucial skill we look for is the ability to handle complexity by breaking problems into manageable sub-problems. For anything non-trivial, you’ll want to avoid writing one giant, monolithic function. Feel free to define helper functions, helper classes, and other abstractions to reach a working solution. You can leverage design patterns or other programming idioms as well. Ideally, your solution will be well-factored and as a result easy to read, understand, and prove correct.
• Delay the implementation of your helper functions. (this serves a corollary to the previous point) Write out the signature, and make sure you understand the contract your helper will enforce, but don’t implement it right away. This serves a number of purposes: (1) it shows that you’re familiar with abstractions (by treating the method as an API); (2) it allows you to maintain momentum towards the overall solution; (3) it results in fewer context-switches for your brain (you can reason about each level of the call stack separately); and (4) your interviewer may grant you the implementation for free, if he or she considers it trivial.
• Don’t get caught up in trivialities. At Palantir we are much more interested in your general problem solving and coding abilities than your recall of library function names or obscure language syntax. If you can’t remember exactly how to do something in your chosen language, make something up and just explain to your interviewer that you would look up the specifics in the documentation. Likewise, if you utilize an abstraction or programming idiom which admits a trivial implementation, don’t be afraid to just write out the interface and omit the implementation so you can concentrate on more important aspects of the problem (e.g., “I’m going to use a circular buffer here with the following interface without writing out the full implementation”).

Once you have a solution

• Think about edge cases. Naturally, you should strive for a solution that’s correct in all observable aspects. Sometimes there will be a flaw in the core logic of your solution, but more often your only bugs will be in how you handle edge cases. (This is true of real-world engineering as well.) Make sure your solution works on all edge cases you can think of. One way you can search for edge-case bugs is to…
• Step through your code. One of the best ways to check your work is to simulate how your code executes against a sample input. Take one of your earlier examples and make sure your code produces the right result. Huge caveat here: when mentally simulating how your code behaves, your brain will be tempted to project what it wants to happen rather than what actually says happen. Fight this tendency by being as literal as possible. For example, if you’re calculating a string index with code like str.length()-suffix.length(), don’t just assume you know where that index will land; actually do the math and make sure the value is what you were hoping for.
• Explain the shortcuts you took. If you skipped things for reasons of expedience that you would otherwise do in a “real world” scenario, please let us know what you did and why. For example, “If I were writing this for production use, I would check an invariant here.” Since whiteboard coding is an artificial environment, this gives us a sense for how you’ll treat code once you’re actually on the job.

As an addendum, here are a few suggestions for books we like about the art of software construction:

Clean Code: A Handbook of Agile Software Craftsmanship – Robert C. Martin
Code Complete: A Practical Handbook of Software Construction – Steve McConnell
The Practice of Programming – Brian Kernighan, Rob Pike
Design Patterns: Elements of Reusable Object-Oriented Software – Erich Gamma, et al.
Effective Java – Joshua Bloch

A few months back, Kevin introduced us to the Hedgehog Programming language – (here’s the post if you missed it).

The Palantir Finance programming language — Hedgehog as we know it — is an interpreted, statically typed, object-oriented language. With a syntax that’s based loosely on Java, it mixes roughly Java-style semantics and a few idiosyncrasies that make it a really interesting case study in language design. It’s built to be extremely efficient for batch operations on time series, which is the heavy lifting in financial analysis.

In this video, Eugene and Dave, two of the engineers that work on the language and platform features needed to support it, give a talk that goes into a number of areas around the Hedgehog language, including why we needed to build a language, how it makes the platform more powerful, how we built dev tools into the UI to make debugging easier, and a bunch of the nitty-gritty features that go into the strange (but fitting) beast that is the Hedgehog Language.

As a final note: this is one of things that I love about working at Palantir Technologies. We study a problem pretty hard before we decide that we need to re-invent the wheel – and then when we do, we go all out. It’s one of the benefits of working with the incredibly talented and motivated folks here. When someone says, “well, we need to build a programming language. No, we’re sure,” we just roll up our sleeves and do it. We can add it to the list of: JMX monitoring system, refined Lucene search engine, speeding up Map-Reduce-like systems to interactive time, and implementing our own GIS platform.

public abstract class ForwardingCollection<E> extends ForwardingObject implements Collection<E> {
  @Override protected abstract Collection<E> delegate();

  public boolean add(E element) {
    return delegate().add(element);
  }

  // ... (more overridden methods) ...
}

Things worth noting:

• This class overrides the delegate() method to return a more specific type.
• The class contains no instance variables and does not have to be marked Serializable.

Now, here’s an example of how to implement a decorator using a forwarding class (also from Google Collections):

  public class ConstrainedCollection<E> extends ForwardingCollection<E> {
    private final Collection<E> delegate;
    private final Constraint<? super E> constraint;

    public ConstrainedCollection(Collection<E> delegate, Constraint constraint) {
      this.delegate = checkNotNull(delegate);
      this.constraint = checkNotNull(constraint);
    }
    @Override protected Collection<E> delegate() {
      return delegate;
    }
    @Override public boolean add(E element) {
      constraint.checkElement(element);
      return super.add(element);
    }
    @Override public boolean addAll(Collection<? extends E> elements) {
      return super.addAll(checkElements(elements, constraint));
    }
  }

This class implements a collection that checks a constraint on all elements added to the collection. Note that writing the decorator is easy given the forwarding class — only the methods relevant to the decorator need to be overridden.

One thing about being a developer on the Palantir Finance product that doesn’t get nearly enough publicity is the fact that we have our own programming language. I’m pretty excited about it so let me repeat, with emphasis: we have our own programming language. Yeah, it’s awesome. All those late hours you spent in the lab working on your final project in compilers: turns out they’re actually good for something other than getting into grad school.

Building this language ourselves — as opposed to, say, using an existing language that already just works — wasn’t an easy decision. In fact, it wasn’t even a single decision. We wracked our collective brain dozens of times trying to think of a better approach. But every which way we sliced it, the problems we needed to solve always pointed to building our own language. I still question this decision sometimes, but on the whole I’m very happy with how things have turned out.

The Palantir Finance programming language — Hedgehog as we know it — is an interpreted, statically typed, object-oriented language. With a syntax that’s based loosely on Java, it mixes roughly Java-style semantics and a few idiosyncrasies that make it a really interesting case study in language design. It’s built to be extremely efficient for batch operations on time series, which is the heavy lifting in financial analysis. It also allows you to dynamically add methods to a class from outside the class itself (conceptually similar to Ruby’s Mixins) — you define the function and its input type, and when you type the dot operator, your new method is auto-completed alongside all the “native” methods. Hedgehog also has a vast number of effectively global constants: all the stocks, bonds, and other financial instruments that are essential to the user experience, but that make for quite a design challenge.

I’m not a language guy myself, so instead of continuing to geek out over the core language features, I want to geek out about an emergent property that’s truly unique to the Hedgehog language. But first I’m going to back up and talk about something else that’s really important to us at Palantir: user experience. (I’ll get back to languages I promise.)

There’s a UX principle that says your interface should be “low threshold, high ceiling”. That is, it should be easy for the user to get started, but also able to do powerful things. This is actually a corollary of a more general principle: that your interface should strive for the optimal learning curve. My first CS professor explained this with a set of three diagrams, each representing one of the major OS families. I don’t remember exactly how he drew these diagrams at the time, but an updated version of them might look like this:

The x-axis of each curve represents “wizardry,” a measure of the user’s technical sophistication. The y-axis represents the power of the system — how much the user can accomplish at a given level of wizardry.

The best of the three curves, my prof argued, was the third curve. The first learning curve is great for providing incentives to learn. Each unit of effort spent to increase your wizardry yields an appropriate amount of reward or power. The drawback is that it’s hard for new users to do anything useful; its reward threshold is too high. The middle curve has a lower threshold and is better for novice users, but will frustrate an intermediate user because of the great plateau in the middle. (This might represent a place where the GUI isn’t powerful enough for the tasks you want to accomplish but scripting is still too difficult, leaving no way to express your commands) The third curve, however, is the best of both worlds: a low threshold and a smooth trajectory to the top.

Now let’s apply this back to our topic at hand, programming languages. Specifically, what does the learning curve look like for learning a first language? (Once you’ve learned one, of course, the rest come pretty easily.)

If your experience of learning to program was anything like mine, the first few projects in your first language were painful. You could sense the power further up the curve — it’s what convinced you to stick with CS — but simple tasks took a lot more effort than they should have, at the beginning.

Hedgehog on the other hand — our little homebrew that will someday have its own Wikipedia page — has the smoothest learning curve I’ve ever seen in a programming language. That’s the emergent property I wanted to talk about, because it’s a thing of beauty. You can get started with Hedgehog right away and accomplish quite a bit — without even knowing that you’re “programming” and the slope on the curve stays relatively constant throughout your trajectory.

We didn’t realize it at the time, but we were probably destined to create a low-threshold, high-ceiling language with a smooth learning curve, due to the nature of our user base. Financial analysts are impatient, and they still need to perform many kinds of complicated analysis. They definitely don’t have the time or inclination to spend a semester learning how to program. The solution to their problem is Hedgehog.

Allow me to illustrate with one of the earliest things a user might type into the expression bar:

And that’s it. The user types a ticker symbol and he gets a chart of IBM’s stock price. At no point did he have to wonder about variables or types or #includes. This experience is so frictionless he probably doesn’t even realize he’s writing code in a programming language. He just starts with what he knows, and the system gives him what he wants.

It starts to get interesting as you move further up the curve. Take this user input:

Of course that innocent dot between “IBM” and “volume” means a method invocation to anyone who’s familiar with C++ or Java. But to a new Palantir Finance user it simply means, “Let me access all the types of data associated with IBM.” Conceptually painless.

Or how about this one?

The volume/1000 expression is an anonymous method acting in the scope of a Stock object; it’s syntactic sugar for return this.volume()/1000;. But by allowing the user to strip away all the unnecessary syntax, we make learning the language that much easier.

I could go on tracing the curve here (I’ve only scratched the surface), but I hope I’ve made my point: we coax new users into writing code by making it look as much as possible like performing operations that they already intuitively understand. This is one of the benefits of creating a domain-specific language — we got the richness of the domain for free, and all the understanding that comes with it — and then we went above and beyond the simplification of a traditional DSL to really pare down the complexity of the language for novice users.

From simple beginnings like the ones I’ve shown here, it doesn’t take our users long at all to cross the threshold to more intermediate-level work, such as chaining function calls together or creating their own methods. As far as the high ceiling goes, we’re still working on it, but the language is currently capable of producing not only a quine, as one of our candidates showed us (yes, we ended up hiring him), but also code that can generate studies like the one below:

So Hedgehog has a low threshold and a smooth learning curve, and the ceiling is high enough that our users can do some really serious information processing with it — tasks that would make their other tools break down and cry. But there’s still a lot of interesting work for us to do, especially in pushing the language’s ceiling higher (developing better interactive debugging; working with large objects efficiently) — and as always, making it faster.

If you’d like to see the Hedgehog Programming Language in action, you can sign up for an account at Palantir JoyRide. the Palantir Finance public demo.

This is a response to Ari’s awesome post on human-computer symbiosis. Ari and I were chatting about the equation he developed and I was wondering if there were some further refinements that are possible… let’s take a look:

We are attempting to understand the total analytic capability for a given task a of a human-computer team. Analytic capability in this case probably means:

(1)

Where A is the answer to the analytic problem in question and t_A is the time needed to arrive at the answer based on the inputs available. In the case of chess, A could be the optimum next move given all previous information and t_A would be how long it takes to decide on this move.

Read on for a look at how this generalizes in human-computer symbiotic systems.

In the case of the human-computer team, we know that a is going to be a function of both the human’s analytical capability h and the computer’s analytical capability c (where both h and c have units of answers/time). In the limit case we know that:

(2)

(3)

Or in plain English, if there is no human present, the total analytic capability is simply the analytic capability of the computer. So the naïve solution would be that:

(4)

(4) clearly meets the limiting cases described in (2) and (3). Kasparov noticed a mixing function where the ability of the human and computer to work together becomes the dominant term — we might call this the mixing capability for the given task or m. Including this phenomenon, the total analytic capability (4) would be re-defined as:

(5)

where m has the property that:

(6)

Thus maintaining the limits expressed in (2) and (3) and adhering to the observation that if there is no human or computer component then there will be no mixing advantage. A naïve solution to this constraint would be simple linear mixing:

(7)

where M (units of time per answer) is the mixing efficiency and will be primarily based on the type of task being solved — some analytical tasks lend themselves to a combined process more than others (for example, multiplying 20 digit numbers does not really benefit from the intuition of a human so the ability of a human and computer to perform this task is merely their additive ability).

What Kasparov noticed is that the mixing was primarily based on the quality of the process rather than the analytical power of either the human or computer separately. This seems to imply that we must somehow account for the fact that the quality of the human-computer interface is responsible for the quality of the mixing. This can be modeled as a unitless friction of interaction f_i that impedes the ability of the human and computer to work together.

Equation (7) can thus be re-written as:

(8)

In this case, the maximum value for the mixing capability is realized when the friction of interaction goes to zero. This mixing capability is the same as the equation Ari developed (less the coefficient which is necessary to maintain consistent units throughout).

We can now re-write our analytic capability in (5) as:

(9)

Below, see a plot of this function over a range of values for h, c and f_i:

As can clearly be seen from this functional plot (note the vertical scale), the effect of interface friction dominates over the other terms whenever both the human and computer can make important contributions to the task at hand. The conclusion can be drawn that the most effective way to solve analytical problems is to minimize the friction of the human-computer interface; or to put it another way: optimal analytical systems are those that are built specifically to maximize the ability of the human to leverage the ability of the computer.

I am certain there is still the possibility for further refinement, for example:

(10)

Here at Palantir, a lot of our automatic tests are full-chain tests. A backend server is fired up, client code runs against it, and everything runs much like a production environment. This makes intuitive sense because it’s a faithful approximation of how the system will run in the field.

However, there are some disadvantages to this:

• Full-pass tests don’t always localize the problem. Tests on a client class might fail even if it was the service that behaved incorrectly.
• These full-pass tests are relatively slow. Client code is running against an actual remote service. If a client is being tested, the server code still has to do work — sometimes a lot of work — even if that isn’t the focus of the test.
• The constraints of the test are loose. Full-chain tests can mostly only see whether the operation finished correctly. It’s much harder to figure out whether the operation was done efficiently and without making unnecessary service calls.
• They’re very little setup flexibility. If you want an RPC to return a specific value, you have little choice but to have your test get the service into a state where it can return that value. This is easy in some cases, but prohibitively difficult in others.
• Client tests are forced to share any non-determinism leaked from the service. For example, under real conditions, a request to call A might respond before call B, and sometimes the other way around. This can result in flaky tests or tests that don’t always simulate the conditions you want to exercise.

What’s to be done? Fortunately, there’s an option that handles these cases elegantly. We also test with jMock, a library that dynamically generates mock objects from arbitrary interfaces. These mock objects can be configured to check that particular methods are called with particular inputs a particular number of times, and then give prescribed responses.

Hit the link to see a concrete example of jMock in action.

jMock in action

Let’s say I want to test my object viewer page in Palantir Web, but I don’t want to fire up a dispatch server at all. First, I create my mock service object.

Mockery context = new Mockery();
final PalantirService service = context.mock(PalantirService.class);

Then, I set the expectations of my mock object. In this case, I want to tell my mock object to expect a call to PalantirService.getObject() and PalantirService.getDataSources(). getObject() will return a specific object. Any call made to the service apart from these will make the test fail.

context.checking(new Expectations() {{
        oneOf(service).getObject(realm.getId(), myObject.getId());
        will(returnValue(myObject));
        oneOf(service).getDataSources(myObject.getDataSources());
}});

Now, I create the object I want to test and inject the service.

ObjectViewController controller = new ObjectViewController();
controller.setService(service);

And then we fire away.

ModelMap model = new ModelMap();
controller.doGet(myObject.getId(), model);

Now that the controller (the class we’re exercising) has gone off and populated the model, we check to see that the model is populated correctly. Just like we would in any other test.

assertEquals(myObject.getName(), model.get("objectName"));
assertEquals(myObject, model.get("object"));

But in addition, we also assert that the expectations specified above were satisfied.

context.assertIsSatisfied();

Not only can we be sure that the right calls were made with the right parameters, but we can also be sure that no calls besides the expected calls were made. So the next time you want more speed or control over your tests, take a look at jMock or another framework like it. It’s a powerful tool in the effort to test your best!

[A number of weeks ago, we published a post on the search technology used by Palantir. That post covered raising the memory efficiency of a couple of operations. This is part two of that series.]

The most familiar use of search engines is to index documents made available on the Internet via the hypertext transfer protocol. Forgotten names like AltaVista, names not-yet-really-learned like Bing, and, of course, Google come to mind.

This one, massive use case has a couple of properties that I’d like to highlight:

• Asynchronous indexing and querying – web search engines tend to use crawlers and indexers to build up an index of the web. After each crawl is finished, the new index is brought online for use by the query engine.
• Lack of access controls – all the data in the index is available to any query. In fact, most queries are (from the standpoint of the index) completely anonymous.

Palantir: not a web search engine

Search technology is just one part of what makes up a Palantir system. For us, it’s a way to quickly retrieve Palantir objects in a Palantir system, it’s not the whole of the application.

I’d like to highlight a couple of differences from the web search engine case. A Palantir system needs the following properties:

• Realtime indexing and querying – we need information to be available immediately as it changes in the system.
• Leak-proof access controls – we need the search engine to help us make sure that we don’t have information leaking across access control boundaries.

Hit the link to read more about these topics.

Realtime indexing

The Palantir platforms implement realtime indexing: as soon as an analyst changes an object in the system, it needs to be available to query. This could be a change to data in the object or a change to the security tags on the object.

From a programming perspective, this is pretty straightforward: a Palantir transaction will not commit until the search engine is finished indexing the new data.

From a search engine operational perspective, this induces some challenges. Asynchronous indexing allows the search engines to bring online a highly optimized static form of the index. Contrast that with realtime indexing, where every cycle spent optimizing the index is removing cycles from serving other queries and there is likely a human waiting for the optimizing process to finish.

When using the static index, a query only accesses one, optimized index file which then points to the documents containing the results. However, as changes and additions are indexed into the system, there is a lot of overhead to merging them into the master index.

Instead of merging and optimizing on every change, Lucene can keep around a number of smaller indexes that hold all the fresh entries. These are fixed-size append-only segments that are much cheaper to write to than the optimized and merged form of the index. So basically, these ‘dynamic’ indexes are linear lists of single-document indexes. When the search engine goes to run a query it has to follow this simple (yet expensive) algorithm:

• Query the static, merged index, accumulating results. (this part is reasonably fast)
• For each of the dynamic indexes:
  • Open the file, incurring IO overhead.
  • Query each single-document index and look for additional records or newer records that supersede one of the existing found results.

You can see how the overhead of this can quickly get pretty large as the number of dynamic indexes grows: it grows linearly with number of new indexed records. Compare that with the optimized index, which should be close to constant time for any given query.

To get around this, the indexer will only allow a certain number of these dynamic indexes to accumulate before it kicks off a background merge job. During the merge job, we take a noticeable performance hit, but by batching up the merge run we amortize the overhead away for an overall performance win. This hybrid mode didn’t require us to write any new code, but just to tune Lucene to give us the performance profile we wanted.

Preventing Information Leaks

The Palantir data platform has a fairly sophisticated security model baked in (see The White Videos for a more in-depth look at the security model). One of the features that we have implemented is the ability to show a narrower view of an object based the user’s permissions: the user only sees the slice of the data that they have been granted access to. Part of the complexity in implementing this was that we can’t even hint that the other, hidden data exists at all.

Search engines ranks their results by relevance, showing the matches to the query that it believes to be most relevant first. One common way to make these relevance calculations is by comparing the length of the search term or phrase to the length of the term that it matched. Consider the search term ‘king’: it will match the following phrases:

• “I’m the king of the world!”
• “King salmon are often found in the Pacific Northwest and are also known as Chinook salmon.”
• “Yes, my king.”

Using a length-computed relevance, the phrase, “Yes, my king.” is the most relevant.

Getting back to the Palantir object model: for each distinct set of permissions that an object has, we compute a different object label based on the properties that are visible to that particular slice. These multiple titles all go into the search engine. If we were to compute relevance based on the length of the phrase that matched, and the shortest match on the object is shorter than the match that is actually visible to us, we could return the object with a higher-than-obvious relevance. If we were to do that, we’d be leaking information, namely that there’s data on this object that the user making the query is not privy to. (Note that filtering of objects that aren’t at all visible to the user is done in a higher layer after the results have been accumulated and ranked by the search engine.)

Given this problem, there are two approaches one can take:

1. Store all the information needed to decide which labels are visible to the user running the query and then use only the visible labels when calculating the relevance of a match. Note that is a pretty expensive operation.
2. Don’t use the length of match to compute relevance. Note that skipping a relevance calculation is, obviously, a very cheap thing do.

Which do we do? Both.

When matching against object labels, the length metric actually lets us discern between better and worse matches. So in that case, we incur the cost of this calculation in order to return higher quality results.

However, when matching against things like document bodies, the ratio of the size of the match to the size of the search term starts to have less meaning but still has the possibility of leaking information in the query results. For fields like this, we turn off the relevance calculations based on length of match. The upshot is the we don’t have to store the permissions information in the index nor incur the cost of the permissions/views calculation for these fields.

A heartfelt thank you

To be clear, this post highlights the ways in which our search code diverges from the main Lucene code base. We’re huge fans of Lucene and have great respect for the developers that built and maintain what is probably the world’s greatest open-source search engine.

A Palantir cluster seamlessly integrates many pieces of proven technology. One of them is our customized version of the venerable Java search engine, Lucene. Search engine technology tends to be optimized for the common use case of indexing web documents (or similar information architectures) where you have a few search terms in each query and many, many documents as results. We want to leverage the inverted index capabilities of Lucene, but our data access patterns are a bit different than the typical use case: we need things like pervasive range-querying, different types of relevance, and dynamic views of the data based on security constraints. So in building our data platform, we’ve run into some interesting challenges that are pretty unique in the information retrieval realm, specifically:

1. Raising memory efficiency
2. Real-time indexing
3. Preventing information leaks across access boundaries in an efficient manner

I’ll cover (1) in this post and (2) and (3) in a later post, due out in about two weeks. (Note: part 2 is available here)

Hit the link and we’ll delve into this topic.

Raising memory efficiency

We’ve addressed the issue of resource constraints, generally, in our earlier post: Bandwidth isn’t cheap. Disk isn’t cheap. CPU isn’t cheap. In that post, we posited “RAM to the rescue”:

On the other hand, some things in a SCIF are comparatively cheap. We never use boxes with less than 32GB of memory, and, in fact, lots of sites use 128GB of memory. RAM requires negligible power and cooling, and compared to disk, it’s relatively simple to install. It’s also easy to reconfigure the setup to use the additional memory.

While this is true, no matter how much RAM you buy, your users will find a way to use it all — search is no exception. In many of our environments, the search processes share hardware with other processes in the Palantir cluster, so while the OS may have 128 GB of RAM available, the search process’s VM has substantially less available to it. Compare this to a cluster of dedicated search nodes, where each node will have indexes sized to fit specifically into the memory available.

The upshot is that we needed to modify parts of Lucene to deal with tighter memory constraints than it was designed for.

Priority queue results accumulation

Most systems that implement search include some notion of paging through the results. We use a multi-level paging system, with the search server maintaining a server-side page for each query and serving smaller client-facing pages from.

Vanilla Lucene uses the following algorithm for accumulating search results:

1. Load all matching results.
2. Sort by some relevance metric(s).
3. Return the top n results.

The results are cached as a server-side page in case the client wants to load more than the first n results. You can see where this could run into trouble: if the total number of matching documents is high, that’s a lot of wasted RAM while we winnow it down to the size of the server page. So we use the following algorithm:

1. Construct a priority queue of constrained size with priority computed using the chosen relevance metric
2. Stream through the results, inserting into the queue
3. Return the set of results in the priority queue

Now we never need more RAM than the size of a server-side page to serve results. The downside is that if the client wants more than one server-side page, we have to run the search — in its entirety — twice (ouch). To avoid the first set of results, we adjust the priority queue to kick out all results that were in the first page based on relevance metric.

Using bitsets to optimize range queries

A range query can return a result set of very high cardinality – a range is a very compact way of describing a large set of matching terms (even if they are discrete values, like dates). One way to think about a range query of, say, 10 <= age <= 15, is that it expands to age = 10 OR age = 11 OR age = 12 OR age = 13 OR age = 14 OR age = 15. Rather than treat range queries in any special way, Lucene just does this expansion of the range and runs the query like a normal query.

Internally, Lucene stores a list of metadata nodes, ordered by document id, of each document that matches a given term. The algorithm goes something like this:

1. Open the document id lists for all matching terms
2. Walk the list pointers for each potential match such that you accumulate all the metadata for a given document.
3. Pass all this metadata up to the query processor which decides:
   1. Does this document match the overall query? (remember that terms can be inverted)
   2. Use term frequency taken from the metadata to calculate the relevance.

This structure and attendant algorithm has some nice properties:

• All documents are processed in a set order.
• Everything is known about a document all at once.
• It terminates in a single linear scan.

… and has one very nasty property:

• All of the term value buckets that match the range must be open simultaneously.

This is not a big deal for most English language queries. However, for large ranges and the like, there can be thousands or even millions of terms.

The semantics of range queries have an interesting feature: a document that matches the range twice is not more relevant than one that matches once. (Contrast this with a simple term query: multiple matches do indicate higher relevance). Being able to discard the accounting of how many time we match the range leads to a huge win:

1. We only need a single bit to represent a match
2. We can process a single term value bucket at a time instead of holding all buckets open in memory.

Our search engine accumulates range queries into bitset objects, allowing for a very compact representation of results. We need much less memory than we did before since we only load one term value bucket at a time. And the algorithm is simpler: no more walking pointers or O(n) check before figuring out which pointer moves next.

The next episode

Tune in for Palantir: search with a twist (part two) in a few weeks. I’ll cover the following topics:

• Real-time indexing
• Preventing information leaks across access boundaries in an efficient manner. (see Jason’s Multi-Level Security post over on the Palantir Government Analysis Blog for a high-level look at why these feature are important. and check out Bob McGrew’s “Access Control Model” White Video for in-depth look at how we apply security to our object model.)

Here at Palantir we use test-driven development (or TDD for short). Integrated tools like Eclipse and JUnit simplify writing and running unit tests. However, once you need to test a broader swath of functionality, it’s time to write functional, integration, and system tests. While technically not ‘unit testing’, the testing framework that JUnit provides is basically the same infrastructure that you want to leverage for writing these more involved types of testing.

When you’re developing enterprise software, functional testing often means getting your clients to talk to your servers. For the main Palantir Government product, we integrate the process of bringing the server up and down with the Ant scripts that run our automated unit tests: our testing tasks bring up the server, run the test suite, and then kill the server. This works great and produces nice results.

When I started working on our authentication server, the pattern that we had used before didn’t work for me. While the Palantir Government tests ran with a single, static configuration file, I needed to run the authentication server with multiple configurations in the course of running through the all the different functional tests. I determined that I needed a way to programmatically bring the server up and down for testing. In JUnit parlance, I needed a way to programmatically launch the server component as part of my setup() function for my unit tests and stop it in my teardown().

With my itch-to-scratch firmly in hand (or some other mixed metaphor), I set out to figure out how to invoke new Java processes from inside a unit test. The solution I came up with (with source code and examples) after the jump.

The Six Ingredients

So there are six ingredients that go into spawning a new VM:

• The classpath to use for the new VM
• The name of the class to run
• The directory to be used as the current directory for the process
• The command line arguments to pass to the process
• The set of Java system properties to use for this process
• The environment to pass to the process

Let’s look at each item individually.

Classpath

The classpath will tell the spawned VM where to load classes from. In JavaInvoke, we use the existing classpath (from the spawning VM) as a starting point and then prepend any new entries to allow overriding the classpath for the spawned VM.

This takes a lot of the tedium out of having to figuring out what to put in the classpath. Most likely, you want something similar to what you already have, if not completely identical.

We get the classpath from System.getProperty("java.class.path") and can add new entries by prepending the new entry, using the value of File.pathSeparatorChar as the entry delimiter. Using File.pathSeparatorChar makes the code cross-platform friendly (since the path separator is ‘;’ on Windows and ‘:’ on Unix (Linux, Solaris, OS/X, etc.).

Caveat: if you change the working directory and your original classpath was constructed using relative paths, you’ll probably have trouble getting anything to run (since your classpath will no longer point to right locations).

Class name

Pretty simple: what do you want to run in the spawned VM? The class must have a static void main(String args[]) defined, and it must be available for loading via the classpath.

Working Directory

If it should be different from the current working directory (CWD) of the running process, then set it and JavaInvoke will change it in the environment.

Command line arguments

If the process needs any command line arguments, including VM options, specify them in a string array. Note that not all of these arguments will necessarily make it to your main method, since the VM executable will parse it first and remove the VM arguments, passing through the program arguments.

Java System Properties

System properties can be used to control many aspects of how a VM runs. You can set them programmatically in your code or you can set set them on the command line by passing -Dkey=value. Our JavaInvoke implementation will take a Map of properties as a convenience argument; all it does is rewrite the map into the command line.

Process environment

This is an operating-system level construct. This is the set of environment variables, also in a Map that you would like merged with the current environment. This would be the place that you set things like LD_LIBRARY_PATH on Unix.

Dealing with input and output

So you might ask the question, “where does the output from the process go?” Or more troubling, “How do I send the process some input?” The Java Process object has methods to deal with this, allowing you to get streams that give you access to the input, output, and error streams of spawned process. That API is straight-forward to deal with, just like any other use of the java.io streams.

However, we want to make the typical case really easy: pulling the output from the spawned process back to the parent that spawned it. To that end, we add into the mix a class called OutputPiper. It fires up a thread that pulls all input from the spawned process, tags it with an identifier, and then outputs to the spawner’s stdout/stderr.

OutputPiper

(as extracted from ProcessSpawner.java)

	public static class OutputPiper extends Thread  {
		InputStream in;
		PrintStream out;
		String tag = null;

		public OutputPiper(String tag, InputStream in,PrintStream out) {
			this.in = in;
			this.out = out;
			this.tag = tag;
			// make sure that we don't keep the VM alive
			this.setDaemon(true);
			this.setName("OutputPiper-" + tag);
			out.println("Starting output piper for tag: " + tag);
			this.start();
		}

		@Override
		public void run() {
			try {
				BufferedReader reader = new BufferedReader(new InputStreamReader(in));
				String line = null;
				do {
					line = reader.readLine();
					if(line != null) {
						out.println(tag + ": " + line);
					}
				}while(line != null);
			}
			catch (Exception e) {
				//
			}
			out.println("Output piper exiting for tag: " + tag);
		}

		public static OutputPiper createOutputPiper(String tag, InputStream in, PrintStream out) {
			OutputPiper rc = new OutputPiper(tag, in,out);
			return rc;
		}
	}

Outpiper extends Thread so that all the output will arrive back to the controlling process in a timely manner. For each given process, we spawn off two OutputPipers, one for stdout and one for stderr, corresponding to the Process.getInputStream() and the Process.getErrorStream().

ProcessSpawner & JavaInvoke

There are two key classes in the example:

• ProcessSpawner.java – Essentially a wrapper around ProcessBuilder, a generic process spawner that makes it simple to invoke processes that that use OutputPipers to forward their output back to their parent. This class allows you to specify the working directory, process environment, and command line for the process to be invoked.
• JavaInvoke.java – a specialized subclass of ProcessSpawner, this class makes spawning new VMs a piece of cake, doing the necessary translation for Java system properties, setting the proper classpath environment variable with potential overrides, and fills in the fully qualified class name to run.

The Example & Source Code

I’ve put together a running example that implements a trivial client and server in JUnit test. The setup() method spawns the server and then the tests run the client code against the server, tearing it down after each test. It’s available in the PalantirVMSpawnerExample.zip zip file. Unzip it, run the run.sh or run.bat script as appropriate. It should generate output that looks like this:

-----------------------------------------------------
Starting test testAck
INFO [main] JavaInvoke - CLASSPATH=./lib/devblog-vmspawner.jar
INFO [main] ProcessSpawner - Build process spawner for the following command line:
INFO [main] ProcessSpawner - /home/pteng/java/i586/jdk1.5.0_14/jre/bin/java com.palantir.blog.processspawner.Server
Starting output piper for tag: server-stdout
Starting output piper for tag: server-stderr
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: [Socket Handler2]: Got message: some message
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: [Socket Handler3]: Got message: SHUTDOWN
Output piper exiting for tag: server-stdout
Output piper exiting for tag: server-stderr
Finished test testAck
-----------------------------------------------------
-----------------------------------------------------
Starting test testShutdown
INFO [main] JavaInvoke - CLASSPATH=./lib/devblog-vmspawner.jar
INFO [main] ProcessSpawner - Build process spawner for the following command line:
INFO [main] ProcessSpawner - /home/pteng/java/i586/jdk1.5.0_14/jre/bin/java com.palantir.blog.processspawner.Server
Starting output piper for tag: server-stdout
Starting output piper for tag: server-stderr
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: Spawning socket handler
server-stdout: Waiting for connection
server-stdout: [Socket Handler3]: Got message: SHUTDOWN
Output piper exiting for tag: server-stdout
Output piper exiting for tag: server-stderr
Took 3 ms to send shutdown.
Took 335 ms for process to die.
Finished test testShutdown
-----------------------------------------------------
SUCCESS: all 2 tests passed

The source is included in the zip file, but if you wanted to look at it or link to it on the web, here are the classes involved:

• Client.java
• Example.java
• JavaInvoke.java
• ProcessSpawner.java
• Server.java
• ServerSpawningTest.java

And as an added bonus, there’s an Ant build.xml that will let you tweak and rebuild the demo yourself.

Comments and questions welcome. Enjoy.