Late last year, we were honored to be invited to talk at Reflections|Projections, ACM@UIUC’s annual student-run computing conference. We decided to bring a talk about Horizon, our system for doing aggregate analysis and filtering across very large amounts of data. The video of the talk was posted a few weeks back on the conference website.

Horizon started as research project / technology demonstrator built as part of Palantir’s Hack Week – a periodic innovation sprint that our engineering team uses to build brand new ideas from whole cloth. It was then used by the Center For Public Integrity in their Who’s Behind The Subprime Meltdown report. We produced a short video on the subject, Beyond the Cloud: Project Horizon, released on our analysis blog. Subsequently, it was folded into our product offering, under the name Object Explorer.

In this hour-long talk, two of the engineers that built this technology tell the story of how Horizon came to be, how it works, and show a live demo of doing analysis on hundreds of millions of records in interactive time.

[A number of weeks ago, we published a post on the search technology used by Palantir. That post covered raising the memory efficiency of a couple of operations. This is part two of that series.]

The most familiar use of search engines is to index documents made available on the Internet via the hypertext transfer protocol. Forgotten names like AltaVista, names not-yet-really-learned like Bing, and, of course, Google come to mind.

This one, massive use case has a couple of properties that I’d like to highlight:

• Asynchronous indexing and querying – web search engines tend to use crawlers and indexers to build up an index of the web. After each crawl is finished, the new index is brought online for use by the query engine.
• Lack of access controls – all the data in the index is available to any query. In fact, most queries are (from the standpoint of the index) completely anonymous.

Palantir: not a web search engine

Search technology is just one part of what makes up a Palantir system. For us, it’s a way to quickly retrieve Palantir objects in a Palantir system, it’s not the whole of the application.

I’d like to highlight a couple of differences from the web search engine case. A Palantir system needs the following properties:

• Realtime indexing and querying – we need information to be available immediately as it changes in the system.
• Leak-proof access controls – we need the search engine to help us make sure that we don’t have information leaking across access control boundaries.

Hit the link to read more about these topics.

Realtime indexing

The Palantir platforms implement realtime indexing: as soon as an analyst changes an object in the system, it needs to be available to query. This could be a change to data in the object or a change to the security tags on the object.

From a programming perspective, this is pretty straightforward: a Palantir transaction will not commit until the search engine is finished indexing the new data.

From a search engine operational perspective, this induces some challenges. Asynchronous indexing allows the search engines to bring online a highly optimized static form of the index. Contrast that with realtime indexing, where every cycle spent optimizing the index is removing cycles from serving other queries and there is likely a human waiting for the optimizing process to finish.

When using the static index, a query only accesses one, optimized index file which then points to the documents containing the results. However, as changes and additions are indexed into the system, there is a lot of overhead to merging them into the master index.

Instead of merging and optimizing on every change, Lucene can keep around a number of smaller indexes that hold all the fresh entries. These are fixed-size append-only segments that are much cheaper to write to than the optimized and merged form of the index. So basically, these ‘dynamic’ indexes are linear lists of single-document indexes. When the search engine goes to run a query it has to follow this simple (yet expensive) algorithm:

• Query the static, merged index, accumulating results. (this part is reasonably fast)
• For each of the dynamic indexes:
  • Open the file, incurring IO overhead.
  • Query each single-document index and look for additional records or newer records that supersede one of the existing found results.

You can see how the overhead of this can quickly get pretty large as the number of dynamic indexes grows: it grows linearly with number of new indexed records. Compare that with the optimized index, which should be close to constant time for any given query.

To get around this, the indexer will only allow a certain number of these dynamic indexes to accumulate before it kicks off a background merge job. During the merge job, we take a noticeable performance hit, but by batching up the merge run we amortize the overhead away for an overall performance win. This hybrid mode didn’t require us to write any new code, but just to tune Lucene to give us the performance profile we wanted.

Preventing Information Leaks

The Palantir data platform has a fairly sophisticated security model baked in (see The White Videos for a more in-depth look at the security model). One of the features that we have implemented is the ability to show a narrower view of an object based the user’s permissions: the user only sees the slice of the data that they have been granted access to. Part of the complexity in implementing this was that we can’t even hint that the other, hidden data exists at all.

Search engines ranks their results by relevance, showing the matches to the query that it believes to be most relevant first. One common way to make these relevance calculations is by comparing the length of the search term or phrase to the length of the term that it matched. Consider the search term ‘king’: it will match the following phrases:

• “I’m the king of the world!”
• “King salmon are often found in the Pacific Northwest and are also known as Chinook salmon.”
• “Yes, my king.”

Using a length-computed relevance, the phrase, “Yes, my king.” is the most relevant.

Getting back to the Palantir object model: for each distinct set of permissions that an object has, we compute a different object label based on the properties that are visible to that particular slice. These multiple titles all go into the search engine. If we were to compute relevance based on the length of the phrase that matched, and the shortest match on the object is shorter than the match that is actually visible to us, we could return the object with a higher-than-obvious relevance. If we were to do that, we’d be leaking information, namely that there’s data on this object that the user making the query is not privy to. (Note that filtering of objects that aren’t at all visible to the user is done in a higher layer after the results have been accumulated and ranked by the search engine.)

Given this problem, there are two approaches one can take:

1. Store all the information needed to decide which labels are visible to the user running the query and then use only the visible labels when calculating the relevance of a match. Note that is a pretty expensive operation.
2. Don’t use the length of match to compute relevance. Note that skipping a relevance calculation is, obviously, a very cheap thing do.

Which do we do? Both.

When matching against object labels, the length metric actually lets us discern between better and worse matches. So in that case, we incur the cost of this calculation in order to return higher quality results.

However, when matching against things like document bodies, the ratio of the size of the match to the size of the search term starts to have less meaning but still has the possibility of leaking information in the query results. For fields like this, we turn off the relevance calculations based on length of match. The upshot is the we don’t have to store the permissions information in the index nor incur the cost of the permissions/views calculation for these fields.

A heartfelt thank you

To be clear, this post highlights the ways in which our search code diverges from the main Lucene code base. We’re huge fans of Lucene and have great respect for the developers that built and maintain what is probably the world’s greatest open-source search engine.

A Palantir cluster seamlessly integrates many pieces of proven technology. One of them is our customized version of the venerable Java search engine, Lucene. Search engine technology tends to be optimized for the common use case of indexing web documents (or similar information architectures) where you have a few search terms in each query and many, many documents as results. We want to leverage the inverted index capabilities of Lucene, but our data access patterns are a bit different than the typical use case: we need things like pervasive range-querying, different types of relevance, and dynamic views of the data based on security constraints. So in building our data platform, we’ve run into some interesting challenges that are pretty unique in the information retrieval realm, specifically:

1. Raising memory efficiency
2. Real-time indexing
3. Preventing information leaks across access boundaries in an efficient manner

I’ll cover (1) in this post and (2) and (3) in a later post, due out in about two weeks. (Note: part 2 is available here)

Hit the link and we’ll delve into this topic.

Raising memory efficiency

We’ve addressed the issue of resource constraints, generally, in our earlier post: Bandwidth isn’t cheap. Disk isn’t cheap. CPU isn’t cheap. In that post, we posited “RAM to the rescue”:

On the other hand, some things in a SCIF are comparatively cheap. We never use boxes with less than 32GB of memory, and, in fact, lots of sites use 128GB of memory. RAM requires negligible power and cooling, and compared to disk, it’s relatively simple to install. It’s also easy to reconfigure the setup to use the additional memory.

While this is true, no matter how much RAM you buy, your users will find a way to use it all — search is no exception. In many of our environments, the search processes share hardware with other processes in the Palantir cluster, so while the OS may have 128 GB of RAM available, the search process’s VM has substantially less available to it. Compare this to a cluster of dedicated search nodes, where each node will have indexes sized to fit specifically into the memory available.

The upshot is that we needed to modify parts of Lucene to deal with tighter memory constraints than it was designed for.

Priority queue results accumulation

Most systems that implement search include some notion of paging through the results. We use a multi-level paging system, with the search server maintaining a server-side page for each query and serving smaller client-facing pages from.

Vanilla Lucene uses the following algorithm for accumulating search results:

1. Load all matching results.
2. Sort by some relevance metric(s).
3. Return the top n results.

The results are cached as a server-side page in case the client wants to load more than the first n results. You can see where this could run into trouble: if the total number of matching documents is high, that’s a lot of wasted RAM while we winnow it down to the size of the server page. So we use the following algorithm:

1. Construct a priority queue of constrained size with priority computed using the chosen relevance metric
2. Stream through the results, inserting into the queue
3. Return the set of results in the priority queue

Now we never need more RAM than the size of a server-side page to serve results. The downside is that if the client wants more than one server-side page, we have to run the search — in its entirety — twice (ouch). To avoid the first set of results, we adjust the priority queue to kick out all results that were in the first page based on relevance metric.

Using bitsets to optimize range queries

A range query can return a result set of very high cardinality – a range is a very compact way of describing a large set of matching terms (even if they are discrete values, like dates). One way to think about a range query of, say, 10 <= age <= 15, is that it expands to age = 10 OR age = 11 OR age = 12 OR age = 13 OR age = 14 OR age = 15. Rather than treat range queries in any special way, Lucene just does this expansion of the range and runs the query like a normal query.

Internally, Lucene stores a list of metadata nodes, ordered by document id, of each document that matches a given term. The algorithm goes something like this:

1. Open the document id lists for all matching terms
2. Walk the list pointers for each potential match such that you accumulate all the metadata for a given document.
3. Pass all this metadata up to the query processor which decides:
   1. Does this document match the overall query? (remember that terms can be inverted)
   2. Use term frequency taken from the metadata to calculate the relevance.

This structure and attendant algorithm has some nice properties:

• All documents are processed in a set order.
• Everything is known about a document all at once.
• It terminates in a single linear scan.

… and has one very nasty property:

• All of the term value buckets that match the range must be open simultaneously.

This is not a big deal for most English language queries. However, for large ranges and the like, there can be thousands or even millions of terms.

The semantics of range queries have an interesting feature: a document that matches the range twice is not more relevant than one that matches once. (Contrast this with a simple term query: multiple matches do indicate higher relevance). Being able to discard the accounting of how many time we match the range leads to a huge win:

1. We only need a single bit to represent a match
2. We can process a single term value bucket at a time instead of holding all buckets open in memory.

Our search engine accumulates range queries into bitset objects, allowing for a very compact representation of results. We need much less memory than we did before since we only load one term value bucket at a time. And the algorithm is simpler: no more walking pointers or O(n) check before figuring out which pointer moves next.

The next episode

Tune in for Palantir: search with a twist (part two) in a few weeks. I’ll cover the following topics:

• Real-time indexing
• Preventing information leaks across access boundaries in an efficient manner. (see Jason’s Multi-Level Security post over on the Palantir Government Analysis Blog for a high-level look at why these feature are important. and check out Bob McGrew’s “Access Control Model” White Video for in-depth look at how we apply security to our object model.)

At Palantir, we work in Silicon Valley, read High Scalability, and think of web companies like Facebook and Google as our peers. Most of the time, this is exactly the right recipe for bringing disruptive innovation into the intelligence community. Sometimes, though, it’s misleading – when discussing a design decision, it’s received knowledge that “Disk is cheap.” or “CPU is cheap”. For a web company with a deployment in a commercial data center (or its own data center), this received knowledge is correct. But for a company that ships distributed systems instead of hosting them, and for whom the deployment environment is the kind of locked-down server room in which classified data can reside, these assumptions couldn’t be more false.

At Palantir, we are almost never able to host our customers’ data – typically, as the data is very sensitive, we are not even allowed to see it! Our customers’ highly sensitive data has to reside in a Secure Compartmented Information Facility or SCIF – a building which has been built to be resistant to attempts to access the information within, whether through active or passive measures. The network inside a SCIF is physically separated – “airgapped” – from the public Internet to prevent information leakage. As the entire rationale for such facilities is to prevent information leakage, moving information into or out of one is a tightly regulated process, almost always requiring a human to be in the loop.

Bandwidth is narrow

Bandwidth in and out of a data center is cheap. Bandwidth in and out of a SCIF is not – and this manifests in surprising ways. First off, what does it take to get data into a SCIF? First, the data has to be downloaded from wherever it’s hosted and burned to a CD. Then, someone has to carry it into the SCIF and find a security officer to approve adding it to the network. Finding the security officer can take anywhere from 10 minutes to an entire day. Once you’ve found the security officer, he has to run a virus scan on the CD, which can run at a rate of roughly 20 minutes per 100MB.

If you look at the entire process, you can model our connection into the SCIF as averaging about an 8 hour latency and 640 Kbps bandwidth. That’s about the bandwidth of a slow DSL line and the latency of a radio connection to Pluto. (Actually, it’s somewhat slower.) There’s also a big non-linearity at 700MB, which is the amount of data that fits on a single CD. For instance, this non-linearity is the big reason why we prefer to send patches to our customers rather than full distributions, which are slightly less than a gigabyte including dependencies – and thus why it’s worth it to us to build a system for automating patch application rather than simply replacing jar files by hand.

Disks are expensive

Similarly, if you are running a data warehouse, disk is cheap. You can buy a 1 TB, 7200 RPM disk for about $100, which is perfect for the kind of large, serial reads or writes that a data warehousing workflow requires. However, Palantir uses disk for our database and our search engine, both of which have an OLTP-style usage pattern. As opposed to a data warehouse access pattern, which emphasizes full table scans, OLTP emphasizes random access and therefore requires fast disk. To get 1TB at 15k RPMs costs about $1000, and requires a disk array rather than a single disk. In order to keep the disk fast, you also want to leave it only about 20% full, which overall makes fast disk about 50 times more expensive than slow disk. Most importantly, however, installing a disk array requires trained personnel, a special approval process, and reconfiguring the system to use the new disks, which is a fairly complicated and error-prone process.

CPUs are hot

Finally, in a commercial data center, CPU is the cheapest resource of all. In a secure server room, however, it can be quite expensive. Each CPU or additional box requires more power and cooling. If the room is nearly full, adding that extra box may require building out an entirely new server room, which can cost months and hundreds of thousands of dollars just for an office building. Building a server room in a SCIF is much more expensive and prohibitively time-consuming.

RAM to the rescue

On the other hand, some things in a SCIF are comparatively cheap. We never use boxes with less than 32GB of memory, and, in fact, lots of sites use 128GB of memory. RAM requires negligible power and cooling, and compared to disk, it’s relatively simple to install. It’s also easy to reconfigure the setup to use the additional memory.

The upshot

The design guidelines that follow from this are simple: build a system that is as autonomous as possible and scales down as well as it scales out.

All these statistics are compiled from our day-to-day experiences in the office environment of a SCIF. Deploying to soldiers in the field makes the issues involved in deploying to a SCIF seem minor. Of course, that’s what makes what we do fun.

Recently, we were experiencing major performance problems with loading documents from the database. Profiling did not isolate a single cause; everything (including unrelated, background operations) seemed slow. So, we started logging garbage collection, and found that we were collecting garbage at a rate of 20GB/min!

Profiling revealed that the worst offender, by far, was OracleStatement.prepareAccessors(). Interestingly, it only caused a problem when our result set included a LOB. For such queries, it allocates a 1MB object, regardless of whether the query returned any results at all.

Google searches revealed others who saw similar problems when accessing LOBs, but no solutions other than upgrading or changing drivers. We were already using the latest Oracle JDBC driver, and reverting to earlier drivers did not help. Switching drivers did solve the problem; however, pushing the change to production would require extensive testing to ensure that we were not trading in one problem for another (or more).

I was about to start conducting these tests when John discovered that we were setting the OracleConnection parameter “defaultRowPrefetch” to 1000. This parameter determines how many rows are pulled back from the database on each round-trip, and increasing this value from its default of 10 will normally yield a performance gain. As an experiment, I set the value to 1, and re-profiled memory allocation. The amount of memory allocated by OracleStatement.prepareAccessors() decreased by about three orders of magnitude. Thus, it appears that when a query can return a LOB, Oracle’s JDBC driver allocates approximately “rowPrefetch” KB of memory, even when zero rows are returned.

The Solution

Returning the “defaultRowPrefetch” parameter to 10 did rid us of our garbage collection problems. However, because this is a global setting, reverting it reduced the performance of many other queries which returned many rows with no LOBs. The prospect of setting “rowPrefetch” on a per-query basis was unappetizing, to say the least, but the performance loss was significant. In the end, we altered how we retrieve rows from the database so that the fetch size geometrically increases as we pull results back from the database.

Specifically, the first batch we retrieve contains at most 10 rows, after which we increase the batch size to 20. Once we’ve retrieved 20 more rows, we increase the fetch size to 40, and so on. In this way, we never allocate large amounts of memory for queries which return few (or no) results, but we still quickly ramp up to a large fetch size.

For large queries which returned no LOBs, this solution is still slower than when “defaultRowPrefetch” was 1000. However, the slowdown on those queries was minor, overall system performance was substantially improved, and, importantly, the changes did not require any per-query tuning.