Note: this third installment in our series on doing your best in interviews. Previously: “How to Rock an Algorithms Interview” and “The Coding Interview”.

One interview that candidates often struggle with is the systems design interview. Even if you know your algorithms and write clean code, that code needs to run on a computer somewhere — and then things quickly get complicated. A truly unbelievable amount of complexity lies beneath something as simple as visiting Google in your browser. While most of that complexity is abstracted away from the end user, as a system designer you have to face it head on, and the more you can handle, the better.

At Palantir, many of our teams give a systems design interview along with an algorithms interview and a couple of coding interviews. We don’t expect anyone to be an expert at all three disciplines (although some are). We’re looking for generalists with depth — people who are good at most things, and great at some. If systems design isn’t your strength, that’s okay, but you should at least be able to talk and reason competently about a complex system.

Read on to learn about what we’re looking for and how you can prepare.

We’re measuring three things

Nominally, this interview appears to require knowledge of systems and a knack for design — and it does. What makes it interesting, though, and sets it apart from a coding or an algorithms interview, is that whatever solution you come up with during the interview is just a side effect. What we actually care about is the process.

In other words, the systems design interview is all about communication.

This reflects what actually working at Palantir is like. As engineers we have a tremendous amount of freedom. We aren’t asked to implement fully-specced features. Instead we take ownership of open-ended problems, and it’s our job to come up with the best solution to each. We need people we can trust to do the right thing without a lot of supervision — people who can own large projects and take them consistently in the right direction. Invariably, this means being able to communicate effectively with the people around you. Working on problems with huge scope isn’t something you can do in a vacuum.

It’s an open-ended conversation

Usually we’ll start by asking you to design a system that performs a given task. The prompt will be simple, but don’t be fooled — these problems are wide and bottomless, and the point of the interview is to see how much volume you can cover in 45 minutes.

For the most part, you’ll be steering the conversation. It’s up to you to understand the problem. That might mean asking questions, sketching diagrams on the board, and bouncing ideas off your interviewer. Do you know the constraints? What kind of inputs does your system need to handle? You have to get a sense for the scope of the problem before you start exploring the space of possible solutions. And remember, there is no single right answer to a real-world problem. Everything is a tradeoff.

Topics

Systems are complex, and when you’re designing a system you’re grappling with its full complexity. Given this, there are many topics you should be familiar with, such as:

• Concurrency. Do you understand threads, deadlock, and starvation? Do you know how to parallelize algorithms? Do you understand consistency and coherence?
• Networking. Do you roughly understand IPC and TCP/IP? Do you know the difference between throughput and latency, and when each is the relevant factor?
• Abstraction. You should understand the systems you’re building upon. Do you know roughly how an OS, file system, and database work? Do you know about the various levels of caching in a modern OS?
• Real-World Performance. You should be familiar with the speed of everything your computer can do, including the relative performance of RAM, disk, SSD and your network.
• Estimation. Estimation, especially in the form of a back-of-the-envelope calculation, is important because it helps you narrow down the list of possible solutions to only the ones that are feasible. Then you have only a few prototypes or micro-benchmarks to write.
• Availability and Reliability. Are you thinking about how things can fail, especially in a distributed environment? Do know how to design a system to cope with network failures? Do you understand durability?

Remember, we’re not looking for mastery of all these topics. We’re looking for familiarity. We just want to make sure you have a good lay of the land, so you know which questions to ask and when to consult an expert.

How to prepare

How do you get better at something? If your answer isn’t along the lines of “practice” or “hard work,” then I have a bridge to sell you. Just like you have to write a lot of code to get better at coding and do a lot of drills to get really good at basketball, you’ll need practice to get better at design. Here are some activities that can help:

• Do mock design sessions. Grab an empty room and a fellow engineer, and ask her to give you a design problem, preferably related to something she’s worked on. Don’t think of it as an interview — just try to come up with the best solution you can. Design interviews are similar to actual design sessions, so getting better at one will make you better at the other.
• Work on an actual system. Contribute to OSS or build something with a friend. Treat your class projects as more than just academic exercises — actually focus on the architecture and the tradeoffs behind each decision. As with most things, the best way to learn is by doing.
• Do back-of-the-envelope calculations for something you’re building and then write micro-benchmarks to verify them. If your micro-benchmarks don’t match your back-of-the-envelope numbers, some part of your mental model will have to give, and you’ll learn something in the process.
• Dig into the performance characteristics of an open source system. For example, take a look at LevelDB. It’s new and clean and small and well-documented. Read about the implementation to understand how it stores its data on disk and how it compacts the data into levels. Ask yourself questions about tradeoffs: which kinds of data and sizes are optimal, and which degrade read/write performance? (Hint: think about random vs. sequential writes.)
• Learn how databases and operating systems work under the hood. These technologies are not only tools in your belt, but also a great source of design inspiration. If you can think like a DB or an OS and understand how each solves the problems it was designed to solve, you’ll be able to apply that mindset to other systems.

Final thought: relax and be creative

The systems design interview can be difficult, but it’s also a place to be creative and to take joy in the imagining of systems unbuilt. If you listen carefully, make sure you fully understand the problem, and then take a clear, straightforward approach to communicating your ideas, you should do fine.

Good luck!

Late last year, we were honored to be invited to talk at Reflections|Projections, ACM@UIUC’s annual student-run computing conference. We decided to bring a talk about Horizon, our system for doing aggregate analysis and filtering across very large amounts of data. The video of the talk was posted a few weeks back on the conference website.

Horizon started as research project / technology demonstrator built as part of Palantir’s Hack Week – a periodic innovation sprint that our engineering team uses to build brand new ideas from whole cloth. It was then used by the Center For Public Integrity in their Who’s Behind The Subprime Meltdown report. We produced a short video on the subject, Beyond the Cloud: Project Horizon, released on our analysis blog. Subsequently, it was folded into our product offering, under the name Object Explorer.

In this hour-long talk, two of the engineers that built this technology tell the story of how Horizon came to be, how it works, and show a live demo of doing analysis on hundreds of millions of records in interactive time.

public abstract class ForwardingCollection<E> extends ForwardingObject implements Collection<E> {
  @Override protected abstract Collection<E> delegate();

  public boolean add(E element) {
    return delegate().add(element);
  }

  // ... (more overridden methods) ...
}

Things worth noting:

• This class overrides the delegate() method to return a more specific type.
• The class contains no instance variables and does not have to be marked Serializable.

Now, here’s an example of how to implement a decorator using a forwarding class (also from Google Collections):

  public class ConstrainedCollection<E> extends ForwardingCollection<E> {
    private final Collection<E> delegate;
    private final Constraint<? super E> constraint;

    public ConstrainedCollection(Collection<E> delegate, Constraint constraint) {
      this.delegate = checkNotNull(delegate);
      this.constraint = checkNotNull(constraint);
    }
    @Override protected Collection<E> delegate() {
      return delegate;
    }
    @Override public boolean add(E element) {
      constraint.checkElement(element);
      return super.add(element);
    }
    @Override public boolean addAll(Collection<? extends E> elements) {
      return super.addAll(checkElements(elements, constraint));
    }
  }

This class implements a collection that checks a constraint on all elements added to the collection. Note that writing the decorator is easy given the forwarding class — only the methods relevant to the decorator need to be overridden.

[A number of weeks ago, we published a post on the search technology used by Palantir. That post covered raising the memory efficiency of a couple of operations. This is part two of that series.]

The most familiar use of search engines is to index documents made available on the Internet via the hypertext transfer protocol. Forgotten names like AltaVista, names not-yet-really-learned like Bing, and, of course, Google come to mind.

This one, massive use case has a couple of properties that I’d like to highlight:

• Asynchronous indexing and querying – web search engines tend to use crawlers and indexers to build up an index of the web. After each crawl is finished, the new index is brought online for use by the query engine.
• Lack of access controls – all the data in the index is available to any query. In fact, most queries are (from the standpoint of the index) completely anonymous.

Palantir: not a web search engine

Search technology is just one part of what makes up a Palantir system. For us, it’s a way to quickly retrieve Palantir objects in a Palantir system, it’s not the whole of the application.

I’d like to highlight a couple of differences from the web search engine case. A Palantir system needs the following properties:

• Realtime indexing and querying – we need information to be available immediately as it changes in the system.
• Leak-proof access controls – we need the search engine to help us make sure that we don’t have information leaking across access control boundaries.

Hit the link to read more about these topics.

Realtime indexing

The Palantir platforms implement realtime indexing: as soon as an analyst changes an object in the system, it needs to be available to query. This could be a change to data in the object or a change to the security tags on the object.

From a programming perspective, this is pretty straightforward: a Palantir transaction will not commit until the search engine is finished indexing the new data.

From a search engine operational perspective, this induces some challenges. Asynchronous indexing allows the search engines to bring online a highly optimized static form of the index. Contrast that with realtime indexing, where every cycle spent optimizing the index is removing cycles from serving other queries and there is likely a human waiting for the optimizing process to finish.

When using the static index, a query only accesses one, optimized index file which then points to the documents containing the results. However, as changes and additions are indexed into the system, there is a lot of overhead to merging them into the master index.

Instead of merging and optimizing on every change, Lucene can keep around a number of smaller indexes that hold all the fresh entries. These are fixed-size append-only segments that are much cheaper to write to than the optimized and merged form of the index. So basically, these ‘dynamic’ indexes are linear lists of single-document indexes. When the search engine goes to run a query it has to follow this simple (yet expensive) algorithm:

• Query the static, merged index, accumulating results. (this part is reasonably fast)
• For each of the dynamic indexes:
  • Open the file, incurring IO overhead.
  • Query each single-document index and look for additional records or newer records that supersede one of the existing found results.

You can see how the overhead of this can quickly get pretty large as the number of dynamic indexes grows: it grows linearly with number of new indexed records. Compare that with the optimized index, which should be close to constant time for any given query.

To get around this, the indexer will only allow a certain number of these dynamic indexes to accumulate before it kicks off a background merge job. During the merge job, we take a noticeable performance hit, but by batching up the merge run we amortize the overhead away for an overall performance win. This hybrid mode didn’t require us to write any new code, but just to tune Lucene to give us the performance profile we wanted.

Preventing Information Leaks

The Palantir data platform has a fairly sophisticated security model baked in (see The White Videos for a more in-depth look at the security model). One of the features that we have implemented is the ability to show a narrower view of an object based the user’s permissions: the user only sees the slice of the data that they have been granted access to. Part of the complexity in implementing this was that we can’t even hint that the other, hidden data exists at all.

Search engines ranks their results by relevance, showing the matches to the query that it believes to be most relevant first. One common way to make these relevance calculations is by comparing the length of the search term or phrase to the length of the term that it matched. Consider the search term ‘king’: it will match the following phrases:

• “I’m the king of the world!”
• “King salmon are often found in the Pacific Northwest and are also known as Chinook salmon.”
• “Yes, my king.”

Using a length-computed relevance, the phrase, “Yes, my king.” is the most relevant.

Getting back to the Palantir object model: for each distinct set of permissions that an object has, we compute a different object label based on the properties that are visible to that particular slice. These multiple titles all go into the search engine. If we were to compute relevance based on the length of the phrase that matched, and the shortest match on the object is shorter than the match that is actually visible to us, we could return the object with a higher-than-obvious relevance. If we were to do that, we’d be leaking information, namely that there’s data on this object that the user making the query is not privy to. (Note that filtering of objects that aren’t at all visible to the user is done in a higher layer after the results have been accumulated and ranked by the search engine.)

Given this problem, there are two approaches one can take:

1. Store all the information needed to decide which labels are visible to the user running the query and then use only the visible labels when calculating the relevance of a match. Note that is a pretty expensive operation.
2. Don’t use the length of match to compute relevance. Note that skipping a relevance calculation is, obviously, a very cheap thing do.

Which do we do? Both.

When matching against object labels, the length metric actually lets us discern between better and worse matches. So in that case, we incur the cost of this calculation in order to return higher quality results.

However, when matching against things like document bodies, the ratio of the size of the match to the size of the search term starts to have less meaning but still has the possibility of leaking information in the query results. For fields like this, we turn off the relevance calculations based on length of match. The upshot is the we don’t have to store the permissions information in the index nor incur the cost of the permissions/views calculation for these fields.

A heartfelt thank you

To be clear, this post highlights the ways in which our search code diverges from the main Lucene code base. We’re huge fans of Lucene and have great respect for the developers that built and maintain what is probably the world’s greatest open-source search engine.

Use Cases & Constraints

We decided on three kinds of questions we wanted to answer:

• What is the health of the deployment?
  • Example: What errors have occurred in the last 24 hours?
• Which parts of the platform are our users engaged with?
  • Example: How much time do users spend in each application?
• How is our server performing over time?
  • Example: What is the average wait on a search query?

The chief constraint was that we build our platform on Log4J. We already use Log4J all over the project, so converting the logging was out of the question. Besides, Log4J provides a guideline for the kind of metadata our events should support, and Log4J makes it easy to record events to a database.

That left us with two problems to solve: how to store structured data with a Log4j message, and how to analyze the collected data.

Analysis is the easy part: just use Palantir! After all, a sequence of logging events has a lot in common with a time series. The rest is explained below.

Recording Structured Data

Consider the problem of plotting usage by a user on a given day. The simplest approximation is to log an event every time an application is closed, and provide the time spent in the application with that event. Posting the information as an unstructured String–e.g. “Andrew spent 46 seconds in Chart”–makes it difficult to later extract the data for analysis. To solve this problem we developed the class RichLogEntry.

RichLogEntrys contain a human-readable message and tagged data in the form of a set of key/value pairs, such as {duration, 46}, {user, Andrew}. This adds to the up-front cost since log messages become more complex, but the benefit is that the analysis engine can easily and generically access data in events.

Furthermore, RichLogEntry plays nicely with existing Log4J infrastructure. Loggers in Log4J already accept an arbitrary Object to pass to Appenders, and Log4J’s default Appenders call toString() on the Objects provided. For RichLogEntry the toString() is simply the human-readable message. So a call to the logging framework with a RichLogEntry would look like (pseudo-Java):

logger.info( (“Andrew just spent 46 seconds in Chart”,
{“duration” : 46.0, “application” : “Chart”, “user” : “Andrew”}) );


For most Appenders this would produce the human-readable String, but our custom Appender knows how to store the tagged data for later analysis.

Example: Application Usage

We implemented the above (i.e. log a “duration” message each time a Palantir application loses focus), and hooked in the data with a Palantir Data Provider plugin.

In the image below, we’re using our Explorer application to analyze the logging data. Our filter framework combines filtering and visualization into a single application. The image contains three filters from top to bottom, each containing a blue title bar. The results of each filter are fed into the filter below it:

• The top filter separates messages by application and displays statistics for each. We’ve selected the Explorer application, so its 144 messages will be fed into the next filter.
• The middle filter has a histogram of the number of seconds each Explorer instance was active (in log scale). Each “bucket” represents a range of durations, and its height is the number of messages with that duration. It looks like I usually spend around 10 seconds in Explorer before switching windows. We’re selecting the gray part of the histogram to avoid skewing our results for the times I’ve gone AFK with Palantir running.
• The bottom filter counts the number of log events over time.

In Palantir Finance, filters such as these can be saved and used anywhere in the platform. Let’s do that, and compare my usage to my coworker Eric L’s. Creating a new set of filters for Eric is easy–I just modify a single filter above to specify him instead of me (the filter isn’t shown for simplicity’s sake), and then save a new copy. Our Chart application is a good place to view the two series side by side:

Of course, I’m the harder worker!

Conclusion

Our logging framework is complete, and we’ve found many new use cases. We use the framework to:

• analyze performance metrics across builds of both Palantir products
• automatically compile usage reports on deployed installations
• import and explore exotic event data sets by running the events through Log4J

Building the Log4J analysis framework was valuable, fun and easy; and it demonstrates the flexibility of Palantir Finance for working with arbitrary data sets.

A Palantir cluster seamlessly integrates many pieces of proven technology. One of them is our customized version of the venerable Java search engine, Lucene. Search engine technology tends to be optimized for the common use case of indexing web documents (or similar information architectures) where you have a few search terms in each query and many, many documents as results. We want to leverage the inverted index capabilities of Lucene, but our data access patterns are a bit different than the typical use case: we need things like pervasive range-querying, different types of relevance, and dynamic views of the data based on security constraints. So in building our data platform, we’ve run into some interesting challenges that are pretty unique in the information retrieval realm, specifically:

1. Raising memory efficiency
2. Real-time indexing
3. Preventing information leaks across access boundaries in an efficient manner

I’ll cover (1) in this post and (2) and (3) in a later post, due out in about two weeks. (Note: part 2 is available here)

Hit the link and we’ll delve into this topic.

Raising memory efficiency

We’ve addressed the issue of resource constraints, generally, in our earlier post: Bandwidth isn’t cheap. Disk isn’t cheap. CPU isn’t cheap. In that post, we posited “RAM to the rescue”:

On the other hand, some things in a SCIF are comparatively cheap. We never use boxes with less than 32GB of memory, and, in fact, lots of sites use 128GB of memory. RAM requires negligible power and cooling, and compared to disk, it’s relatively simple to install. It’s also easy to reconfigure the setup to use the additional memory.

While this is true, no matter how much RAM you buy, your users will find a way to use it all — search is no exception. In many of our environments, the search processes share hardware with other processes in the Palantir cluster, so while the OS may have 128 GB of RAM available, the search process’s VM has substantially less available to it. Compare this to a cluster of dedicated search nodes, where each node will have indexes sized to fit specifically into the memory available.

The upshot is that we needed to modify parts of Lucene to deal with tighter memory constraints than it was designed for.

Priority queue results accumulation

Most systems that implement search include some notion of paging through the results. We use a multi-level paging system, with the search server maintaining a server-side page for each query and serving smaller client-facing pages from.

Vanilla Lucene uses the following algorithm for accumulating search results:

1. Load all matching results.
2. Sort by some relevance metric(s).
3. Return the top n results.

The results are cached as a server-side page in case the client wants to load more than the first n results. You can see where this could run into trouble: if the total number of matching documents is high, that’s a lot of wasted RAM while we winnow it down to the size of the server page. So we use the following algorithm:

1. Construct a priority queue of constrained size with priority computed using the chosen relevance metric
2. Stream through the results, inserting into the queue
3. Return the set of results in the priority queue

Now we never need more RAM than the size of a server-side page to serve results. The downside is that if the client wants more than one server-side page, we have to run the search — in its entirety — twice (ouch). To avoid the first set of results, we adjust the priority queue to kick out all results that were in the first page based on relevance metric.

Using bitsets to optimize range queries

A range query can return a result set of very high cardinality – a range is a very compact way of describing a large set of matching terms (even if they are discrete values, like dates). One way to think about a range query of, say, 10 <= age <= 15, is that it expands to age = 10 OR age = 11 OR age = 12 OR age = 13 OR age = 14 OR age = 15. Rather than treat range queries in any special way, Lucene just does this expansion of the range and runs the query like a normal query.

Internally, Lucene stores a list of metadata nodes, ordered by document id, of each document that matches a given term. The algorithm goes something like this:

1. Open the document id lists for all matching terms
2. Walk the list pointers for each potential match such that you accumulate all the metadata for a given document.
3. Pass all this metadata up to the query processor which decides:
   1. Does this document match the overall query? (remember that terms can be inverted)
   2. Use term frequency taken from the metadata to calculate the relevance.

This structure and attendant algorithm has some nice properties:

• All documents are processed in a set order.
• Everything is known about a document all at once.
• It terminates in a single linear scan.

… and has one very nasty property:

• All of the term value buckets that match the range must be open simultaneously.

This is not a big deal for most English language queries. However, for large ranges and the like, there can be thousands or even millions of terms.

The semantics of range queries have an interesting feature: a document that matches the range twice is not more relevant than one that matches once. (Contrast this with a simple term query: multiple matches do indicate higher relevance). Being able to discard the accounting of how many time we match the range leads to a huge win:

1. We only need a single bit to represent a match
2. We can process a single term value bucket at a time instead of holding all buckets open in memory.

Our search engine accumulates range queries into bitset objects, allowing for a very compact representation of results. We need much less memory than we did before since we only load one term value bucket at a time. And the algorithm is simpler: no more walking pointers or O(n) check before figuring out which pointer moves next.

The next episode

Tune in for Palantir: search with a twist (part two) in a few weeks. I’ll cover the following topics:

• Real-time indexing
• Preventing information leaks across access boundaries in an efficient manner. (see Jason’s Multi-Level Security post over on the Palantir Government Analysis Blog for a high-level look at why these feature are important. and check out Bob McGrew’s “Access Control Model” White Video for in-depth look at how we apply security to our object model.)

At Palantir, we build distributed software. When deployed at a customer site, our platform consists of several servers running on, and distributed across, a cluster of machines. When I first joined the company, deploying and managing our platform was tedious and time consuming. Need to install servers? One by one, login to the machines where they need to go, lay down their requisite files and manually configure them such that they can work together. Have to bring down a deployment for scheduled maintenance? One by one, and in the correct order, login to the machines where the servers reside and shut them down. Want to change the private keys and certificates used to secure communication between servers? Well, you get the point.

From a customer perspective, the complexity associated with the administration of distributed software represents a significant challenge. Not providing tools to help reduce that complexity impacted the overall usability of our platform. Furthermore, from a Palantir perspective, a non-trivial portion of our resources were being devoted to deploying and managing instances of our platform, both externally (by Forward Deployed Engineers working directly with our customers) and internally (by development, QA and support staff working to maintain and improve our product). Could we be more efficient? No doubt. Given our intense focus on customer satisfaction and the desire to grow / scale our business, action was necessary.

To see how we solved this problem, read on.

We stepped back a bit, taking time to reflect on our situation and understand the problem. Based upon our experience, what key areas would a solution need to address? We settled on the following:

1. Lifecycle management.
   1. Ease initial deployment and upgrade.
   2. Handle coordinated starting, stopping and restarting.
2. Configuration management.
   1. Track which servers are installed on what machines.
   2. Provide centralized management of server configuration information.
3. Automation.
   1. Support encoding common management tasks based on best practices.

In addition to those three key areas, we also identified several important requirements. A couple that definitely warrant mention:

1. Security.
2. Extensibility.

After getting a good sense of what needed to be accomplished, we put effort into investigating if an existing solution would fit the bill. For a variety of reasons (i.e., available feature set, licensing constraints, etc.), we never found a good match. We did, however, come across several open source building blocks that could, when composed appropriately, combine to form the foundation of a homegrown solution. The Config Server was born.

Architecture

The Config Server works with remote agents to enable centralized deployment management. The diagram presented above provides an overview of our management infrastructure. Below is a brief discussion of each key component of our architecture.

• Agent – Agents are installed on every machine in a deployment. They are lightweight background processes that sit around waiting to execute commands submitted by the Config Server, interacting directly with the services installed on a given machine. Instead of implementing our own agent solution, we decided to leverage existing technology, the open source peer-to-peer Software Testing Automation Framework (STAF). From its homepage:
  

  The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework designed around the idea of reusable components, called services (such as process invocation, resource management, logging, and monitoring). STAF removes the tedium of building an automation infrastructure, thus enabling you to focus on building your automation solution. The STAF framework provides the foundation upon which to build higher level solutions, and provides a pluggable approach supported across a large variety of platforms and languages.

  We added support for two-way SSL to STAF to enhance the security of our management infrastructure (specifically, to allow us to implement authorization based on self-signed certificates). But beyond that, no modification was necessary. STAF provides us with a robust solution for remote process invocation and file management, both absolutely essential for centralized deployment management.

• Agent Manager – The Agent Manager provides lifecycle and configuration management functionality for the agents in a deployment. It interacts with remote machines through SSH, using the open source Trilead SSH for Java library.
• Config Registry – The Config Registry maintains and provides access to all of the information the Config Server has about a deployment. It consist of the following:
  • Agent Registry – The Agent Registry contains information about all of the agents in a deployment.
  • Service Registry – The Service Registry keeps track of all of the services in a deployment.
  • Config Repository – The Config Repository is a central store for configurations of the agents and services in a deployment.
  • Package Repository – The Package Repository holds all of the service packages that can be installed in a deployment.
  • Plugin Repository – The Plugin Repository houses all of the plugins that are available for use in the Config Server. Plugins are used by the Security Manager, Service Manager and Task Manager.
• Security Manager – We secure our servers and management infrastructure using public key cryptography. The Security Manager handles the generation and packaging of private keys and certificates. We perform private key and certificate generation using the Bouncy Castle Crypto APIs for Java. Packaging is taken care of by plugins in the Plugin Repository. For example, one plugin packages private keys and certificates into JKS files for use with Java, while another packages them into PEM files for use with OpenSSL.
• Service – Services represent the software installed on the machines in a deployment that drive our platform. They correspond to the servers we’ve built and the 3rd party offerings on which they depend (i.e., databases, entity extractors, etc.).
• Service Manager – The Service Manager interacts with agents to provide lifecycle and configuration management functionality for the services in a deployment. The actual mechanics of lifecycle and configuration management vary from to service to service. For example, starting service A might require invoking one script, while starting service B might require invoking another. For each type of service in a deployment, the Plugin Repository contains a corresponding plugin that embeds the necessary management logic. The Service Manager works with those plugins to get its job done.
• Task Manager – Managing a deployment requires performing tasks that go beyond lifecycle and configuration management for its constituent agents and services (i.e., log aggregation, database user creation, etc.). Such tasks are implemented as plugins. They make things happen by communicating with agents and / or directly with machines via SSH. The Task Manager interacts with the Plugin Manager to load tasks and coordinate their execution.

Functionality

How did we do with respect to our stated needs?

• Lifecycle management – The Agent Manager and Service Manager provide centralized lifecycle management. Initial deployment and upgrades, as well as starting, stopping and restarting servers, can all be handled directly through the Config Server.
• Configuration management – The Config Repository of the Config Server maintains information about deployments and provides centralized configuration management. The Agent Manager and Service Manager support the remote retrieval and application of agent and service configuration.
• Automation – The Config Server’s functionality is exposed via a clean and consistent Java API. Common management tasks can be automated by writing code against that API.

And what about some of our more important requirements?

• Security – All communication in our management infrastructure is secured using two-way SSL. A simple authorization mechanism, implemented using self-signed certificates, ensures that only the authorized entities (most notably, the Config Server), can execute commands through agents. Client access to the data maintained, and functionality exposed, by the Config Server requires password-based authorization.
• Extensibility – The Config Server can be extended to support new types of services and perform new tasks by implementing plugins and dropping them in the Plugin Repository.

Future

In the space of a few months, we built the Config Server to address several key needs and requirements related to the management of our platform. Our work has already begun to pay dividends. Looking ahead, there are several things we would like to do:

• Add support for low-level system management and configuration related to our platform (i.e., user and group management, firewall configuration, etc.).
• Implement multi-deployment management with support for features like staging, mirroring and migration.
• Autonomic Computing, integrating with our monitoring solution to implement platform self-management.

While we’ve accomplished a fair amount, plenty of work remains. We look forward to enhancing our Config Server and its associated infrastructure as we strive to make our platform one that is not only powerful and a pleasure to use, but also easy to manage and maintain.

Distributed systems are complex. Getting them right is hard, and when things don’t go right, it can be difficult to understand what went wrong. In an environment like ours, a good monitoring system isn’t just nice to have; it’s a critical component necessary for understanding behavior and diagnosing problems.

We had three primary goals for the initial monitoring system: graphing of time-series data, alerting on event triggers, and notifications to users. Furthermore, as a product company, we had a design goal of a simple, intuitive (yet powerful and flexible) solution.

Before starting, we did a quick survey of existing open-source packages. Unfortunately, nothing we found quite fit our needs, given our specific requirements of security, protocol, licensing, and integrability into our product. Given that, we made the decision to forge ahead and build our own; we try not to re-invent the wheel but it seemed to make sense here.

For an in-depth look at the architecture of the Monitoring Server and components we used to build it, read on…

Architecture

At the highest level, a two-tiered architecture made the most sense. The back-end, standalone server component would be responsible for collecting, processing, and exposing data through an API. The front-end component would be web-based portlets integrated into our existing management interface.

The server architecture was designed to allow generic components to work together, with everything connected up via Spring. While we started with JMX as our collection method for monitoring data, the architecture sees this as just one pluggable component, with multiple data backends supported. A Spring webservices API allows the front-end portlets to query and manipulate the components at each level.

For our first shipping release, we’ve only shipped the JMX backend, and so this is what production architecture looks like for now:

Components

Any time you choose build instead of buy, there’s a lot of work to be done to get the full set of functionality you need. Fortunately, the Java platform has an extremely rich set of freely available projects and libraries, and we leveraged many of them for the back-end:

• JMX: the core of our system, the Java Management Extensions is a standard for managing and monitoring applications. We use JMX to instrument and monitor our own servers, and because it’s an adopted standard, we gain access to MBeans exposed by third-party components as well.
• rrd4j: round-robin databases (RRDs) are an excellent storage format for time-series data, and RRD4J is a pure Java implementation of the legendary RRDTool. The round-robin format allows for a fixed size file, since older data is overwritten as newer data arrives. The multi-resolution aspect of the files provides long historical views without a space premium. For example, an RRD can contain a high resolution series for recent information and a low resolution series for long-term data.
• HSQLDB: a lightweight, native Java, SQL database that can be run in-process. We use HSQLDB to store all non–time-series information, such as metadata about metrics we’re monitoring.
• Quartz: an open source job scheduling system, we use Quartz primarily for scheduling Alerts. Alerts run periodically to check for a condition, and notify if triggered. Each Alert’s wait period is specified by the user, and fortunately, with Quartz it’s easy to schedule many Alerts at different frequencies.
• Groovy: self-described as “an agile dynamic language for the Java Platform,” Goovy is integrated into our alerting system. Alerts can contain Groovy scriptlets, which give us the expressiveness to create Alerts such as “alert if a metric’s average value over the past 5 minutes is greater than X,” or “alert if the variation of a set of metrics’ values across all servers of type Y is greater than Z.”
• JavaMail: a full-featured email framework. Supports SSL/TLS secure connection protocols, which our clients require.
• JAXB: a simple-to-use Java to XML API, JAXB allows us to convert XML into Java objects (and vice-versa). We use JAXB for parsing configuration files and persisting objects into HSQLDB.
• Spring: a framework for developing enterprise Java applications, Spring is the foundation for our monitoring server.

  Having never used a component framework before, using Spring’s Inversion of Control and Dependency Injection paradigms to build an application turned out to be a pleasant and educational experience. While it enforced discipline in using interfaces, it rewarded us with the ability to easily swap implementations of a component. For example, switching to a HSQLDB-based data store required only a single-line edit, and everything just worked. Seriously.

  We also leveraged Spring early in our development process: we pair-coded interfaces, created stub objects, and then wired everything up in Spring. Once our skeleton was in place, we independently worked on component implementations and swapped them in as they were completed. Later in the cycle, we used Spring in our unit tests to compose our application differently for specific tests, isolating important functionality and using dummy components for non-relevant areas.

User Interface

By moving the user interface into the portlets, we were able to re-skin the fairly ugly native graphing capability that rrd4j provides with a more generic solution that looks good. For comparison, here’s an MRTG style graph produced by rrd4j:

And here’s some graphs from our Monitoring Server (note the portlet UI components for controlling display of the graphs):

While the difference is not that stark, our graphs are much easier on the eyes.

Monitoring Server: present and future

We recently released the monitoring system, and it’s already providing insights into our product’s behavior. We have more features planned: eventing, which will help us track system events such as a server restart or job completion; generating new time-series data from existing data (for example, a series of the rolling standard deviation of a metric, or the number of failure events in the past 24 hours), and Groovy scripting directly against the monitoring server. The last feature is particularly helpful when our engineering team can’t physically access a system due to security restrictions.

From an analysis perspective, we can now start to better understand our system’s behavior, which will help us identify problems before they occur and help steer our development energy going forward. Even the world’s best data analysis software needs a little analysis itself sometimes.