We’re big fans of open source. Libraries from Apache, Google, and various projects hosted on SourceForge.net make up a significant fraction of the third-party code we use to build our products.

We’re proud to be making our first set of open source releases with these two projects: Cinch and Sysmon.

We think it’s the right thing to do, to add our voice to the chorus of developers making software available to freely use, modify, and distribute. These two projects represent our first dip into the open source water – we’re just getting started. As time and other interests allow, we’ll be making other projects available to the dev community.

We’ve chosen the Apache License, Version 2.0 to make our contributions as free from encumberance as possible – our hope is that many people will find them useful and build on top of them just as we have with our own software.

The Projects

Cinch – Cinch makes MVC in Swing easy

Cinch is a Java library for simplifying certain types of GUI code. When developing Swing applications it’s easy to fall into the trap of not separating out Models and Controllers. It’s all too easy to just store the state of that boolean in the checkbox itself, or that String in the JTextField. The design goal behind Cinch was to make it easier to apply MVC than to not by reducing much of the typical Swing friction and boilerplate. Cinch uses Java annotations to reflectively wire up Models, Views, and Controllers.

Already in heavy use inside the Palantir Government product, Cinch changes GUI development in Java to be similar to iOS and OS X’s Cocoa, where annotations are used to bind controls to fields.

Sysmon – A lightweight platform monitoring tool for Java VMs

Sysmon is a lightweight platform monitoring tool. It was designed to gather performance data (CPU, disks, network, etc.) from the host running the Java VM. This data is gathered, packaged, and published via Java Management Extensions (JMX) for access using the JMX APIs and standard tools (such as jconsole). Sysmon can be run as a standalone daemon or as a library to add platform monitoring to any application.

Originally built as component in our Palantir cluster monitoring server, this project should be helpful in scenarios where you need to get data off a host platform and into a VM.

Let us know how we’re doing

We’d love to hear from you on how we’re doing. Aside from the normal outlets to communicate about the projects themselves (see the mailing lists and issue trackers for each project), please feel free to email me directly, Ari Gesher, as the curator of these projects.

Our frontend engineering candidates go through many of the same types of interviews as our backend candidates, i.e., algo and coding. But because they’ll be building the user-facing parts of our system, we want to make sure candidates have some design chops as well. Hence the UI design interview.

Why have a design interview for an engineer?

Perhaps my favorite part of being on the frontend team at Palantir is that we get to work on both the design and the implementation of our own UIs, from inception to finished product. Engineers collaborate on the design of the product with other engineers as well as with designers, and we openly debate the merits of all our decisions. The UI interview simulates this kind of cooperative design and debate. Typically, this means you’ll be asked to design and/or evaluate one or two pieces of UI during the interview, while your interviewer plays both collaborator and critic.

Read on for some helpful points to keep in mind as you work on a design problem with your interviewer.

It’s all about the user

The user is the ultimate arbiter of success for any interface. If users can accomplish their tasks in a simple, efficient, intuitive way, then we’re doing our job right. Therefore, we must keep the user in mind at every stage of interface design.

One common mistake that you can make up front is to assume YOU are the user. Because working on a computer is a solitary activity, it’s easy to forget that everyone experiences an interface in a different way. Depending on what you’re designing, the user may be a complete novice or a seasoned sysadmin.

It’s important to imagine what your users are like. If it helps, give the user a name, age, and occupation. Ask yourself these questions:

• In what context will they use this feature? At work? At home? On a TV from 10 feet away?
• Have they used a similar interface before?
• How savvy are they with computers in general? Do they know how to copy and paste, drag and drop, or open a context menu?

When designing a feature for the interface, walk the user through the feature. Make a rough sketch of the main components (buttons, lists, text blocks) on a whiteboard or sheet of paper. Then simulate how your user might interact with the feature, by drawing in their input and selections.

While sketching your proposed interface, put yourself in the user’s shoes. Ask yourself these questions:

• What will they be doing when they want to start doing X?
• How do they discover the feature?
• What will they want to do after?
• How frequently will the user do X?
• What if X fails to complete properly?

and so on. Once you’ve asked yourself those questions, consider how the answers should impact your design and modify it accordingly.

tl;dr: The user is paramount. Each of your decisions should be justified by its impact on the user.

The interview is highly interactive

Some candidates are reluctant to contradict the interviewer or provide alternative ideas. We want the opposite. If you have a good idea, advocate for your position. I prefer candidates who will contradict me, as long as they back up their ideas with arguments, stories, or examples. The more you can articulate the reason for your beliefs, the better.

Go ahead and keep throwing ideas out there. I’ll shoot down some of them, but don’t be daunted. It may take 10 ideas before we discover a gem. (This is how all design works, not just in an interview setting.)

Be creative, but don’t re-invent the wheel

I’ve seen many candidates jump through incredibly awkward (albeit fancy) hoops just to display some very simple data. If you have a list of data, display it as a list. In general, being familiar with UI conventions is helpful because they encode a lot of hard-earned wisdom.

If you’ve created an interface that will enable the user to do their task as quickly and painlessly as possible, then stop. Don’t weigh down the interface with unnecessary features. As Deiter Rams famously said, “Good design is as little as possible.” This holds true for user interface design just as well as product design. Consider Alan Cooper’s rule of thumb, “A user will take as much time to make a choice as there are choices.” If a user is inundated with options and features, they find the interface more difficult to use. A simple, direct approach is frequently the best one.

If you want a quick and easy measure of the simplicity of a feature, just count the number of clicks to complete the task using your interface. If the user has to switch from mouse to keyboard, count that twice.

How to prepare

If you’ve done some design in the past, especially in a collaborative setting, just come and be yourself — you’ll do fine. But if you’re relatively inexperienced, here are some ways you can improve your design skills:

1. If you’re still in school, take an HCI or infoviz course, especially a project-based course. This will give you some actual design experience, as well as a rich vocabulary and set of concepts for thinking about user interfaces.
2. The design mindset is a muscle, so exercise it every chance you get. Constantly ask yourself, “How could this be different? How could it be better?” The more you ask yourself these questions, the more they’ll become unconscious and automatic. Soon you’ll be wondering about the design of everyday things — in the shower, on the train, and hopefully every time you interact with a piece of software. If you develop the skills to analyze other people’s designs, you’ll be able to apply those skills to your own own.
3. Actually build something and pay attention to the UI. Go through more than one design before (tentatively) settling on the one you’re going to build. Draw wireframes and/or mockups. Iterate constantly, even after you think you’re done.
4. Ask someone to critique your work. If you can find someone with an HCI background (or other design skills), she’ll show you things you couldn’t see yourself. Once this has happened a few times, you’ll start internalizing your critics; you’ll start asking yourself, “What am I missing here?” If you find someone who doesn’t have a design background, treat them like a user. This means taking their feedback very seriously but leaving some room for skepticism. Notoriously, users don’t always know what they want.
5. Read up on UI/UX/HCI/infoviz. There are a lot of good books and blogs out there. Some of the ones that we’ve found particularly helpful are About Face by Alan Cooper, Now You See It by Stephen Few, and Don’t Make Me Think by Steve Krug.

I hope this post will help you design better interfaces, and perhaps I’ll be working with you soon!

Note: this third installment in our series on doing your best in interviews. Previously: “How to Rock an Algorithms Interview” and “The Coding Interview”.

One interview that candidates often struggle with is the systems design interview. Even if you know your algorithms and write clean code, that code needs to run on a computer somewhere — and then things quickly get complicated. A truly unbelievable amount of complexity lies beneath something as simple as visiting Google in your browser. While most of that complexity is abstracted away from the end user, as a system designer you have to face it head on, and the more you can handle, the better.

At Palantir, many of our teams give a systems design interview along with an algorithms interview and a couple of coding interviews. We don’t expect anyone to be an expert at all three disciplines (although some are). We’re looking for generalists with depth — people who are good at most things, and great at some. If systems design isn’t your strength, that’s okay, but you should at least be able to talk and reason competently about a complex system.

Read on to learn about what we’re looking for and how you can prepare.

We’re measuring three things

Nominally, this interview appears to require knowledge of systems and a knack for design — and it does. What makes it interesting, though, and sets it apart from a coding or an algorithms interview, is that whatever solution you come up with during the interview is just a side effect. What we actually care about is the process.

In other words, the systems design interview is all about communication.

This reflects what actually working at Palantir is like. As engineers we have a tremendous amount of freedom. We aren’t asked to implement fully-specced features. Instead we take ownership of open-ended problems, and it’s our job to come up with the best solution to each. We need people we can trust to do the right thing without a lot of supervision — people who can own large projects and take them consistently in the right direction. Invariably, this means being able to communicate effectively with the people around you. Working on problems with huge scope isn’t something you can do in a vacuum.

It’s an open-ended conversation

Usually we’ll start by asking you to design a system that performs a given task. The prompt will be simple, but don’t be fooled — these problems are wide and bottomless, and the point of the interview is to see how much volume you can cover in 45 minutes.

For the most part, you’ll be steering the conversation. It’s up to you to understand the problem. That might mean asking questions, sketching diagrams on the board, and bouncing ideas off your interviewer. Do you know the constraints? What kind of inputs does your system need to handle? You have to get a sense for the scope of the problem before you start exploring the space of possible solutions. And remember, there is no single right answer to a real-world problem. Everything is a tradeoff.

Topics

Systems are complex, and when you’re designing a system you’re grappling with its full complexity. Given this, there are many topics you should be familiar with, such as:

• Concurrency. Do you understand threads, deadlock, and starvation? Do you know how to parallelize algorithms? Do you understand consistency and coherence?
• Networking. Do you roughly understand IPC and TCP/IP? Do you know the difference between throughput and latency, and when each is the relevant factor?
• Abstraction. You should understand the systems you’re building upon. Do you know roughly how an OS, file system, and database work? Do you know about the various levels of caching in a modern OS?
• Real-World Performance. You should be familiar with the speed of everything your computer can do, including the relative performance of RAM, disk, SSD and your network.
• Estimation. Estimation, especially in the form of a back-of-the-envelope calculation, is important because it helps you narrow down the list of possible solutions to only the ones that are feasible. Then you have only a few prototypes or micro-benchmarks to write.
• Availability and Reliability. Are you thinking about how things can fail, especially in a distributed environment? Do know how to design a system to cope with network failures? Do you understand durability?

Remember, we’re not looking for mastery of all these topics. We’re looking for familiarity. We just want to make sure you have a good lay of the land, so you know which questions to ask and when to consult an expert.

How to prepare

How do you get better at something? If your answer isn’t along the lines of “practice” or “hard work,” then I have a bridge to sell you. Just like you have to write a lot of code to get better at coding and do a lot of drills to get really good at basketball, you’ll need practice to get better at design. Here are some activities that can help:

• Do mock design sessions. Grab an empty room and a fellow engineer, and ask her to give you a design problem, preferably related to something she’s worked on. Don’t think of it as an interview — just try to come up with the best solution you can. Design interviews are similar to actual design sessions, so getting better at one will make you better at the other.
• Work on an actual system. Contribute to OSS or build something with a friend. Treat your class projects as more than just academic exercises — actually focus on the architecture and the tradeoffs behind each decision. As with most things, the best way to learn is by doing.
• Do back-of-the-envelope calculations for something you’re building and then write micro-benchmarks to verify them. If your micro-benchmarks don’t match your back-of-the-envelope numbers, some part of your mental model will have to give, and you’ll learn something in the process.
• Dig into the performance characteristics of an open source system. For example, take a look at LevelDB. It’s new and clean and small and well-documented. Read about the implementation to understand how it stores its data on disk and how it compacts the data into levels. Ask yourself questions about tradeoffs: which kinds of data and sizes are optimal, and which degrade read/write performance? (Hint: think about random vs. sequential writes.)
• Learn how databases and operating systems work under the hood. These technologies are not only tools in your belt, but also a great source of design inspiration. If you can think like a DB or an OS and understand how each solves the problems it was designed to solve, you’ll be able to apply that mindset to other systems.

Final thought: relax and be creative

The systems design interview can be difficult, but it’s also a place to be creative and to take joy in the imagining of systems unbuilt. If you listen carefully, make sure you fully understand the problem, and then take a clear, straightforward approach to communicating your ideas, you should do fine.

Good luck!

Note: this part is part two of our series on doing your best in interviews. Part one: “How to Rock an Algorithms Interview”.

Here at Palantir algorithms are important, but code is our lifeblood. We live and die by the quality of the code we ship. It’s no surprise, then, that coding ability is what we stress the most in our interview process. A candidate can get by with mediocre algorithm skills (depending on the role), but no one can skimp on coding.

Suppose you’re confident in your ability to write great software. Your task in a coding interview (of which there will be several) is to show the interviewers that you in fact do have the programming chops — that you’re an experienced coder who knows how to write solid, production-quality code.

This is easier said than done. After all, coding in your favorite IDE from the comfort of $familiar_place is very different from coding on a whiteboard (on a problem you’re totally unfamiliar with) in a pressure-filled 45-minute interview. We realize that the interview environment is not the real world, and we adjust our expectations accordingly. Nonetheless, there are a number of things you can do to put your best foot forward during the interview.

First, though, we’d like to give you a sense for what we look for during a coding interview. Most important is the ability to write clean and correct code — it’s not enough just to be correct. A lot of people will be interacting with your code once you’re on the job, so it should be readable, maintainable, and extensible where appropriate. If your solution is clean and correct, and you produced it in a reasonable amount of time without a lot of help, you’re in good shape. But even if you stumble a bit, there are other ways to demonstrate your ability. As you work, we also watch for debugging ability, problem-solving and analytical skills, creativity, and an understanding of the ecosystem that surrounds production code.

With our evaluation criteria in mind, here are some suggestions we hope will help you perform at your very best.

Before you start coding

• Make sure you understand the problem. Don’t hesitate to ask questions. Specifically, if any of the problem requirements seem loosely defined or otherwise unclear, ask your interviewer to make things more concrete. There is no penalty for asking for clarifications, and you don’t want to miss a key requirement or proceed on unfounded assumptions.
• Work through simple examples. This can be useful both before you begin and after you’ve finished coding. Working through simple examples before coding can give you additional clarity on the nature of the problem — it may help you notice additional cases or patterns in the problem that you would otherwise have missed had you been thinking more abstractly.
• Make a plan. Be wary of jumping into code without thinking about your program’s high-level structure. You don’t have to work out every last detail (this can be difficult for more meaty problems), but you should give the matter sufficient thought. Without proper planning, you may be forced to waste your limited time reworking significant parts of your program.
• Choose a language. At Palantir, we don’t care what languages you know as long as you have a firm grasp on the fundamentals (decomposition, object-oriented design, etc.). That said, you need to be able to communicate with your interviewer, so choose something that both of you can understand. In general, it’s easier for us if you use Java or C++, but we’ll try to accommodate other languages. If all else fails, devise your own pseudo-code. Just make sure it’s precise (i.e. not hand-wavy) and internally consistent, and explain your choices as you go.

While you’re coding

• Think out loud. Explain your thought process to your interviewer as you code. This helps you more fully communicate your solution, and gives your interviewer an opportunity to correct misconceptions or otherwise provide high-level guidance.
• Break the problem down and define abstractions. One crucial skill we look for is the ability to handle complexity by breaking problems into manageable sub-problems. For anything non-trivial, you’ll want to avoid writing one giant, monolithic function. Feel free to define helper functions, helper classes, and other abstractions to reach a working solution. You can leverage design patterns or other programming idioms as well. Ideally, your solution will be well-factored and as a result easy to read, understand, and prove correct.
• Delay the implementation of your helper functions. (this serves a corollary to the previous point) Write out the signature, and make sure you understand the contract your helper will enforce, but don’t implement it right away. This serves a number of purposes: (1) it shows that you’re familiar with abstractions (by treating the method as an API); (2) it allows you to maintain momentum towards the overall solution; (3) it results in fewer context-switches for your brain (you can reason about each level of the call stack separately); and (4) your interviewer may grant you the implementation for free, if he or she considers it trivial.
• Don’t get caught up in trivialities. At Palantir we are much more interested in your general problem solving and coding abilities than your recall of library function names or obscure language syntax. If you can’t remember exactly how to do something in your chosen language, make something up and just explain to your interviewer that you would look up the specifics in the documentation. Likewise, if you utilize an abstraction or programming idiom which admits a trivial implementation, don’t be afraid to just write out the interface and omit the implementation so you can concentrate on more important aspects of the problem (e.g., “I’m going to use a circular buffer here with the following interface without writing out the full implementation”).

Once you have a solution

• Think about edge cases. Naturally, you should strive for a solution that’s correct in all observable aspects. Sometimes there will be a flaw in the core logic of your solution, but more often your only bugs will be in how you handle edge cases. (This is true of real-world engineering as well.) Make sure your solution works on all edge cases you can think of. One way you can search for edge-case bugs is to…
• Step through your code. One of the best ways to check your work is to simulate how your code executes against a sample input. Take one of your earlier examples and make sure your code produces the right result. Huge caveat here: when mentally simulating how your code behaves, your brain will be tempted to project what it wants to happen rather than what actually says happen. Fight this tendency by being as literal as possible. For example, if you’re calculating a string index with code like str.length()-suffix.length(), don’t just assume you know where that index will land; actually do the math and make sure the value is what you were hoping for.
• Explain the shortcuts you took. If you skipped things for reasons of expedience that you would otherwise do in a “real world” scenario, please let us know what you did and why. For example, “If I were writing this for production use, I would check an invariant here.” Since whiteboard coding is an artificial environment, this gives us a sense for how you’ll treat code once you’re actually on the job.

As an addendum, here are a few suggestions for books we like about the art of software construction:

Clean Code: A Handbook of Agile Software Craftsmanship – Robert C. Martin
Code Complete: A Practical Handbook of Software Construction – Steve McConnell
The Practice of Programming – Brian Kernighan, Rob Pike
Design Patterns: Elements of Reusable Object-Oriented Software – Erich Gamma, et al.
Effective Java – Joshua Bloch

We do a lot of interviewing at Palantir, and let me tell you: it’s hard. I don’t mean that we ask tough questions (although we do). I mean that the task of evaluating a candidate is hard.

The problem? Given a whiteboard and one hour, determine whether the person across from you is someone you’d like to work with, in the trenches, for the next n years. A candidate’s performance during an interview is only weakly correlated with his or her true potential, but we’re stuck with the problem of turning the chickenscratch on the whiteboard into an ‘aye’ or ‘nay’. Sometimes it feels like a high-stakes game of reading tea leaves. Believe me we’re doing our best, but we’re often left the nagging worry that we’re passing up brilliant people who just had a bad day or who didn’t click with a particular problem.

In an effort to improve this situation, we wanted to write up a guide that will help candidates make sense of this process, or at least the part known as an Algorithms Interview. At Palantir we ask questions that test for a lot of different skills — coding, design, systems knowledge, etc. — but one of our staple interviews is to ask you to design an algorithm to solve a particular problem.

It usually starts like this:

Given X, figure out an efficient way to do Y.

First: Make sure you understand the problem. You’re not going to lose points asking for clarifications or talking through the obvious upfront. This will also buy you time if your brain isn’t kicking in right away. Nobody expects you to solve a problem in the first 30 seconds or even the first few minutes.

Once you understand the problem, try to come up with a solution – any solution whatever. As long as it’s valid, it doesn’t matter if your solution is trivial or ugly or extremely inefficient. What matters is that you’ve made progress. This does two things: (1) it forces you to engage with the structure of the problem, priming your brain for improvements you can make later, and (2) it gives you something in the bank, which will in turn give you confidence. If you can achieve a brute force solution to a problem, you’ve cleared a major hurdle to solving it in a more efficient way.

Now comes the hard part. You’ve given an O(n^3) solution and your interviewer asks you to do it faster. You stare at the problem, but nothing’s coming to you. At this point, there are a few different moves you can make, depending on the problem at hand and your own personality. Almost all of these can help on almost any problem:

1. Start writing on the board. This may sound obvious, but I’ve had dozens of candidates get stuck while staring at a blank wall. Maybe they’re not visual people, but still I think it’s more productive to stare at some examples of the problem than to stare at nothing. If you can think of a picture that might be relevant, draw it. If there’s a medium-sized example you can work through, go for it. (Medium-sized is better than small, because sometimes the solution to a small example won’t generalize.) Or just write down some propositions that you know to be true. Anything is better than nothing.




2. Talk it through. And don’t worry about sounding stupid. If it makes you feel better, tell your interviewer, “I’m just going to talk out loud. Don’t hold me to any of this.” I know many people prefer to quietly contemplate a problem, but if you’re stuck, talking is one way out of it. Sometimes you’ll say something that clearly communicates to your interviewer that you understand what’s going on. Even though you might not put much stock in it, your interviewer may interrupt you to tell you to pursue that line of thinking. Whatever you do, please DON’T fish for hints. If you need a hint, be honest and ask for one.




3. Think algorithms. Sometimes it’s useful to mull over the particulars of the problem-at-hand and hope a solution jumps out at you (this would be a bottom-up approach). But you can also think about different algorithms and ask whether each of them applies to the problem in front of you (a top-down approach). Changing your frame of reference in this way can often lead to immediate insight. Here are some algorithmic techniques that can help solve more than half the problems we ask at Palantir:
   • Sorting (plus searching / binary search)
   • Divide-and-conquer
   • Dynamic programming / memoization
   • Greediness
   • Recursion
   • Algorithms associated with a specific data structure (which brings us to our fourth suggestion…)




4. Think data structures. Did you know that the top 10 data structures account for 99% of all data structure use in the real world? Probably not, because I just made those numbers up — but they’re in the right ballpark. Yes, on occasion we ask a problem whose optimal solution requires a Bloom filter or suffix tree, but even those problems tend to have a near-optimal solution that uses a much more mundane data structure. The data structures that are going to show up most frequently are:
   • Array
   • Stack / Queue
   • Hashset / Hashmap / Hashtable / Dictionary
   • Tree / binary tree
   • Heap
   • Graph

   You should know these data structures inside and out. What are the insertion/deletion/lookup characteristics? (O(log n) for a balanced binary tree, for example.) What are the common caveats? (Hashing is tricky, and usually takes O(k) time when k is the size of the object being hashed.) What algorithms tend to go along with each data structure? (Dijkstra’s for a graph.) But when you understand these data structures, sometimes the solution to a problem will pop into your mind as soon as you even think about using the right one.




5. Think about related problems you’ve seen before and how they were solved. Chances are, the problem you’ve been presented is a problem that you’ve seen before, or at least very similar. Think about those solutions and how they can be adapted to specifics of the problem at hand. Don’t get tripped up by the form that the problem is presented – distil it down to the core task and see if matches something you’ve solved in the past.




6. Modify the problem by breaking it up into smaller problems. Try to solve a special case or simplified version of the problem. Looking at the corner cases is a good way to bound the complexity and scope of the problem. A reduction of the problem into a subset of the larger problem can give a base to start from and then work your way up to the full scope at hand.

   Looking at the problem as a composition of smaller problems may also be helpful. For example, “find a number in a sorted array which has been shifted cyclically by an unknown constant k” can be solved by (1) first figuring out “k” and then (2) figuring out how to perform binary search on a shifted array).




7. Don’t be afraid to backtrack. If you feel like a particular approach isn’t working, it might be time to try a different approach. Of course you shouldn’t give up too easily. But if you’ve spent a few minutes on an approach that isn’t bearing any fruit and doesn’t feel promising, back up and try something else. I’ve seen more candidates who overcommit than undercommit, which means you should (all else equal) be a little more willing to abandon an unpromising approach.

Incidentally, trying out a few different approaches (rather than sticking with a single approach) tends to work well in interviews, because the problems we choose for an interview usually have many different solutions. Happily, the same is true for the problems we solve on the job =)

“You’re asking us to test our platform’s programming language? How am I supposed to do that?”

My head itches from trying to recall the bits and pieces of what I learned in high school about programming, specifically the semantics of a programming language. Sure, I did a bit of programming for homework assignments in college, but I was no CS major. This was a much different challenge for a QA engineer to test. Compared to an application, a programming language is completely open ended; there are no specifications to test, guidelines to follow, or limits to break.

The Hedgehog language had the basic set of tools laid out for me already: I could declare variables, create data structures, and use loops for iteration. As I was trying out individual usage examples, such as how to structure if statements or how to cast an object to a different type, I realized that this was no way to test something as powerful and flexible as an entire language. It would be like a doctor who claims that since each individual organ works fine, there are no problems with the entire system. This is insufficient: one needs to look at the system as a whole, including examining the interactions between each component. I decided I needed to create much larger and elaborate code samples in order to test the Hedgehog language in a larger scope.

Using the Hedgehog language, I had programmed several algorithms, solving puzzles that would output a number. This was getting bit boring, since once the output value was matched the expected number, there was nothing more to be done. I wanted to create something more dynamic, a toy I could play around and experiment with. And opportunity presented itself in the form of one of our newest tools: the spreadsheet application. With the capability of setting the value of each cell programmatically and then coloring them depending on their value… hmm what could I do with this?

Hedgehog is a powerful tool in coding functions and workflows that directly interact with our applications. Most of the time, the language is used to write expressions for an input value, create custom metrics that return values after a set of calculations, or even to set inputs, calculate, and save documents. Given the language’s ability to integrate with Spreadsheet, the capabilities of the Hedgehog language can literally be visually shown to the user, resulting in some stunning displays. Below are three examples I’ve coded in Palantir Finance’s own language: calculating and drawing the Mandelbrot fractal, simulating Conway’s Game of Life, and solving a Nonograms puzzle.

The Mandelbrot fractal

The Mandelbrot set is one of the most recognized images among fractals. Fractals are unique images due to their self-similarity, such that they have infinite detail: as you zoom in on a fractal border, detailed structures will continue to appear indefinitely.

Since the Mandelbrot set is an example of an escape time fractal, we can write a function that counts the number of iterations it takes for a given point to escape a threshold value. Applying this function to each point within a grid of specified range and resolution, we can generate a field of values to be set in a spreadsheet. After resizing the spreadsheet cells into small squares and applying conditional formatting, specifically a heat map that colors the cell based on its value, the below images can be generated:

Fig M1. Mandelbrot Fractal drawn from (-1.5, -1) to (0.5, 1), with the top left quadrant visible

Fig M2. Mandelbrot Fractal drawn from (0.27205, 0.00451) to (0.27505, 0.00751)

Fig M3. Mandelbrot Fractal with manually created rainbow heat map

The Game of Life

John Conway’s Game of Life is a simple cellular automaton that shows the evolution of an initial state of cells governed by a set of rules. In the classic scenario, cells can be alive or dead, indicated by a filled or empty cell respectively. When evaluating the next generation of cells, each cell checks the number of live neighbors among its adjacent 8 cells. If the cell is alive with 2 or 3 live neighbors, it remains alive. If a dead cell has 3 live neighbors, it becomes alive. All other scenarios, the cell evolves into a dead cell.

In this example, we rely heavily on spreadsheet’s cell dependency tree. Cells are able to reference the value of other cells within their expression, creating a directed acyclic graph. If the value of a cell changes, all cells that point towards the changed cell need to recalculate their value. In this specific case, the user can change the value of which generation to view, which causes the recalculation of the grid of cells representing the game.

In the spreadsheet below, the initial state, or generation 0, of “Noah’s Ark” is shown. The initial data for Noah’s Ark is derived from an external spreadsheet document containing a grid of 0’s and 1’s.

Below is an animated image of the evolution of Noah’s Ark from generation 0 to generation 10.

One problem encountered is that it takes increasingly longer to calculate a larger generation number. Because the only generation that is saved is the initial generation, calculations must always start from generation 0 when the user changes the generation number. This triggers a lot of redundant calculations: if I am at generation 10 and want to see generation 11, the calculation starts from generation 0, repeats all the same calculations up to generation 10, and finally calculates one more iteration to reach generation 11. To prevent unnecessary calculations, the results of previous calculations can be written to a cache file: a separate spreadsheet document containing the generation data that was calculated previously. When caching is enabled, previous generation calculations can be retrieved so the same calculation never happens twice.

Below is an animated image of a “Gospel Glider Gun” from generation 0 to generation 40, shown in increments of 2 generations. The glider gun will continue to oscillate over time while producing a 5 cell glider object that floats away from the gun diagonally down to the right. Without the caching functionality, the 40^th generation would take approximately 4 times as long as the 10^th generation. By enabling caching, each animation frame takes approximately the same time to calculate since only 2 generations are computed per frame.

Nonograms Solver

Nonograms, sometimes better known as Picross, are a type of logic puzzle where the user is given a blank grid with a set of numbers for each row and column. The numbers represent the number of consecutively filled blocks in the final solution within that row or column. Using these numbers as clues, it can be deduced whether or not a space is filled or not, until the entire grid simultaneously satisfies all given conditions. Solving puzzles is a NP-complete problem: it is very easy to verify a solution by checking if each row and column satisfies its set of numbers, but becomes increasingly difficult to solve larger puzzles. Time it takes to check a puzzle increases linearly with puzzle size, but the time to solve increases much faster due to the increase in number of iterations over each row or column crossed with an increased number of permutation possibilities for each set of numbers per iteration. Solving a 25×25 nonogram puzzle by hand usually takes about 30 minutes for an experienced solver.

Given the initial set of values for each row and column, the algorithm iterates through each row or column individually. The simplest method to solve is to find all possible permutations of the given numbers over the length of the row or column and check if there are any spaces that are consistently filled or unfilled across all possibilities. Let’s consider a simple example: In a row of 5 spaces, the set of values given is [2,1]. Thus the possible permutations are shown on the right, where 1 is a filled space, -1 is an unfilled space, and 0 is undetermined.

Additional optimizations, such as queuing the order of rows and columns to solve, trimming the solved ends of a range of spaces, and avoiding unnecessary calculations, were made to improve the performance of the algorithm such that the average time to solve a 25×25 puzzle is 5 minutes. This is fairly impressive given that the Hedgehog language is by no means optimized to handle such calculations and array manipulations. Below are some examples of solved puzzles:

Fig N1. 10x10 nonogram solution, solved in less than a second

Fig 2. 20x20 nonograms solution, solved in 20 seconds

Fig N3. 25x25 nonograms solution, solved in 3 minutes

The initial spark that ignited this series of code examples started from just a simple curiosity of, “I wonder if it would be possible to write this in Hedgehog?” In a way, it was also an exercise for me to learn more about the Hedgehog language, extending the use of its capabilities and libraries of metrics, in order to inject more complicated expressions while testing other applications on our platform. Throughout the process of developing the finished product, I was able to expose issues with the language that may have gone unnoticed with smaller, simpler test cases.

But of course, the driving motivation behind all this was to have fun, which is a big part of what it means to be working at Palantir.

We talk a lot about Human-Computer Symbiosis on this blog – it’s a systems design approach that guides us in our construction of our technology stacks. Given that, we’re always on the lookout for example of HCS systems built by other people.

Here’s an unlikely example: the layout of names in the memorial was made according to ‘meaningful adjacencies’ (as described by Jer Thorp in his blog post, All The Names: Algorithmic Design and the 9/11 Memorial):

The project was to design an algorithm for placement of names on the 9/11 memorial in New York City. In architect Michael Arad‘s vision for the memorial, the names were to be laid according to where people were and who they were with when they died – not alphabetical, nor placed in a grid. Inscribed in bronze parapets, almost three thousand names would stream seamlessly around the memorial pools. Underneath this river of names, though, an arrangement would provide a meaningful framework; one which allows the names of family and friends to exist together. Victims would be linked through what Arad terms ‘meaningful adjacencies’ – connections that would reflect friendships, family bonds, and acts of heroism. through these connections, the memorial becomes a permanent embodiment of not only the many individual victims, but also of the relationships that were part of their lives before those tragic events.

Read on for details on the approach they used and how it embodies HCS architecture (not to mention, a video of their tool in action).

The post goes deep into detail on how they built the information system to model and prototype the layout of the names. In end, they let the computers track the adjacency issues while the humans handled the overall aesthetics of not just the overall memorial, but the details of how the names were laid out:

It would be misleading to say that the layout for the final memorial was produced by an algorithm. Rather, the underlying framework of the arrangement was solved by the algorithm, and humans used that framework to design the final result. This is, I think, a perfect example of something that I’ve believed in for a long time: we should let computers do what computers do best, and let humans do what humans do best. In this case, the computer was able to evaluate millions of possible solutions for the layout, manage a complex relational system, and track a large set of important variables and measurements. Humans, on the other hand, could focus on aesthetic and compositional choices. It would have been very hard (or impossible) for humans to do what the computer did. At the same time, it would have been very difficult to program the computer to handle the tasks that were completed with such dedication and precision by the architects and the memorial team.

To quote J.R. Licklider, whose Man-Computer Symbiosis paper is widely considered to be the seminal work on the how and why of building HCS systems:

Men will set the goals and supply the motivations, of course, at least in the early years. They will formulate hypotheses. They will ask questions. They will think of mechanisms, procedures, and models. They will remember that such-and-such a person did some possibly relevant work on a topic of interest back in 1947, or at any rate shortly after World War II, and they will have an idea in what journals it might have been published. In general, they will make approximate and fallible, but leading, contributions, and they will define criteria and serve as evaluators, judging the contributions of the equipment and guiding the general line of thought.

In addition, men will handle the very-low-probability situations when such situations do actually arise. (In current man-machine systems, that is one of the human operator’s most important functions. The sum of the probabilities of very-low-probability alternatives is often much too large to neglect. ) Men will fill in the gaps, either in the problem solution or in the computer program, when the computer has no mode or routine that is applicable in a particular circumstance.

The information-processing equipment, for its part, will convert hypotheses into testable models and then test the models against data (which the human operator may designate roughly and identify as relevant when the computer presents them for his approval). The equipment will answer questions. It will simulate the mechanisms and models, carry out the procedures, and display the results to the operator. It will transform data, plot graphs (“cutting the cake” in whatever way the human operator specifies, or in several alternative ways if the human operator is not sure what he wants). The equipment will interpolate, extrapolate, and transform. It will convert static equations or logical statements into dynamic models so the human operator can examine their behavior. In general, it will carry out the routinizable, clerical operations that fill the intervals between decisions.

WTC Names Arrangement Tool from blprnt on Vimeo.

A few months back, Kevin introduced us to the Hedgehog Programming language – (here’s the post if you missed it).

The Palantir Finance programming language — Hedgehog as we know it — is an interpreted, statically typed, object-oriented language. With a syntax that’s based loosely on Java, it mixes roughly Java-style semantics and a few idiosyncrasies that make it a really interesting case study in language design. It’s built to be extremely efficient for batch operations on time series, which is the heavy lifting in financial analysis.

In this video, Eugene and Dave, two of the engineers that work on the language and platform features needed to support it, give a talk that goes into a number of areas around the Hedgehog language, including why we needed to build a language, how it makes the platform more powerful, how we built dev tools into the UI to make debugging easier, and a bunch of the nitty-gritty features that go into the strange (but fitting) beast that is the Hedgehog Language.

As a final note: this is one of things that I love about working at Palantir Technologies. We study a problem pretty hard before we decide that we need to re-invent the wheel – and then when we do, we go all out. It’s one of the benefits of working with the incredibly talented and motivated folks here. When someone says, “well, we need to build a programming language. No, we’re sure,” we just roll up our sleeves and do it. We can add it to the list of: JMX monitoring system, refined Lucene search engine, speeding up Map-Reduce-like systems to interactive time, and implementing our own GIS platform.

Late last year, we were honored to be invited to talk at Reflections|Projections, ACM@UIUC’s annual student-run computing conference. We decided to bring a talk about Horizon, our system for doing aggregate analysis and filtering across very large amounts of data. The video of the talk was posted a few weeks back on the conference website.

Horizon started as research project / technology demonstrator built as part of Palantir’s Hack Week – a periodic innovation sprint that our engineering team uses to build brand new ideas from whole cloth. It was then used by the Center For Public Integrity in their Who’s Behind The Subprime Meltdown report. We produced a short video on the subject, Beyond the Cloud: Project Horizon, released on our analysis blog. Subsequently, it was folded into our product offering, under the name Object Explorer.

In this hour-long talk, two of the engineers that built this technology tell the story of how Horizon came to be, how it works, and show a live demo of doing analysis on hundreds of millions of records in interactive time.

public abstract class ForwardingCollection<E> extends ForwardingObject implements Collection<E> {
  @Override protected abstract Collection<E> delegate();

  public boolean add(E element) {
    return delegate().add(element);
  }

  // ... (more overridden methods) ...
}

Things worth noting:

• This class overrides the delegate() method to return a more specific type.
• The class contains no instance variables and does not have to be marked Serializable.

Now, here’s an example of how to implement a decorator using a forwarding class (also from Google Collections):

  public class ConstrainedCollection<E> extends ForwardingCollection<E> {
    private final Collection<E> delegate;
    private final Constraint<? super E> constraint;

    public ConstrainedCollection(Collection<E> delegate, Constraint constraint) {
      this.delegate = checkNotNull(delegate);
      this.constraint = checkNotNull(constraint);
    }
    @Override protected Collection<E> delegate() {
      return delegate;
    }
    @Override public boolean add(E element) {
      constraint.checkElement(element);
      return super.add(element);
    }
    @Override public boolean addAll(Collection<? extends E> elements) {
      return super.addAll(checkElements(elements, constraint));
    }
  }

This class implements a collection that checks a constraint on all elements added to the collection. Note that writing the decorator is easy given the forwarding class — only the methods relevant to the decorator need to be overridden.