As we roll into the peak of internship season, it seems like a worthwhile time to talk about just what it is that interns do in their time at Palantir: our software engineering interns are full members of the development team from the day they arrive. During their time with us, they design, implement, and test their projects while alongside the full time engineers.
We believe in hitting the ground running; in fact, before you get your badge on the first day, you must commit code. But don’t worry, this all takes place under the watchful eyes of your mentor: a full-time engineer who’s there to guide through everything from our development environment, how to write specifications, to software architecture, to how to figure out which cafe has the best lunch.
You can broadly break down the projects the Palantir interns worked on into a number of categories:
Geospatial Enhancements – adding in new geospatial features at various levels of our software stack.
Join Us In 2012!
We’re currently hiring for our 2012 intern class as well as full time positions. For those interested in working on the sorts of projects mentioned here, you’ll want to apply for this job:
From those from non-traditional schools that have co-op programs that run during the year, we offer those as well. Go ahead and apply for the appropriate intern position and make a note of when you’d like your co-op to be in your application.
Read on to dive into a large sampling of software engineering projects our interns worked on this past summer. Read the rest of this entry »
We’re big fans of open source. Libraries from Apache, Google, and various projects hosted on SourceForge.net make up a significant fraction of the third-party code we use to build our products.
We’re proud to be making our first set of open source releases with these two projects: Cinch and Sysmon.
We think it’s the right thing to do, to add our voice to the chorus of developers making software available to freely use, modify, and distribute. These two projects represent our first dip into the open source water – we’re just getting started. As time and other interests allow, we’ll be making other projects available to the dev community.
We’ve chosen the Apache License, Version 2.0 to make our contributions as free from encumberance as possible – our hope is that many people will find them useful and build on top of them just as we have with our own software.
Cinch is a Java library for simplifying certain types of GUI code. When developing Swing applications it’s easy to fall into the trap of not separating out Models and Controllers. It’s all too easy to just store the state of that boolean in the checkbox itself, or that String in the JTextField. The design goal behind Cinch was to make it easier to apply MVC than to not by reducing much of the typical Swing friction and boilerplate. Cinch uses Java annotations to reflectively wire up Models, Views, and Controllers.
Sysmon – A lightweight platform monitoring tool for Java VMs
Sysmon is a lightweight platform monitoring tool. It was designed to gather performance data (CPU, disks, network, etc.) from the host running the Java VM. This data is gathered, packaged, and published via Java Management Extensions (JMX) for access using the JMX APIs and standard tools (such as jconsole). Sysmon can be run as a standalone daemon or as a library to add platform monitoring to any application.
Originally built as component in our Palantir cluster monitoring server, this project should be helpful in scenarios where you need to get data off a host platform and into a VM.
Let us know how we’re doing
We’d love to hear from you on how we’re doing. Aside from the normal outlets to communicate about the projects themselves (see the mailing lists and issue trackers for each project), please feel free to email me directly, Ari Gesher, as the curator of these projects.
Comic courtesy of XKCD, via Creative Commons License
Our frontend engineering candidates go through many of the same types of interviews as our backend candidates, i.e., algo and coding. But because they’ll be building the user-facing parts of our system, we want to make sure candidates have some design chops as well. Hence the UI design interview.
Why have a design interview for an engineer?
Perhaps my favorite part of being on the frontend team at Palantir is that we get to work on both the design and the implementation of our own UIs, from inception to finished product. Engineers collaborate on the design of the product with other engineers as well as with designers, and we openly debate the merits of all our decisions. The UI interview simulates this kind of cooperative design and debate. Typically, this means you’ll be asked to design and/or evaluate one or two pieces of UI during the interview, while your interviewer plays both collaborator and critic.
Read on for some helpful points to keep in mind as you work on a design problem with your interviewer. Read the rest of this entry »
One interview that candidates often struggle with is the systems design interview. Even if you know your algorithms and write clean code, that code needs to run on a computer somewhere — and then things quickly get complicated. A truly unbelievable amount of complexity lies beneath something as simple as visiting Google in your browser. While most of that complexity is abstracted away from the end user, as a system designer you have to face it head on, and the more you can handle, the better.
At Palantir, many of our teams give a systems design interview along with an algorithms interview and a couple of coding interviews. We don’t expect anyone to be an expert at all three disciplines (although some are). We’re looking for generalists with depth — people who are good at most things, and great at some. If systems design isn’t your strength, that’s okay, but you should at least be able to talk and reason competently about a complex system.
Read on to learn about what we’re looking for and how you can prepare.
Here at Palantir algorithms are important, but code is our lifeblood. We live and die by the quality of the code we ship. It’s no surprise, then, that coding ability is what we stress the most in our interview process. A candidate can get by with mediocre algorithm skills (depending on the role), but no one can skimp on coding.
Suppose you’re confident in your ability to write great software. Your task in a coding interview (of which there will be several) is to show the interviewers that you in fact do have the programming chops — that you’re an experienced coder who knows how to write solid, production-quality code.
This is easier said than done. After all, coding in your favorite IDE from the comfort of $familiar_place is very different from coding on a whiteboard (on a problem you’re totally unfamiliar with) in a pressure-filled 45-minute interview. We realize that the interview environment is not the real world, and we adjust our expectations accordingly. Nonetheless, there are a number of things you can do to put your best foot forward during the interview.
First, though, we’d like to give you a sense for what we look for during a coding interview. Most important is the ability to write clean and correct code — it’s not enough just to be correct. A lot of people will be interacting with your code once you’re on the job, so it should be readable, maintainable, and extensible where appropriate. If your solution is clean and correct, and you produced it in a reasonable amount of time without a lot of help, you’re in good shape. But even if you stumble a bit, there are other ways to demonstrate your ability. As you work, we also watch for debugging ability, problem-solving and analytical skills, creativity, and an understanding of the ecosystem that surrounds production code.
With our evaluation criteria in mind, here are some suggestions we hope will help you perform at your very best.
Comic courtesy of XKCD, via Creative Commons License
We do a lot of interviewing at Palantir, and let me tell you: it’s hard. I don’t mean that we ask tough questions (although we do). I mean that the task of evaluating a candidate is hard.
The problem? Given a whiteboard and one hour, determine whether the person across from you is someone you’d like to work with, in the trenches, for the next n years. A candidate’s performance during an interview is only weakly correlated with his or her true potential, but we’re stuck with the problem of turning the chickenscratch on the whiteboard into an ‘aye’ or ‘nay’. Sometimes it feels like a high-stakes game of reading tea leaves. Believe me we’re doing our best, but we’re often left the nagging worry that we’re passing up brilliant people who just had a bad day or who didn’t click with a particular problem.
In an effort to improve this situation, we wanted to write up a guide that will help candidates make sense of this process, or at least the part known as an Algorithms Interview. At Palantir we ask questions that test for a lot of different skills — coding, design, systems knowledge, etc. — but one of our staple interviews is to ask you to design an algorithm to solve a particular problem.
It usually starts like this:
Given X, figure out an efficient way to do Y.
First: Make sure you understand the problem. You’re not going to lose points asking for clarifications or talking through the obvious upfront. This will also buy you time if your brain isn’t kicking in right away. Nobody expects you to solve a problem in the first 30 seconds or even the first few minutes.
Once you understand the problem, try to come up with a solution – any solution whatever. As long as it’s valid, it doesn’t matter if your solution is trivial or ugly or extremely inefficient. What matters is that you’ve made progress. This does two things: (1) it forces you to engage with the structure of the problem, priming your brain for improvements you can make later, and (2) it gives you something in the bank, which will in turn give you confidence. If you can achieve a brute force solution to a problem, you’ve cleared a major hurdle to solving it in a more efficient way.
Now comes the hard part. You’ve given an O(n^3) solution and your interviewer asks you to do it faster. You stare at the problem, but nothing’s coming to you. At this point, there are a few different moves you can make, depending on the problem at hand and your own personality. Almost all of these can help on almost any problem:
Start writing on the board. This may sound obvious, but I’ve had dozens of candidates get stuck while staring at a blank wall. Maybe they’re not visual people, but still I think it’s more productive to stare at some examples of the problem than to stare at nothing. If you can think of a picture that might be relevant, draw it. If there’s a medium-sized example you can work through, go for it. (Medium-sized is better than small, because sometimes the solution to a small example won’t generalize.) Or just write down some propositions that you know to be true. Anything is better than nothing.
Talk it through. And don’t worry about sounding stupid. If it makes you feel better, tell your interviewer, “I’m just going to talk out loud. Don’t hold me to any of this.” I know many people prefer to quietly contemplate a problem, but if you’re stuck, talking is one way out of it. Sometimes you’ll say something that clearly communicates to your interviewer that you understand what’s going on. Even though you might not put much stock in it, your interviewer may interrupt you to tell you to pursue that line of thinking. Whatever you do, please DON’T fish for hints. If you need a hint, be honest and ask for one.
Think algorithms. Sometimes it’s useful to mull over the particulars of the problem-at-hand and hope a solution jumps out at you (this would be a bottom-up approach). But you can also think about different algorithms and ask whether each of them applies to the problem in front of you (a top-down approach). Changing your frame of reference in this way can often lead to immediate insight. Here are some algorithmic techniques that can help solve more than half the problems we ask at Palantir:
Sorting (plus searching / binary search)
Divide-and-conquer
Dynamic programming / memoization
Greediness
Recursion
Algorithms associated with a specific data structure (which brings us to our fourth suggestion…)
Think data structures. Did you know that the top 10 data structures account for 99% of all data structure use in the real world? Probably not, because I just made those numbers up — but they’re in the right ballpark. Yes, on occasion we ask a problem whose optimal solution requires a Bloom filter or suffix tree, but even those problems tend to have a near-optimal solution that uses a much more mundane data structure. The data structures that are going to show up most frequently are:
Array
Stack / Queue
Hashset / Hashmap / Hashtable / Dictionary
Tree / binary tree
Heap
Graph
You should know these data structures inside and out. What are the insertion/deletion/lookup characteristics? (O(log n) for a balanced binary tree, for example.) What are the common caveats? (Hashing is tricky, and usually takes O(k) time when k is the size of the object being hashed.) What algorithms tend to go along with each data structure? (Dijkstra’s for a graph.) But when you understand these data structures, sometimes the solution to a problem will pop into your mind as soon as you even think about using the right one.
Think about related problems you’ve seen before and how they were solved. Chances are, the problem you’ve been presented is a problem that you’ve seen before, or at least very similar. Think about those solutions and how they can be adapted to specifics of the problem at hand. Don’t get tripped up by the form that the problem is presented – distil it down to the core task and see if matches something you’ve solved in the past.
Modify the problem by breaking it up into smaller problems. Try to solve a special case or simplified version of the problem. Looking at the corner cases is a good way to bound the complexity and scope of the problem. A reduction of the problem into a subset of the larger problem can give a base to start from and then work your way up to the full scope at hand.
Looking at the problem as a composition of smaller problems may also be helpful. For example, “find a number in a sorted array which has been shifted cyclically by an unknown constant k” can be solved by (1) first figuring out “k” and then (2) figuring out how to perform binary search on a shifted array).
Don’t be afraid to backtrack. If you feel like a particular approach isn’t working, it might be time to try a different approach. Of course you shouldn’t give up too easily. But if you’ve spent a few minutes on an approach that isn’t bearing any fruit and doesn’t feel promising, back up and try something else. I’ve seen more candidates who overcommit than undercommit, which means you should (all else equal) be a little more willing to abandon an unpromising approach.
Incidentally, trying out a few different approaches (rather than sticking with a single approach) tends to work well in interviews, because the problems we choose for an interview usually have many different solutions. Happily, the same is true for the problems we solve on the job =)
“You’re asking us to test our platform’s programming language? How am I supposed to do that?”
My head itches from trying to recall the bits and pieces of what I learned in high school about programming, specifically the semantics of a programming language. Sure, I did a bit of programming for homework assignments in college, but I was no CS major. This was a much different challenge for a QA engineer to test. Compared to an application, a programming language is completely open ended; there are no specifications to test, guidelines to follow, or limits to break.
The Hedgehog language had the basic set of tools laid out for me already: I could declare variables, create data structures, and use loops for iteration. As I was trying out individual usage examples, such as how to structure if statements or how to cast an object to a different type, I realized that this was no way to test something as powerful and flexible as an entire language. It would be like a doctor who claims that since each individual organ works fine, there are no problems with the entire system. This is insufficient: one needs to look at the system as a whole, including examining the interactions between each component. I decided I needed to create much larger and elaborate code samples in order to test the Hedgehog language in a larger scope.
Using the Hedgehog language, I had programmed several algorithms, solving puzzles that would output a number. This was getting bit boring, since once the output value was matched the expected number, there was nothing more to be done. I wanted to create something more dynamic, a toy I could play around and experiment with. And opportunity presented itself in the form of one of our newest tools: the spreadsheet application. With the capability of setting the value of each cell programmatically and then coloring them depending on their value… hmm what could I do with this?
Hedgehog is a powerful tool in coding functions and workflows that directly interact with our applications. Most of the time, the language is used to write expressions for an input value, create custom metrics that return values after a set of calculations, or even to set inputs, calculate, and save documents. Given the language’s ability to integrate with Spreadsheet, the capabilities of the Hedgehog language can literally be visually shown to the user, resulting in some stunning displays.Below are three examples I’ve coded in Palantir Finance’s own language: calculating and drawing the Mandelbrot fractal, simulating Conway’s Game of Life, and solving a Nonograms puzzle. Read the rest of this entry »
We talk a lot about Human-Computer Symbiosis on this blog – it’s a systems design approach that guides us in our construction of our technology stacks. Given that, we’re always on the lookout for example of HCS systems built by other people.
The project was to design an algorithm for placement of names on the 9/11 memorial in New York City. In architect Michael Arad‘s vision for the memorial, the names were to be laid according to where people were and who they were with when they died – not alphabetical, nor placed in a grid. Inscribed in bronze parapets, almost three thousand names would stream seamlessly around the memorial pools. Underneath this river of names, though, an arrangement would provide a meaningful framework; one which allows the names of family and friends to exist together. Victims would be linked through what Arad terms ‘meaningful adjacencies’ – connections that would reflect friendships, family bonds, and acts of heroism. through these connections, the memorial becomes a permanent embodiment of not only the many individual victims, but also of the relationships that were part of their lives before those tragic events.
Read on for details on the approach they used and how it embodies HCS architecture (not to mention, a video of their tool in action). Read the rest of this entry »
The Palantir Finance programming language — Hedgehog as we know it — is an interpreted, statically typed, object-oriented language. With a syntax that’s based loosely on Java, it mixes roughly Java-style semantics and a few idiosyncrasies that make it a really interesting case study in language design. It’s built to be extremely efficient for batch operations on time series, which is the heavy lifting in financial analysis.
In this video, Eugene and Dave, two of the engineers that work on the language and platform features needed to support it, give a talk that goes into a number of areas around the Hedgehog language, including why we needed to build a language, how it makes the platform more powerful, how we built dev tools into the UI to make debugging easier, and a bunch of the nitty-gritty features that go into the strange (but fitting) beast that is the Hedgehog Language.
As a final note: this is one of things that I love about working at Palantir Technologies. We study a problem pretty hard before we decide that we need to re-invent the wheel – and then when we do, we go all out. It’s one of the benefits of working with the incredibly talented and motivated folks here. When someone says, “well, we need to build a programming language. No, we’re sure,” we just roll up our sleeves and do it. We can add it to the list of: JMX monitoring system, refined Lucene search engine, speeding up Map-Reduce-like systems to interactive time, and implementing our own GIS platform.
Late last year, we were honored to be invited to talk at Reflections|Projections, ACM@UIUC’s annual student-run computing conference. We decided to bring a talk about Horizon, our system for doing aggregate analysis and filtering across very large amounts of data. The video of the talk was posted a few weeks back on the conference website.
Horizon started as research project / technology demonstrator built as part of Palantir’s Hack Week – a periodic innovation sprint that our engineering team uses to build brand new ideas from whole cloth. It was then used by the Center For Public Integrity in their Who’s Behind The Subprime Meltdown report. We produced a short video on the subject, Beyond the Cloud: Project Horizon, released on our analysis blog. Subsequently, it was folded into our product offering, under the name Object Explorer.
In this hour-long talk, two of the engineers that built this technology tell the story of how Horizon came to be, how it works, and show a live demo of doing analysis on hundreds of millions of records in interactive time.
Disclaimer: Palantir Technologies is not affiliated with, endorsed or sponsored by Palantir.net, Inc.
Palantir.net's website is located at www.palantir.net
Scholarship for Women & Minorities in Computer Science
