July 23rd, 2010 | Ari
Cyber security is a hot topic, especially in national security circles. The world has witnessed a number of high-profile incidents in the past two years that have been notable for sharing three very important aspects:
- they were targeted attacks, carried out against specific institutions
- they were politically motivated, and, inconclusively, appear to be state-sponsored
- they used multiple-step, multi-vectors attacks and managed to evade existing security countermeasures
This deviates from the types of attacks that IT-centric approaches have sought to defend networks against. Traditional approaches neutralize the perceived threats against a network with a host of countermeasures: firewalls, malware scanners, automated network vulnerability scanning, patch policies, and intrusion detection systems. The network defenses can learn new tricks when the administrators update the signatures, or, for certain types of data, employ a Bayesian inference strategy (as has been employed to fight spam). This approach does a good job of protecting against untargeted attacks as well as weak targeted attacks.
Full network defense requires human analysts looking at anomalies at a level above the automated countermeasures. Check out the rest of this post to take a look at how human-driven, computer-aided analysis is a game changer in cyber security.
Read the rest of this entry »
June 2nd, 2010 | Asher
This is a response to Ari’s awesome post on human-computer symbiosis. Ari and I were chatting about the equation he developed and I was wondering if there were some further refinements that are possible… let’s take a look:
We are attempting to understand the total analytic capability for a given task a of a human-computer team. Analytic capability in this case probably means:
(1)
Where A is the answer to the analytic problem in question and tA is the time needed to arrive at the answer based on the inputs available. In the case of chess, A could be the optimum next move given all previous information and tA would be how long it takes to decide on this move.
Read on for a look at how this generalizes in human-computer symbiotic systems.
Read the rest of this entry »
April 5th, 2010 | Ari
[Editor's Note: an edited version of this post first appeared on O'Reilly's Radar blog.]
The prologue was an earthquake of unexpected magnitude and location that left 250,000 dead.
As computer scientists and technologists, we’re used to dealing with large numbers in the abstract. Expressed in human terms, the mind-boggling numbers of 250,000 dead, 300,000 injured and over 1 million people left homeless are hard to comprehend.
Hit the link to read more about how effective data management and analysis is crucial to recovery efforts and see specific examples of data about the situation in Haiti modeled in Palantir Government.
Read the rest of this entry »
March 8th, 2010 | Ari
As we build our platforms and applications following a human-computer symbiosis approach, we keep an ear to the ground for interesting examples that illuminate new techniques or validate our approach in some empirical way.
One of the areas that we’re interested is in the overall friction of analysis systems. The systems that we build are built on commodity hardware — we’re not building faster computers and yet we can deliver orders-of-magnitude better performance on analysis tasks than existing solutions. How do we do this? By building software in such a way that it reduces the friction experienced at the boundaries between the computing power, the analyst, and the source data.
Chess as analysis laboratory
Chess is, at its heart, a predictive venture. The player attempts to anticipate their opponent’s moves, planning their own moves accordingly, with the straightforward goal of finding a sequence of piece moves that force checkmate.
This game is, in its ideal form, analysis. (The moves made are the logical extension of the analysis.) The data are clean, the problem is well-defined and everyone plays by the same rules. There are even well-defined metrics for ranking chess players by skill — a better chess player is a better chess-game analyst.
In the realm of evaluation of analysis systems, this is as about as good as it gets in terms of designing controlled experiments to study the relative strengths of different analysis systems.
Garry Kasparov, widely considered to be the greatest chess player of all time, recently wrote a review of Diego Rasskin Gutman’s book, Chess Metaphors: Artificial Intelligence and the Human Mind.
The review is excellent and covers a lot of ground. However, one particular anecdote stood out as a very interesting example of human-computer symbiosis (emphasis added):
In 2005, the online chess-playing site Playchess.com hosted what it called a “freestyle” chess tournament in which anyone could compete in teams with other players or computers. Normally, “anti-cheating” algorithms are employed by online sites to prevent, or at least discourage, players from cheating with computer assistance. (I wonder if these detection algorithms, which employ diagnostic analysis of moves and calculate probabilities, are any less “intelligent” than the playing programs they detect.)
Lured by the substantial prize money, several groups of strong grandmasters working with several computers at the same time entered the competition. At first, the results seemed predictable. The teams of human plus machine dominated even the strongest computers. The chess machine Hydra, which is a chess-specific supercomputer like Deep Blue, was no match for a strong human player using a relatively weak laptop. Human strategic guidance combined with the tactical acuity of a computer was overwhelming.
The surprise came at the conclusion of the event. The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.
After the jump, we look at this finding in a more generalized way and map it onto the Palantir approach.
Read the rest of this entry »
November 22nd, 2009 | Steve
Here at Palantir, a lot of our automatic tests are full-chain tests. A backend server is fired up, client code runs against it, and everything runs much like a production environment. This makes intuitive sense because it’s a faithful approximation of how the system will run in the field.
However, there are some disadvantages to this:
- Full-pass tests don’t always localize the problem. Tests on a client class might fail even if it was the service that behaved incorrectly.
- These full-pass tests are relatively slow. Client code is running against an actual remote service. If a client is being tested, the server code still has to do work — sometimes a lot of work — even if that isn’t the focus of the test.
- The constraints of the test are loose. Full-chain tests can mostly only see whether the operation finished correctly. It’s much harder to figure out whether the operation was done efficiently and without making unnecessary service calls.
- They’re very little setup flexibility. If you want an RPC to return a specific value, you have little choice but to have your test get the service into a state where it can return that value. This is easy in some cases, but prohibitively difficult in others.
- Client tests are forced to share any non-determinism leaked from the service. For example, under real conditions, a request to call A might respond before call B, and sometimes the other way around. This can result in flaky tests or tests that don’t always simulate the conditions you want to exercise.
What’s to be done? Fortunately, there’s an option that handles these cases elegantly. We also test with jMock, a library that dynamically generates mock objects from arbitrary interfaces. These mock objects can be configured to check that particular methods are called with particular inputs a particular number of times, and then give prescribed responses.
Hit the link to see a concrete example of jMock in action.
Read the rest of this entry »
November 6th, 2009 | Ari
If you’ve taken the time to peruse the Palantir Government analysis blog, you’ve seen numerous examples of Palantir Government as applied to interesting problems; they are recorded screen captures of our analysis desktop client. It’s a showcase of useful, meaningful, and compelling visual and semantic tools being used to do analysis on a wide range of datasets.
What enabled this analysis? Aside from the obvious hard work of our UI and analysis tools teams, it’s the flexibility and power of the Palantir data platform. More than just a scalable datastore, the Palantir data platforms act as robust and clean abstractions on top of data.
One of the early architecture decisions that we made when building both Palantir Government and Palantir Finance was to separate the respective data platforms from the end-user applications used to actually perform analysis. More than just following the client-server model, this separation made the data servers in both products into generic intelligence infrastructure for analytic problems, with our clients acting as analysis applications on top of those platforms.
And so, one way to look at our data platform is as an operating system for analytic applications. In this post we’ll explore the history of operating systems, understand why they’re so important and see how the Palantir data servers deliver the same potential to revolutionize the writing of analysis software that operating systems did to the writing of general programs for computers.
Read the rest of this entry »
October 27th, 2009 | Ari
[A number of weeks ago, we published a post on the search technology used by Palantir. That post covered raising the memory efficiency of a couple of operations. This is part two of that series.]
The most familiar use of search engines is to index documents made available on the Internet via the hypertext transfer protocol. Forgotten names like AltaVista, names not-yet-really-learned like Bing, and, of course, Google come to mind.
This one, massive use case has a couple of properties that I’d like to highlight:
- Asynchronous indexing and querying – web search engines tend to use crawlers and indexers to build up an index of the web. After each crawl is finished, the new index is brought online for use by the query engine.
- Lack of access controls – all the data in the index is available to any query. In fact, most queries are (from the standpoint of the index) completely anonymous.
Palantir: not a web search engine
Search technology is just one part of what makes up a Palantir system. For us, it’s a way to quickly retrieve Palantir objects in a Palantir system, it’s not the whole of the application.
I’d like to highlight a couple of differences from the web search engine case. A Palantir system needs the following properties:
- Realtime indexing and querying – we need information to be available immediately as it changes in the system.
- Leak-proof access controls – we need the search engine to help us make sure that we don’t have information leaking across access control boundaries.
Hit the link to read more about these topics.
Read the rest of this entry »
September 29th, 2009 | Ari
Software engineering is a craft that blends science and art. This fact is easy to overlook as the artistic aspects are often eclipsed by discussions of the science and technology behind what we do.
This is not one of those times: the art in software engineering is most evident when building compelling visual interfaces, something Palantir knows a thing or two about.
A demo reel is an industry term in the movie business — a short reel that acts as a portfolio when applying for jobs, a highlight reel of the author’s visual career. We’re not in the movie business, we’re in the software business. We do, however, use moving pictures to tell stories, stories backed by data — this is our demo reel: two-and-a-half minutes of data visualization and user interface eye-candy (It has pounding music — you may want to put on headphones or turn down your speakers.):
The movie will take a few seconds to load. It’s 800×600, so expanding to full-screen is suggested. We’ve done our best to create a streamable-yet-good-looking video. The compression artifacts are there, but shouldn’t be too distracting. In a real Palantir client, there are no compression artifacts and everything looks even better than it does here.
The Palantir family of products is much more that just pretty pictures; we have the underlying intelligence infrastructure to make those realtime animations possible and (more importantly) meaningful. That said, we sure do think they’re pretty.
By the way, if you’re interested in the progression of our interfaces, this not the first time we’ve posted eye candy: we posted a set of updated screenshots a little over a year ago; think of this as the next installment in the series.
And yes, it’s really all Java Swing.
August 26th, 2009 | Andrew C.
In a previous post, Eric W. covered how we analyze polled system health information. Now we’ll look at pushed information, in the form of logging events.
Use Cases & Constraints
We decided on three kinds of questions we wanted to answer:
- What is the health of the deployment?
- Example: What errors have occurred in the last 24 hours?
- Which parts of the platform are our users engaged with?
- Example: How much time do users spend in each application?
- How is our server performing over time?
- Example: What is the average wait on a search query?
The chief constraint was that we build our platform on Log4J. We already use Log4J all over the project, so converting the logging was out of the question. Besides, Log4J provides a guideline for the kind of metadata our events should support, and Log4J makes it easy to record events to a database.
That left us with two problems to solve: how to store structured data with a Log4j message, and how to analyze the collected data.
Analysis is the easy part: just use Palantir! After all, a sequence of logging events has a lot in common with a time series. The rest is explained below.
Read the rest of this entry »
August 24th, 2009 | Ari
We put up a post last year on the 2008 VAST Grand Challenge. Well, the IEEE VAST Challenge 2009 is over and the awards are in. We had another strong year, scoring two awards:
- Grand Challenge: Analyst’s Tool Choice (Of 48 submissions, only 3 Grand Challenge awards were given)
- Intuitive Traffic Visualization and Video Description of the Analysis Process
Some background on the event: three years ago, the IEEE began an annual conference called VAST (Visual Analytics in Science and Technology). The VAST symposium focuses on the fundamental research contributions and real-world application of visual analytics. As a part of the conference, the VAST Challenge allows teams to compete on delivering analytic solutions against a synthetic real-world dataset.
A selection of choice quotes from the judges:
- An award for “highly usable integrated exploration environment”, “efficient analytic exploration platform” or something along these lines would be appropriate.
- Survey Question: How much novelty do you see in this submission (data processing, visualization, interaction, hypothesis generation or evaluation, overall process, etc.)? Answer: More so than novelty was the extremely efficient solution approach to this challenge, much more so than other solutions.
- The submission shows two things very clearly: One, it shows the analytical process as being a multi-faceted, simultaneous processing of different information that is quite common among analysts. Two, it shows how multiple perspectives can be displayed on a single monitor, enabling the analyst to visualize what his mind is analyzing. Outstanding!
Our submission
And finally, our submission to the Grand Challenge. Here we have our overview video, with a link to the full video below:
For an in-depth look at the data and techniques used to make this a reality, check out our full submission in Finding a Mole: Cyber Counter Intelligence on the Palantir Analysis Blog.
