Carl juggles with his creation

One of the things our customers love to do is print our beautiful object graphsландшафт and tape them to the wall for discussion. What they hate to do is print 30 pages, line them up, and tape them to a poster one at a time. So we bought a plotter, and I started plotting.

I needed to print directly to a Java Graphics object. Unfortunately, the available information on large output printing from Java is thin at best. While there are lots of ways to successfully place ink on paper, I was only able to find one that reliably lets the application pick odd paper sizes that plotters use, like 24×19.7 inches. (The term “plotter” used to mean something with pens for printing blueprints and such. Now it just means a large format printer, commonly printers that can use roll paper as a source.)

One of the first things you’ll learn when you start working with printing in Java is that a language intended to be all things to all people (i.e., cross-platform) is utterly lousy at tasks highly specific to a given environment, such as printing. It will not surprise you to hear that native print services on Windows are pretty different from those available on a Mac, which themselves are pretty different from the CUPS system common to Unix systems.

So, by and large, you are reduced to the least common denominator of printing. Part and parcel of this least common denominator is agreeing on what constitutes a piece of paper and sticking to it. This is fine for people thinking, “My paper is 8.5 inches wide by 11 inches tall.” It poses a bit of a problem for people with plotters who are thinking, “My paper is 24 inches wide by as many damned inches tall as I need.” Even relatively powerful programs like PhotoShop or GIMP don’t seem to support plotters well. I believe Photoshop works by specifying the exact paper size you want to use, but any technique in which the easiest solution for the user is to pull out a calculator does not meet with my approval.

Advice in a nutshell

A tutorial by MartyHall provides some valuable insights into printing which I applied to my printing component. While it didn’t address my specific issues, props to a well-done starting point.

I’m starting with the assumption that you’ve got yourself a class that implements the Printable interface so that once the Java printing subsystem is coerced into accepting the appropriate paper size, Printable.print(Graphics, PageFormat, int) will be called correctly and your job will print properly. Since my usage was screen-visible as well as printable, I simplified and asserted that one screen pixel is equivalent to one point (1/72 inch), which is how Java prepares the scaling on the Graphics object passed to Printable.print(…), thus making printouts look about the size of onscreen displays. Then, knowing that the printout needed, for example, 1200 vertical pixels/points, I can automatically calculate that I want to use a paper size 18″ tall or so (1200 / 72 dpi = 16.67″ + margins).

Here’s a summary of the critical steps:

1. Do the math yourself to get the paper size required in points (1/72 of an inch)
2. Implement the Pageable interface to report page count (usually one) and the PageFormat using the computed paper size.
3. Create a PrinterJob configured with the correct PrintService
4. Set your Pageable on the PrinterJob instance
5. Optionally display print setup dialog (PrinterJob.printDialog()) recognizing that changes to orientation or paper size will be ignored.
6. call PrinterJob.print()

Generally, don’t use the standard dialogs

This might seem contrary to desired behavior, but I found that giving the user access to the usual native or cross-platform print dialogs can be a bit sticky because those are places where the user can alter the selected paper; in order to properly print to roll paper sources, we must override any paper settings the dialogs might provide. Ultimately this can be confusing for the user, but that’s an unfortunate side effect of supporting plotters.

Don’t take any advice too seriously

We still choose to display the dialog because complex printers like plotters have many options that the users may need to alter and would otherwise have no way to modify.

StickFigure: looking at plotting in Palantir

This work didn’t happen in a vacuum: we have a need to print large graphs for display. Being able to print them in a large format is one of the simplest and most beloved methods of collaboration.

To give an example of why plotter printing is interesting, we created a stick figure on the graph in Palantir. After creating this monstrosity, we went go print it and examined the layout. The first layout was with letter-size paper, and just by looking at the head, we could see that it was going to be a lot of pages:

Zoomed-in view of the head of the stick figure. (click for full image)

Zooming out to view the entire graph, it’s clear that it’s going to be 35 pages!

Full pagination of the stick figure. (click for full image)

Even switching to the largest standard paper size only gets us to two sheets:

Paginated out to largest standard paper size, still two pages (click for full image)

When we switch over to using the plotter layout, we finally get to the one sheet we were looking for:

Paginated onto a plotter roll. (click for full image)

And finally, Mr. StickFigure comes to life:

Carl juggles with his creation

Palantir has entered the 2008 VAST Challenge. We present an in-depth look at one of our challenge solutions as the first public example of the Palantir platform in action.

Two years ago, the IEEE began an annual conference called VAST (Visual Analytics in Science and Technology). The VAST symposium focuses on the fundamental research contributions and real-world application of visual analytics. As a part of the conference, the VAST Challenge allows teams to compete on delivering analytic solutions against a synthetic real-world dataset.

Each year, the organizers build a very vast (pun intended) dataset from scratch. The data is entirely fictional but mirrors real-world use cases and scenarios. This year’s dataset is about a new religious movement that started on an imaginary Caribbean island (cleverly titled Isla del Sueño, or “Island of Dreams”) situated between Florida and Cuba. There are four subsets of the synthetic data: the Wikipedia page for the movement and its associated edit and discussion pages; landing and Coast Guard interdiction records for boats leaving Isla del Sueño for Florida/Mexico; cell phone records from the island; and RFID tracking data from people in a building that was attacked with an IED.

The types of questions asked in the problem sets are qualitative questions that require answers backed by data. These are the sorts of questions that don’t yield answers using a machine-learning/data-mining approach nor can an unassisted human get these answers by simple inspection of the data. They require some sort of human-computer symbiosis to solve.

To solve the VAST problems, we assembled an ad-hoc team of analysts — composed of a mix of engineers, in-house professional analysts, and one senior executive — and asked them to use the Palantir Government software to extract insights from the data.

The results speak for themselves: the complete set of Palantir’s VAST solutions are available here.

Read on for an in-depth look at how we deconstructed and solved one of the problems.

Analyzing Cell Phone Networks or “Where’s Waldo Ferdinando?”

Mini-Challenge 3 asks a couple of questions about the command and control structure of the religious movement by examining cell phone data from the island:

• What is the Catalano/Vidro social network, as reflected in the cell phone call data, at the end of the time period?
• Characterize the changes in the Catalano/Vidro social structure over the ten day period.

We started out with a CSV containing 9,385 rows that look like this, logging ten-days of calls:

A tabular format like this is about as far from “visual” or “analytic” as one can get; not even a social-network-analysis-savant could glean something interesting from the data in that form. In addition to the spreadsheet, we were given the approximate geographic locations of each cell tower and the following intelligence (from the contest’s synthetic dataset):

“We have medium confidence that Ferdinando Catalano is identifier 200. Close relatives and associates he would be calling would include David Vidro, Juan Vidro, Jorge Vidro, and Estaban Catalano. We believe Ferdinando would call brother Estaban most frequently. We also believe that David Vidro coordinates high-level Paraiso activities and communications. ”

Using nothing but that spreadsheet and the power of Palantir, we were able to uncover secretive attempts by member of the Paraiso movement’s highest echelon to conceal their cell phone traces by switching devices.

Most importantly, the analytic team did this without needing to be experts in the technical aspects of social network analysis, database queries, or Java. Instead, Palantir allows users — people who have a deep understanding of how the world works but are not technically savvy — to get answers backed by data. So answers to high-level questions such as, “Who were these people talking to during those last three days of network silence?” are actually easy to extract from the dataset. We realize that this sounds awfully hand-wavy, abstract, and vague, but the videos and explanations show exactly how it’s done.

Here’s the two-minute long Mini Challenge 3 Video and full explanation. The ten-minute long Palantir VAST 2008 Grand Challenge video gives a brief overview of Palantir and then presents all four solutions.

Palantir Finance, our product that has not yet been released, brings a similar level of power, flexibility, and high-level analytics to the realm of finance. Stay tuned for a sneak peek of those capabilities in the coming months.

If you found that video as much fun as we do, check out the other demonstrations of our ability to make data come to life. The “Boat Video” about migration data does an especially good job of showing off Palantir’s relational analysis abilities and its open-platform geospatial integration. Each of the write-ups includes additional screenshots that aren’t in the videos.

Welcome to the future of analysis!

 
 

Time Filtering

The graph is linked to the histogram which allows view filtering based on time ranges.

Financial Dashboard

The dashboard allows a financial analyst to follow a configurable set of widgets delivering real-time financial data.

Instrument Groups

The Instrument Groups tool lets analysts build up dynamic groups of financial instruments using expressive queries, filters, and calculations

Flows

More impressive in an animation than still image, the flows tool allow for the intuitive visualization of resources flowing on between entities. The configurable threshold enables the filtering of the ‘noise’ of smaller payments.

]]> http://blog.palantirtech.com/2008/07/04/palantir-screenshots-round-two/feed/ http://blog.palantirtech.com/2008/06/23/pkcs12/ http://blog.palantirtech.com/2008/06/23/pkcs12/#comments Mon, 23 Jun 2008 08:00:57 +0000 Ari http://blog.palantirtech.com/?p=94

Some of our customers are pretty serious about data security. To that end, our products need to support and integrate with SSL for both data security and authentication. SSL is very neat technology, but there is a dizzying maze of standards to navigate to figure how to get it all to work.

It turns out that in this age of Google, the fastest way to figure out how to do something is often to Google for key terms and hope that someone has put the relevant details in a blog post somewhere. In trying to figure out how to set up keys on a SunOne Directory Server for testing our LDAP integration, I needed to figure out how to get keys into PKCS#12 format after generating them with OpenSSL.

I’ll spare you the gory details of what it took to figure out how to do this; suffice to say there was some trial and error mixed with a bunch of RTFM. After the jump, the full howto.

Introduction to PKCS#12

PKCS#12 is used by a number of different vendors as their standard key-exchange format for key management, most notably IE and the SunOne/Netscape products. OpenSSL, while remaining the Swiss Army-Knife of crypto tools, doesn’t use PKCS#12 as a native format. However it knows how to convert to it.

The overall strategy here is to convert to PKCS#12 as a last step; we do normal key generation and signing using OpenSSL and then convert the results into PKCS#12 format.

To get a key-pair generated and signed by a certificate authority and then ready for import into one of these tools follow these steps (user input has been bolded):

Generate the key

# openssl genrsa -out  example-key.pem 2048
Generating RSA private key, 2048 bit long modulus
..................................+++
...........................................................................................+++
e is 65537 (0x10001)

Your RSA, 2048-bit public/private key pair now reside in the file named example-key.pem.

Generate the Certificate Signing Request (CSR)

Next we generate the CSR normally.

# openssl req -new -key example-key.pem -out example.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:US
State or Province Name (full name) [Berkshire]:California
Locality Name (eg, city) [Newbury]:Palo Alto
Organization Name (eg, company) [My Company Ltd]:Palantir Technologies
Organizational Unit Name (eg, section) []:Palantir TechBlog Examples
Common Name (eg, your name or your server's hostname) []:example.palantirtech.com
Email Address []:example@palantirtech.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

A certificate signing request for the public key in example-key.pem is now in example.csr and will look something like this if you look at the contents:

—–BEGIN CERTIFICATE REQUEST—–
MIIC/zCCAecCAQAwgbkxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
MRIwEAYDVQQHEwlQYWxvIEFsdG8xHjAcBgNVBAoTFVBhbGFudGlyIFRlY2hub2xv
Z2llczEfMB0GA1UECxMWUEcgQmFja2VuZCBFbmdpbmVlcmluZzEaMBgGA1UEAxMR
ZmxpbnQueW9qb2UubG9jYWwxJDAiBgkqhkiG9w0BCQEWFXJlZ3NAcGFsYW50aXJ0
ZWNoLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMWmxedFCqsx
ypNNYTUbcwVxtTUFMS6pP0y78GAabFvDQHW48IQXgyMrqlLl4//dI7EO/K6+ZdEJ
Go6fuXW5jd/1RuxQRkCsfdVUrrhghVOywpSeWCzwraFniPQOqstQHuWdfygFkpVZ
xsY3pu9cVvs827wvjHZXPpmLRZu5xDom+r8aAdjrQdkCCqbzdNFcxnxdHM03LMCz
BJcAUSMjhvynBRNgzB01bOJOk+72lAuaEhjF0TMvnRMo16mhd/dnaHkqqW+N8q9m
yTH1cTrNrdgJKkJ44K1sSJhFVMgkC2R4GK6RmprwIDVD8YkYLG+Iahw18xQKbCn1
8NomTXbZBAUCAwEAAaAAMA0GCSqGSIb3DQEBBQUAA4IBAQBGR/1jScSVekhy/aR2
xePJcO4jzmMVW4NfxOknxLkDvuDhbP7wGWPtspLXNf45mc3AXtkNgQTwO0siZHyy
HaSLo4BuB4A62ofZfTVn6KYVzxuwskkLRdhlHA6mpM01hMBhmzbIZDyAZqyWQAJ5
rC0Xl+ai/AAdLXO664qR3f2awXCUhJ/vYg7Qw4FHCLeoKcWxJfq9X9Ho5UJaFYTW
6QVFKeODF4Y2vMnBx63RcgOIuJXI38ZK4+k1ONKuB75IhbIu3DtMTIv3kYYtfd6G
P5BXlmgQgxVnFJcDprjEjHEUY4B+0vaWbDqy7U6fucT7EJ+qrSfk0Hf89aaQPKFf
DUwz
—–END CERTIFICATE REQUEST—–

This CSR is sent of to your CA of choice and you get back a certificate, essentially a signature for your public key by the CA.

The contents of the file, PEM-encoded, will look something like this:

—–BEGIN CERTIFICATE—–
MIIDdDCCAlwCAxAAIDANBgkqhkiG9w0BAQQFADB/MR4wHAYDVQQKExVQYWxhbnRp
ciBUZWNobm9sb2dpZXMxEjAQBgNVBAcTCVBhbG8gQWx0bzETMBEGA1UECBMKQ2Fs
aWZvcm5pYTELMAkGA1UEBhMCVVMxJzAlBgNVBAMTHlBhbGFudGlyIENlcnRpZmlj
YXRlIEF1dGhvcml0eTAeFw0wODAyMjYwNjQ5NTVaFw0xMzAyMjQwNjQ5NTVaMH8x
CzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMR4wHAYDVQQKExVQYWxh
bnRpciBUZWNobm9sb2dpZXMxHzAdBgNVBAsTFlBHIEJhY2tlbmQgRW5naW5lZXJp
bmcxGjAYBgNVBAMTEWZsaW50Lnlvam9lLmxvY2FsMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAxabF50UKqzHKk01hNRtzBXG1NQUxLqk/TLvwYBpsW8NA
dbjwhBeDIyuqUuXj/90jsQ78rr5l0Qkajp+5dbmN3/VG7FBGQKx91VSuuGCFU7LC
lJ5YLPCtoWeI9A6qy1Ae5Z1/KAWSlVnGxjem71xW+zzbvC+Mdlc+mYtFm7nEOib6
vxoB2OtB2QIKpvN00VzGfF0czTcswLMElwBRIyOG/KcFE2DMHTVs4k6T7vaUC5oS
GMXRMy+dEyjXqaF392doeSqpb43yr2bJMfVxOs2t2AkqQnjgrWxImEVUyCQLZHgY
rpGamvAgNUPxiRgsb4hqHDXzFApsKfXw2iZNdtkEBQIDAQABMA0GCSqGSIb3DQEB
BAUAA4IBAQB0RbDK5D/6ndxSBermRwmUNkUFbSh+e6dB8g4GEnRiRKReoPaBkJeG
UMZU2Rv9xZZBO+vN5USHbY8syfGNdHPNn8nndD5PJk4InBfBgcxlepzVK46sHuar
+VRERk6b7RnP3vvbaBIAEG4h1+bdm5CANCexo8yptJp6m8d3AKmqv+mjTmoLc3Y7
yTFVzCj8AyMJOHg1jE+paQSWDwe/Hya/01a8g6HaCLIVeLuHO0IWmIZ1oK+TiFt8
H7cSCmNwGa9oojK3P9PH7sJbVsV1uhm0vYn0Qdmb7toWu1eKit1NRyeEUFlc2T9B
h3N944gca4k+eMcAOrR7iBWUDwXMTVCr
—–END CERTIFICATE—–

Stick it in its own file, example.crt.

Create the PKCS#12 key

Concatenate the cert with the key file and then have OpenSSL convert it to PKCS#12

# cat example-key.pem example.crt > example.pem
# openssl pkcs12 -export -in example.pem -out example.pkcs12 -name “example”
Enter Export Password: (password is not echoed here, but you must enter something)
Verifying - Enter Export Password: (password is not echoed here, but you must enter something)

Verify it worked

As they say: if you didn’t test it, it doesn’t work. So now we verify that another tool that claims to ingest PCKS#12,

verify that it worked with a different tool. Here I use Java’s keytool:

# keytool -v -list -storetype pkcs12 -keystore example.pkcs12
Enter keystore password:  password (whatever you set it to above)

Keystore type: pkcs12
Keystore provider: SunJSSE

Your keystore contains 1 entry

Alias name: flint
Creation date: Feb 26, 2008
Entry type: keyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=example.palantirtech.com, OU=Palantir TechBlog Examples, O=Palantir Technologies, ST=California, C=US
Issuer: CN=Palantir Certificate Authority, C=US, ST=California, L=Palo Alto, O=Palantir Technologies
Serial number: 100020
Valid from: Mon Feb 25 22:49:55 PST 2008 until: Sat Feb 23 22:49:55 PST 2013
Certificate fingerprints:
         MD5:  C2:4E:B0:62:D8:06:FB:10:77:A5:37:6C:C8:2F:2A:AF
         SHA1: D2:B4:6B:0C:9D:3B:A4:94:B9:BF:25:E5:57:D6:96:FA:FB:84:A6:A7

*******************************************
*******************************************

You’re now good to go. As usual: additions, oversights, and comments welcome.

The notion of time is central to both of our products at Palantir, and there are many instances in which the user needs to specify a certain point in time. Although there are simple ways to create choosers (you could use a JSpinner that uses a SpinnerDateModel or simply use multiple JComboBox objects), I decided to experiment with writing some more visual time chooser components. These components are fairly experimental — they aren’t used in either product yet and I coded them up pretty quickly so I could get some feedback.

You can see these choosers in action in our office furniture in Bulgaria
webstart demo. The source code is available in the office furniture in Bulgaria
JAR.

If anyone has any feedback or suggestions as to how these choosers could be improved (or any ideas on how to make a better time chooser altogether) please leave a comment and let me know!

Meanwhile, if you want to know a little bit more about these choosers and how I went about designing them, read on…

The value of data analysis as a career was recently emphasized by Hal Varian in the Freakonomics blog in The New York Times. Hal is an internationally known economist who is currently serving as Google’s Chief Economist while on leave from his three professorships at the University of California at Berkeley.

Q: Your job sounds extremely interesting. What jobs would you recommend to a young person with an interest, and maybe a bachelors degree, in economics?

A: If you are looking for a career where your services will be in high demand, you should find something where you provide a scarce, complementary service to something that is getting ubiquitous and cheap. So what’s getting ubiquitous and cheap? Data. And what is complementary to data? Analysis. So my recommendation is to take lots of courses about how to manipulate and analyze data: databases, machine learning, econometrics, statistics, visualization, and so on. [emphasis added]

Following the discovery that our offices were the birthplace of Java (or least the place where it had its childhood), I invited James Gosling to come visit. For those that don’t know who James Gosling is, he’s more-or-less the father of Java. Java started as a project of James Gosling’s in 1991; today, 17 years later, he’s still at Sun, in charge of guiding the Java platform into the future.

How does one invite such a luminary to come visit one’s offices? One guesses what his email address is and sends him an email out of the blue:

James,

My name is Ari Gordon-Schlosberg, an engineer at Palantir Technologies. I recently became interested in the storied history of our current facilities at 100 Hamilton Ave. in Palo Alto. As Java programmers, our engineering team is really excited to be working in the same place that gave the world Java.

You may not have heard of Palantir, but we’re working on some pretty interesting problems, using Java to build large-scale analysis applications that really push forward the state-of-the-art. We’ve won some accolades for our use of Swing by Romain Guy. If you felt like dropping by the next time you’re in the valley, we’d love to have you come by, see your old digs, and take a peek at what we’re working on.

Sincerely,

Ari Gordon-Schlosberg

To quote the Microsoft Program Manager’s book of proverbs: 90% of making things happen is sending email.

So James dropped by one Thursday for demos, lunch, and schmoozing with our engineers.

The first order of business was to demo our software to James. We got a bunch of the senior engineers together and showed him an abbreviated demo of both Palantir Government and Palantir Finance. We focused less on the problem-space aspects of the software and more on how we’re using Java to build the application. We went over how both of our apps are completely written in Java and that our GUIs are built with custom Swing components.

The most memorable part of the conversation went something like this:

LEAD DEV: So… what do you think of our applications?

GOSLING: It makes me want to weep.

LEAD DEV: Uh… ?

GOSLING: Yeah, we’ve been working on this infrastructure for years to be able to build applications like this and finally someone is doing it.

The rest of the visit was spent talking about Java, its history and its future. Topics ranged from why it’s hard to get dinosaurs like cable companies and mobile carriers to use modern technology to some of the complication in building an optimizing JIT compiler.

After lunch, I walked him to the elevator to see him off. We said our goodbyes and he stepped into the elevator, which was already occupied by the mailman making his rounds. As the doors closed, I hear the mailman say to James:

“Well, I haven’t seen you around here in a while.”

Palantir started in a very small office on Sand Hill Rd. For a time (while between offices), all work was done in a founder’s bedroom. In late 2005, the Palantir moved to medium-sized, nondescript suite on Page Mill Rd., just across the street from Hewlett-Packard in south Palo Alto. I joined a few months later, as the fifteenth employee. Since that time, we’ve sextupled in size and we’re currently pushing a triple-digit headcount!

We almost ran out of space last year, but managed to find a fantastic office space in downtown Palo Alto, at the intersection of Hamilton and Alma. At the time we acquired the space, we were told that it had been the home of Digital Equipment Corporation’s Western Research Lab. I mentioned this to an acquaintance of mine who mentioned, offhandedly, that this same set of offices was where Java was developed.

A few weeks ago, I got curious about this potentially dangling reference and decided to see if I could transmute it from rumor into fact. It sure would be neat if the fundamental technology that underlies our software was first developed in the very offices in which we work.

Some quick research confirmed the rumors and they’re all true! After the jump, the history of 100 Hamilton Ave.

1982-1992: DEC’s Western Research Lab

Before there was a Google, Microsoft, Apple, Sun, or Internet, there was Digital Equipment Corporation (aka DEC). DEC built the PDP series of mini-computers that Unix was originally developed on. In later years, DEC built AltaVista, the first search engine that returned useful results quickly. They did this to show off the power of their Alpha microprocessor, which was one of the first “commodity” 64-bit processors.

In the style of most tech behemoths, DEC funded a number of laboratories where researchers could work on basic science that would hopefully be turned into profitable technologies. In 1982, DEC opened the Western Research Laboratory at 100 Hamilton Ave., in Palo Alto.

Their raison d’être is spelled out in the front matter from one of their technical reports:

The Western Research Laboratory (WRL) is a computer systems research group that was founded by Digital Equipment Corporation in 1982. Our focus is computer science research relevant to the design and application of high performance scientific computers. We test our ideas by designing, building, and using real systems. The systems we build are research prototypes; they are not intended to become products.

As you may know, DEC was purchased by Compaq in 1998, who was then bought in some bizarre reverse-merger fashion by Hewlett-Packard. The full list of their published research is available on the HP Labs website.

People might assume that researchers in this field might be a dry humorless lot. Not so! In 1988, WRL publishes Technical Note 13: Characterization of Organic Illumination Systems, an experiment into the incandescent properties of three varieties of pickles, mandarin oranges, and bok choy. The report includes instructions on making your own pickles as well as a discussion of the practical considerations of making glowing cucumbers commercially viable:

9. Practical considerations

The primary advantage of pickles as light bulbs is that they can be eaten, either before or after providing illumination. Thus they are to be preferred for long sea voyages. Pickles are also organically grown and so do not contribute to pollution. However, whereas incandescent lamps can be manufactured by a single machine at a rate of 20 or 30 per minute [4], proper pickling takes several weeks and requires careful control [5]. Thus the challenge to economically exploit the rediscovery of pickle light sources comes down to developing techniques for the massive growing of cucumbers and efficient vast vats for pickling. This may be an excellent industry for the Developing world.

For those of you wondering what an incandescent pickle looks like, YouTube is here to help you out:

The legacy of this experiment lived on inside DEC for years. From the Wikipedia page on the DEC Alpha:

Officially, the Alpha processors were designated the DECchip 21×64 series, the “21″ signifying the 21st century, and the “64″ indicating 64 bits, with the middle digit corresponding to the generation of the Alpha architecture. Internally, Alpha processors were also identified by EV numbers, EV officially standing for “Extended VAX” but having an alternative humorous meaning of “Electric Vlasic”, giving homage to the Electric Pickle experiment at Western Research Lab.

The Western Research Lab ultimately produced some slightly more useful innovations, like AltaVista (in 1995, after they had moved around the corner).

1993-1997: Javasoft

Sometime in 1992, DEC moved the Western Research Lab a few blocks away to 250 University Ave. The world did not yet know Java. At that time, the Java team was known as “The Green Team” and were working in relative seclusion from the rest of Sun, on the fourth floor of the Bank of America building on Sand Hill Road.

Around 1993, the Green Team demoed the *7 (”star seven”), a hand-held computer running an early version of Java (then known as “Oak”) on top of a stripped down version of SunOS.

Building on the momentum of that demo, the Green Team formed FirstPerson, a wholly owned subsidiary of Sun. FirstPerson was working on building and promoting the Oak technology (not yet officially named Java), focusing primarily on the cable set-top box market. The team expanded to about 100 people and moved into the offices that are currently occupied by Palantir. As one of the earliest Java developers recalls:

We moved the company to the best location in the entire south bay. The old DEC Western Research Lab at 100 Hamilton Avenue in Palo Alto. This is ironically where the OSF, Open Software Foundation was formed by the “Hamilton Group” before they came up with what Scott always called “Oppose Sun Forever”. Six blocks of the finest restaurants in the Bay Area, CalTrans across the street for the San Fran commuters, and the nicest looking office buildings around. Things were looking up.

Javasoft occupied 100 Hamilton for a very exciting four years before being re-assimilated into the Sun mothership. During that time Java was named, introduced, released to world, and worked through three major revisions of the platform (1.0, 1.1, and 1.2).

This interview with Sami Shalo from JavaWorld’s October 1996 issue (no longer available online) has the details:

JavaWorld: What is your recollection of the meeting in which the word “Java” was chosen as the name of Sun’s new programming environment? Where was it held and when?

Shaio: Oh, boy. Well let me look through my extensive archives. The meeting was held at the “TestPattern” conference room in “PAL2,” which is the building at 100 Hamilton Ave., where all the Java-Internet thing happened. It’s also the infamous building where OSF started, when it was known as the “Hamilton Group.” I believe the meeting was held in perhaps January of 1995.

2007 - ?: Palantir Technologies

Palantir started operations at 100 Hamilton in October of 2007. We love our new home, especially having lunch out on our third-floor kitchen balcony. In the grandest Silicon Valley tradition, we’re living the dream of working hard and playing hard, moving the state of the art forward, one line of code at a time.

We often use worker threads to do some long-running process, so often run into two issues using SwingUtilities invokeLater/invokeAndWait, and have developed wrapper code to deal with it. One issue is executing code from both worker threads and the Event Dispatch Thread (EDT). invokeLater and invokeAndWait both throw exceptions if executed on EDT. Second, invokeAndWait isn’t guaranteed — interruption on the calling thread will resume execution before the job is finished. The remainder of this post shows the code we used to solve these issues.

Reducing repetitive code

We often write functions that are executed from the Event Dispatch Thread (EDT) and also from other threads. Since these methods will throw an exception if you’re already on the EDT, it makes sense for us to wrap them in a quick safety check so we avoid an extra three lines of code every place we want to use it. Thus, a first minor improvement:

Note the use of invokeLater — that will be important to the next section. invokeAndWait wouldn’t work the same.

invokeAndWait doesn’t wait

The second gotcha with invokeAndWait is that it doesn’t always. Wait, that is. If a thread is waiting for a Runnable to execute on the EDT and that thread is interrupted, execution of the waiting thread will resume immediately, BEFORE FINISHING the Runnable. I had assumed this meant we failed to wait for the EDT, that the job was never executed and should I execute invokeAndWait again. This was the wrong interpretation. Turns out, all that happened was waiting on completion of the job was interrupted. The Runnable is still enqueued on the EDT and will still execute. Depending on how the timing works out, the EDT Runnable may even be executing at the same time. I just have no idea when it will finish. Very inconvenient.

invokeAndWait throws InterruptedException so you can learn of this situation and deal with it, but it’s not really something you can solve after the error has already occurred. So the following is our solution:

We create a FutureTask to wrap the provided Runnable. We can await completion of the FutureTask via FutureTask.get() which tells us when the provided Runnable has finished (successfully or thrown exceptions, or whatever). After the Runnable finishes, if there were any interruptions during execution, we re-interrupt Thread.currentThread() so the caller knows an interruption occurred and can choose to perform followup action.

Happy Swinging.

-Carl

The following is my attempt at describing what we do at a high level without oversimplifying. I hope that after reading this a candidate will ‘get’ what we’re about, or at least understand enough not to apply tiny labels to our expansive vision.

The problem: implementing analysis

At Palantir we specialize in analysis.

Yes, that’s painfully abstract, and I’ll get to it in a second.

In real-world terms, we are building a software platform that enables people to take whatever data is relevant to them and understand it more easily and thoroughly than ever before, using concepts that they already understand. And we are applying this vision, at first, to solving problems in the finance sector and the government intelligence community.

The first important thing to note is that we don’t actually do the analysis ourselves. We don’t devise winning trading strategies and we don’t catch terrorists. We write software that enables other people to pull off these feats. These people, experts in their respective fields, are called analysts.

So what exactly do analysts do? What is analysis?

Analysis is everything necessary to extract insight from information.

Let’s break that down a bit.

Information is easy: It’s data. It lives in a relational database or as files indexed on a hard drive, and you can easily run queries against it. It comes in two forms, structured and unstructured. And there is a lot of it in the modern world - too much, actually, for current tools to make sense of.

Insight is trickier. Insight is something only a person can generate, and understanding this is critical for any organization that wants to do analysis right. Thus the challenge of data analysis is how to bring vast amounts of information into productive contact with human intelligence. In other words, the challenge is how to enable the analyst.

From the analyst’s perspective there are five essential features of an analysis platform:

1. First, and most important, the analyst should be in control. In other words, the primary way of interacting with an analysis tool should be human-driven queries. While automated approaches can complement a human-driven approach, there simply is no substitute for human intelligence. Unless you put a person behind the wheel, the system can never be flexible or creative enough to uncover truly original insight. Artificial Intelligence just isn’t there yet.
2. Ability to summarize large data sets. Some of this is what has traditionally been called data mining: the largely automated approach—using machine learning or other statistical techniques—of processing lots of data at once and extracting nuggets that capture something interesting about the data. Unlike Palantir, traditional approaches have focused almost exclusively on this aspect of analysis.
3. Ability to visualize large data sets. Here the analyst wants interesting and informative ways of viewing data graphically, to make it easier for him to digest. The analyst wants more than just a summary of the data; he wants a nuanced view of what’s going on inside these data sets: What’s the overall shape of the distribution? What are the outliers? What are important structures within the data?
4. Ability to iterate rapidly. This means enabling the analyst to ask a question, get the answer, and then quickly ask either a variant on the initial question or a follow-up question that depends on the answer to the initial question. This rapid, iterative process allows the analyst to quickly test out hypotheses and develop theories about what’s going on in the data, and by extension to discover what’s going on in the world.
5. Ability to collaborate with other analysts. Getting a handle on a terabyte of data, especially when it comprises multiple data types, is definitely more than a one-person job. Any organization that’s serious about understanding the world needs a team of analysts that can work together as more than the sum of its parts. This requires the ability for one analyst to effortlessly share the results of his analysis with his colleagues.

The Palantir approach

That’s what analysis looks like to the analyst, or rather what it should look like in an ideal world. (Current tools fall far short of this vision.) So what do we do at Palantir in order to make analysis this smooth and easy?

You could say that we help summarize large data sets, in the sense that we have to provide the analyst with a rich library of techniques and algorithms. You could also say that we do visualization, in the sense that we have to provide the analyst with a set of interesting and informative ways of visualizing their data. We do both of these things, and we have to be creative and solve hard problems in order to add value in these areas. But we do a lot more than that.

Probably the most central hard problem that we address in trying to enable the analyst is data modeling, the process of figuring out what data types are relevant to a domain, defining what they represent in the world, and deciding how to represent them in the system. At Palantir we make sure our data model (ontology) is both flexible and dynamic, and that it mirrors the concepts people naturally use when reasoning about the domain. This is no small challenge, but we’re already making it a reality. In finance our basic data types include financial instruments, dates, portfolios, indices, and strategies—the same things that financial researchers think about, talk about, and reason with. In the intelligence product our basic data types include people, places, and events (all with associated properties), which is exactly the way we all represent the world in our minds.

Data modeling, data summarization, and data visualization are the core disciplines for approaching large data sets. Human-driven queries, rapid iteration, and collaboration are multipliers, taking the power unlocked by the core disciplines to the next level. When these pieces are brought together in a coherent system, the result is in an analysis platform both very generic and very powerful.

This is what we mean when we say that we’re changing the way people approach data. Welcome to the future of analysis.