This is a response to Ari’s awesome post on human-computer symbiosis. Ari and I were chatting about the equation he developed and I was wondering if there were some further refinements that are possible… let’s take a look:

We are attempting to understand the total analytic capability for a given task a of a human-computer team. Analytic capability in this case probably means:

(1)

Where A is the answer to the analytic problem in question and t_A is the time needed to arrive at the answer based on the inputs available. In the case of chess, A could be the optimum next move given all previous information and t_A would be how long it takes to decide on this move.

Read on for a look at how this generalizes in human-computer symbiotic systems.

In the case of the human-computer team, we know that a is going to be a function of both the human’s analytical capability h and the computer’s analytical capability c (where both h and c have units of answers/time). In the limit case we know that:

(2)

(3)

Or in plain English, if there is no human present, the total analytic capability is simply the analytic capability of the computer. So the naïve solution would be that:

(4)

(4) clearly meets the limiting cases described in (2) and (3). Kasparov noticed a mixing function where the ability of the human and computer to work together becomes the dominant term — we might call this the mixing capability for the given task or m. Including this phenomenon, the total analytic capability (4) would be re-defined as:

(5)

where m has the property that:

(6)

Thus maintaining the limits expressed in (2) and (3) and adhering to the observation that if there is no human or computer component then there will be no mixing advantage. A naïve solution to this constraint would be simple linear mixing:

(7)

where M (units of time per answer) is the mixing efficiency and will be primarily based on the type of task being solved — some analytical tasks lend themselves to a combined process more than others (for example, multiplying 20 digit numbers does not really benefit from the intuition of a human so the ability of a human and computer to perform this task is merely their additive ability).

What Kasparov noticed is that the mixing was primarily based on the quality of the process rather than the analytical power of either the human or computer separately. This seems to imply that we must somehow account for the fact that the quality of the human-computer interface is responsible for the quality of the mixing. This can be modeled as a unitless friction of interaction f_i that impedes the ability of the human and computer to work together.

Equation (7) can thus be re-written as:

(8)

In this case, the maximum value for the mixing capability is realized when the friction of interaction goes to zero. This mixing capability is the same as the equation Ari developed (less the coefficient which is necessary to maintain consistent units throughout).

We can now re-write our analytic capability in (5) as:

(9)

Below, see a plot of this function over a range of values for h, c and f_i:

As can clearly be seen from this functional plot (note the vertical scale), the effect of interface friction dominates over the other terms whenever both the human and computer can make important contributions to the task at hand. The conclusion can be drawn that the most effective way to solve analytical problems is to minimize the friction of the human-computer interface; or to put it another way: optimal analytical systems are those that are built specifically to maximize the ability of the human to leverage the ability of the computer.

I am certain there is still the possibility for further refinement, for example:

(10)

Here at Palantir, a lot of our automatic tests are full-chain tests. A backend server is fired up, client code runs against it, and everything runs much like a production environment. This makes intuitive sense because it’s a faithful approximation of how the system will run in the field.

However, there are some disadvantages to this:

• Full-pass tests don’t always localize the problem. Tests on a client class might fail even if it was the service that behaved incorrectly.
• These full-pass tests are relatively slow. Client code is running against an actual remote service. If a client is being tested, the server code still has to do work — sometimes a lot of work — even if that isn’t the focus of the test.
• The constraints of the test are loose. Full-chain tests can mostly only see whether the operation finished correctly. It’s much harder to figure out whether the operation was done efficiently and without making unnecessary service calls.
• They’re very little setup flexibility. If you want an RPC to return a specific value, you have little choice but to have your test get the service into a state where it can return that value. This is easy in some cases, but prohibitively difficult in others.
• Client tests are forced to share any non-determinism leaked from the service. For example, under real conditions, a request to call A might respond before call B, and sometimes the other way around. This can result in flaky tests or tests that don’t always simulate the conditions you want to exercise.

What’s to be done? Fortunately, there’s an option that handles these cases elegantly. We also test with jMock, a library that dynamically generates mock objects from arbitrary interfaces. These mock objects can be configured to check that particular methods are called with particular inputs a particular number of times, and then give prescribed responses.

Hit the link to see a concrete example of jMock in action.

jMock in action

Let’s say I want to test my object viewer page in Palantir Web, but I don’t want to fire up a dispatch server at all. First, I create my mock service object.

Mockery context = new Mockery();
final PalantirService service = context.mock(PalantirService.class);

Then, I set the expectations of my mock object. In this case, I want to tell my mock object to expect a call to PalantirService.getObject() and PalantirService.getDataSources(). getObject() will return a specific object. Any call made to the service apart from these will make the test fail.

context.checking(new Expectations() {{
        oneOf(service).getObject(realm.getId(), myObject.getId());
        will(returnValue(myObject));
        oneOf(service).getDataSources(myObject.getDataSources());
}});

Now, I create the object I want to test and inject the service.

ObjectViewController controller = new ObjectViewController();
controller.setService(service);

And then we fire away.

ModelMap model = new ModelMap();
controller.doGet(myObject.getId(), model);

Now that the controller (the class we’re exercising) has gone off and populated the model, we check to see that the model is populated correctly. Just like we would in any other test.

assertEquals(myObject.getName(), model.get("objectName"));
assertEquals(myObject, model.get("object"));

But in addition, we also assert that the expectations specified above were satisfied.

context.assertIsSatisfied();

Not only can we be sure that the right calls were made with the right parameters, but we can also be sure that no calls besides the expected calls were made. So the next time you want more speed or control over your tests, take a look at jMock or another framework like it. It’s a powerful tool in the effort to test your best!

Model–View–Controller (MVC) is an architectural pattern used in software engineering. Successful use of the pattern isolates business logic from user interface considerations, resulting in an application where it is easier to modify either the visual appearance of the application or the underlying business rules without affecting the other. In MVC, the model represents the information (the data) of the application; the view corresponds to elements of the user interface such as text, checkbox items, and so forth; and the controller manages the communication of data and the business rules used to manipulate the data to and from the model.

They go on to show the classic triangle diagram and how it’s baked into various GUI and web frameworks. There’s only one clause in the entire article that hints at something deeper: “Though MVC comes in different flavors…”

Different flavors indeed. In fact MVC is not just a pattern but a whole family of patterns: MVC, MVA, MVP, PAC, Model-Delegate…. It very quickly gets very hairy.

In this article I want to describe one of MVC’s lesser-known variants, the Model-View-Adapter (MVA) pattern, and talk about its advantages over traditional MVC in the context of a Java Swing application.

Architecture

The best place to start is with an architecture diagram. While vanilla MVC is a triangle:

MVA puts the Adapter in a position to strictly mediate between Model and View:

Here a solid line represents a direct relationship while a dashed line represents an indirect relationship via the Observer pattern. Put another way, the Adapter holds a pointer both to the Model and to the View and directly calls methods on both. At the same time, it attaches itself as a listener both to the Model and to the View in order to receive events. It receives property change events from the Model and action events (checkbox ticked, text entered, etc.) from the View, and then routes appropriate changes to the other side. The Adapter is entirely responsible for keeping the Model and the View in sync; the Model and View are both relatively dumb structures, knowing nothing about the other.

The advantages to organizing code this way are:

• All “moving parts” are centralized in one place, the Adapter. No worrying about where to add a listener; no hunting around to find isolated listeners.
• Separation of concerns between the View and the Adapter. The View is responsible for layout and visual presentation while the Adapter is responsible for synchronization and the dynamic aspects of the user interface.
• Better decoupling between Models and Views. Specifically, the View doesn’t need to know anything about the Model.

Additionally, while it will never be possible to fully componentize any variant of the MVC pattern, MVA is more amenable to componentization and thus more of its implementation can be centralized (in a single class) and reused. Once componentized, we can augment the basic functionality with things like:

• Automatic registration and unregistration of listeners when the View enters and exits the Swing component hierarchy, thereby preventing certain kinds of memory leaks.
• Automatic unregistration of listeners when the program shuts down. This can help free up resources like realtime subscriptions.
• Method for swapping a new Model object in for an old Model object.
• Ability to execute a task without listeners attached, to help prevent event-action-event loops.

The downside to using MVA over MVC is that the Adapter tends to take on a lot of the responsibility and can get quite complicated. But in my experience that can be mitigated by having good conventions about which pieces (M, V, A) are allowed to communicate with which other pieces and at what times. Enforcing predictable control flow goes a long way toward managing complexity.

Read on for a code-level description of our implementation of the MVA pattern.

Palantir MVA Implementation

Our half-componentization of MVA resides in a single abstract class named Adapter:

public abstract class Adapter<ViewType extends Component, ModelType> {
// constructor
protected Adapter(ViewType view, ModelType model); { ... }

/**
* Attach listeners to the View's subcomponents (checkboxes etc.).
* Listeners should be stored as member variables in the Adapter
* subclass.
*/
protected abstract void registerViewListeners();

/**
* Detach the same listeners (member variables) that were
* attached in registerViewListeners().
*/
protected abstract void unregisterViewListeners();

/**
* Attach listener(s) to the Model.
*/
protected abstract void registerModelListeners();

/**
* Detach the same listeners (member variables) that were
* attached in registerModelListeners().
*/
protected abstract void unregisterModelListeners();

/**
* Bring the View fully in synch with the Model.  Typically
* this involves querying state from the Model and
* reconfiguring subcomponents of the View accordingly.
*/
protected abstract void fullSynchronize();

protected ModelType getModel() { ... }
protected ViewType getView() { ... }

// other methods elided
}

New View components that want to stay synchronized with a Model must instantiate a subclass of Adapter and implement the abstract methods. The Adapter parent class (itself an example of the Template Method design pattern) will then call into the appropriate abstract methods at the appropriate times. For example, after the View is constructed, as soon as it’s displayed in the Swing component hierarchy the Adapter parent class will automatically call fullSynchronize() (whose implementation should bring the View in line with the Model) and then registerViewListeners() and registerModelListeners(), so the Adapter is poised to react to events. Likewise, when the View is removed from the component hierarchy (when its containing frame is closed, say), both unregisterViewListeners() and unregisterModelListeners() will be called. This can help ensure that no memory will be leaked when a long-life-cycle object (like a system-wide singleton) retains a pointer to a short-life-cycle object (the View) via the Observer pattern.

Dealing With Listener Loops

One problem that confronts UI developers is the problem of “listener loops”: infinite loops that result when the View fires an event, the Adapter (or Controller) responds to it by setting some property on the Model, and an event is propagated from the Model back to the View, starting the whole cycle over again.

One way to combat this is to make sure your Model only fires events when the value that’s being set on the Model is different from the value currently stored in the Model. (This will cut off the infinite loop after one and a half cycles.) It’s a good practice but often isn’t enough, especially when your system is multithreaded and events start to queue up. You can sometimes get into situations where an M-V-C triplet will thrash forever between two different values for one of the Model’s properties.

Our solution to this problem is a protected method (on our Adapter base class) called runWithoutViewListeners:

/**
* Guarantees that the job r will be run:
*    - on the Swing thread
*    - with Model listeners attached
*    - with View listeners DEtached
*/
public final void runWithoutViewListeners(final Runnable r) { ... }

The implementation of this method checks to make sure the view listeners are attached when it’s called, detaches them via a call to unregisterViewListeners(), invokes the Runnable, then reattaches the view listeners via a call to registerViewListeners(). The code inside the Runnable can then make whatever changes it wants to the View without perturbing the Model downstream. Listener loop averted!

More To Come

I hope that’s given you some sense of the territory out there in the wide world of MVC-variants. In a week or two, Derek will show off some of the work he’s done on the M piece of the MVA triad related to “event bubbling.” Stay tuned!

One of the things our customers love to do is print our beautiful object graphs and tape them to the wall for discussion. What they hate to do is print 30 pages, line them up, and tape them to a poster one at a time. So we bought a plotter, and I started plotting.

I needed to print directly to a Java Graphics object. Unfortunately, the available information on large output printing from Java is thin at best. While there are lots of ways to successfully place ink on paper, I was only able to find one that reliably lets the application pick odd paper sizes that plotters use, like 24×19.7 inches. (The term “plotter” used to mean something with pens for printing blueprints and such. Now it just means a large format printer, commonly printers that can use roll paper as a source.)

One of the first things you’ll learn when you start working with printing in Java is that a language intended to be all things to all people (i.e., cross-platform) is utterly lousy at tasks highly specific to a given environment, such as printing. It will not surprise you to hear that native print services on Windows are pretty different from those available on a Mac, which themselves are pretty different from the CUPS system common to Unix systems.

So, by and large, you are reduced to the least common denominator of printing. Part and parcel of this least common denominator is agreeing on what constitutes a piece of paper and sticking to it. This is fine for people thinking, “My paper is 8.5 inches wide by 11 inches tall.” It poses a bit of a problem for people with plotters who are thinking, “My paper is 24 inches wide by as many damned inches tall as I need.” Even relatively powerful programs like PhotoShop or GIMP don’t seem to support plotters well. I believe Photoshop works by specifying the exact paper size you want to use, but any technique in which the easiest solution for the user is to pull out a calculator does not meet with my approval.

Advice in a nutshell

A tutorial by MartyHall provides some valuable insights into printing which I applied to my printing component. While it didn’t address my specific issues, props to a well-done starting point.

I’m starting with the assumption that you’ve got yourself a class that implements the Printable interface so that once the Java printing subsystem is coerced into accepting the appropriate paper size, Printable.print(Graphics, PageFormat, int) will be called correctly and your job will print properly. Since my usage was screen-visible as well as printable, I simplified and asserted that one screen pixel is equivalent to one point (1/72 inch), which is how Java prepares the scaling on the Graphics object passed to Printable.print(…), thus making printouts look about the size of onscreen displays. Then, knowing that the printout needed, for example, 1200 vertical pixels/points, I can automatically calculate that I want to use a paper size 18″ tall or so (1200 / 72 dpi = 16.67″ + margins).

Here’s a summary of the critical steps:

1. Do the math yourself to get the paper size required in points (1/72 of an inch)
2. Implement the Pageable interface to report page count (usually one) and the PageFormat using the computed paper size.
3. Create a PrinterJob configured with the correct PrintService
4. Set your Pageable on the PrinterJob instance
5. Optionally display print setup dialog (PrinterJob.printDialog()) recognizing that changes to orientation or paper size will be ignored.
6. call PrinterJob.print()

Generally, don’t use the standard dialogs

This might seem contrary to desired behavior, but I found that giving the user access to the usual native or cross-platform print dialogs can be a bit sticky because those are places where the user can alter the selected paper; in order to properly print to roll paper sources, we must override any paper settings the dialogs might provide. Ultimately this can be confusing for the user, but that’s an unfortunate side effect of supporting plotters.

Don’t take any advice too seriously

We still choose to display the dialog because complex printers like plotters have many options that the users may need to alter and would otherwise have no way to modify.

StickFigure: looking at plotting in Palantir

This work didn’t happen in a vacuum: we have a need to print large graphs for display. Being able to print them in a large format is one of the simplest and most beloved methods of collaboration.

To give an example of why plotter printing is interesting, we created a stick figure on the graph in Palantir. After creating this monstrosity, we went go print it and examined the layout. The first layout was with letter-size paper, and just by looking at the head, we could see that it was going to be a lot of pages:

Zoomed-in view of the head of the stick figure. (click for full image)

Zooming out to view the entire graph, it’s clear that it’s going to be 35 pages!

Full pagination of the stick figure. (click for full image)

Even switching to the largest standard paper size only gets us to two sheets:

Paginated out to largest standard paper size, still two pages (click for full image)

When we switch over to using the plotter layout, we finally get to the one sheet we were looking for:

Paginated onto a plotter roll. (click for full image)

And finally, Mr. StickFigure comes to life:

Carl juggles with his creation

Following the discovery that our offices were the birthplace of Java (or least the place where it had its childhood), I invited James Gosling to come visit. For those that don’t know who James Gosling is, he’s more-or-less the father of Java. Java started as a project of James Gosling’s in 1991; today, 17 years later, he’s still at Sun, in charge of guiding the Java platform into the future.

How does one invite such a luminary to come visit one’s offices? One guesses what his email address is and sends him an email out of the blue:

James,

My name is Ari Gordon-Schlosberg, an engineer at Palantir Technologies. I recently became interested in the storied history of our current facilities at 100 Hamilton Ave. in Palo Alto. As Java programmers, our engineering team is really excited to be working in the same place that gave the world Java.

You may not have heard of Palantir, but we’re working on some pretty interesting problems, using Java to build large-scale analysis applications that really push forward the state-of-the-art. We’ve won some accolades for our use of Swing by Romain Guy. If you felt like dropping by the next time you’re in the valley, we’d love to have you come by, see your old digs, and take a peek at what we’re working on.

Sincerely,

Ari Gordon-Schlosberg

To quote the Microsoft Program Manager’s book of proverbs: 90% of making things happen is sending email.

So James dropped by one Thursday for demos, lunch, and schmoozing with our engineers.

The first order of business was to demo our software to James. We got a bunch of the senior engineers together and showed him an abbreviated demo of both Palantir Government and Palantir Finance. We focused less on the problem-space aspects of the software and more on how we’re using Java to build the application. We went over how both of our apps are completely written in Java and that our GUIs are built with custom Swing components.

The most memorable part of the conversation went something like this:

LEAD DEV: So… what do you think of our applications?

GOSLING: It makes me want to weep.

LEAD DEV: Uh… ?

GOSLING: Yeah, we’ve been working on this infrastructure for years to be able to build applications like this and finally someone is doing it.

The rest of the visit was spent talking about Java, its history and its future. Topics ranged from why it’s hard to get dinosaurs like cable companies and mobile carriers to use modern technology to some of the complication in building an optimizing JIT compiler.

After lunch, I walked him to the elevator to see him off. We said our goodbyes and he stepped into the elevator, which was already occupied by the mailman making his rounds. As the doors closed, I hear the mailman say to James:

“Well, I haven’t seen you around here in a while.”

class GenericDataSource extends DataSource implements Comparable<dataSource> {
    public int compareTo(DataSource o) {
        if(o != null){
            return getName().compareTo(o.getName());
        }
        return 1;
    }
    @Override
    public boolean equals(Object o) {
        if(o == null) return false;
        if(o instanceof DataSource)
            return this.getLocator().equals(((DataSource)d).getLocator());
        return false;
    }
}

According to the documentation for Comparable:

It is strongly recommended, but not strictly required that (x.compareTo(y)==0) == (x.equals(y)). Generally speaking, any class that implements the Comparable interface and violates this condition should clearly indicate this fact. The recommended language is “Note: this class has a natural ordering that is inconsistent with equals.”

It is clear that this uses two totally different schemes for equals and compareTo. One uses the locator field and the other uses the name field. This means that this class has an inconsistent compareTo.

How would failing to have my natural ordering be consistent effect me? You would expect the following code to be true for collections that don’t contain null, but this is not the case for classes where the natural ordering is not preserved.

new HashSet<Type>(collection).size() == new TreeSet<Type>(collection).size()

Another issue with this compareTo function has to do with its handling of null. Since the compareTo function is reflexive even for null, the javadoc states that null must be handled by an NPE.

In the foregoing description, the notation sgn(expression) designates the mathematical signum function, which is defined to return one of -1, 0, or 1 according to whether the value of expression is negative, zero or positive. The implementor must ensure sgn(x.compareTo(y)) == -sgn(y.compareTo(x)) for all x and y. (This implies that x.compareTo(y) must throw an exception if and only if y.compareTo(x) throws an exception.)

In short: in general, make sure that equals and compareTo are implemented in a coherent and consistent manner and it will save you some subtle and annoying headaches.

Date formats are not synchronized. It is recommended to create separate format instances for each thread. If multiple threads access a format concurrently, it must be synchronized externally.

Suggestions on handling synchronization:

• Use the Joda Time libraries if possible. They are thread safe. Joda Time is fairly easy to port to from existing JDK compatible code.
• Synchronize calls to format() manually. You probably want to write a wrapper class/function to do this, and make sure nobody calls the original format() method by accident. This seems like it should be safe, but see http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=422833.
• Use thread-local storage, e.g.

private final static ThreadLocal<SimpleDateFormat> shortTimeFormat =
     new ThreadLocal<SimpleDateFormat>() {
            protected SimpleDateFormat initialValue() {
                 return new SimpleDateFormat("HH:mm");
             }
      };

N.B. In general, you should be very careful when using ThreadLocal storage, especially when using thread pools (your data won’t be automatically garbage collected and may be visible to other Threads—a security risk) or when storing references to Thread objects in ThreadLocal storage (which might confuse the GC). However, in this case, we should be okay.

There comes a time in every developer’s life when they need to write code that processes some XML. Lately, we’ve seen the proliferation of APIs that make XML processing easier, like JAXB (Java API for XML Binding). However, when speed and scale are required, chances are you’re going to need to roll your own processor. Before I continue, let me clear up some terminology, when I say “processor”, I mean the code of yours that’s wrapped around a SAX (tutorial), DOM (tutorial), or an XPP (tutorial) parser, not the guts of the parser itself.

At the end of the day, that’s the interesting part of what you’re doing – the grammar of your data model rather than the minutiae of start and end tags. Building a processor is the interface between the data interchange format and the internal data model of your application.

Click through for a tour of XML parsers and a look at a novel technique for encoding processors that use pull parsers (as usual, we’ve included a WebStart demo, as well as a jar file containing the compiled example along with all of its source code).

XML Parsers

Each of the three most common types of XML parsers has its own strengths and weaknesses. DOM, in particular, does not scale well. The DOM technique of reading the entire document into memory and creating first-class language objects for every explicit and implied element makes it slow and resource intensive. Allegorically, this is balanced against the ease of programming against the DOM API, although I personally find the DOM API to be a bit cumbersome. The upshot is that DOM is not well suited for large documents (because of its memory requirements) or applications that need low-latency and overall throughput (since you a) don’t see the DOM until the whole thing is done being built and b) objects are expensive to construct).

SAX Parsing scales very well, but it uses a callback pattern which can be awkward to code against for large grammars. This problem has been addressed in various ways. Gianluigi Colaiacomo of IBM wrote this article detailing a method that ends up looking a lot like pull parsing for SAX: Simplify document handler programs with the SAX parser.

In the case of XML Pull Parsing (excellent tech brief on XPP here), it’s optimized for parsing tasks that require all elements in a document to be processed and leads to an event stream style of programming that is much more intuitive and easy-to-follow than SAX. In the past few years, the work on pull parsing has come together into a standard called StAX, or Streaming API for XML.

StAX has more or less replaced XmlPull as the standard for XML streaming, but the XPP parsers still exist and are very, very fast. XPP became our parser of choice when looking at things like real-time serialization of data and high-speed import of object graphs from XML for our integration interfaces.

Life On Earth

First, let’s set the stage: rather than going into the detail and complexity of the Palantir XML formats, I’ve designed a somewhat simpler example to make this all easier to digest. Note that the LifeOnEarth schema document, source code, and example LifeOnEarth instance document are included in the JAR file.

In this example, what will we be encoding? Organisms! That’s right, biological taxonomy. The biological taxonomy is the set of official Latin names for all the living things in the world. It’s basically a straight hierarchy, with one notable exception: Phylum vs. Division. Plant biologists use the term division to describe the same broad morphological grouping that their zoological counterparts call a phyla.

This is great for our example: after all, who wants to have a straight hierarchy? We need something a little out of the ordinary to make things interesting.

The XML schema: lifeOnEarth

The idea here is that we’ll build an XML Schema document describing an instance document format for encoding information about the biological taxonomy. We make extensive use of complexType elements in the schema to afford us a sort of code reuse. The Class->Family->Order->Genus->Species hierarchy logically repeated underneath Phylum and Division in the taxonomy model. However, so as to be as DRY as possible, we use complex type declarations instead of defining complex types in-line when declaring the elements . It’s a lot like using declared types instead of anonymous inner classes in Java.

Using the complexType elements, we define the names, attributes, and allowed members for every step of the hierarchy. We then define the document itself by declaring a single top level element, lifeOnEarth:

<xsd:element name="lifeOnEarth" type="tns:lifeOnEarth"></xsd:element>

You can see that the element lifeOnEarth is of complexType lifeOnEarth; the same name in different namespaces. The type lifeOnEarth is defined thusly:

<xsd:complextype name="lifeOnEarth">
            <xsd:sequence>
                        <xsd:element name="domain" type="tns:domain" maxoccurs="unbounded"></xsd:element>
            </xsd:sequence>
</xsd:complextype>

An element of type lifeOnEarth contains one or more elements (sequence, in XSD parlance) of type domain. An element of type domain contains a sequence of elements of type kingdom. So on and so forth, all the way down to species (with the double hierarchy for the phylum/division duality).

Here’s the schema document:

<?xml version="1.0" encoding="UTF-8"?>

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
            targetNamespace="http://www.palantirtech.com/schema/examples/lifeOnEarth"
            xmlns:tns="http://www.palantirtech.com/schema/examples/lifeOnEarth"
            elementFormDefault="qualified">
    <xsd:annotation>
        <xsd:documentation>This schema is based on alpha taxonomy, he science of describing, categorizing and naming organisms.  For a good overview, see http://en.wikipedia.org/wiki/Taxon</xsd:documentation>
    </xsd:annotation>
    <xsd:complexType name="lifeOnEarth">
        <xsd:sequence>
            <xsd:element name="domain" type="tns:domain" maxOccurs="unbounded"></xsd:element>
        </xsd:sequence>
    </xsd:complexType>
    <xsd:complexType name="species">
        <xsd:attribute name="commonName" type="xsd:string"/>
        <xsd:attribute name="name" type="xsd:string"/>
    </xsd:complexType>
    <xsd:complexType name="genus">
        <xsd:sequence minOccurs="0">
            <xsd:element name="species" type="tns:species" maxOccurs="unbounded"></xsd:element>
        </xsd:sequence>
        <xsd:attribute name="name" type="xsd:string"/>
    </xsd:complexType>
    <xsd:complexType name="family">
        <xsd:sequence minOccurs="0">
            <xsd:element name="genus" type="tns:genus" maxOccurs="unbounded"></xsd:element>
        </xsd:sequence>
        <xsd:attribute name="name" type="xsd:string"/>
    </xsd:complexType>
    <xsd:complexType name="order">
        <xsd:sequence minOccurs="0">
            <xsd:element name="family" type="tns:family" maxOccurs="unbounded"></xsd:element>
        </xsd:sequence>
        <xsd:attribute name="name" type="xsd:string"/>
    </xsd:complexType>
    <xsd:complexType name="class">
        <xsd:sequence minOccurs="0">
            <xsd:element name="order" maxOccurs="unbounded" type="tns:order"></xsd:element>
        </xsd:sequence>
        <xsd:attribute name="name" type="xsd:string"/>
    </xsd:complexType>
    <xsd:complexType name="phylum">
        <xsd:sequence minOccurs="0">
            <xsd:element name="class" type="tns:class" maxOccurs="unbounded"></xsd:element>
        </xsd:sequence>
        <xsd:attribute name="name" type="xsd:string"/>
    </xsd:complexType>
    <xsd:complexType name="division">
        <xsd:sequence minOccurs="0">
            <xsd:element name="class" type="tns:class" maxOccurs="unbounded"></xsd:element>
        </xsd:sequence>
        <xsd:attribute name="name" type="xsd:string"/>
    </xsd:complexType>
    <xsd:complexType name="kingdom">
        <xsd:choice maxOccurs="unbounded" minOccurs="0">
            <xsd:element name="phylum" type="tns:phylum"></xsd:element>
            <xsd:element name="division" type="tns:division"></xsd:element>
        </xsd:choice>
        <xsd:attribute name="name" type="xsd:string"/>
    </xsd:complexType>
    <xsd:complexType name="domain">
        <xsd:sequence minOccurs="0">
            <xsd:element name="kingdom" type="tns:kingdom" maxOccurs="unbounded"></xsd:element>
        </xsd:sequence>
        <xsd:attribute name="name" type="xsd:string"/>
    </xsd:complexType>
    <xsd:element name="lifeOnEarth" type="tns:lifeOnEarth"></xsd:element>
</xsd:schema>

The upside of making the schema document comes from being able to leverage existing validation mechanisms. Using a validator, any document can be verified to be a well-form instance of the schema. This allows the processor to be able to assume valid documents, greatly reducing the error handling burden on the developer when writing the processor. Additionally, if you’re writing both sides of the XML transaction, you can validate the output of the renderer to perform early error detection. And finally, the validator is the perfect thing for using in unit testing related to your XML processing. Sun included a simple validation example in the Javadocs for the javax.xml.validation package. For a more in-depth look at using validation, see LifeOnEarthSchema.java, part of this example that shows how create a shared schema object (using the singleton pattern) from an XSD file loaded from the classpath.

Encoding the XML schema’s FSA (peanut butter)

So it turns out that processing XML, just like any parsing task, can be represented by Finite State Automata. Finite state automatas (FSAs) can be encoded by a list of valid states and the valid transitions between states. Since XML is nicely hierarchical, encoding the information is fairly straightforward. A tag name maps to a state in the FSA and its valid transitions are to any of its parents or any of its children.

With this in mind, I sat down to design our XML formats for integration, realizing that I needed a way to represent all the states the parser could be in as it parsed the document. That context would drive how the parser events were handled and the routing of parsed data to the proper places.

So I started thinking: “I’ll bet I can do this really cleanly with enums.” Here’s what I came up with:

We encode all the states of the parser into a Java Enum. The Enum constructor takes two arguments: one or more ParserStates that are valid parent states and the text that corresponds to the text representation of the XML tag in the document. Since most states only have one parent, we overload the constructor for convenience and readability to only take a single state instead of an array.

After the constructors, we define an instance method that will check that a passed state is a valid transition from this state and some static members and initialization to allow us to map strings passed by the XML parser into the Enum objects.

Here’s the Enum’s definition:

	private enum ParserState {

		GROUND(new ParserState[]{},""),
		LIFE_ON_EARTH(GROUND,"lifeOnEarth"),
		DOMAIN(LIFE_ON_EARTH,"domain"),
		KINGDOM(DOMAIN,"kingdom"),
		PHYLUM(KINGDOM,"phylum"),
		DIVISION(KINGDOM,"division"),
		CLASS(new ParserState[]{DIVISION,PHYLUM},"class"),
		ORDER(CLASS,"order"),
		FAMILY(ORDER,"family"),
		GENUS(FAMILY,"genus"),
		SPECIES(GENUS,"species");

		/**
		 * Array of parent states to this one.
		 */
		ParserState parents[] = new ParserState[]{};
		/**
		 * Tag name for this state.
		 */
		String tagName = null;

		/**
		 * Constructor for ParserState with a single parent state.
		 * @param parent
		 * @param tagName
		 */
		ParserState(ParserState parent,String tagName){
			this.parents = new ParserState[]{parent};
			this.tagName = tagName;
		}

		/**
		 * Constructor for ParserState with a multiple parent states.
		 * @param parents
		 * @param tagName
		 */
		ParserState(ParserState[] parents,String tagName){
			this.parents = parents;
			this.tagName = tagName;
		}

		/**
		 * Checks whether it is valid to transition to the
		 * specified state from this state.
		 * @param newState
		 * @return
		 */
		public boolean checkValid(ParserState toState){
			if(this.equals(toState))
				return true;

			for(int i = 0; i < toState.parents.length ; i++){
				if(toState.parents[i].equals(this)){
					return true;
				}
			}
			for(int i = 0; i < this.parents.length; i++){
				if(this.parents[i].equals(toState)){
					return true;
				}
			}
			return false;
		}

		public String getTagName() {
			return tagName;
		}

		// End enum instance methods and variables

		// Static methods, variable, and intializations
		static Map<string,ParserState> tagLookup = new HashMap<string,ParserState>();

		/*
		 * This code executes after the enums have been constructed.
		 *
		 * Because of order of execution when initializing an enum,
		 * you can't call static functions in an enum constructor.
		 * (They are constructed before static initialization).
		 *
		 * Instead, we use a static initializer to populate the lookup
		 * hashmap after all the enums are constructed.
		 */
		static {
			for(ParserState state : ParserState.values()){
				registerState(state);
			}
		}

		/**
		 * Maps a tag name to a ParserState
		 * @param tagName
		 * @return the ParserState for that tag.
		 */
		public static ParserState lookupStateForTag(String tagName){
			return tagLookup.get(tagName);
		}

		private static void registerState(ParserState state){
			tagLookup.put(state.tagName, state);
		}

	}

XML Pull Parsing (chocolate)

The idea with XML Pull Parsing is that the document can be represented as a stream of events describing the document content. You pull the events off the stream as you process them (hence the name). These events come in five flavors: START_DOCUMENT, END_DOCUMENT, START_TAG, END_TAG, TEXT.

The tags names are pretty easy to understand intuitively, except for TEXT, which is the event for free text inside of an element.

An example document like this:

<foo> bar <baz>jones</baz> bleargh</foo>

… would produce events in the following order: START_DOCUMENT, START_TAG, TEXT, START_TAG, TEXT, END_TAG, TEXT, END_TAG, END_DOCUMENT.

Pretty much every XML pull parser-based processor has a loop that looks like this at their core:

	protected void processDocument() throws XmlPullParserException, IOException, PalantirException {

		// pull first event
		int eventType = parser.getEventType();

		do { // core loop
			switch(eventType){

			case XmlPullParser.START_DOCUMENT:
				log.debug("Start document");
				break;

			case XmlPullParser.START_TAG:
				processStartElement();
				break;

			case XmlPullParser.END_TAG:
				processEndElement();
				break;

			case XmlPullParser.TEXT:
				processText();
				break;

			// never called, here for completeness
			case XmlPullParser.END_DOCUMENT:
				log.debug("End document");
				break;

			}

			eventType = parser.next();

		} while (eventType != XmlPullParser.END_DOCUMENT);
	}

So the next logical thing to take a look at is the processStartElement() method. It turns out that here’s where the real peanut-butter & chocolate synergy comes into play. In this case, we call getName() on the parser object and it will tell us the name of the tag just started.

We want to:

• Check that it’s a valid state transition.
• Change the state of the parser.
• Dispatch to the appropriate method to deal with this kind of tag.

Here’s the method (note that stateStack is of type Stack<ParserState>):

	public void processStartElement() throws PalantirException, XmlPullParserException {

		String tagName = parser.getName();

		// check for state transition
		ParserState newState = ParserState.lookupStateForTag(tagName);
		if(newState != null){
			// we know it's a valid tag
			if(parserState.checkValid(newState)){
				// change FSA state
				stateStack.push(parserState);
				parserState = newState;
			}
			else{
				// invalid transition
				String message = "Got illegal state transition in parser, suspect malformed XML.\n" +
				"At line " + parser.getLineNumber() + ", col " + parser.getColumnNumber() + ".\n" +
				"Invalid state transition: " + parserState.toString() + " -> " + newState.toString();
				log.error(message);
				throw new PalantirException(message);
			}
		}
		else{
			// unknown tag: ignore
		}

		// dispatch to handling method
		dispatch();
	}

You can see how we leverage the encoding in the ParserState Enum to make this a very simple method. Since we’re encoding tag-to-state mappings, we can easily lookup the appropriate state for this tag (go back and look at the static initializer if I lost you there). Once we know the proposed new state, we can check whether or not the state transition is a valid one, throwing an exception on an invalid state (which implies a malformed document). Finally, we call dispatch(), which is nothing but a big switch statement using all the possible states from the Enum:

	private void dispatch() throws XmlPullParserException, PalantirException {

		// logging statements removed for readability

		switch (parserState) {
		case DOMAIN:
			processDomain();
			break;
		case KINGDOM:
			processKingdom();
			break;
		case PHYLUM:
			processPhylum();
			break;
		case DIVISION:
			processDivision();
			break;
		case CLASS:
			processClass();
			break;
		case FAMILY:
			processFamily();
			break;
		case ORDER:
			processOrder();
			break;
		case GENUS:
			processGenus();
			break;
		case SPECIES:
			processSpecies();
			break;
		default:
			break;
		}
	}

The methods for doing the processing for TEXT and END_TAG methods are much simpler but also leverage the dispatch() method:

	public void processEndElement() throws XmlPullParserException, PalantirException {
		dispatch();
		// already know to be valid, since transition
		// validity is bi-directional
		parserState = stateStack.pop();
	}

	public void processText() throws XmlPullParserException, PalantirException {
		dispatch();
	}

What you end up with is a very straightforward, readable, and performant dispatching framework in your processor. The code practically writes itself! All that’s left to do is to define each of the tag processing methods, which will have to understand how to process the attributes (START_TAG), any free text (TEXT), and finally close up any data structures which the tag data is being recorded to. They will all loosely follow this form:

	void processTag() throws XmlPullParserException {

		switch (parser.getEventType()) {
		case XmlPullParser.START_TAG:
			// handle attributes and tag existence here
			break;
		case XmlPullParser.END_TAG:
			// handle close-up here
			break;
		case XmlPullParser.TEXT:
			// handle free text here
			break;
		}
	}

This ends up being really nice since all the logic about each tag is encapsulated in a single method, facilitating debugging, modifications, and having other people read and actually understand your code (not the easiest thing to do with SAX Parser callbacks!).

Putting it all together

So the demo that shows this off parses the instance document that’s filled with entries like this:

 <tns:domain name='Bacteria'>
    <tns:kingdom name='Monera'>
      <tns:phylum name='Proteobacteria'>
        <tns:class name='Proteobacteria'>
          <tns:order name='Enterobacteriales'>
            <tns:family name='Enterobacteriaceae'>
              <tns:genus name='Escherichia'>
                <tns:species name='E. coli' commonName="E. coli"/>
            </tns:genus>
          </tns:family>
        </tns:order>
      </tns:class>
    </tns:phylum>
  </tns:kingdom>
</tns:domain>

It then spits out strings describing each species to our Swing-based console, like this:

Lifeform 'E. coli':
Bacteria : Monera : Proteobacteria : Proteobacteria : Enterobacteriales : Enterobacteriaceae : Escherichia : E. coli

Check it out and let me know about any questions or improvements or errors in the comments.

What if what you really wanted was the color from one image and the alpha channel from another? You’d be out of luck, but for the talents of Brien. Here’s what you normally get with a standard AlphaComposite.SRC_OVER sort of technique. In the following two examples, the icon is opaque and the rectangle is partially opaque black fading to transparency.

What we needed looks more like this:

Read on to find out how we did it, and why.

While it seems like this might be overkill, this was only a simple example. In practice, we wanted a complex circular fade applied to another relatively complex graphic that couldn’t be easily replaced with primitive graphics operations. With SourceAlphaComposite.java, we were able to produce two graphics to get exactly the effect we wanted. Be on the lookout for another cool usage of this class in the next posting.

Assuming your image ColorModel supports transparency, the transparency of any given pixel is encoded in the bit patterns of the raster. We picked an image type that makes things easy: in a BufferedImage of type BufferedImage.TYPE_4BYTE_ABGR the opacity of the pixel may be found in the fourth byte of the pixel information. We extend the interface CompositeContext and implement its compose() method. The method has this fingerprint:

public void compose(Raster src, Raster dstIn, WritableRaster dstOut)

The algorithm is simple:

1. blit the dstIn argument to the output, dstOut.
2. Loop over each pixel in the src argument, overwriting its opacity information into the corresponding pixel in dstOut.

A word of warning, however: since this class involves direct manipulation of the alpha channel of a BufferedImage, and not all image types support alpha, let alone in the same way, we picked the image types most convenient to us. Our code example only works with BufferedImages of type BufferedImage.TYPE_4BYTE_ABGR or BufferedImage.TYPE_INT_ARGB; implementation of support for other image types should be pretty straightforward.

In any case, check out the source code and let me know if you have any questions, comments, or enhancements to share.

package com.palantir.ui;

import java.awt.Composite;
import java.awt.CompositeContext;
import java.awt.RenderingHints;
import java.awt.image.BufferedImage;
import java.awt.image.ColorModel;
import java.awt.image.Raster;
import java.awt.image.WritableRaster;

/**
 * A {@link Composite} implementation that uses the destination RGB and the source alpha.
 * This can be used to perform alpha gradients.
 */
public class SourceAlphaComposite implements Composite, CompositeContext {
	public static SourceAlphaComposite createComposite(final BufferedImage bimage) throws UnsupportedBufferException {
		return createComposite(bimage.getType());
	}

	/**
	 * This factory currently supports <code>TYPE_4BYTE_ABGR</code> and <code>TYPE_INT_ARGB</code>.
	 */
	public static SourceAlphaComposite createComposite(int type) throws UnsupportedBufferException {
		switch ( type ) {
			case BufferedImage.TYPE_4BYTE_ABGR:
			case BufferedImage.TYPE_INT_ARGB:
				return new SourceAlphaComposite( 3 );

			default:
				throw new UnsupportedBufferException();
		}
	}

	private final int alphaIndex;

	public SourceAlphaComposite(final int alphaIndex) {
		if ( alphaIndex < 0 ) {
			throw new IllegalArgumentException( "There is no way a negative index will work." );
		}

		this.alphaIndex = alphaIndex;
	}

	public int getAlphaIndex() {
		return alphaIndex;
	}

	public CompositeContext createContext(
		final ColorModel srcColorModel, final ColorModel dstColorModel, final RenderingHints hints
	) {
		return this;
	}

	public void dispose() {
    	// Do nothing
    }

    public void compose(final Raster src, final Raster dstIn, final WritableRaster dstOut) {
    	final int
    		w = dstOut.getWidth(),
    		h = dstOut.getHeight();

    	final int n = src.getNumBands();
    	final int[] spixel = new int[ n ], opixel = new int[ n ], dpixel = new int[ n ];

    	for ( int x = 0; w > x; x++ )
    		for ( int y = 0; h > y; y++ ) {
    			src.getPixel( x, y, spixel );
    			dstIn.getPixel( x, y, dpixel );

    			// Use the destination color (except use the source alpha, below):
    			System.arraycopy( dpixel, 0, opixel, 0, opixel.length );

    			final int dalpha = dpixel[ alphaIndex ];
    			if ( 0 != dalpha ) {
	    			// Use the source alpha:
	    			opixel[ alphaIndex ] = dalpha * spixel[ alphaIndex ] / 0xFF;
    			}

    			dstOut.setPixel( x, y, opixel );
    		}
    }

    /**
     * Exception we throw for images that we don't support.
     */
    public static class UnsupportedBufferException extends Exception {
		public UnsupportedBufferException() {
		}

		public UnsupportedBufferException(final Exception cause) {
			super( cause );
		}
	}
}

One fairly well established technique is to write a DocumentFilter, and when insertString() or replace() is called, validate the added text and truncate as necessary to ensure the database field length is not exceeded.

Now the fun part. What happens when you try to store your comments on N’Ko, Mongolian, Bopomofo (phonetic markers, now commonly used as an input character set for Mandarin), or even ancient Viking runes? You get two choices, store as ASCII or ISO-8859-1 (aka Latin-1), or whatever, and you lose data. Oops. Or convert to UTF-16 or UTF-8. Hm. Wait a minute, now the value (in bytes) is somewhere between 1-3 times as many bytes as the original String length. So, how do you limit the text field to the number of bytes the database will permit? If you picked UTF-16, it’s pretty simple, divide the database limit by two. But it’s pretty wasteful of space, usually. On the other hand, you can’t predict exactly how many bytes the UTF-8 representation needs until you try it out.

The following algorithm will produce a String which, if converted to supplied Charset, will be no more than maxBytes in length. It could be less, depending on the charset chosen and the text being trimmed. This happens because it removes whole characters at once, which may trim several bytes, jumping you from 1 byte over the limit to two under.

public static String limitStringByBytes(String string, int maxBytes, String encoding) {
if(string == null)
return string;
int i = string.length() – 1;
int shaveBytes = computeByteLength(string,encoding) – maxBytes;
while ( shaveBytes > 0 && i >= 0 ) {
shaveBytes -= computeByteLength( string.charAt( i ), encoding );
i–;
}
if( (i+1) <= 0 )
return “”;
else if( (i+1) >= string.length() )
return string;
else
return string.substring(0, i + 1 );
}

As a final note (thanks to the comments by one of our faithful and numerous readers), we would like to acknowledge that we have indeed ignored the existence of the supplementary planes of Unicode mappings, sticking to the Basic Multilingual Plane in this example. This avoids the even more intricate hassle of dealing with surrogate pairs. If one of these rather obscure character encodings (Byzantine Music Symbols, Phoenician, or my personal favorite, Deseret [editors note: yeah, I didn't know what it was either. Wikipedia to the rescue], for example) should appear, it’s possible that they might be truncated mid-character. According to the Unicode standard, this is an error, but also a very unlikely situation to encounter. Free Palantir t-shirt to the first person who posts a working example that properly deals with surrogates.