At Google we used protocol buffers for nearly everything. They were the basis of SSTables, which were the basis of bigtable, which was the basis of other storage infrastructure. RPCs would also use protocol buffers to define service interfaces and data structures that were passed between machines. Since leaving Google, I've found that there are a couple practical alternatives to Protocol Buffers such as JSON (or BSON) and Thrift. Schema-less serialization protocols are great for most applications, but when a datastructure is used by many services and is somewhat complex, I've found that agreeing on a schema is helpful.

The most common choices for schema-ful serialization schemes are Thrift and Google's open-sourced version of Protocol Buffers, of which Protocol Buffers' documentation is orders-of-magnitude better. Thrift provides some nice features such as client and server implementations (not just serialization), support for richer datastructures, and Javascript bindings. Despite these advantages, it can be a pain to figure out how to use Thrift because of spotty documentation.

To that end, I've just created an extremely simple wrapper library and example that demonstrates how to use Thrift in clojure.

Given a thrift definition like this: The only thing which requires any mental thought is defining your handler functions, like so: That's (nearly) all there is to it.

The example is https://github.com/ithayer/plaid-penguin-example, and you can see the README for how to run the example. The base library (which is about 20 lines) is at https://github.com/ithayer/plaid-penguin, and can also be gotten from clojars here: The main goal of this library is just to document what you need to start a server and client so that you don't have to go digging.

Another neat trick is that you can use the bean function in clojure to get a clojure map from the Thrift data structures, like so: Just wanted to show how easy this was. If you're interested in working with python, clojure and thrift to help people with a painful problem, we're hiring at ReadyForZero.


React to this post on hacker news.