J. Brisbin

Just another Wordpress.com weblog

RabbitMQ as a NoSQL distributed cache

leave a comment »

Part of what I’ve been doing with the cloud-friendly Tomcat session manager is basically implementing my own asynchronous distributed object cache. At the moment, this functionality is tightly coupled to what I’m doing inside the session Store. But in making some changes recently to add Spring Security integration and make working with Spring Security 3.0 a little easier, I noticed that there’s a lot of what I’m doing inside the session Store that could simply be abstracted into its own package and used as a standalone distributed cache.

The concept is simple and I think the code will be straightforward. Instead of synchronously loading an object from a data store (which is configured on the back end to shard its data or do other kinds of distributed load-balancing and failover replication), code would request the object be loaded asynchronously and provide a callback to be executed when the load is complete. This would actually simplify and make my own code quite a bit more robust and it would add another voice to the important area of our industry that is getting a lot of focus at the moment, that of distributed caching.

Terracotta looks like a killer app in all ways. I’d love to be able to use something I didn’t write myself to solve a lot of our problems. But I spent all our money on VMware support and new servers. There’s nothing left to go chasing proprietary and heavy solutions to our problems. I’ll use OpenSource or software I’ve written myself–or I’ll do something else entirely. A distributed data cache backed by RabbitMQ will be relatively lightweight (probably not at first, as I often have to strip things out to get to my lightweight goal) and I’m sure quite fast. It will transparently allow for sharding and aggregating data with no additional configuration. Since queue subscribers get load-balanced anyway, there’s no need to figure out some way to split up objects because they’ll be spread over however many listeners I put on those queues. I can partition data by using different RabbitMQ servers and combinations of queues and exchanges.

I’m starting work on this right away since I’ll be on vacation next week and, geek that I am, will likely not be able to pull myself away for long. Expect to see something on GitHub week after next!

Advertisements

Written by J. Brisbin

July 7, 2010 at 7:14 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: