J. Brisbin

Just another Wordpress.com weblog

Posts Tagged ‘cloud

Tomcat/tcServer session manager with attribute replication

leave a comment »

I’d like to think that my programming projects don’t suffer from philosophical schizophrenia but that I simply move from point A to point B so fast it just looks that way. Unfortunately, sometimes I have to come face-to-face with this and accept it for what it is: my programming projects suffer from philosophical schizophrenia.

I say this because I’ve been hacking on some changes to the virtual/hybrid cloud Tomcat and tcServer session manager and I went through several stages while I was trying to solve some basic problems.

For one, how do I get a serialized object in the cloud when I don’t know where it is? RabbitMQ comes to my rescue here because I think I’ve finally settled on the most efficient way to answer this question: bind a queue to a topic exchange using the object’s key or identifier (in this case, a session ID). Then any messages for that object automagically get routed to the right VM (the one that has that object in memory). I’m thinking this idea can be extended to create an asynchronous, RabbitMQ-backed, Map implementation.

Now that I have the object, how do I keep it in sync with the “master”? In my case, I send a replication message out whenever a session’s (set|remove)Attribute methods are called and the value objects differ. One notable problem that I don’t see being easily overcome (but thankfully, doesn’t apply to my scenario) is if there are listeners on sessions. I don’t have RabbitMQ wired into the internal Catalina session event mechanism. I could add that at some point, but for the moment, I think this kind of dumb saving/loading via RabbitMQ messages will work for what we’re doing.

I’ve now switched back to a ONEFORALL type operation mode which means there is only one real session object that resides in an internal map on the server who first created it. Whenever another server sees a request for this session, it will send a load message to this queue every time it needs it. That last part is important: it loads a session object every time it needs it. When code sets an attribute on server TC3, that attribute is replicated back to the original server (TC1) so subsequent session loads get that updated object. I’m still trying to wrap my head around how I want to handle replication in case of node failures. No good answer on that one, yet.

REPLICATED mode is my next task. In simplifying this code, I focussed on getting the ONEFORALL mode working right to begin with. Now I can go back and make it more performant by cranking out a little extra code to handle replication events.

Initial smoke tests seem to indicate this works pretty well. Session load times in my testing were around 20-60 milliseconds. Depending on the location of your RabbitMQ server and your network topology, you might experience different results. I’m sanding off a few rough edges now and I’ll be testing this on a new set of cloud servers we’re setting up as part of a new vSphere/vCenter-managed cloud.

As always, the code is available on GitHub:

http://github.com/jbrisbin/vcloud/tree/master/session-manager/

Advertisements

Written by J. Brisbin

May 4, 2010 at 6:35 pm

Distributed Atomicity in the cloud with RabbitMQ

leave a comment »

The private cloud Tomcat/tcServer session manager I’m working on has a huge job cut out for it. Maintaining the state of an object that exists in possibly more than one location at any given point in time is not an easy task, I know. To be honest, if it weren’t for my Midwestern stubbornness, I might not take the time to work through these hefty issues. I might follow the path of least resistance, like most of the industry has done so far.

I just don’t like the idea of sticky sessions. I look at my pool of tcServer instances as one big homongenous group of available resources. In my mind, there should be no distinction made between machines running in different VMs–or even on different hardware. They should exist and cooperate together as a single unit.

But in “replicated” mode, each server has a copy of the object. This is great for failover and it makes the session manager extremely performant. But yet another sticky wicket rears its ugly head. How do I protect this object and make sure it gets updated properly before someone else has a chance to operate on it?

Call it distributed atomicity if you want–the idea being that an object exists within the context of a cloud of compute resources (in this case, a Tomcat/tcServer user session object) and needs to be updated with all the right attributes when code in a different physical process operates on that object. I’m attacking this problem by implementing a form of distributed atomicity that uses RabbitMQ to send the contents of newly-added attributes to any interested parties throughout the cloud. I already replicate the session object by grabbing it with a Valve, just before the request is completed. This session object gets serialized to the cloud before the response is sent, the idea being that this particular object will be updated in all the places it is needed before another server has a chance to operate on that object.

By using the messaging infrastructure of RabbitMQ, I can at least make updates to this object reasonably atomic. Now the question becomes: where does this object live? For performance reasons, it’s probably not realistic to have just one object to share among web application servers. In the case of Tomcat/tcServer, the internal code is requesting the session object so often (multiple times during a single request) that each server simply has to cache a session object for the length of the user’s request.

A tool like ZooKeeper might be helpful in this case. If code has to set an attribute on a session object, the session would set a barrier in ZooKeeper that lets other code know it is in the process of being altered. Once setAttribute() is finished, a message is then sent with the serialized attribute. The other interested parties could alter its local copy of the object with the updated attribute until it receives a full replication of the object. Would the second, full replication be superfluous? At this point I can’t say. In the interest of completeness, I feel compelled to issue a second replication event, but in the interest of performance and bandwidth conservation, I wonder if its really necessary.

I’m far from finished with the cloud-based session manager. I’m trying to get it to a stable point so that I can migrate my cloud away from sticky sessions. The “replicated” mode seems to work fine; and I’m okay with sending too many messages–I’d rather have that than have too few and end up with page loads blocking because the session can’t be tracked down.

Distributed, asynchronous programming isn’t easy. It isn’t for the faint of heart or those with pesky bosses breathing down their necks to meet arbitrary and usually unhelpful deadlines. It also doesn’t help if you’re not a bona-fide genius. I often feel a little out of my league given the number of CompSci grads that are doing fantastic work in this interesting and growing segment of the industry. But I’m stubborn enough to keep plugging away when I should probably give up.

Written by J. Brisbin

April 22, 2010 at 6:17 pm

Tomcat/tcServer cloud session manager now has “replicated” mode

leave a comment »

I’ve updated my virtual/hybrid cloud Tomcat/tcServer session manager to use two different modes of operation. The default mode is what I’ve described previously. The new mode of operation is called “replicated” and it, as the name implies, replicates the user’s session object to every node consuming events on that exchange. This might be the whole cloud, it might not, depending on how you have your exchanges configured.

I’m working on code to only replicate the session if it sees changes in the MD5 signature of the serialized session. Otherwise, it’ll conserve your bandwidth and not replicate the session until it has to. Until then, though, the entire session gets replicated after every request. Excessive? Maybe. 🙂

I’m also trying a different approach to loading user sessions. Rather than contacting the direct queue of the node that advertises itself as the owner of that session, I’m sending a load message to the fanout exchange. This way, dedicated replicator/failover consumers can also respond to load requests in case a node goes down unexpectedly.

At the moment, there’s still no persisting of sessions to disk since a server replicates all its sessions off that node when it’s going down. I’m not sure I really need to dump a node’s sessions to disk when it goes down. I think I want to have dedicated consumers for that purpose.

With dedicated failover consumers, when new servers come up, they get the list of current sessions from the failover node. I don’t see that restoring things from disk would add significant functionality to this store. If you feel differently, be sure and let me know. It wouldn’t be difficult to implement a disk-based persistence mechanism for restarts.

You can checkout the source code from github:

git clone git://github.com/jbrisbin/vcloud.git

The only other change is that you now need to add a special replication valve that calls the replicateSession() method after each request invocation.

<Valve className="com.jbrisbin.vcloud.session.CloudSessionReplicationValve"/>

Written by J. Brisbin

April 20, 2010 at 4:08 pm

Posted in The Virtual Cloud

Tagged with , ,

Securing services within a hybrid, private cloud

leave a comment »

It seems to me there is an inverse relationship between security and functionality. To gain functionality in my hybrid/virtual cloud environment, I have to sacrifice security. Or do I?

To be honest, I’m not entirely certain yet. I keep running into problems I need to solve within my cloud that I can’t easily address. Case in point: keeping configuration files in sync.

In order to keep a configuration file in sync, I have to provide some mechanism to overwrite a local file with the contents of a remote file (either one that’s downloaded for that purpose, or comes as the body of a RabbitMQ message). Even though our user pool is limited to only employees of the company, that still exposes (at least theoretically) a vulnerability in that a user with access to our LAN could manipulate the file over-writing process to inject their own script onto a cloud-based machine. To combat this, I’m only exposing limited files for updating based on a key value (rather than the full, real path). Even if a malicious user was able to force a config file to be updated with the contents of their script, it would only be of a limited number of files, and those files would not have their direct paths exposed because the updater takes a key value which maps to the real local path inside the update consumer and this map is never exposed to users.

Assuming this malicious user was able to get their script on a server, they’d still have to invoke it somehow. Ensuring that my config file updater, which is meant to only update configuration files, writes their updates to the filesystem with the executable bit turned off, the script would not be executable.

After a configuration file has been updated, it’s very likely a service will need to be restarted. Exposing some functionality to restart services also gives me pause because I’ll be running commands in the shell at the behest of a message which I can’t really be certain isn’t coming from a malicious user. By using the key-based method previously described, though, I can severly limit the number of commands that might potentially be run. Exposing only the key value means the full path to the script is not derived from the data coming from the message but from data already living on the machine which is executing the command. If a malicious user was able to manipulate this side of the system, they’d still only be able to invoke restarts on specific services.

To fully exploit the system, a malicious user would have to inject their own configuration into a service (maybe by putting in Proxy directives to an Apache configuration file or something) and invoke a restart of the service. To do this, they’d need to:

  1. Intercept a message containing the key of the configuration file to inject their own configuration into.
  2. Intercept a message containing the key of the service that needs restarting after injecting their own configuration.
  3. Send the appropriately-crafted message through the message broker which is secured with a username and password.

We use an internal firewall here to segregate our AS/400 from the rest of the LAN for PCI compliance reasons. I suspect we could do the same thing for our cloud so that a malicious user would first have to gain access to a machine behind the internal firewall before it could contact the RabbitMQ server to even exploit these services.

I honestly don’t know what security best-practices are going to look like for hybrid/virtual private clouds. Thankfully, I’m not the only one asking these questions. As the cloud environment commands a greater share in the market, I’m sure people who are much smarter than I am can offer more practical solutions than a simple firewall and username/password security plan.

Written by J. Brisbin

April 19, 2010 at 3:16 pm

Posted in The Virtual Cloud

Tagged with , ,

Publish Tomcat/tcServer lifecycle events into the cloud with RabbitMQ

with one comment

Once of the common tasks in any cloud environment is to manage membership lists. In the case of a cloud of Tomcat or SpringSource tcServer instances, I wrote a simple JMX MBean class that exposes my tcServer instances to RabbitMQ and serves two functions:

  1. Expose the calling of internal JMX methods to management tools that send messages using RabbitMQ.
  2. Expose the Catalina lifecyle events to the entire cloud.

To maintain a membership list of tcServer instances, I now just have to listen to the events exchange and respond to the lifecycle events I’m interested in:

def members = []
mq.exchange(name: "vcloud.events", type: "topic") {
  queue(name: null, routingKey: "#") {
    consume onmessage: {msg ->
      def key = msg.envelope.routingKey
      def msgBody = msg.bodyAsString
      def source = msg.envelope.routingKey[msgBody.length() + 1..key.length() - 1]
      println "Received ${msgBody} event from ${source}"
      if ( msgBody == "start" ) {
        members << source
      } else if ( msgBody == "stop" ) {
        members.remove(source)
      }

      return true
    }
  }
}

Starting and stopping the tcServer instance yields this in the console:

Received init event from instance.id
members=[]
Received before_start event from instance.id
members=[]
Received start event from instance.id
members=[instance.id]
Received after_start event from instance.id
members=[instance.id]
Received before_stop event from instance.id
members=[instance.id]
Received stop event from instance.id
members=[]
Received after_stop event from instance.id
members=[]

It seems to me one of the defining characteristics of cloud computing versus traditional clusters is the transparency between runtimes and what used to be separate servers. To that end, I’ve exposed the inner workings of my tcServers both to other servers of their kind in the cloud, and to sundry management and monitoring tools I may choose to write in the future.

If you’re concerned with security, opening up the JMX MBeans of your server may give you pause. Fair enough. In my case, that’s not as big of a concern because these servers are protected from the outside world. Only LAN and WAN users can access these servers, so I don’t mind exposing JMX methods to trivially-secured message brokers, particularly if it gives me this kind of inexpensive and direct control over the services I’m exposing to the cloud.

Written by J. Brisbin

April 15, 2010 at 9:32 pm

Federation in the cloud with RabbitMQ and batch messages

with 2 comments

Our remote locations are connected to the central data warehouse through a fantastically complicated and expensive DSL and partial T1 WAN. The partial T1 was only a 56k connection! Yep, you read that right. We were pumping megabytes of data every day through a 56k connection. Most of those have been converted to DSL now, so that severe bandwidth limitation has been eased, but it’s still a limitation and we still have to be bandwidth-conscious when developing new applications that require transmission of data across the WAN.

To solve this problem with our PostgreSQL installations, the C++ guru we had at the time (who abandoned us for a cushy–and well-paying–position with a financial services firm on Wall street) wrote a handy little utility to accept SQL statements as input, run them against the database, then zip up the results and send them back to the requestor. The SQL results had to be zipped up because of the severe bandwidth limitation.

In moving to RabbitMQ, I want to make sure that we still have a way to batch and zip up multiple messages, which we distribute to the consumers here in the data center, then have the batcher consume all the results, batch those results up, and send the whole zip file back to the requestor. It’s a little complicated (I hope not unnecessarily) but a good example of doing “federation” in a cloud (cluster/hybrid cloud/virtual cloud/…who can decide which term fits, anyway?) environment.

The Batch Message Handler

To help me with writing handlers for different message types, I have a generic QueueingMessageListener whose sole responsibility is to process incoming messages and pass them off to the appropriate Handler subclass. One of these handlers works with batch messages.

These batch messages don’t have to be created in Java, either. Here’s some Python code which creates a “batch” message to send to my Java-language consumers:


dta = {
  'start': 1,
  'limit': 20,
  'datasource': 'as400-development',
  'sql': 'SELECT count(*) AS total FROM MYLIB.MYFILE WHERE CONO=?',
  'params': [ 1 ]
}
jsql = json.dumps(dta)

out = StringIO()
zfile = zipfile.ZipFile(out, 'w')

now = datetime.now()
zinfo_date = (now.year, now.month, now.day, now.hour, now.minute, now.second)
zinfo = zipfile.ZipInfo('test-1', zinfo_date)
zfile.writestr(zinfo, jsql)

now = datetime.now()
zinfo_date = (now.year, now.month, now.day, now.hour, now.minute, now.second)
zinfo = zipfile.ZipInfo('test-2', zinfo_date)
zfile.writestr(zinfo, jsql)

zfile.close()

If you’re not familiar with Python, don’t panic! All this code does is create a zip file, serialize a Python dictionary into JSON, and add the same text to two different entries within the zip file. This will ultimately acheive the same end as sending the message two differents times, directly to the SQL executor service.

I call them “batch” messages, but they’re really just zip files containing a text file of JSON-encoded data (the body of the real message) and having a filename in the zip file that corresponds to the “request id” (or correlationId, if you prefer the RabbitMQ-centric terminology). The batcher unzips the message and extracts the contents, creating one new message per zip entry:

ByteArrayInputStream bytesIn = new ByteArrayInputStream(bytes);
ZipInputStream in = new ZipInputStream(bytesIn);
try {
  ZipEntry entry;
  while (null != (entry = in.getNextEntry())) {
    String subRequestId = entry.getName();
    byte[] buff = new byte[4096];
    bytesOut = new ByteArrayOutputStream();
    for (int bytesRead = 0; bytesRead > -1; bytesRead = in.read(buff)) {
      bytesOut.write(buff, 0, bytesRead);
    }
    JsonParser p = new MappingJsonFactory().createJsonParser(bytesOut.toByteArray());
    Map request = p.readValueAs(Map.class);
    if (null != request) {
      Message msg = new StandardMessage(null, bytesOut.toByteArray());
      msg.setRequestId(subRequestId);
      msg.setSecurityKey(getSecurityKey());
      msg.setType(getType());
      msg.setRequest(request);
      messages.add(msg);
    }
  }
} catch (IOException e) {
  log.error(e.getMessage(), e);
}

Delegation Execution

The batch message handler doesn’t do any real work itself, it only unzips the messages which are zip files and dumps the individual entries into their respective queues.

This does create a problem, though, with the results. I’ve made it configurable so that a request can send a batch message and receive results for each of those embedded requests one-by-one, as they are ready. This is a little easier because the batch message handler consumer doesn’t have to create a temporary queue and listen for results itself.

If you want the results back as a zip file, though, the batch message handler just declares an “anonymous” queue and sets that as the RepyTo. To use this feature, the requestor has to provide a timeout value, otherwise the batch message handler won’t know how long to wait if it doesn’t receive a response for every request.

Assuming all goes well…

… and the batch message handler gets an equal number of responses for the number of requests it sent out, it zips everything into a response file, and dumps that zip file into the requestor’s response queue.

Written by J. Brisbin

April 15, 2010 at 8:14 pm

Posted in RabbitMQ, The Virtual Cloud

Tagged with , ,

Tomcat/tcServer Deployer using RabbitMQ

with one comment

I was prototyping a RabbitMQ-based web application deployer for SpringSource tcServer 6.0 today using the Groovy DSL for RabbitMQ. I’m starting to think this DSL is more than a simple administrative tool, though, because it took me about an hour (hour-and-a-half, tops) to create a consumer that listens to events emitted by the build server (in our case, TeamCity 5.0), downloads the new war file from the special URL and calls a couple very convenient JMX methods within tcServer to redeploy the application.

Read the rest of this entry »

Written by J. Brisbin

April 14, 2010 at 9:53 pm

Posted in RabbitMQ, The Virtual Cloud

Tagged with , ,