J. Brisbin

Just another Wordpress.com weblog

Cloud Artifact Deployment with RabbitMQ and Ruby

leave a comment »

Running a hybrid or private cloud is great for your scalability but can get a little dodgy when it comes to deploying artifacts onto the various servers that need them. To show how I’m solving this problem, I’ve uploaded my Ruby scripts that monitor and deploy artifacts that have been staged by the automated processes on my continuous integration server, TeamCity. In order to make it fairly secure, it will not deploy arbitrary artifacts. Anything you want automatically deployed must be explicitly configured as to the URL from which to download the artifact and the path to which you want it copied (or unzipped/untarred).

The Parts

There are a couple moving parts here. You need a RabbitMQ server, of course. You also need a couple servers to deploy things to. I use three instances of SpringSource tcServer (basically Tomcat 6.0) per Ubuntu 10.04 virtual machine. So this script needs to deploy the same file to three different locations. I also need to deploy HTML files to my Apache server’s document root. As an aside: Apache has now been relegated to only serving static resources and PHP pages and is no longer the out-in-front proxy server. I’ve switched to HAProxy. I love it. More on that in a future post.

The Scripts

I haven’t included the script that actually publishes the notifications yet. That’s a Python script at the moment (Ruby is so much more fun to program in than Python :). It looks like this:


#!/usr/bin/python

import os, sys, hashlib
from amqplib import client_0_8 as amqp

EXCHANGE = 'vcloud.deployment.events'

def get_md5_sum(filename):
  if not os.path.exists(filename):
    return None

  md5 = hashlib.md5()
  try:
    with open(filename, 'r') as f:
      bytes = f.read(4096)
      while bytes:
        md5.update(bytes)
        bytes = f.read(4096)
  except IOError:
  # Probably doesn't exist
    pass
  return md5.hexdigest()

def send_deploy_message(queue=None, artifact=None, unzip=False):
	if not queue is None and not artifact is None:
		md5sum = get_md5_sum('/var/deploy/artifacts/%s' % sys.argv[2])
		#print 'MD5: %s' % md5sum

		mq_conn = amqp.Connection(host='rabbitmq', userid='guest', password='guest', virtual_host='/')
		mq_channel = mq_conn.channel()
		mq_channel.exchange_delete(EXCHANGE)
		mq_channel.exchange_declare(EXCHANGE, 'topic', durable=True, auto_delete=False)
		mq_channel.queue_declare(queue, durable=True, auto_delete=False, exclusive=False)
		mq_channel.queue_bind(queue=queue, exchange=EXCHANGE)
		msg = amqp.Message(artifact, delivery_mode=2, correlation_id=md5sum, application_headers={ 'unzip': unzip })
		mq_channel.basic_publish(msg, exchange=EXCHANGE, routing_key='')

if __name__ == '__main__':
	send_deploy_message(queue=sys.argv[1], artifact=sys.argv[2], unzip=sys.argv[3])

I’ll be converting this to Ruby at some point soon.

You can check out the Ruby scripts themselves on Github: http://github.com/jbrisbin/cloud-utils-deployer

The Deployment Chain

When our developers check anything into our Git repository, TeamCity sees that change and commences to build the project and automagically stage those artifacts onto the development server. This deployment requires no manual intervention. We always want development to use the latest bleeding edge of our application code. Once we’ve had a chance to test those changes and we’re ready to push them to production, I have a configuration in TeamCity that calls the above Python script. The developer can just click the button and it publishes a message to RabbitMQ announcing the availability of that project’s artifacts (of which there’s likely several). We haven’t decided how often we want the actual deployment to happen, but for the moment a cron job runs at 7:00 A.M. every morning on all the running application servers (it should also be run from an init.d script to catch servers that have been down and are behind on their artifacts). That script is the “monitor” script. It simply subscribes to a queue with the same name as the configuration section in the monior.yml YAML file:


myapp.war:
  :deploy: deploy -e %s

The “%s” placeholder in the “:deploy” section (the preceding colon is significant in Ruby) will be replaced by the name of the artifact as pulled from the body of the message. It may or may not correspond to the queue name. It doesn’t have to because it’s simply an arbitrary key in the deploy.yml file.

The “deploy” script is where all the fun happens. Via command-line switches, you can turn on or off the ETag matching and MD5 sum matching it does to keep from redeploying something that it’s already deployed (it keeps track in its own cache files).

First, the deployment script has to download the resource to a temporary file:


request = Net::HTTP::Get.new(@uri.request_uri)
load_etags do |etags|
	etag = etags[@name]
	if !@force and !etag.nil?
		request.initialize_http_header({
			'If-None-Match' => etag
		})
	end

	response = @http.request(request)
	case response
		when Net::HTTPSuccess
			# Continue to download file...
			$log.info(@name) { "Downloading: #{@uri.to_s}..." }
			bytes = response.body
			require "md5"
			@hash = MD5.new.update(bytes).hexdigest
			# Write to temp file, ready to deploy
			@temp_file = "/tmp/#{@name}"
			File.open(@temp_file, "w") { |f| f.write(bytes) }
			# Update ETags
			etags[@name] = response['etag']

			outdated = true
		when Net::HTTPNotModified
			# No need to download it again
			$log.info(@name) { "ETag matched, not downloading: #{@uri.to_s}" }
		else
			$log.fatal(@name) { "Error HTTP status code received: #{response['code']}" }
	end

	if @use_etags
		save_etags(etags)
	end
end

This method returns a true|false depending on if it thinks the resource is out-of-date or not. The deployment script then calls the “deploy!” method, which attempts to either copy the resource (if it’s say, a WAR file) or unzip the resource to the pre-configured path (if it’s say, a “.tar.gz” file of static HTML resources or a “.zip” file of XML definitions). The deployer decides whether to try to unzip or untar based on the extension. If it’s “.tar.gz” it will run the “tar” command. If it’s anything else, it will try to unzip it. This isn’t configurable, but might be a good project for someone if they want to use “.tbz2” files or something! šŸ™‚

Permissions

The user you run this as matters. I have the log file set to “/var/log/cloud/deployer.log”. This is configurable in the sense that you can download the source code and change it in the constant where it’s defined (cloud/logger.rb). Your user should also have write permission to a directory named “/var/lib/cloud/”. You can change this (at the moment) only by editing the “cloud/deploy.rb” file and changing the constants. There’s only so many hours in the day. Just didn’t have time to make it fully configurable. I’d love some help on that, though, and would gladly accept patches!

Still to come…

I just haven’t had time to make it a true Gem yet. That’s my intention, but at this point, on a Friday afternoon, I’m thinking it’ll be next yet before that’s done. UPDATE: Done! This is now on RubyGems.org.

As always, the full source (Apache licensed) is on Github:

http://github.com/jbrisbin/cloud-utils-deployer

I’d love to hear what you think.

Advertisements

Written by J. Brisbin

June 11, 2010 at 8:55 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: