At work, we've been using AWS OpsWorks lately for a lof of our new infrastructure. It is a pretty neat tool to setup a deployment pipeline for your applications and allows for quiet a lot of customization (through the use of chef-solo recipes). How to use OpsWorks is out of the scope of this blog post. What I did want to talk about however, was how to use Jenkins and Boto to automate instance creation/deployment. The use case here is that not everyone has direct access to the AWS console or has keys setup to issue commands via the CLI. We still need to allow these devs to be able to deploy their code (be it a specific branch or simply the master branch) to an instance resembling production and ensure that things work correctly (and down the road, run automation tests).
Luckliy, with the help of Boto and the Python script plugin for Jenkins, this is relatively easy. You obviously need to have a Jenkins server running along with the Python script plugin installed. Additionally, you need to install the boto library (I installed it globally) - usually via pip (sudo pip install boto).
Once these requirements are met, make the required modifications to the script below, add a build step to execute a python script and voila.
The process this script follows is documented inline, but the gist is that we do deployments by creating new instances and deleting old ones instead of running the deploy command on an existing instance. This does take longer but I feel it's cleaner than deploying multiple times to the same instance.
import boto
import time
import sys
import os
# Fill in the various required fields below (should be self explanatory)
AWS_ACCESS_KEY = "FILL_ME"
AWS_SECRET_KEY = "FILL_ME"
STACK_ID = "FILL_ME"
APP_ID = "FILL_ME"
LAYER_IDS = ["FILL_ME"]
INSTANCE_TYPE = "m1.medium"
# We allow users to build 'parametrized' builds - they can specify the branch they want to use
# to build the new instance. The APP is updated to use that branch and once the new instance is created,
# it's updated again to rever to master (this piece is hardcoded)
# This is the format OpsWorks expects (they should clean it up and make the dict be more pythonic)
APP_SOURCE = {"Url": "FILL_ME", "Type": "git", "Revision": os.getenv("BRANCH")}
APP_SOURCE_MASTER = APP_SOURCE.copy()
APP_SOURCE_MASTER['Revision'] = "master" # default branch should always be master
opsworks = boto.connect_opsworks(aws_access_key_id=AWS_ACCESS_KEY, aws_secret_access_key=AWS_SECRET_KEY)
# Print out the instances running presently
online_instances = map(lambda x: "Hostname: %s, DNS: %s, Status: %s" % (x['Hostname'], x['PrivateDns'] if 'PrivateDns' in x else '', x['Status']), opsworks.describe_instances(stack_id=STACK_ID)['Instances'])
print "Online Instances: " + str(online_instances)
print "Deploying branch: %s" % APP_SOURCE["Revision"]
# Update the app to use our specific branch
opsworks.update_app(app_id=APP_ID, app_source=APP_SOURCE)
print "Adding another instance"
# Creat the instance (we'll later have to start it too)
new_inst = opsworks.create_instance(stack_id=STACK_ID, layer_ids=LAYER_IDS, instance_type=INSTANCE_TYPE)
print "New Instance ID: %s. Starting it now" % new_inst["InstanceId"]
# As promised, start the instance.
opsworks.start_instance(new_inst["InstanceId"])
print "Start command sent. Checking Status now"
# We need to start 'describing' the instance to check it's status. Depending on the type of instance created
# it can take a while to come up (m1.medium usually take 15-20 mins for me - anecdotal evidence though).
inst = opsworks.describe_instances(instance_ids=[new_inst["InstanceId"]])['Instances'][0]
print "New Instance Hostname: %s, Status: %s" % (inst['Hostname'], inst['Status'])
# Wait until the instance is either up or the setup failed.
while inst['Status'] not in ["online", "setup_failed"]:
sys.stdout.flush()
inst = opsworks.describe_instances(instance_ids=[new_inst["InstanceId"]])['Instances'][0]
print "Status is: %s. Sleeping for 30 seconds" % inst['Status']
time.sleep(30)
# Use this to specify (to Jenkins) whether the job failed or not.
EXIT_CODE = 0
if inst['Status'] == "online":
print "New instance started. DNS: %s" % inst['PrivateDns']
elif inst['Status'] == "setup_failed":
print "Error creating instance. See the Opsworks console for more information."
EXIT_CODE = 1
# Get the log file URL to show
log_file = opsworks.describe_commands(instance_id=inst['InstanceId'])['Commands'][0]['LogUrl']
print "Log file for this instance is %s" % log_file
print "Reverting the revision back to %s" % APP_SOURCE_MASTER["Revision"]
opsworks.update_app(app_id=APP_ID, app_source=APP_SOURCE_MASTER)
print "All Done"
# Exit with either success (0) or failure (1).
sys.exit(EXIT_CODE)
Once this script is in place, you should also check the 'This build is parametrized' checkbox and select a String Parameter with the name Branch and default value with master. Save the job - you should now be able to click the Build with parameters option presented by Jenkins, optionally change the branch and then run the job. If things go well, Jenkins should give you the PrivateDns URL for the instance.