« HE:labs

Automated zero downtime deployment to AWS Elastic Beanstalk for Rails with EbDeployer

Postado por Thiago Marano em 19/05/2015

Wouldn't it be awesome if it were so easy and straightforward to configure a zero downtime deployment to AWS? Well, it is!

Why zero downtime?

Wow! Didn't you know? If you haven't done anything about it, when you deploy a new version of a web application there is a downtime window when the web server is unaccessible. If this application is critical to its users, this might be a problem. Or if you are on the road to Continuous Delivery, or just need to deploy frequently. Or if (GOD HEAR THESE WORDS!) you would lose so much money if not able to answer user requests. Zero downtime is something you might eventually want.

The solution

Ever present applications have the quality of making a new version of the application to instantly become available to its users. It usually involves starting a new instance of the application with the new version, and then switch traffic to it after it boots and smokes tests confirm it is good to go. This switch is usually acomplished via a DNS or load balance change. This approach is sometimes called A/B deployment or Blue/Green Deployment.

AWS Elastic Beanstalk

AWS already has a service that manages application deployments and environments. Amazon Elastic Beanstalk is Amazon's cloud solution take on Heroku-style application server management and deployment. It enables you to quickly deploy, monitor and scale web applications. It does that by leveraging AWS EC2, S3 and RDS services. It means the infra structure is transparent and you can inspect and fine tune any of its resources - application server, databases and load balancers.

Why is it better than Heroku? (I knew you were going to ask that)

It is not. It is a different beast and it will suit you better given different requirements. Elastic Beanstalk gives you total control of your resources because it is built on top of AWS. But simple scenarios might require more configuration than Heroku. While complex server management operations like auto scaling are handled with simple and customizable configurations. It is also cheaper. And has a much more mature billing model, allowing you to do things like reserve instances in advance, and setup alerts when costs are forecasted to get too high.

EbDeployer is the tool of choice!

The standard way to deploy to Elastic Beanstalk is using the eb deploy command that performs an inplace update of the running application. It doesn't handle zero downtime deployments though. And you are left to build yourself the tooling needed to manage multiple environments.

The gem EbDeployer was built by the awesome Mingle Team with the purpose to automate zero downtime deployment scenarios. It handles environment creation, update, smoke test, the DNS switch to the new stack and persistent resources using a shared CloudFormation stack.

Setup the configuration

First download and install the aws command line tool. You can get it with brew install awscli. The following command will prompt you to enter your AWS credentials.

1 aws configure

Add the following to your Gemfile.

1 gem 'eb_deployer'

Then, run the commands:

1 bundle install
2 rails generate eb_deployer:install

This will create the following configuration files:

1 .ebextensions/01_postgres_packages.config # Packages needed to install the `pg` postgres gem. 
2 config/eb_deployer.yml # It configures eb_deployer options as well as Elastic Beanstalk.
3 config/rds.json # The CloudFormation stack configuration with a Postgres RDS instance.
4 lib/tasks/eb_deployer.rake # Deployment tasks for you to customize.

Files dropped at .ebextensions folder will run in alphabetical order on deployment.

It will also add the pg gem to the Gemfile in case it is not present.

The following is added to database.yml.

1 production:
2   adapter: postgresql
3   database: <%= ENV['DATABASE_NAME'] || 'example_app' %>
4   host: <%= ENV['DATABASE_HOST'] || 'localhost' %>
5   port: <%= ENV['DATABASE_PORT'] || 5432 %>
6   username: <%= ENV['DATABASE_USERNAME'] || 'my_user' %>
7   password: <%= ENV['DATABASE_PASSWORD'] %>
8   min_messages: ERROR

The eb_deployer.yml file contains a bunch of useful comments about options you might want to add and/or tweak. Taking the time to read it will give you a great idea of eb_deployer capabilities. Let's take a look at it without the comments.

 1 application: example_app
 3 common:
 4   solution_stack_name: 64bit Amazon Linux 2015.03 v1.3.1 running Ruby 2.1 (Passenger Standalone)
 6   smoke_test: |
 7     curl_http_code = "curl -s -o /dev/null -w \"%{http_code}\" http://#{host_name}"
 8     Timeout.timeout(600) do
 9       until ['200', '301', '302'].include?(`#{curl_http_code}`.strip)
10         sleep 5
11       end
12     end
14   option_settings:
15     - namespace: aws:autoscaling:launchconfiguration
16       option_name: InstanceType
17       value: m1.small
18     - namespace: aws:elasticbeanstalk:application:environment
19       option_name: DATABASE_NAME
20       value: example_app
21     - namespace: aws:elasticbeanstalk:application:environment
22       option_name: DATABASE_USERNAME
23       value: example_app
24     - namespace: aws:elasticbeanstalk:application:environment
25       option_name: DATABASE_PASSWORD
26       value: PleaseChangeMe
28   resources:
29     template: config/rds.json
30     inputs:
31       DBName: example_app
32       DBUser: example_app
33       DBPassword: PleaseChangeMe
34     outputs:
35       RDSPassSecurityGroup:
36         namespace: aws:autoscaling:launchconfiguration
37         option_name: SecurityGroups
38       RDSHost:
39         namespace: aws:elasticbeanstalk:application:environment
40         option_name: DATABASE_HOST
41       RDSPort:
42         namespace: aws:elasticbeanstalk:application:environment
43         option_name: DATABASE_PORT
45 environments:
46   dev:
47     strategy: inplace-update
48   production:
49     option_settings:
50       - namespace: aws:elasticbeanstalk:application:environment
51         option_name: SECRET_KEY_BASE
52         value: 52f6fb66e2e8306dff7e5a0964f46df25239e377ceab35ba29966c1292862a1e00ca2e09333527b10427902fe287

There are a few of things to note here before we proceed.

The common section is common (DUH!) to the environments under the environments section. The default configuration contains two environments: dev and production. Configurations defined under those environments will inherit from common, and can be used to override or define environment specific configurations.

The option_settings section is a list of Elastic Beanstalk options passed directly to the AWS client. You can find the full list of available options at Elastic Beanstalk options documentation. Note that the value field is expected to be a string, so if you have a number or boolean you need surrounded it with quotes, like value: '1' and value: 'true'.

The resources section will setup a persistent CloudFormation stack using rds.json as the stack template configuration. This template can be customized if you need other resources. For our purpose we won't touch the template. The outputs defined here are available to your application as environment variables, this is how your application knows where to connect to the database.

Tailor it to your needs

The first thing you might want to customize is the solution_stack_name, depending on the ruby version, web server and OS you want to use. The full list of available platforms can be found at Supported Platforms.

You can use the version_label option under common to give a meaningful name to the version reported by amazon to be running on each environment. The following snipet will concatenate the application name, the git SHA and an environment variable name meant to be outputed from a CI build number. Here I've called it VERSION_LABEL_SUFFIX.

1 version_label: example_app-<%= `git log -n1 | awk '/^commit/ {print $2; exit;}' | cut -c 1-10`.strip %>-<%= ENV['VERSION_LABEL_SUFFIX'] %>

The default configuration will perform the upgrade with the following strategy:

1 strategy: blue-green

Note that each application environment you have, in our case 'dev' and 'production', there will be two Elastic Beanstalk environments, one active and another inactive. Thus you can configure the dev environment to run with a single Elastic Beanstalk environment, and perform inplace updates. This can be achieved with:

1 strategy: inplace

Unfortunatly a new medium sized application was taking something around 10 minuts to deploy, more than the smoke test 8 minutes default wait. To turn around this issue we increase the wait to 15 minutes. Under option_sessings:

1 - namespace: aws:elasticbeanstalk:command
2   option_name: Timeout
3   value: "900"

One last thing I am going to suggest, but not less important - do setup a proper smoke test suite to ensure the deployment sanity. The smoke_test script will run after deployment. And the CNAME switch will only proceed if it finishes successfully. It helps avoiding a broken version to get released.

Environment variables

Environment variables with API keys, secrets and other stuff can be achieved with:

1 environments:
2   staging:
3     option_settings:
4       - namespace: aws:elasticbeanstalk:application:environment
5         option_name: S3_BUCKET
6         value: example_app-dev.assets

To improve deployment speed we can skip asset precompile, and do it when packaging. The configuration is done with the enviroment variable:

1 - namespace: aws:elasticbeanstalk:application:environment
3   value: "true"

Although this should be the default on Elastic Beanstalk ruby platforms, I've found it necessary to explicity setup rack and rails environments.

1 - namespace: aws:elasticbeanstalk:application:environment
2   option_name: RACK_ENV
3   value: production
5 - namespace: aws:elasticbeanstalk:application:environment
6   option_name: RAILS_ENV
7   value: production

SSH keys

If you ever need to login to your EC2 webserver, you can setup SSH keys this way:

1 aws ec2 create-key-pair --key-name=example_app-dev-keypair

Below is the configuration setup the dev environment to use the created key.

1 environments:
2   dev:
3     option_settings:
4       - namespace: aws:autoscaling:launchconfiguration
5         option_name: EC2KeyName
6         value: example_app-dev-keypair

The configuration below allows your machine to receive SSH connections from any host.

1 option_settings:
2     - namespace: aws:autoscaling:launchconfiguration
3       option_name: SSHSourceRestriction
4       value: tcp, 22, 22,

Package and deploy tasks

Next, we want to edit eb_deployer.rake. I've tweaked the original code to do the following: support running from MacOS, precompile assets before building package, and add git the SHA and CI build number to package name. It looks like this:

 1 namespace :eb do
 2   def eb_deployer_env
 3     ENV["EB_DEPLOYER_ENV"] || "dev"
 4   end
 6   def eb_deployer_package
 7     git_sha = `git log -n1 | awk '/^commit/ {print $2; exit;}' | cut -c 1-10`.strip
 8     "tmp/pkg/example_app-#{git_sha}-#{ENV['VERSION_LABEL_SUFFIX']}.zip"
 9   end
11   desc "Remove the package file we generated."
12   task :clean do
13     sh "rm -rf tmp/pkg"
14   end
16   desc "Build package for eb_deployer to deploy to a Ruby environment in tmp/pkg directory."
17   task :package => [:clean, :environment] do
18     sh "mkdir -p tmp/pkg"
19     to_exclude = %w(tmp .git coverage .idea .DS_Store).map { |file| "--exclude='*#{file}*'" }.join(' ')
20     sh "bundle exec rake assets:precompile"
21     sh "zip -9 -r #{to_exclude} '#{eb_deployer_package}'  ."
22   end
24   desc "Deploy package we built in tmp directory. default to dev environment, specify environment variable EB_DEPLOYER_ENV to override, for example: EB_DEPLOYER_ENV=production rake eb:deploy."
25   task :deploy => [:package] do
26     sh "eb_deploy --package '#{eb_deployer_package}' --environment #{eb_deployer_env}"
27   end
29   desc "Destroy Elastic Beanstalk environments. It won't destroy resources defined in eb_deployer.yml. Default to dev environment, specify EB_DEPLOYER_ENV to override."
30   task :destroy do
31     sh "eb_deploy --destroy --environment #{eb_deployer_env}"
32   end
33 end


By now you should be able to deploy the first version of your app. This usually takes around 10 minutes.

1 EB_DEPLOYER_ENV=dev rake eb:deploy

A new Elastic Beanstalk environment should have been crated with the application deployed! The next deploy will do the same and you will end up with two stacks. From there on, deployments will be made on the inactive stack, and a switch will proceed to swap the old stack with the new one.

Did I tell you custom environments are one configuration away?

As you might have notice, I've mentioned the CI a couple of times. This solution was implemented with a CI in mind. Nevertheless, the deploy command can be ran by anyone with proper permissions. As overlooked as it might be, I find this to be very useful when the need for a separeted environment arises. For example, when you need to spike something out. Or you gonna do disruptive changes like upgrade frameworks and libraries. Think rails upgrades, database upgrades, performance tests, features that a toggle wouldn't handle. By the time you actually need to branch the repository.

Final considerations

I feel responsible to remember the reader that if he followed this guide, there will be more work.

There is still the fact that this setup has a single database instance. It means that database changes cannot be destructive. They need to be preceeded by code that can handle both old and new schemas. Only after the database is migrated, the transitional code can be removed by a subsequent deploy. Background jobs are another concern because workers might need to be smart enough to know if they are on the inactive stack and should stop processing jobs.

I hope I was able to present tools that enable you to deal with real world application resilience needs.

Other resources

First, I recommend you to go to Li Xiao's Deploy Service by EbDeployer for some history and a more detailed look at EbDeployer.

Deploying Versions with Zero Downtime is Amazon recommendations on how to perform a zero downtime deploy on Elastic Beanstalk. The approach is similar to that of EbDeployer, it uses a CNAME switch to replace the old stack with the new one. This is more like a DIY instructions as oposed to the out-of-box goodness of EbDeployer. Also, it misses the configuration for persistent resources using CloudFormation.

You could also try to achieve similar behaviour using Elastic Beanstalk BatchSize command option. It would involve telling the load balancer to update one server at a time, given you have multiple instances. You can read more at Elastic Beanstalk options documentation.

The blog post Lighting fast, zero-downtime deployments with git, capistrano, nginx and Unicorn is a somewhat old, maybe outdated, article on how to have similiar bahaviour using two unicorn processes and smart workers. Note that some people on the comments have found issues with this approach.

On How to set up a Rails 4.2 app on AWS with Elastic Beanstalk and PostgreSQL. It is evident that even a single application instance setup is not as straightforward as with EbDeployer.

Faster Rails 3 deployments to AWS Elastic Beanstalk is Vinicius Horewicz's journey on reducing Elastic Beanstalk deployment down close to 1 minute.

By the way, on Heroku you can achieve a similar zero downtime deployment behaviour using the preboot feature.


Sabia que nosso blog agora está no Medium? Confira Aqui!