M uch like we refactor code for maintenability, every now and then it pays to take a look at our infrastructure and see if there is anything we can improve. For example, recently I decided to shutdown our Postgres & Redis instances and move over to AWS RDS and Elasticache. I also wanted to upgrade all our web frontend servers and queue workers to the latest Ubuntu LTS and a bunch of other tasks that required reimaging servers. Having previously suffered with Chef/Puppet, I decided to take a look at Ansible.

First thing I struggled with was how to get Ansible to play nice with EC2. Ansible has the concept of inventory and in most examples, this is a file with hardcoded hosts. However, there does exist a bundled plugin to query resources on AWS. The second thing I struggled with is figuring out a directory structure that makes sense and what are some best practices. So I’m going to give you a quick rundown on how I setup our Ansible deployment for Sidelines.

Directory Structure

|-env
|-group_vars/
|---all/
|---tag_environment_production/
|-----secrets.yml
|---tag_environment_staging/
|-----secrets.yml
|-plugins/
|---inventory/
|-----ec2.py
|-----ec2.ini
|-roles/
|---<rolename>
|-----defaults/
|-----handlers/
|-----tasks/
|-----templates/
|-----files/
|-web-uwsgi-playbook.yml
|-dbserver-playbook.yml

A few things going on here. Ansible is installed in a virtualenv, I hate having system-wide python packages. The plugins/inventory directory contains ec2.py and ec2.ini. Also, pay special attention to the subdirectories under group_vars, we’ll get to those in a sec.

EC2 Resources & Conventions

To make Ansible use the EC2 plugin, edit ansible.cfg and make sure the line hostfile = plugins/inventory/ec2.py is there.

Now there are two conventions I follow:

• Production instances are tagged with environment = production and staging instances with environment = staging. An instance cannot be both production and staging
• I separate the functional roles of instances using EC2 security groups. This forces me to also think about security and only open up ports that are absolutely necessary

Security

We have a number of passwords/keys and it becomes impossible and insecure to try and remember them all. Ansible offers a clean solution for this called the Ansible vault. I encourage you to read more about it, I’ll only point out that I simply put all secrets in an encrypted file called secrets.yml. I maintain two such files, one for staging and one for production. If you put each in the right directory, Ansible will pick up the right one. How does Ansible know though?

Production vs Staging

Recall the convention that all production instances are tagged with environment=production. The ec2.py inventory plugin creates a ton of dynamic groups so you can address your EC2 instances any way you deem suitable. Therefore, it will create a dynamic group tag_ for each tag in your system, a security_group for each security group, etc. With that in mind, and the two conventions we talked about above, addressing all my uWSGI machines using Ansible syntax is as simple as putting hosts: security_group_web-uwsgi:&tag_environment_{{environment}} in my playbook. To execute the playbook I use ansible-playbook web-uwsgi-playbook.yml -v --extra-vars env=production --ask-vault-pass for production and ansible-playbook web-uwsgi-playbook.yml -v --extra-vars env=staging --ask-vault-pass for staging.

Ansible’s learning curve is not very steep. After overcoming a few frustrations and wrapping my head around writing DSL in YAML, I’ve found Ansible to be a valuable tool and will certainly be using it more going forward.