Will Thames’ tech blog

Immutable Kubernetes configuration with Ansible

2019-01-28T20:00:00+10:00

This post touches on a key component of my Managing Kubernetes is Easy With Ansible talk that I gave at AnsibleFest 2018. Since giving that talk, I’ve also solved some of the unforeseen consequences, and go into further detail here.

There are a number of problems associated with managing configuration (in the form of configmaps and secrets) through the default mechanism.

Changes to configmaps or secrets are not picked up by the pods that rely on them unless the related deployment is also updated
Rolling back a deployment to a previous version does not roll back the associated configuration

We can solve these problems through immutable configmaps and secrets that are named based on their contents, such that if the contents change, their name changes.

The kubectl tool has part of a solution for that, in that you can pass --append-hash when creating a configmap or secret, but only when using kubectl create, it’s not useful when applying resource definitions from files. However, this idea was the inspiration to add an append_hash parameter to Ansible’s k8s module, and this is part of the solution.

Another part of the solution is the k8s_config_resource_name filter plugin, which takes a configmap definition and returns the full name with the hash added.

At this point we can define dicts of configmaps and secrets, and also refer to the full resource name in deployments or similar resources.

In inventory we might have something like:

kube_resource_configmaps:
  env: "{{ lookup('template', kube_resource_template_dir + 'env-configmap.yml') | from_yaml }}"
kube_resource_secrets:
  env: "{{ lookup('template', kube_resource_template_dir + 'env-secrets.yml') | from_yaml }}"

which then gets referenced in the deployment with:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ kube_resource_name }}
  namespace: {{ kube_resource_namespace }}
spec:
  template:
    spec:
      containers:
        - envFrom:
            - configMapRef:
                name: {{ kube_resource_configmaps.env | k8s_config_resource_name }}
            - secretRef:
                name: {{ kube_resource_secrets.env | k8s_config_resource_name }}

Its best to create the secrets and configmaps first, as the pods created by the deployment will depend on them being present to be able to start - then you can use the wait functionality that the k8s module gains with Ansible 2.8 to check the pods start correctly.

At this point, we have met our original goals - changing a configmap or secret will change its name, and so the deployment will change to point to the new name, so new pods will be created using the new configuration. The replicaset to which kubectl rollback deploy will roll back will contain the old configuration names, and so rolling back a deployment will also roll back the configuration.

However, my presentation ignores two issues (the first I was aware of but hadn’t yet solved, the second my colleague highlighted as an obstacle to moving to the new way of working)

there is no garbage collection of configmaps that were referenced by replicasets but are no longer used
diffs no longer work - because the configmaps and secrets change name, it’s hard to compare against the previous version of the configmap or secret.

The solution to both is simple to describe, and harder to implement with the current tools, but possible.

Kubernetes has the concept of resource owners: if a resource that owns another resource is deleted, the owned resource is also deleted. So pods are owned by replicasets which are in turn owned by deployments - deleting the deployment removes all replicasets associated with the deployment, and all pods associated with those replicasets.

We can use this to add owner references to configmaps and secrets - we find the replicaset associated with the recent deployment, and set that resource as the owner in the configmap/secret. This means that when replicasets are retired (Kubernetes keeps ten replicasets by default, but this is configurable with spec.revisionHistoryLimit) the associated configmaps and secrets are also retired.

For diffs, first we lookup the deployments.kubernetes.io/revision annotation of the previous deployment and the current deployment. We then have to find all replicasets (as there’s no way to search by annotation, unfortunately) and then select the previous replicaset and current replicaset using the same annotation.

Adding a new label, such as kube_resource_prefix, to our configmaps allows us to iterate over our kube_resource_configmaps dict (the label_selectors argument to k8s_facts is useful here), each time looking for all configmaps with the label of the current configmap, and then finding the configmap with the owner reference set to the previous replicaset and the configmap owned by the current replicaset. Exactly the same technique holds for secrets too. Once we’ve found the before and after, we can display the differences using the debug module (an explicit diff module might be useful).

I’ve updated the ansible-role-kube-resource role with this new functionality so that it’s easier to use, it does rely on all resources having kube_resource_name and kube_resource_namespace correctly set. All of the owner reference based functionality relies on replicasets updating when deployments change - so isn’t currently useful for e.g. statefulsets.

In other news on that role, I’ve now added a molecule test suite, inspired by Jeff Geerling (@geerlingguy)’s talk at AnsibleFest and his super helpful molecule blog post. It relies on an existing kubernetes platform being set up and configured, but given that Docker comes with Kubernetes (on some platforms at least), and that minikube exists, that shouldn’t be insurmountable.

There are quite a few features from Ansible 2.8 that are used in the role but have been backported into the role - kubernetes resource validation, waiting for resources to meet the desired state, append_hash and k8s_config_resource_name for example. Even if you choose not to use the role, you can make use of the same techniques through setting and using module_utils and filter_plugins configuration directives in ansible.cfg.

connection: local vs delegate_to: localhost

2018-07-01T00:00:00+10:00

Performing tasks locally is a common operation when working with an API of some kind—typical use cases are cloud services, network devices, cluster management. There are three ways of achieving this in Ansible: connection: local, delegate_to: localhost and local_action. The last is rarely seen these days and can be deemed equivalent to delegate_to: localhost in terms of advantages and disadvantages, but with the additional disadvantage of being a very unusual style, adding a readability penalty.

In a previous post I talked about the runner pattern which allows better use of inventory for different scenarios even when the controller is localhost. connection: local behaves very differently if the host is localhost or a ‘runner’ host, which is surprising.

The main difference between connection and delegate_to is that connection can be used at a play or task level, whereas delegate_to operates at a task level only. This means that if you have a playbook with fifty tasks, each will need delegate_to set. Worse, if you’re using someone else’s role, you’ll have to hope they’ve provided for this eventuality.

The problem with connection: local for the runner pattern is that it assumes that it’s an entirely new connection and will use the system python rather than what ever python you prefer to use. Situations where this is a problem include when using virtualenvs to install python libraries or on OS X where most rely on python from brew. In this case, you might run pip install library, run Ansible and find that that library can’t be found because it’s looking in the wrong place.

To demonstrate this, I wrote a boto3_facts module, which shows python location and version as well as boto3 and botocore versions.

- hosts: localhost
  gather_facts: no

  tasks:
  - name: localhost without explicit connection
    boto3_facts:

- hosts: fakehost
  gather_facts: no

  tasks:
  - name: runner host using delegate_to
    boto3_facts:
    delegate_to: localhost

- hosts: fakehost
  gather_facts: no

  tasks:
  - name: runner host using local_action
    local_action:
      module: boto3_facts

- hosts: fakehost
  connection: local
  gather_facts: no

  tasks:
  - name: runner host using local connection
    boto3_facts:

$ ansible-playbook boto3_facts.yml -v -i fakehost,
Using /Users/will/tmp/ansible/boto3_facts/ansible.cfg as config file

PLAY [localhost] *****************************************************************************************************************

TASK [localhost without explicit connection] *************************************************************************************
ok: [localhost] => changed=false
  boto3_version: 1.7.42
  botocore_version: 1.10.42
  python: /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
  python_version: |-
    2.7.14 (default, Mar  9 2018, 23:57:12)
    [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]

PLAY [fakehost] ******************************************************************************************************************

TASK [runner host using delegate_to] *********************************************************************************************
ok: [fakehost -> localhost] => changed=false
  boto3_version: 1.7.42
  botocore_version: 1.10.42
  python: /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
  python_version: |-
    2.7.14 (default, Mar  9 2018, 23:57:12)
    [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]

PLAY [fakehost] ******************************************************************************************************************

TASK [runner host using local_action] ********************************************************************************************
ok: [fakehost -> localhost] => changed=false
  boto3_version: 1.7.42
  botocore_version: 1.10.42
  python: /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
  python_version: |-
    2.7.14 (default, Mar  9 2018, 23:57:12)
    [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]

PLAY [fakehost] ******************************************************************************************************************

TASK [runner host using local connection] ****************************************************************************************
ok: [fakehost] => changed=false
  python: /usr/bin/python
  python_version: |-
    2.7.10 (default, Oct  6 2017, 22:29:07)
    [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)]

PLAY RECAP ***********************************************************************************************************************
fakehost                   : ok=3    changed=0    unreachable=0    failed=0
localhost                  : ok=1    changed=0    unreachable=0    failed=0

The easiest way to fix this is to set ansible_python_interpreter: "{{ ansible_playbook_python }}". My preferred approach is in group_vars/all if all tasks run locally, or group_vars/runner if using the runner pattern—but, as with below, at playbook vars level also works.

In conclusion, I much prefer connection: local for the runner pattern now that ansible_python_interpreter can be set dynamically.

- hosts: fakehost
  connection: local
  vars:
    ansible_python_interpreter: "{{ ansible_playbook_python }}"
  gather_facts: no

  tasks:
  - name: runner host using local connection and ansible_python_interpreter set
    boto3_facts:

PLAY [fakehost] ******************************************************************************************************************

TASK [runner host using local connection and ansible_python_interpreter set] *****************************************************
ok: [fakehost] => changed=false
  boto3_version: 1.7.42
  botocore_version: 1.10.42
  python: /usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
  python_version: |-
    2.7.14 (default, Mar  9 2018, 23:57:12)
    [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]

Managing Multiple AWS Consoles With Multi Account Containers

2018-02-28T00:00:00+10:00

While there are ways of managing multiple AWS account consoles in a single browser (such as assuming roles to access other accounts), various constraints might prevent that (e.g. accounts owned by third parties to which access should be segregated).

Most people in this situation have done the ‘browser juggle’, where you might have Firefox open with another window in Safe mode, Chrome open with another window in Incognito mode, and no doubt further browsers depending upon your OS.

With Firefox’s Multi Account Container add-on this can be a thing of the past. This add on makes it easy to manage multiple AWS accounts in a single Firefox window.

First, set up a container per account by clicking the Multi Account Container button, usually to the right end of the add ons bar, (or pressing <Ctrl>-.) and then Edit Containers. Give each account a separate colour if possible (as the colour appears as a line under the tab title, making it easier to distinguish which tab corresponds to which account).

Each time you need to use the console in a different account, long-click the + new tab button (also available via File > New Container Tab and, I’ve just learned, <Ctrl>+. and then <Tab> to select through the containers)

You’ll still to need to log in to the appropriate account for the container in the normal way. Even if you close all the container tabs, and later reopen the console in that same container, it’ll remember your session—so you’ll be back in the console as long as your session hasn’t expired.

Note that you can also elect to always open a URL in a particular container (not so useful for multiple AWS consoles, useful for segregating off sites whose business model depends on knowing what you’re doing, yes, I mean Facebook—particularly if you then default all other sites to a new container, or even just clear all cookies)

Using updated modules, libraries and plugins with stable Ansible

2017-12-12T20:00:00+10:00

This page was updated 2020-05-16 to incorporate how to use collections to the same effect

This page was updated on 2019-04-07 to improve `module_utils` information and add plugin information

There are many reasons to want to use newer modules than a chosen stable Ansible core release:

Feature enhancements don’t get backported to stable branches
Non-security bug fixes only tend to get backported one version — which means if say 2.N.0 hasn’t had all the core bugs ironed out yet, you might not get the benefit of module bug fixes while you remain on 2.N-1.0
Some improvements only exist in PR form. Some improvements only exist in branches made by combining multiple PRs.^† Some improvements are very handy but so experimental they’re not even ready for a PR!^†

When that happens, thankfully you don’t have to run off your own megamerge branch of ansible^†.

Which ever of the two approaches below you take, I recommend keeping a README.md file in the library directory. For modules, it looks a bit like:

|Module                     | PR                                            | Notes           |
|---------------------------|-----------------------------------------------|-----------------|
|cloudfront_distribution.py | https://github.com/ansible/ansible/pull/31284 | Unmerged        |
|ec2_placement_group.py     | https://github.com/ansible/ansible/pull/33139 | Available in 2.5|

but the same log is useful for plugins and collections.

Keeping track of why I’m using non-core code allows me to remove it when the desired functionality becomes available in newer Ansible or collections releases.

The collections approach

Using ansible collections to incorporate updates from baseline can be a sensible approach when you need several changes at once or just want to track latest or a branch.

You can use ansible-galaxy to install collections. I always recommend pinning to a specific version to avoid surprises. If you don’t want to use ansible-galaxy, you can always just commit the results of installing the collection (or just update a submodule) in the collections subdirectory below your playbooks (other locations are available through configuring collections_paths - take note of both the plurals, I wasted hours investigating why ANSIBLE_COLLECTION_PATHS wasn’t working).

mkdir collections
git clone git@github.com:ansible-collections/community.kubernetes collections/community.kubernetes

This will get you the latest merged commits from the official community.kubernetes (as opposed to ansible-galaxy collection install -p collections community.kubernetes which gets you the latest released version) - but you can also clone a fork and choose a branch to get as yet unmerged commits (e.g. if you’re keen to include an unmerged pull request)

The selective approach

Use this when you don’t necessarily want to update every module or plugin in a collection which might introduce unexpected behaviours in modules you’re not looking to update

My approach for this is to use the default library directory feature — create a library directory in the top level of your playbooks repository, and put any modules that you need but aren’t yet in the version of ansible you’re using there.

If you’re using modules that rely on updates to module_utils shared libraries, you can set the module_utils config directive in ansible.cfg (./module_utils is an undocumented default) and copy the relevant module_utils files into your codebase as well. The layout of your module_utils directory should reflect that of ansible. For example, if you need updates to the k8s module, you’ll probably need to copy one or both of k8s/common.py and k8s/raw.py from lib/ansible/module_utils to your module_utils/k8s directory. You will also need an empty __init__.py at the bottom level directory.

module_utils/
└── k8s
    ├── __init__.py
    ├── common.py
    └── raw.py

Similarly, you can optionally set filter_plugins, (which has ./filter_pluginsas an undocumented default), lookup_plugins (which has ./lookup_plugins as an undocumented default), etc. to point to updated plugins — I tend to use plugins/lookup, plugins/filter etc. to reflect Ansible’s codebase structure (but I didn’t know about the defaults until doing some tests for the update to this page).

One other point worth noting is that if you’re already using roles for your logic, you can put updates of modules, libraries and plugins in your library, module_utils and plugins directories at the top level of the role. You’ll need a versioning strategy for your role that reflects Ansible versions so that you can remove published changes later on (for example my Kubernetes role has a v2.8 branch without the module_utils/k8s tree) with published v2.7-1 and v2.8-1 tags. The need for an __init__.py doesn’t seem to be as apparent when using a role (i.e. my tests pass on my Kubernetes-role without needing an __init__.py)

^† — I’ve been there.

Generating Inventory

2017-11-01T00:00:00+10:00

As mentioned in yesterday’s blogpost, using a combination of environments, applications and operations can cause a cartesian explosion in hosts and groups to manage.

Even 10 applications in 3 environments over 2 operations can lead to sixty hosts, plus likely as many groups.

For example, we use a structure that looks like:

[environment:children]
application-environment

[application-environment:children]
operation-application-environment

[operation-application:children]
operation-application-environment

[operation-application-environment]
operation-application-environment-runner

[runner]
operation-application-environment-runner

[application:children]
application-environment
operation-application

[operation:children]
operation-application

And that’s just one host! Admittedly using groups that will be reused many times.

This problem has always been solvable using a script to generate inventory, but Ansible 2.4’s inventory plugin architecture, combined with the inspiration from the constructed plugin caused me to simplify my problem through creating a generator plugin. If the plugin proves popular I’ll likely raise a PR soon.

The generator plugin is then installed by putting it into our ansible playbooks repo under plugins/inventory/generator.py, and updating ansible.cfg to include

[defaults]
inventory_plugins = plugins/inventory

[inventory]
enable_plugins = generator,host_list,script,yaml,ini

The above inventory can be expressed with the inventory plugin using:

# inventory.config file in YAML format
plugin: generator
strict: False
hosts:
    name: "{{ operation }}-{{ application }}-{{ environment }}-runner"
    parents:
      - name: "{{ operation }}-{{ application }}-{{ environment }}"
        parents:
          - name: "{{ operation }}-{{ application }}"
            parents:
              - name: "{{ operation }}"
              - name: "{{ application }}"
          - name: "{{ application }}-{{ environment }}"
            parents:
              - name: "{{ application }}"
              - name: "{{ environment }}"
      - name: runner
layers:
    operation:
        - build
        - launch
    environment:
        - dev
        - test
        - prod
    application:
        - web
        - api

We are already using this to reduce 100+ line static inventory host files (our biggest as-yet unreduced file is 500 lines!) to something very similar to the above.

The major benefit is, as always, reducing repetition - expanding this to more environments or applications is then a matter of adding a single element to the appropriate layer, rather than adding the appropriate groups and hosts in 10 different places.

ansible-inventory-grapher copes fine with the results (I put the effort in to finally fixing it so that I could validate the result)

ansible-inventory-grapher -i inventory/generator.config all -a 'rankdir=LR;' -q | dot -Tpng | display png:-

Making The Most Of Inventory

2017-10-31T00:00:00+10:00

When using Ansible to consume APIs such as cloud services, the logic runs from the controller machine. As a result, people tend to think that as this runs locally, using hosts: localhost is the best option.

In reality, using localhost as your host loses you a lot of power. To avoid this, we can use a ‘runner’ pattern, where a runner is a local target on which you can hang inventory.

In our environment we tend to be either managing whole environments (e.g. provisioning an entire network) and infrastructure components within them (load balancers, database servies) or specific applications.

For environments, we might have a hierarchy that looks a little like:

[non_prod_account:children]
dev
test

[dev:children]
dev-runner

[test:children]
test-runner

[runner:children]
dev-runner
test-runner

or, in picture form:

That way, when we use hosts: test-runner we can pick up all the inventory associated with the test group - the CIDR prefix of networks, the tag to use for Environment, DNS zones, and many many other things (in reality a lot of our variables come from the all group but with the env property used in populating variables - so our VPC name is {{ env }}-{{ cidr_prefix }}.

To avoid needing to add connection: local to plays using runners, create a runner group vars file (e.g. inventory/group_vars/runners) and add ansible_connection: local.

With applications, and the right inventory structure, we can do things like create all the loadbalancers with:

- hosts: application:&{{ env }}:&runner
  connection: local

  roles:
  - loadbalancer

Each application group contains a (possibly empty) list of loadbalancers, each item in the list containing the configuration used when calling something like elb_application_lb module.

Using the runner pattern allows us to avoid a whole bunch of variable files, and structure inventory in a logical hierarchical fashion, making use of inheritance to minimise repetition of data.

One disadvantage of the runner pattern is that it creates a ridiculous combinatorial explosion of hosts and groups in inventory files. In my next post, I’ll show a very simple solution to that.

Oh, and ansible-inventory-grapher is currently working reasonably well against Ansible 2.4 at last (I need to tidy some magic variables from the host before it’s totally ready for release, but it’s on the ansible-2.4 branch if you’re keen)

So You Want To Test AWS Modules For Ansible

2017-07-17T00:00:00+10:00

This page was updated on 2018-02-28 to better document IAM policy changes, the aliases file, YAML anchors for testing credentials

You’re a (prospective) contributor to Ansible, and you have some great improvements to make to an existing module or a brand new module. As a conscientious developer, you know that having tests will ensure that you don’t break existing behaviour, and that other people’s future enhancements won’t break your desired behaviour. The standard tests for AWS modules are integration tests as most of them rely on creating some resources in AWS, updating them, and then cleaning up afterwards.

I’ll start this post from absolute first principles—I’ll use a shiny new Fedora 26 vagrant VM.

Setting up Ansible for development

vagrant init fedora/26-cloud-base
vagrant up
vagrant ssh

The easiest way to ensure we have all the dependencies for Ansible installed is then to use dnf to install ansible.

sudo dnf install ansible

Next, clone the your fork of the source code for ansible:

ssh-keygen
eval `ssh-agent -s`
ssh-add
# Add the generated key to your github account
sudo dnf install git
git clone git@github.com:YOURUSER/ansible
cd ansible
git remote add upstream https://github.com/ansible/ansible
git pull upstream devel

Note that if you’re doing this on a host with any other ssh keys, it’s worth generating a github specific ssh key (using e.g. ssh-keygen -f ~/.ssh/github) and setting up your .ssh/config appropriately:

Host github.com
  IdentityFile ~/.ssh/github

At this point, ansible --version should give the released version of Ansible (at the time of writing, this was 2.3.1.0). To use the development version (not recommended for production use), do:

source hacking/env-setup

And ansible --version should now show the development version (currently 2.4.0)

Install docker

Add docker and set it up so that you don’t have to be root to run it. There are some security implications with this—being in the docker group is effectively equivalent to root access.

sudo dnf install docker
sudo groupadd docker
sudo gpasswd -a $USER docker
sudo systemctl restart docker
newgrp docker

Setting up AWS account

Install the relevant command line tools (you can do this in a virtualenv if you prefer—it depends if you’d need to test with different versions of boto3 etc). The boto library is currently still required for quite a few modules, as well as the common code used to connect to AWS.

pip install --user botocore boto3 boto awscli

Ensure you have an IAM administrative user with API access. DO NOT use this user for Ansible, or put its credentials anywhere near your git repo! I use boto profiles for all of the AWS accounts that I use, and don’t have a default profile so that I always have to choose.

Note: DO NOT use your AWS root user—if you don’t have a suitable user yet, create a new IAM user with the AdministratorAccess managed policy (or similar) attached

Ensure a profile for this IAM user exists in ~/.aws/credentials and then run:

export ADMIN_PROFILE=$your_profile_name
ansible-playbook hacking/aws_config/setup-iam.yml -e iam_group=ansible_test -e profile=$ADMIN_PROFILE -e region=us-east-2 -vv

You don’t actually have to set profile or region at all if you don’t need them—region defaults to us-east-1, but you can only choose us-east-2 as an alternative at this time.

You’ll now need to go into AWS console, create an IAM user (called e.g. ansible_test) and make them a member of the newly created group (ansible_test if you used that with iam_group in While you’re there, create an API credential for the test user. This can all be automated:

aws iam create-user --user-name ansible_test --profile $ADMIN_PROFILE
aws iam add-user-to-group --user-name ansible_test --group_name ansible_test --profile $ADMIN_PROFILE
aws iam create-access-key --user-name ansible_test --profile $ADMIN_PROFILE

Note: This can be done through Ansible’s iam module too. It’s just simpler to document the steps with the CLI as they fit on three lines rather than one very long ansible adhoc command line or a larger ansible playbook.

Using the information from the new credential, you can do:

cp test/integration/cloud-config-aws.yml.template test/integration/cloud-config-aws.yml

and then update that file to include this new secret key and access key. It’s also worth adding the following for region if you’re not using us-east-1

aws_region: us-east-2
ec2_region: "{{ aws_region }}"

Note: The credentials in ~/.aws/credentials and cloud-config-aws.yml should be completely different. A good way to test this is:

aws sts get-caller-identity --profile=$ADMIN_PROFILE

ansible -m shell -a 'AWS_SECRET_ACCESS_KEY={{ aws_secret_key }} AWS_ACCESS_KEY_ID={{ aws_access_key }} aws sts get-caller-identity' \
  -e @test/integration/cloud-config-aws.yml localhost

The first of these should return your administrator user. The second of these should return your Ansible test user. If this isn’t the case, check the previous steps (and let me know if I can improve the documentation!).

Ensuring your user has the correct IAM policies

ansible-playbook hacking/aws_config/setup-iam.yml -e region=us-east-2 \
  -e profile=$ADMIN_PROFILE -e iam_group=ansible_test -vv

If you need additional IAM policies for your module, you’ll need to add these to a new or an existing policy file in hacking/aws_config/testing_policies. Note that there is a maximum of 10 policies per group, so we’re now using consolidated policies (e.g. compute, storage, network) etc. rather than a policy per service.

Adding the necessary policies here is a good way of documenting the new policies that you need, which will also help Ansible’s AWS account administrators add the policies needed by the CI account.

Running an integration test suite

Check that you can run the integration tests

ansible-test integration --docker -v ec2_group

The first run will be quite slow as you need to pull down all the required containers. Future runs will be much quicker, particularly if you pass --docker-no-pull to speed things up (it’s worth updating the containers fairly regularly, of course)

Writing your own tests

For your module, you will need:

test/integration/module_name/aliases
test/integration/module_name/tasks/main.yml

Previously, I said here that `meta/main.yml` was required, but it isn't—most of the AWS module test suites don't need to specify any dependencies at all.

aliases

cloud/aws
posix/ci/cloud/groupX/aws

Without the cloud/aws in the aliases file, the cloud-config-aws.yml doesn’t get picked up.

The second alias is used to parallelise the AWS integration tests across multiple test runners. There are currently five groups (group1 through group5)—just choose one - we can always rebalance if needed.

If you want to reuse a test suite for multiple modules (e.g. the aws_waf_web_acl suite also tests aws_waf_condition, aws_waf_rule and aws_waf_facts), add those other dependent modules into aliases as well.

tasks/main.yml

This should contain all of the tasks needed for the tests. Most actions are coupled with an assert task that the test has had the desired effect.

The main tasks should be in a block with an accompanying always that cleans up to avoid things being left behind by the tests. Clean up tasks should always ignore errors.

Before running any tasks, set up the account credentials in a reusable YAML anchor for use in each AWS task.

- block:
  - name: set up AWS credentials
    set_fact:
      aws_connection_info: &aws_connection_info
        aws_region: "{{ aws_region }}"
        aws_access_key: "{{ aws_access_key }}"
        aws_secret_key: "{{ aws_secret_key }}"
        security_token: "{{ security_token }}"
    no_log: yes

  - name: create resource
    aws_module:
      <<: *aws_connection_info
      state: present
      name: "{{ resource_prefix }}-more-name"
      ...
    register: aws_module_result

  - name: check that resource was created
    assert:
      that:
       &mdash;aws_module_result.changed
       &mdash;aws_module_result.name == "{{ resource_prefix }}-more-name"
    ...

- always:
  - name: remove resource
    aws_module:
      <<: *aws_connection_info
      state: absent
      name: "{{ resource_prefix }}-more-name"
      ...
    ignore_errors: yes

Things worth checking

Create a resource, assert that the returned properties are as expected
Run that task again, check that nothing changed
Update a resource, check that the returned properties are changed
Run that task again, check that nothing changed
If the task has purge_tags or similar, check that works as expected
Run the task again in check mode
Delete the resource

Troubleshooting

While docker is quite useful for quickly running tests and cleaning up after itself, occasionally you might need to get the debugger out to see why your module isn’t doing what you expect.

At this point, it’s likely easiest to create a quick test playbook

- hosts: localhost
  vars_files:
  - test/integration/cloud-config-aws.yml

  roles:
  - test/integration/targets/aws_module

And run this with ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook test-playbook.yml -vvv. You can then follow the ansible debugging instructions.

I tend to use epdb to put in a breakpoint where the module is going wrong (or before the module has gone wrong) and analyse from there.

So my steps are:

python /path/to/module explode
cd /path/to/debug_dir
Edit ansible_module_module_name.py, add import epdb; epdb.st() at the relevant location
python /path/to/module execute
python -c 'import epdb; epdb.connect()'

Tidy up

Detach the policies from your test IAM group—this will leave the test group and user in place but with zero privileges.

export GROUP=$your_iam_group
for policy_arn in `aws iam list-attached-group-policies --group-name $GROUP --profile $ADMIN_PROFILE --query 'AttachedPolicies[].PolicyArn' --output text`
do aws iam detach-group-policy --group-name $GROUP --policy-arn $policy_arn --profile $ADMIN_PROFILE
done

Shut down docker and vagrant

sudo systemctl stop docker
exit
vagrant halt

Conclusion

This post is designed to assist people with setting up the various steps needed to test and debug AWS modules for Ansible. There are almost certainly errors, omissions, and things that I could learn to do better.

If you have any suggestions for improvement, please raise an issue or PR on https://github.com/willthames/willthames.github.io or just let me know on Twitter or email (links below). Thanks!

Thanks

Thanks to Moritz Grimm for pointing out a couple of typos. These are now fixed.

An Introduction to Code Reviews

2016-11-07T21:30:00+10:00

Most software development teams have long been doing code reviews, and while it’s not uncommon amongst system administrators, it’s not universally practised.

Why do code reviews

Constructive code reviews are a means of ensuring the quality of code is consistent across the team – and typically all code gets raised to a much higher standard as a result. Through code reviews, team members can obtain feedback from their peers on their contributions before merge, as well as learn from the feedback on other colleagues’ contributions. Code review also promotes a shared understanding of the entire codebase amongst the team, making it easier for everyone to contribute. Additionally it provides visibility of changes to the codebase to everyone, which helps to avoid errors, and reduce wasteful or overly complicated code being added to the codebase.

When should code reviews happen

Code reviews should happen before any commit gets merged into a production codebase. Sometimes this principle needs to be bypassed (e.g. an emergency fix for production when no reviewer is available) but such changes should be still be reviewed retrospectively.

Ideally, code would undergo some kind of continuous integration testing prior to review. The sophistication of such testing can range from basic syntax checking up to full-scale test deployments. If it passes this automated checking, then it’s worth spending human effort on a review.

Where do code reviews happen

Most social version control platforms (Github, Bitbucket, etc.) have the ability to do Pull Requests (or Merge Requests in Gitlab’s case). This is the easiest (but not necessarily most robust) way to have them.

Dedicated code review platforms such as Gerrit, Crucible may also be used – it doesn’t really matter what tool you use as long as you are able to comment at the line level, at the general level, and provide some indicator of approval (e.g. a +1 or shipit)

Refer to the documentation for your version control platform, and perhaps your internal documentation for your change workflow, for how to submit reviews.

For the Reviewer: Reviewing the code

First of all, you have to understand the change you’re reviewing. If you don’t understand the change, you can’t positively review the change. A good commit message should contain the purpose of the change, and if necessary, how that change is being achieved. If, after reading the commit messages, you don’t understand what is going on, you should ask for the commit message to be improved.

When you understand the change, there are a few key levels of code review:

syntax – is the code well formatted, meeting indentation and whitespace standards and free from parse errors.
style – is the code idiomatic for the language, are common error-prone patterns for that language avoided, is the code easily understood
functional – does the code do what it’s supposed to do
architecture – is the code required, are there any improvements to be made through abstractions, reuse of other code. Does the code tie in with the rest of the codebase.

The first two of these are excellent candidates for automated checks – particularly as from a reviewer’s point of view, they’re really tedious to review, and from a reviewee’s point of view, they can feel like nitpicking. If the code has to meet such automated checks before it even gets to review, then the human element can be saved for the deep structural thought. ansible-review is an example of such a tool for Ansible; most languages and CM frameworks have similar tools.

Comment on specific lines of code if you can to say where the code doesn’t meet standards or could be improved. General feedback on the change as a whole can typically be provided as a comment without referencing a specific line.

Assume best intentions, and try and address the code rather than the person writing the code. Criticism should never be personal.

Code reviews should be objective where possible. There are always subjective preferences in any code base, but such preferences should be decided at a team level beforehand, and then be well documented – by pointing to such documentation in the code review, the feeling of subjectivity can be avoided. As you come across undocumented preferences, determine that they are what the team wish to use, and document them.

If you are satisfied that there are no blocking issues with the change, signify your approval in the appropriate way.

I prefer to let the code contributor accept the change if possible, in case there are any last minute issues that they notice. In some tools or under some permission schemes, this may not be allowed, and others may have to merge the result.

For the Reviewee: Prepare for code reviews

For a contributor:

ensure that your commit messages explain what you are trying to achieve and why.
adhere to the standards of your code base.
assume best intentions from the reviewer.
realise that a code review is not a battle, and try not to take criticism of your code personally. However, if criticism is personal, then you should say so.
try and reduce conflict resulting from misunderstandings – see if you can clear up such misunderstandings, either in the review, in the commit messages or through talking it through with the reviewer.

Getting started

If you don’t currently do code reviews, and you’re not using pull requests for contributing code, and you don’t have documented standards, all of the above might seem a little daunting.

Not having standards is a bit of a chicken and egg situation – without reviewing code, often preferences exist but aren’t expressed anywhere (some of our preferences have been implicit for years until a new contributor comes along and does something off the wall, and we realise it needs to be explicit).

One way to start might be to just ask a colleague to give you feedback on your recent commits. This might help to start discovering preferences, and then these can be documented. From there, you’ll likely find that code review tools provide a much easier way to provide feedback, because you can associate your comments with a line of code very easily.

In our global team of 20+ sysadmins, we actually use a code review process for improving our standards and best practices – all new standards must be accepted by at least two colleagues, and all best practice suggestions must get at least one +1. This is intended to ensure that no one feels that standards are imposed upon them. In a small co-located team a 2 minute chat might suffice instead!

Ansible Brisbane October 2016 talk

2016-10-25T21:30:00+10:00

At Ansible Brisbane October 2016, I gave a talk on Automating Ansible Code Reviews

DevOps Days Singapore Ansible Workshop

2016-10-09T20:00:00+10:00

My three hour workshop on Ansible from Zero to Best Practices