Continuous integration and deployment using Gitlab, Webhook and Ansible

jjpk.me

There is something particularly pleasant in being able to update a production application simply by running git push on one's workstation. I mean, there used to be a time when updating an app meant saving the code and updating the production environment over FTP! Sure, if you were a cool kid working with other cool kids, you used SVN or git in order to keep track of code changes, but for a long time, going from a repository update to a production update took time and required human intervention, especially since a lot of the web was using shared hosting and FTP was the only way to access your web root.

Nowadays however, anyone who mentions such an approach to production management will probably do so in a pub, just cracking jokes with the other devs after a long week of ticket closing. As a matter of fact, we have reached a point where it is possible to automate absolutely everything between a developer's push and the arrival of the release onto the production servers. In this article, I will be going through the approach I have chosen for this small website. For this recipe, you will need:

  • A Gitlab account for version control.
  • Ansible, an orchestration tool which can also easily be used for deployment scripting.
  • adnanh's webhook server to connect the previous two components.

An minimal Flask application

For this article, I will be using the minimal application used as an example on Flask's website.

Hello, World!

# helloworld.py
from flask import Flask

def hello_world():
    return 'Hello, World'

def create_app():
    new_app = Flask(__name__)
    new_app.add_url_rule('/', 'index', view_func=hello_world)
    return new_app

app = create_app()

This app has a single route, / which returns the text/plain response "Hello, World". You can save this into a Python file, and use the Flask development server to test it:

$ FLASK_APP=helloworld.py flask run

Note that in order for this to work, you will need to have the Flask Python package installed. Flask rightfully recommends the use of Python virtual environments when developing Flask apps, and you can read more about the installation process on their website. Once you have the server running, check out http://127.0.0.1:5000/ and you should see the plain Hello World.

Testing

In order to slightly complexify our integration process, let's also write tests for this application.

# tests.py
from helloworld import create_app
from flask_testing import TestCase
import unittest


class HelloWorldTest(TestCase):
    def create_app(self):
        app = create_app()
        app.config['TESTING'] = True
        return app

    def test_only_route(self):
        r = self.client.get('/')
        assert r.status_code == 200
        assert 'Hello, World' in str(r.data)


if __name__ == '__main__':
    unittest.main()

Here, we are using Python's unittest module along with Flask-Testing. Here's a quick summary of our dependencies and setup:

$ python3 -m virtualenv venv
$ source venv/bin/activate
$ pip3 install flask flask-testing

$ mkdir flaskapp # contains helloworld.py and tests.py
$ cd flaskapp
$ python3 tests.py # run the tests
$ FLASK_APP=helloworld.py flask run # run the development server

Version control and continuous integration

The next step will be to get our app's code on a git remote. Note that here, I am assuming the code is open. It is however absolutely possible to set up our pipeline for private repositories. For Gitlab, this can easily be done using Deploy Keys.

Setting up the repository

This is pretty straightforward. Let's create the repository, add the remote, commit and push. I'm using the URL for my own Flask app here.

$ cd flaskapp
$ git init
$ git remote add origin git@gitlab.com:jjpk-me/flask.git
$ git add .
$ git commit -m 'Bringing in the great Hello World feature.'
$ git push --set-upstream origin master

You should be able to see your changes on your repo's web page.

Continuous integration (CI)

CI is an umbrella term for the set of processes which an application undergoes to go from source code to production package. For a Node app, for example, this will include building the Javascript and CSS assets, among other things. In our case, all we need to do to prep our app for deployment is run the tests. When those fail, the app should obviously be held back and production should remain unchanged.

On GitLab, CI can be configured using a .gitlab-ci.yml file located at the root of your repository. GitLab offers an enormous number of features through this file, so feel free to check out their docs for more information. Here we will be keeping things simple: our CI pipeline has one job, test the app.

default:
  image: python:3

test:
  script:
    - python3 -m pip install flask flask-testing
    - PYTHONPATH=. python flaskapp/tests.py

CI jobs are run in containers, so the first block is where I choose the base image, Python 3. I then define a single job called test and the associated script. Note that the initial working directory for jobs is a copy of our git working tree, so we have access to our files. All I need to do here is install our dependencies and run the tests. If these exit with status 0, the pipeline succeeds and will so appear on your CI/CD dashboard.

You can now test your pipeline: commit the CI file and check your CI/CD dashboard. You should see a running pipeline with one job. If all goes well, you should see your pipeline go green after a few minutes.

$ cd flaskapp
$ git add .gitlab-ci.yml
$ git commit -m 'Adding CI.'
$ git push

Continuous deployment

There are a myriad of ways to handle deployment to production. The most basic approaches typically involve shell scripts: you define the process which downloads the latest release (or commit with a successful pipeline) and updates the production files. Every time a pipeline succeeds, you run this script and voilà!

Shell scripting, however, is pretty rudimentary. There are a lot of error and edge cases to consider, and that can end up taking a lot more time than you first expected. I don't think I can recall a case where I used shell scripts and did not end up with at least twice the line count I initially thought I would.

This is where Ansible comes in. Ansible is an orchestration tool which allows you to describe system setups and deployment processes in YAML files, called playbooks. The idea is: whenever the app updates upstream, this playbook should run to stop the app, pull the code and restart it all.

There are many ways to install Ansible, however chances are you will be able to find a binary package for your distribution. Once you have installed it, you should have a handful of Ansible executables available, including ansible-playbook, which we will be using.

The deployment playbook

For our simple app, deployment is quite straightforward. First, we suspend the app. Then, we make sure the git working tree we have on the production machine is up-to-date. Finally, we restart the app.

# flaskapp.book.yml
---
- hosts: 127.0.0.1
  connection: local
  tasks:

    - name: stop the Flask app
      systemd:
        name: flaskapp
        scope: user
        state: stopped

    - name: fetch the latest Flask app updates
      git:
        repo: git@gitlab.com:jjpk-me/flask.git
        dest: /home/flaskapp/app

    - name: restart the Flask app
      systemd:
        name: flaskapp
        scope: user
        state: started

Note that I am making a few assumptions here:

  • The Flask app has a dedicated UNIX account with a home directory at /home/flaskapp.
  • There is a lingering user systemd instance for this account.
  • The app is systemd-managed.

There are of course quite a few other ways to do things, but I thought this would keep things simple. You can always tweak it after all. Once you're satisfied with your playbook, you can attempt the initial deployment manually with:

$ ansible-playbook -Ki 127.0.0.1, flaskapp.book.yml # as the Flask app user

Deployment webhooks

Now, at this point, deployment is still nowhere near continuous. When the app is updated, nothing happens and I need to manually run the playbook as the Flask app user.

This is not what we want: we need automation. Here's how we are going to get there:

  1. We are going to expose a REST endpoint on the production server.
  2. Whenever this endpoint is queried, the playbook will run.
  3. We will set up a GitLab webhook to have it GET the endpoint after every successful pipeline.

For the REST endpoint, I wholeheartedly recommend adnanh's webhook server, an extremely convenient and flexible piece of Go software designed for just this purpose. Again, you should be able to find binary packages for it: check out the installation instructions here.

The webhook server is configured through hook descriptions: basically, a JSON file which describes the HTTP endpoint, a set of conditions to be met by every request and a command to be run when a valid request comes in. Here is the one we will be using.

/* flaskapp-hooks.json */
[
  {
    "id": "flaskapp",
    "execute-command": "ansible-playbook",
    "command-working-directory": "/home/flaskapp",
    "pass-arguments-to-command": [
      { "source": "string", "name": "-i" },
      { "source": "string", "name": "127.0.0.1," },
      { "source": "string", "name": "flaskapp.book.yml" }
    ],
    "response-message": "Redeploying the Flask app",
    "trigger-rule": {
      "and": [{
        "match": {
          "type": "value",
          "value": "*****",
          "parameter": {
            "source": "header",
            "name": "X-Gitlab-Token"
          }
        }
      },{
        "match": {
          "type": "value",
          "value": "success",
          "parameter": {
            "source": "payload",
            "name": "object_attributes.status"
          }
        }
      },{
        "match": {
          "type": "value",
          "value": "master",
          "parameter": {
            "source": "payload",
            "name": "object_attributes.ref"
          }
        }
      }]
    }
  }
]

Here, I define a single hook (endpoint /flaskapp) which runs the ansible-playbook executable with a few arguments. The -i 127.0.0.1 is necessary unless you've set up an Ansible inventory for the Flask app user. The trigger-rule entry defines the conditions to be met by a request for it to trigger the hook:

  • The request must come with a X-Gitlab-Token header value which matches a predefined secret (*****).
  • The request's JSON payload must have an object_attributes.status key with value success (the pipeline must have succeeded, GitLab will be sending us all sorts of pipeline events).
  • The request's JSON payload must have an object_attributes.ref key with value master (the pipeline must have run on the master branch, Gitlab will be sneding us events for all branches).

Now you can start your webhook server:

$ webhook -hooks flaskapp-hooks.json

The final stage in our process is to set up Gitlab: we need to have it send requests to our webhook server for pipeline events. On Gitlab, on your repository page, what we are looking for is in Settings > Webhooks. There, you can create a new hook with the URL to your webhook server. Remember to specify the secret token ***** otherwise your requests will be rejected and the playbook will never run! You can then check Pipeline events and save!

A minor change

To test our set up, let's make a small tweak to the Flask app:

$ sed 's/Hello, World/Hello, World!/' -i helloworld.py
$ git add helloworld.py
$ git commit -m 'Adding the missing and vital exclamation mark.'
$ git push

Once your push is complete, the following should happen:

  1. Gitlab will receive the push.
  2. CI will trigger, your app will be tested and the pipeline should succeed.
  3. Gitlab will send a GET request to your webhook server.
  4. The latter will recognise the secret token and run its command.
  5. The Ansible playbook begins to run: the app stops, updates and restarts.
  6. The update appears online!