Testing Travis CI workflows locally in Docker


#1

Habitat uses the free travis-ci.org service (rather than the paid travis-ci.com) one, we’re not able to easily log into a CI instance and debug travis-specific issues. Since spinning up new runs takes awhile and consumes limited resources, debugging travis locally is useful. Travis has documentation for running a container based docker image locally, but it’s rather sparse, so this document describes the rest of the details necessary for debugging habitat CI tests.

Prerequisites

  • Docker installed

Travis docker image setup

➤ docker run --privileged --name travis-debug -it -u travis travisci/ci-amethyst:packer-1512508255-986baf0 /bin/bash -l

The --privileged option is necessary for the studio to successfully bind mount directories such as /dev. The name travis-debug is arbitrary and the 6dd8f2496746 identifier is automatically generated. The choice of docker image was a best-guess for what matched the instance log output for existing travis builds for habitat. For example:

instance: travis-job-b2e18128-2f18-4c14-b371-f199e1b366aa travis-ci-amethyst-trusty-1512508224-986baf0 (via amqp)

The travis docker images can be found on Docker Hub.

Install travis-build tool

This is broken into chunks to indicate where the commands are expected to generate output.

travis@6dd8f2496746:~$ mkdir .travis
travis@6dd8f2496746:~$ cd .travis/
travis@6dd8f2496746:~/.travis$ git clone https://github.com/travis-ci/travis-build.git
travis@6dd8f2496746:~/.travis$ cd travis-build/
travis@6dd8f2496746:~/.travis/travis-build$ gem install travis

This next set of steps is necessary to work around a travis/support (LoadError) known issue.

travis@6dd8f2496746:~/.travis/travis-build$ bundle install
travis@6dd8f2496746:~/.travis/travis-build$ bundler add travis
travis@6dd8f2496746:~/.travis/travis-build$ bundler binstubs travis
travis@6dd8f2496746:~/.travis/travis-build$ cd

Set up habitat sources

travis@6dd8f2496746:~$ git clone https://github.com/habitat-sh/habitat.git habitat-sh/habitat
travis@6dd8f2496746:~$ cd habitat-sh/habitat/
travis@6dd8f2496746:~/habitat-sh/habitat$ git checkout branch/commit/to/test
  • Modify .travis.yml since matrix and env/global sections will be ignored
  • Remove branches since we’re running the tests manually
  • Move the contents of include to the top level since we’re explicitly choosing the configuration to run
  • Remove the openssl command which generates the builder-github-app.pem since our environment lacks $encrypted_f0fe831d6b31_key and $encrypted_f0fe831d6b31_iv (we’ll install this file manually)

For example:

branches:
  only:
    - master
    - /^sentinel.+$/
    - /^acceptance_deploy.+$/
    - /^test_development-.*$/
    - /^\d+\.\d+\.\d+$/

os: linux
env:
  global:
    - PATH=$HOME/.cargo/bin:$PATH
    # Habitat Rust program components
    - _RUST_HAB_BIN_COMPONENTS="components/airlock|components/hab|components/hab-butterfly|components/launcher|components/pkg-export-docker|components/pkg-export-kubernetes|components/pkg-export-helm|components/sup"

matrix:
  include:
    - language: rust
      env:
        - COMPONENTS=srv
        - AFFECTED_DIRS="Cargo\.lock|$_RUST_BLDR_BIN_COMPONENTS|$_RUST_BLDR_LIB_COMPONENTS"
      rust: stable
      sudo: required
      addons:
        apt:
          sources:
            - kalakris-cmake
          packages:
            - build-essential
      cache:
        apt: true
        cargo: true
        directories:
          - "$HOME/pkgs"
      before_install:
        - source ./support/ci/rust_env.sh
        - openssl aes-256-cbc -K $encrypted_f0fe831d6b31_key -iv $encrypted_f0fe831d6b31_iv -in ./support/ci/builder-github-app.pem.enc -out /tmp/builder-github-app.pem -d
      script:
        - ./test/builder-api/test.sh

becomes

os: linux

language: rust
env:
  - COMPONENTS=srv
  - AFFECTED_DIRS="Cargo\.lock|$_RUST_BLDR_BIN_COMPONENTS|$_RUST_BLDR_LIB_COMPONENTS"
rust: stable
sudo: required
addons:
  apt:
    sources:
      - kalakris-cmake
    packages:
      - build-essential
cache:
  apt: true
  cargo: true
  directories:
    - "$HOME/pkgs"
before_install:
  - source ./support/ci/rust_env.sh
script:
  - ./test/builder-api/test.sh

Use travis-build to turn .travis.yml into a bash script

travis@6dd8f2496746:~/habitat-sh/habitat$ ~/.travis/travis-build/bin/travis compile > ci.sh

Fix up ci.sh on account of our .travis.yml modifications:

  • Search for branch to find the clone command and add branch/commit/to/test like so:
    travis_cmd git\ clone\ --depth\=50\ --branch\=\'branch/commit/to/test\'\ https://github.com/habitat-sh/habitat.git\ habitat-sh/habitat --echo --retry --timing
  • Search for Setting environment variables from .travis.yml and add travis_cmd export commands for any env/global entries removed from the original .travis.yml. For example:
env:
  global:
    # Habitat Rust program components
    - _RUST_HAB_BIN_COMPONENTS="components/airlock|components/hab|components/hab-butterfly|components/launcher|components/pkg-export-docker|components/pkg-export-kubernetes|components/pkg-export-helm|components/sup"

becomes:

travis_cmd export\ _RUST_HAB_BIN_COMPONENTS\=\"components/airlock\|components/hab\|components/hab-butterfly\|components/launcher\|components/pkg-export-docker\|components/pkg-export-kubernetes\|components/sup\" --echo

Note that characters need to be escaped in the context of travis_cmd.

  • Create /tmp/builder-github-app.pem and/or /tmp/habitat-srv-admin from the credentials in 1Password to account for the openssl call that was removed
  • Optionally replace sudo apt-get update -qq 2>&1 >/dev/null with sudo apt-get update so progress of this slow command is observable

This would be a good time to take a snapshot of your container so that you can run the image from there while
iterating on the branch you’re trying to get passing tests:

docker commit --message="Travis before running ci.sh" 6dd8f2496746 travis-debug-snapshot:tag-name

Where 6dd8f2496746 should be the container ID that shows up in the command prompt (or in docker ps). travis-debug-snapshot and tag-name are arbitrary.

Run ci.sh

For the most readable output, just run bash ci.sh, but for debugging purposes bash -x ci.sh is often helpful. If there are failures, you can fix things up, push any necessary changes to your remote branch and then run ci.sh again. This diverges a bit from the true travis workflow since the workspace is no longer pristine, but it’s much faster to iterate on this way. Once you have things running the way you want, you can start over from the base travis image. Or, if you made a snapshot as suggested above, you can restart it:

➤ docker run --privileged --name=travis-debug-restore -it -u travis travis-debug-snapshot:tag-name /bin/bash -l

And then just run ci.sh again.