Hardening habitat against upstream changes / failures


I’ve been using habitat for close to a year now to manage the build process for a couple of internal applications. The experience has been mostly good, however we’ve had problems with breaking changes in package dependencies during rebuilds and some problems with bldr being offline, etc. Of course, these frequently seem to happen when we’re trying to get a release out, so it’s become a high visibility problem even though it’s only happen half a dozen times or so. As a result, I’m exploring ways to harden our build process against those issues. I haven’t found anything really in the way of docs that covers best practices around this though, so I’m starting this as a place to collect ideas. In essence, what I’m trying to accomplish is the ability to perform builds reliably and consistently, even if all the habitat infrastructure is unreachable, and to only use updated versions of packages when I explicitly ask for it.

Here’s what I’m thinking so far:

  • Pin package versions - I generally don’t like doing this as it tends to create maintenance issues, and in habitat’s case in particular, I don’t know how it resolves transitive dependencies. I’m wondering if in the normal case this would actually cause more problems than it solves.
  • Cache all dependencies locally - My hope is that if a build has everything it needs it won’t attempt to download deps at build time. In limited experimentation, making this happen is non-obvious. Even if I install depedencies when I create my container that builds happen in, they seem to still be re-downloaded during builds.
  • Run a local bldr - Sort of an extension of the above, but this introduces quite a bit of overhead, so I’d rather not go there if possible.

Thoughts? Am I forgetting anything? Is “offline habitat building” already documented somewhere and I missed it?