Soliciting Ideas for a revived Habitat Kubernetes Operator

We’re looking at reviving the old Habitat Kubernetes Operator and are looking for feedback on how to proceed.

For those of you using Habitat with Kubernetes, what are your biggest problems that an Operator could help with? If you used the old Habitat Operator, which of its capabilities would you like to see in a new one? Which design choices made in the old Habitat Operator would you prefer we left out of a new one?

Thanks!

I would love if I could just feed a codebase + plan path into a CRD object and leave it to the operator to deal with building and loading into a supervisor. Maybe with a “pod” notion like Deployments have so a set of services could be defined with affinity to run together and be a mix of built and downloaded packages, I could replace my habitat-compose pattern with something like that potentially

I’ve come to the conclusion that Habitat has almost no place in Kubernetes. I’ve been using them together for nearly two years and over that time period have used less and less of Habitat.

Here’s what’s left:

  • Packaging / dependency management
  • Config templating
  • user.toml watch+reload

The binding mechanism introduced an unacceptable amount of instability into my systems (entire clustered services having their config re-templated and getting reloaded/restarted if a single pod drops out and again when the new one spawns) and had to be removed before anything was prod-ready. The config and file gossip system were never useful and always ran orthogonal to the (much better) ConfigMap/Secret mechanism.

I’m still struggling with the inability to have some config template changes restart the service and some not.

At this point, I would recommend against adoption of Habitat for any greenfield project targeting Kubernetes deployment. It is still an extremely valuable tool for migration from traditional VM deployments into containers and Kubernetes, but I’d encourage elimination of the useful bridging features as soon as possible post-migration for the reasons outlined above.

That being said, please see the issues I filed against the operator while I still used it. All of these are still valid: https://github.com/habitat-sh/habitat-operator/issues/created_by/HT154

1 Like

I agree with @ht154 on vanilla Habitat not being a great fit for Kubernetes. What might be cool is if Habitat could export your habitat plan into a CRD, and subscribe for application updates.

This is not fully thought out, but in Kubernetes, the k8s control plane should handle application lifecycle management. It does this via a custom CRD per service. If someone could write a plan in Habitat, export it to a CRD, and then the Habitat Operator is only responsible for seeing when the Habitat packages & CRDs have updates and pulling those down and coordinating an update (blue/green, rolling, etc.) using Kubernetes, that could be cool. This sticks with the Habitat “build it once, run it anywhere” without getting in Kubernetes’ way.

1 Like

Hi @tashadrew! :wave:

Yup, that’s exactly the approach that I’m looking at right now. Ultimately, I’d like to find a good way to decompose the functionality of the Supervisor in such a way that lets you pick and choose as appropriate for your platform. If you’re running on bare metal, let the Supervisor handle everything for you; if you’re in Kubernetes, on the other hand, let those capabilities be handled in a native Kubernetes way. Upgrades seemed to be a nice starting point.

2 Likes

Very cool! I can definitely see a lot of value in the Habitat build system, cross platforms.

1 Like