[brainstorming] Building an "HA Loadbalancer" with Nginx and Habitat


#1

Hello,

I’m looking to proof out if there’s a way to replicate existing functionality we currently have in Habitat.

Once common use case I see is using Nginx as a load balancer/reverse proxy in HA mode with Keepalived. It is my hypothesis that using Habitat I can replicate this functionality and make overall configuration easier. But I’m having a bit of trouble getting there…

I started writing a keepalived plan some months ago, but hadn’t really continued as I felt it was replicating some Habitat functionality… (although thinking back, I think there was some issue with a kernel module…? maybe?)

The way we run it, is with two Nginx servers. My understanding is that Habitat requires at least 3 nodes running the same service to be able to form a consensus. I this could easily be overcome by running 3 Nginx nodes instead of the two that we typically run.

The next thing is the floating IP. From an Nginx configuration standpoint, I think just adding a “floating ip” configuration attribute, then writing the nginx config such that only the “leader” configures the floating IP to listen on would probably do the trick…

But where I’m stumped is “how do we tell the network where the FIP lives”? I’m a bit out of my wheelhouse here… It seems that Keepalived uses some implementation of VRRP which is maybe the secret sauce in how the “active” FIP is announced to the world.

Writing this out now (this is the third iteration of this post…) it occurs to me, that I don’t think Habitat has a way of announcing that floating IP, so really we would require the magic of Keepalived for that?

I dunno… maybe the answer is just to use Keepalived. DigitalOcean seems to use Corosync in place of Keepalived…?

Anyway, this is about where I am… I welcome any feedback, comments, questions, thoughts, ideas, insults, jokes, memes, or favorite ice cream flavors…

Thanks!
-Q


#2

Oh this is an interesting idea. I haven’t messed around with nginx HA mode so I would probably need to read up and think about this. I think it kind of also depends on what your goal is. Like, are you going to be running HA between 2x different datacenters with the intention of flipping between them in the case that one DC goes down? Or is it single DC multi-node proxies for in case one of the proxies goes down?


#3

Well… now THAT is an interesting idea… my initial thought was just multi-node proxies in one DC.

GSLB becomes a whole different problem because now we’re not just talking layer 3 and moving an IP between MAC addresses. (at least based on my understanding, IANANE… (Network Engineer))