I’ve wanted to use exabgp along with dnsdist to provide anycasted, highly available recursive DNS servers ever since discovering dnsdist.
This is a fourth in blog series about DNS, specifically awesome things that can be done with dnsdist. This one was inspired by this blog post about exabgp and healthchecks, as well as this custom lua dnsdist load balancing policy.
Routing, Front and Back Ends
- BGP: AS395089 on router, AS65101 w/ exabgp/dnsdist load balancers
- 4x dnsdist load balancers as client facing DNS resolvers
- 4x PowerDNS recursor backends
- Google Public DNS as backup recursor backends
Active node is chosen using MED, lowest wins. This primary and secondary each mirror each others config, and then there is a 3rd on-net that runs both, and lastly one off-vm and off-net running a tunnel back.
Linux is funny about loopback type addresses. I was unable to get Debian to reliably bring up the interfaces I wanted so I ended up just putting them in
rc.local which is a hack, but it works reliably at least.
dnsdist binds to the two local IP addresses (v4 and v6) as well as all of the anycasted addresses (2x v4 and 2x v6). I am also using a custom weighted/tiered load balancer policy that chooses servers with the highest weight first, then the next level of weight if none are available from that weight tier, and so on. That load balancing policy is available here.
exabgp and healthchecks
One sample healthchecked entry:
The command= line is the check, anything which returns non-zero when it fails will work. This makes it very very easy to write a check. Disabling BGP advertisements is as simple as touching the file listed in the disable= line. This can be handy before reboots to take down BGP gracefully and fail over to other servers gracefully.
Experience so far
I’ve had weird issues with what appears to be next-hop and IPv6, which I’ve temporarily resolved by using the fe80 link-local addresses as the next-hop. For whatever reason this seems to fix things. This could be an issue specifically with Mikrotik BGP, and IPv6 next-hops. I’m tempted to add a Cisco router to the mix and peer with that and see if that makes things less strange.