I have a small homelab that’s not nice enough for /r/homelab but is a bit more than just self hosting. Since I’m a decently knowledgeable sysadmin and network engineer, my goal is to build an enterprise-ish environment for myself to tinker around and play inside. This means a lot of my setup is more complicated than it needs to be and I spend a lot of time troubleshooting and debugging my overengineering, so when something breaks my first assumption is that it was something I did. I usually build my stuff to be relatively aelf sufficient when I leave it alone.
But this weekend and today I simply couldn’t find what I broke. I was attempting to move a clunky lets encrypt cert renewal job off of my DNS server to somewhere I could better manage it. Why was it on my DNS server? Because for a while now, dynamic updates only half worked for me. My bind9 server was fully capable and I have a custom nsupdate cronjob to update my DDNS records that I installed on my UDM-Pro. But for whatever reason, as soon as I entered my home network1 it wouldn’t work. Since I thought it better to manage my certs from Proxmox or another internal service, I needed to figure out why this was. I looked high, I looked low, I looked in /etc but there was no configuration error that I could find. I tested the same TSIG key on another machine in my VPC and on my UDM-Pro but there it went without a hitch. The error was weird — NOTIMP — and I couldn’t find anything relevant online. As a last resort I turned to ChatGPT2, but all this confirmed was that there should be no errors with my configuration. It’s conclusion was that it had to be networking.
So i scoured the configuration of my UDM looking for any filtering or traffic rules I had, but nothing was clicking. This wasn’t a connection issue, this is the server telling me that updates were not allowed for this zone. I was clearly hitting the DNS server, right? Well there was nothing in the update logs on the server, so I suspected that for some reason the requests weren’t making it through. So I spun up wireshark on my UDM and on my DNS server, and saw for myself that the dynamic update requests weren’t even reaching the bind server. I would see the update come into the router, and a response from the bind server, so what was responding? This was either some crazy filtering from my ISP — which i knew to be false because updates from the router worked — or my UDM doing something. Finally after some sleep I came back and looked at the UDM cobsole again and it hit me.
Ad block.
I quickly paused it and lo and behold it was blocking my dynamic updates. There was no record of this in the Insights tab; it was just silently absorbing my dynamic updates and masquerading as my name server. I can understand masquerading as name servers due to what its supposed to do, but I have no idea why it would steal my dynamic updates. I wouldn’t think what DNS filtering that enables is fail closed. For being a prosumer company, Ubiquiti’s features always feel halfway implemented to work in most scenarios but never actually developing full support for things. Yes, I brought this onto myself for enabling ad-blocking (it was good while it lasted, I’ll have to reimplement it in a non stupis way) but the fact that it does zero inspection of the DNS opcode before forwarding requests feels dumb.
1I have two “sites”, my homelab and a cloud VPC; critical infra like DNS and mail is hosted in the VPC.
2I minimally use AI for troubleshooting as a last resort to either turn me on a new path to the solution or as a sanity check before I blame a different component.
I fr hate using AI to troubleshoot because I can feel how it makes me lazy, but sometimes using AI is better than banging my head against a wall for 10 hours. And usually i stop once I find a productive line of research or investigation to follow.
Like google and stack overflow before it AI is a tool. If you use any of these and stop researching after the tool gives you an answer, researching to understand why the answer works in the first place, then any of these tools will make you lazy.
But you are human and it is imposible to understand everything so choose your battles.
Also, dude, your setup sounds sick as hell and this is a fantastic writeup. Thank you.
Thank you, it’s a lot of work and I could get by with a lot less but I’d like to essentially have enterprise level everything for me to just fuck around with and provide to friends as i see fit. It’s a bit if a hodgepodge of well implemented stuff stuck together with duct tape and bubblegum but im refining it slowly all the time.
Meh…it’s a tool that needs some heavy regulation, but a tool nonetheless.