LOL Not really, but boy it has been a day. Started at 7:00 am and I finally resolved (?) the issue. In fact I’ve got through every last bit of my network, and at this point in the evening, I actually don’t have a solid reason why the issue was present. Something in my VPN settings glitched, or something got triggered on pFsense and got hung up…something, something with Tailscale. It wasn’t CLoudflare this time. LOL
You ever do so much to a problem that when you ‘fix’ it, you have no real idea what the fix truly was? You ever have a problem and find all the shit you cobbled together in the name of ‘just get it running and back online’? I did, and decided that I would fix that shit too. It took all flippin’ day.
You guys that do this for a living…I salute you! jebus crispies!
ETA: 8 bells and all’s well today.
Sometimes the fix has been done but the effect takes a while. For a cache to age out or a change to propogate. It all depends on what you are working with/on. Or you made a change but forgot to restart a specific service.
Meanwhile even though you did a fix correctly and aren’t aware of it, since it doesn’t seem to work you change something else and break it again inadvertently.
Note to self: Scan lab for hidden cameras
My mind went to this one
OK, here’s a somewhat famous case of email that could only be sent within something over 500 miles, but no further: https://web.mit.edu/jemorris/humor/500-miles
Story of my nerd life 😂
Man, I know that feeling. One thing that helped me better deal with issues like this, was to have a changelog. Basically I write down what a setting was, what I changed it to and a reason. If something goes wrong, I can at least undo what changes I’ve made and see if it helps. It’s not perfect, but it might shave some hours off a RCA.
This is what led me to set up SnipeIT and keep a log on every change on each device. It gets wild as the devices and services increase.
SnipeIT
I’ve seen the app rolling around in Awesome Lists, but always thought it was for larger operations with a lot of infrastructure and not necessarily a home lab. I might look into it seriously.
“So what was the problem in the end?”
“Man, I don’t fucking know.”
- me, every goddamn time
Me with Windows issues.
Was it a Windows update?
Was it the release upgrade (e.g. 24H2 -> 25H2)?
Was it the restart?
Was it a driver update?
Some program specific bug?And due to bad naming, convoluted or outdated documentation or fucky SEO from spam sites pretending to know the fix that did X or Y…
At least with Linux you sort of can find the specific fix in a release from the merge/commit notes.
It’s always DNS
Touch it until it works, then never again while it still does.
Me and the strange electrician type wiring they did to the cat5 Ethernet in the walls where i need to plug my fiber gateway into… 950MBs and no issues so far! First week was hell tho.
I hate, hate, hate when I fix something and I don’t know why the fix worked (or what the fix even was…). I want my suffering to result in something learned so it doesn’t happen again.
I want my suffering to result in something learned so it doesn’t happen again.
Soooo much this. I’m down to learn about technology any day of the week. But when I ‘fix’ something and I don’t know what the ‘fix’ really was, it is a rush of mixed emotions. I am ecstatic relieved the problem is fixed, but left empty not having learned the ‘why it broke’ in the first place. And then I’m always fearful that the problem will gestate in my lab and rear it’s ugly head again at some other inopportune time.
This. If I pay the cost in frustration and anguish and soul-searching and demanding justice from an uncaring god, I want some thing for it. I want documentation. I want my lessons learned from the post incident review. I want something I can hack into mgmtConfig to make sure nothing else will do that too.
Struggling for no payoff is the absolute worst thing.
How I fixed hardware acceleration on my NUC for Jellyfin.
My docs on how to install/enable HuC/GuC:
And to this day I am procrastinating upgrading my main docker host because I can’t be bothered to either inplace upgrade my Debian 10 (or 11) to a current release and then fix that I restart my whole journey and sacrifice a whole weekend on fixing my card house of a homelab/home infrastructure…
When you do it for work, you log what you have changed each time you make a change to try to fix it, and you log what you revert, so you can keep track of what you have tried, what worked, and what didn’t and have a clearer idea of what the solution was.
Sometimes it really does take a while to nail down though, and sometimes it isn’t entirely clear why what worked worked. Especially if you’re a junior network engineer without as much experience.
I made a self-hosted forgejo repository of /etc. Commit messages aren’t always informative, and I’ve never actually gone back to the repository to figure something out, but it’s there, just in case. Me cosplaying a sysadmin.
Yeah, same here ! Can’t believe how useful it is to have a git repo to keep track of changes, even as a non coder/sysadmin.
Simple pull/push commands and I’m now able to keep track of my bash scripts and specific .dot/config files.
To bad there isn’t a way to keep side notes a la Obsidian. Comments in the code are okay, but sometimes I wan’t to breakup the whole command with some notes to get a better understanding !
and we do it at work because it makes things easier in the long run!
great idea to do some of this stuff at home too.
Some issues are like that, you fix the issue presented and find more issues to fix.
I do it for a day job and have had issues that I can’t explain as anything more than restart a service and now it works, I also run my own kit and have had the same type of issue.
I had bizarre DNS issue I could not figure out. It ended up being that I turned on hardware routing/NAT on my OpenWRT box and then forgot about it
It’s always DNS 😂
Sometimes the problem isn’t on your end. The server might be having issues, or your ISP. Have to rule those out too.
I understand.
I learned again for the nth time that home assistant doesnt like refreshing my cert, and I can’t go to the site to refresh the cert unless it has a valid cert…
Maybe I’ll fix it tomorrow. It’s valid again now.
What’s holding you back from visiting the site to fix it? HSTS?
Why not use a reverse proxy?
When I set it up, I did not know better.
Now? Inertia. Nginx already does it for other things, I haven’t bothered to move home assistant over because home assistant works ( all but one days in a cycle)
“inertia” is a much nicer word than “complacency”.
I’ll be borrowing that.
Yeah I’ve got home assistant, a thing built for automating, but I leave that shit to certbot lol
