3 ms·
> For that matter, the whole port-scanning agent/aggr combination shouldn't have needed to exist in theory, but in practice, independent verification was needed
by tetha 3y ago
> For that matter, the whole port-scanning agent/aggr combination shouldn't have needed to exist in theory, but in practice, independent verification was needed.
This is very much tickling me as one of the truths about operational work. I can't count how often I have said something like "Yes, technically we don't need all of that.... but..." Systems tend to become weird in production if many people use them.
Like we have a central component at work and most things have good client side load balancing and retries in place. Thus we decided against adding a server-side load balancer - more software and systems on the path are slower and more pieces that could fail.
Except, as we found out during an outage, ~2-3 critical requests into this central component in a third party component... aren't and just fail hard if they hit a bad node. And then everything escalates into a mess very quickly.
So now we have both kinds of load balancing. Isn't that wonderful?
- eadmund 3y ago> I can't count how often I have said something like "Yes, technically we don't need all of that.... but..." Systems tend to become weird in production if many people use them. Minor note (of the literally/figuratively type): shouldn’t that be ‘yes, theoretically we don’t need all of that, but technically [or “practically”] we do’? I think that ‘technically’ connotes actual correctness.
- tetha 3y agoI just wrote a longer piece, but deleted it because I realized what I was thinking. I think this is the right way around, because I lean towards the "technically correct" side here. "Technically correct" always carries a bit of an ivory tower snobby attitude to me. After all, the tech I'm working with has proven and working client side load balancing and retries, so there is no server side load balancing necessary. That's technically correct if you observe the system in isolation. And it's also correct that the bad software interfacing with it should fix their implementation. It's just not correct in practice if terrible software, implementation bugs, or all manner of things beyond a simple, correct implementation start interfacing with the system. Then even a technically or clinically or theoretically correct system will fall apart because there's mud in places where mud shouldn't be able to get to. After all, a patch in a month won't help me with a preventable outage tomorrow.
- bippihippi1 3y agothat's one of the top rules in programming. check your own errors, even if it probably can't happen because it will.
- bombcar 3y agoIt's the old saying - "the difference between theory and practice is in theory there's no difference, in practice there is." We refer to "unexpected critical things" as "load bearing flasks" - you never know they're load bearing until you get ye flask.
- chatmasta 3y agoTreating your systems like a black box is a good way to make them more robust and discover issues that you otherwise wouldn't notice with internal monitoring that bypasses them via shortcuts. It's also why bug bounties are so effective at uncovering bugs - external observers bring a new perspective.