3 ms·

There is so much abstraction going on now, debugging can be very, very challenging. Many systems are distributed and virtualized, so even trying to create a det

There is so much abstraction going on now, debugging can be very, very challenging. Many systems are distributed and virtualized, so even trying to create a detailed visualization of "what's going on" can require mind-boggling amounts of information. Also, even if you understand and agree with Werner Vogels' mantra "everything fails all the time", it's incredibly challenging to make a truly robust distributed system. There's just so much happening so rapidly, low-probability problems become consistent failures as you scale, and the wrong recovery approach can have non-obvious second-order effects leading to bigger problems.

dhosek 4y ago
My traditional way of understanding a program was to find main() (or it’s equivalent) and find my way through the code from there. In modern codebases, this doesn’t work at all. I do find it a bit amusing that younger devs find it magical that I can trace through code without running it and find bugs.
- atomicnumber3 4y ago
  Yeah, I agree, and I didn't even start all that long ago. For many programs, if you read main, all it does is kick off thread pools and then wait. And so to find the "entry point" you have to know how all the whatever (be it http or whatever) handlers work for this system. And it also feels like this kind of stuff is the first thing people end up DIYing / NIHing so even when languages or stacks have normal ways of doing things, somebody has always come along and muddied the water at some point and now you're stuck with it. Even UI is so damn hard these days because it's all JavaScript with a truly pitiful standard library and needs to handle 50 different screen sizes and browsers and run on both gaming desktops and 30 year old phones running leaked android pre-releases. I'll say it. I miss my days of programming in Java Swing. May GridBagLayout forever bless your pack()s.
- easton 4y ago
  > I do find it a bit amusing that younger devs find it magical that I can trace through code without running it and find bugs. I’m finding at work with the mass reliance on AWS stuff that this skill is super necessary for a lot of code thee days, because if you have to wait 10 minutes for your code to deploy before you can see if it’s right, you need to be able to see bugs before you run it.
- acuozzo 4y ago
  This is why the documentation I always write (especially in rushed cases in which I write no other documentation) is a "where do I start?" guide for newcomers to the codebase I'm leaving behind. I usually call it TREASURE_MAP so that it won't get skipped over like README is sometimes.
jiggawatts 4y ago
The thing that blew my mind is learning that retry loops paradoxically reduce reliability by introducing bi-stable scenarios where a working system at just over 50% load can suddenly enter a death spiral where retries push the load past 100%. This causes more errors, which cause more retries, and so on…
- mr_tristan 4y ago
  That's an excellent example. Mostly because I've probably had to replace simple retry loops more than once after they triggered various kinds of death spirals. Even recently, an engineer on my team spotted a retry technique, where the system tried to check if the DB's replication mechanism was slowing down, then would just re-enqueue the message for later. This ended up doing a bonkers amount of wasted effort because when DB replication started getting overwhelmed, messages would just constantly be re-enqueued and not ever processed. So the queue system ended up just having a bonkers amount of messages that were just popping off the queue, doing nothing, and hopping right back on.
- derangedHorse 4y ago
  Working at scale is truly a different ball game. A 1% yearly failure rate from a service serving around 1m users (a sizable but not huge number in today's age) will end up failing for 1000 different users on average. If the service is responsible for something like payments then 1000 different customers probably will end up calling support, support will have to identify and handle this anticipated failure mode, etc.. Sure error handling strategies like exponential backoff and microservice paradigms like "sagas" try to teach us more error-avoidant approaches, but sometimes it can be a lot to handle.

dhosek 4y ago

My traditional way of understanding a program was to find main() (or it’s equivalent) and find my way through the code from there. In modern codebases, this doesn’t work at all. I do find it a bit amusing that younger devs find it magical that I can trace through code without running it and find bugs.

atomicnumber3 4y ago
Yeah, I agree, and I didn't even start all that long ago. For many programs, if you read main, all it does is kick off thread pools and then wait. And so to find the "entry point" you have to know how all the whatever (be it http or whatever) handlers work for this system. And it also feels like this kind of stuff is the first thing people end up DIYing / NIHing so even when languages or stacks have normal ways of doing things, somebody has always come along and muddied the water at some point and now you're stuck with it. Even UI is so damn hard these days because it's all JavaScript with a truly pitiful standard library and needs to handle 50 different screen sizes and browsers and run on both gaming desktops and 30 year old phones running leaked android pre-releases. I'll say it. I miss my days of programming in Java Swing. May GridBagLayout forever bless your pack()s.
easton 4y ago
> I do find it a bit amusing that younger devs find it magical that I can trace through code without running it and find bugs. I’m finding at work with the mass reliance on AWS stuff that this skill is super necessary for a lot of code thee days, because if you have to wait 10 minutes for your code to deploy before you can see if it’s right, you need to be able to see bugs before you run it.
acuozzo 4y ago
This is why the documentation I always write (especially in rushed cases in which I write no other documentation) is a "where do I start?" guide for newcomers to the codebase I'm leaving behind. I usually call it TREASURE_MAP so that it won't get skipped over like README is sometimes.

jiggawatts 4y ago

The thing that blew my mind is learning that retry loops paradoxically reduce reliability by introducing bi-stable scenarios where a working system at just over 50% load can suddenly enter a death spiral where retries push the load past 100%. This causes more errors, which cause more retries, and so on…

mr_tristan 4y ago
That's an excellent example. Mostly because I've probably had to replace simple retry loops more than once after they triggered various kinds of death spirals. Even recently, an engineer on my team spotted a retry technique, where the system tried to check if the DB's replication mechanism was slowing down, then would just re-enqueue the message for later. This ended up doing a bonkers amount of wasted effort because when DB replication started getting overwhelmed, messages would just constantly be re-enqueued and not ever processed. So the queue system ended up just having a bonkers amount of messages that were just popping off the queue, doing nothing, and hopping right back on.
derangedHorse 4y ago
Working at scale is truly a different ball game. A 1% yearly failure rate from a service serving around 1m users (a sizable but not huge number in today's age) will end up failing for 1000 different users on average. If the service is responsible for something like payments then 1000 different customers probably will end up calling support, support will have to identify and handle this anticipated failure mode, etc.. Sure error handling strategies like exponential backoff and microservice paradigms like "sagas" try to teach us more error-avoidant approaches, but sometimes it can be a lot to handle.