4 ms·

The Case of the Connection Timeout

69 points by yarapavan 6y ago

yarapavan 6y ago
Read about the author’s motivations, design decisions, and plans at https://jvns.ca/blog/2021/04/16/notes-on-debugging-puzzles/ https://jvns.ca/blog/2021/04/16/notes-on-debugging-puzzles/.
barbazoo 6y ago
This is great, I actually had fun solving this one and I learned something too.
saagarjha 6y ago
The biggest takeaway here isn’t the specific network debugging skills used (though they are important, but you’ll notice you can still follow along if you don’t know how to use them). It’s learning that you can solve your own problems, and that there are little experiments you can run to debug issues even if you have no idea what is going on. This is basically teaching the scientific method to programmers, which I think is possibly the single most important skill they could have in their toolbox (computers are mostly deterministic, after all) and it is shocking how many engineers don’t have the ability or desire to use this tool.
- 908B64B197 6y ago
  > This is basically teaching the scientific method to programmers That's the difference between programmers and engineers.
mrlonglong 6y ago
I enjoyed that. Found the issue almost immediately !
crazydoggers 6y ago
Hmmm... but if I had know it was hosted on AWS at the beginning, the security group would have been the first thing I checked rather than curl or tcp dump, especially given the odd port. The syn packet isn’t exactly diagnostic... as is mentioned other things can drop syn packets. For instance if your host file was redirecting to a random IP which was firewalled... same issue. Context is everything, and knowing your system and stacks rather than single tools is important. What this shows me is that partitioning of knowledge where only certain team members understand certain things is the real danger.
- pixl97 6y ago
  Heh, ya, DNS and security groups issues are my number one fix for our AWS customers.
CSSer 6y ago
For those of you who enjoyed this, the author also created two others you can find at https://mysteries.wizardzines.com/ https://mysteries.wizardzines.com/ I really enjoyed "The Case of the DNS Update that Didn't Work" because I've definitely experienced that exact scenario on multiple occasions (no spoilers).
- bombcar 6y ago
  That’s a nice one but I’d add to remove the extraneous results in AWS lest they confuse something in the future. I always try to delete dns configs when moving servers.
LambdaComplex 6y ago
Whenever I face a problem like this, I try to narrow it down to a certain layer of the OSI model as quickly as possible. Is DNS resolution working? Yes, so we have an IP address to connect to. Can I ping that address, or is there a network connectivity problem? Am I able to establish a TCP connection to that address/port using telnet?
- pixl97 6y ago
  Yep. This page describes about half my day job. Our customers will run a windows application on a rather random port, and in the instructions we tell them to open firewall/AWS for it but half the time they forget. I have a pretty decent powershell script where they put their host name and port and the script will print out what it succeeds at and fails on.
WastedLife 6y ago
Neat project. I was tossed for a loop when the tcpdump displayed the loopback interface (pun intended).
- nucleardog 6y ago
  Yep, definitely made me think this was going down a totally different path for a minute.
xchaotic 6y ago
I’m such a nerd. I thought it’s the firewall. And sure enough it was a network rule blocking the traffic.
lazyweb 6y ago
Very nice! Great for junior sysadmins, or people wanting to get into that kind of work. Discovering that the target machine is hosted on AWS halfway through reminds me of previous gigs .. revealing essential information about the infrastructure along the way.
lmilcin 6y ago
Fun game and it kinda shows how this works. People are asking me how I debug issues and how I can seem to be able to always find the cause and I always had trouble explaining the process. I think it is a mix of being a detective and a scientist. Detectives work on clues about stuff that happened and usually can't do experiments and scientists are usually fixed on making predictions and repeatable experiments. Whereas most people who complain they can't debug problems like that try to treat them as if it was about knowing how to solve it. "How did you know it?", "How did you know it is this parameter?", etc. I think the two important abilities needed to be able to be good at debugging are: 1) be curious and comfortable to work with stuff you have no knowledge about -- learn as you go 2) have analytical approach to solving problems and especially understanding of logic
ryukafalz 6y ago
This is honestly brilliant and it has me thinking about trying something similar for documenting the troubleshooting process of internal tools. It's like sitting down with a knowledgeable coworker, but maybe that coworker doesn't work here anymore! Something like this can stick around.

69 points by yarapavan 6y ago

Read about the author’s motivations, design decisions, and plans at https://jvns.ca/blog/2021/04/16/notes-on-debugging-puzzles/ https://jvns.ca/blog/2021/04/16/notes-on-debugging-puzzles/.

barbazoo 6y ago

This is great, I actually had fun solving this one and I learned something too.

saagarjha 6y ago

The biggest takeaway here isn’t the specific network debugging skills used (though they are important, but you’ll notice you can still follow along if you don’t know how to use them). It’s learning that you can solve your own problems, and that there are little experiments you can run to debug issues even if you have no idea what is going on. This is basically teaching the scientific method to programmers, which I think is possibly the single most important skill they could have in their toolbox (computers are mostly deterministic, after all) and it is shocking how many engineers don’t have the ability or desire to use this tool.

908B64B197 6y ago
> This is basically teaching the scientific method to programmers That's the difference between programmers and engineers.

mrlonglong 6y ago

I enjoyed that. Found the issue almost immediately !

crazydoggers 6y ago

Hmmm... but if I had know it was hosted on AWS at the beginning, the security group would have been the first thing I checked rather than curl or tcp dump, especially given the odd port. The syn packet isn’t exactly diagnostic... as is mentioned other things can drop syn packets. For instance if your host file was redirecting to a random IP which was firewalled... same issue. Context is everything, and knowing your system and stacks rather than single tools is important. What this shows me is that partitioning of knowledge where only certain team members understand certain things is the real danger.

pixl97 6y ago
Heh, ya, DNS and security groups issues are my number one fix for our AWS customers.

CSSer 6y ago

For those of you who enjoyed this, the author also created two others you can find at https://mysteries.wizardzines.com/ https://mysteries.wizardzines.com/ I really enjoyed "The Case of the DNS Update that Didn't Work" because I've definitely experienced that exact scenario on multiple occasions (no spoilers).

bombcar 6y ago
That’s a nice one but I’d add to remove the extraneous results in AWS lest they confuse something in the future. I always try to delete dns configs when moving servers.

LambdaComplex 6y ago

Whenever I face a problem like this, I try to narrow it down to a certain layer of the OSI model as quickly as possible. Is DNS resolution working? Yes, so we have an IP address to connect to. Can I ping that address, or is there a network connectivity problem? Am I able to establish a TCP connection to that address/port using telnet?

pixl97 6y ago
Yep. This page describes about half my day job. Our customers will run a windows application on a rather random port, and in the instructions we tell them to open firewall/AWS for it but half the time they forget. I have a pretty decent powershell script where they put their host name and port and the script will print out what it succeeds at and fails on.

WastedLife 6y ago

Neat project. I was tossed for a loop when the tcpdump displayed the loopback interface (pun intended).

nucleardog 6y ago
Yep, definitely made me think this was going down a totally different path for a minute.

xchaotic 6y ago

I’m such a nerd. I thought it’s the firewall. And sure enough it was a network rule blocking the traffic.

lazyweb 6y ago

Very nice! Great for junior sysadmins, or people wanting to get into that kind of work. Discovering that the target machine is hosted on AWS halfway through reminds me of previous gigs .. revealing essential information about the infrastructure along the way.

lmilcin 6y ago

Fun game and it kinda shows how this works. People are asking me how I debug issues and how I can seem to be able to always find the cause and I always had trouble explaining the process. I think it is a mix of being a detective and a scientist. Detectives work on clues about stuff that happened and usually can't do experiments and scientists are usually fixed on making predictions and repeatable experiments. Whereas most people who complain they can't debug problems like that try to treat them as if it was about knowing how to solve it. "How did you know it?", "How did you know it is this parameter?", etc. I think the two important abilities needed to be able to be good at debugging are: 1) be curious and comfortable to work with stuff you have no knowledge about -- learn as you go 2) have analytical approach to solving problems and especially understanding of logic

ryukafalz 6y ago

This is honestly brilliant and it has me thinking about trying something similar for documenting the troubleshooting process of internal tools. It's like sitting down with a knowledgeable coworker, but maybe that coworker doesn't work here anymore! Something like this can stick around.