5 ms·

Traceroute doesn't see 90% of the machines your packet passes through. When your packet leaves a router at some-pop-some-port-wherever, that fiber isn't usuall

Traceroute doesn't see 90% of the machines your packet passes through. When your packet leaves a router at some-pop-some-port-wherever, that fiber isn't usually the same piece of glass that plugs into the next hop. There's a whole chain of amplifiers and possibly multiplexers that handle it between here and there. Some of those provide reliable transport service, giving you the illusion of a fiber that never breaks, despite backhoes doing what backhoes do. Some of those shift the wavelength of your signal, letting you use cheap optics without troubling in the nuances of DWDM that packs your signal alongside dozens of others onto the same long-haul fiber. Some of those just boost the signal, along with all those others on the same fiber. But what all those machines have in common, is that none of them speak IP. None of them touch the payload. None of them are capable of decrementing a hop count. They're "part of the wire" as far as the packet is concerned. In my experience, this leads to two types of network engineers, separated by their understanding of these underlying realities.

zamfi 2y ago
In traceroute’s defense, it is traceroute — for sure it doesn’t tell you anything about the devices that don’t operate at the IP level. Those devices either don’t affect the IP “route” abstraction (e.g., signal boosters) or do so in ways that end up plausibly visible in the next hop. There’s a reason the network layer abstraction is so strong, and an analogy to CPU ISAs here that have a similar strength. TCP, similarly, doesn’t tell you when packets are deduplicated/resent/reordered/etc. — that’s just not part of the presented abstraction. Want that? Use UDP.
- PeterCorless 2y ago
  "It doesn't show me the local digital loop carrier!" "Is the digital loop carrier doing any IP-level routing?" "No, but..."
latchkey 2y ago
> In my experience, this leads to two types of network engineers, separated by their understanding of these underlying realities. What's wrong with that? Certainly, someone with a complete picture is "better", but it is effectively two different types of problems. Do they need to be combined?
- aftbit 2y ago
  I think you're reading the connotation into this. In my opinion, there is nothing wrong with being a higher level engineer. There's plenty of work "above the fold" so to speak. For example, I think it behooves every software engineer to have a general grasp of how CPUs work, what speculative execution is, how CPU caching and invalidation works, etc, but the average webdev doesn't really need to know this, and might run into some abstraction breaking implications only a few times in their career, while debugging a tricky bug or performance regression. I imagine something similar is true for network engineers. Likely many can work for years at a time without worrying about fiber signal repeaters, other than that one weird packet loss issue that ends up getting traced back to a marginal optic in a cable vault somewhere. Of course, none of this applies to the compiler engineers or the people who build the physical network layer. They are in the "second type" of engineer that actually needs to understand this stuff in depth in order to do their jobs on a day to day basis.
  - latchkey 2y ago
    > I think you're reading the connotation into this. I am reading the connotation into this and asking about it. 4 paragraphs of talking only about tech and then it diverged into a personal statement at the end.
    - aftbit 2y ago
      What I meant by that statement is that whatever you're asking about is actually coming from you. I don't think the simple statement in the comment actually contains any connotation. Just like the old programmer joke - there are 10 kinds of people in the world, those who understand binary and those who do not. We just like dividing people into classes because it helps us understand the world. I guess to engage with this a bit more, one reason you might find a negative connotation towards engineers who are ignorant of the underlying details is that it is the larger set. After all, most engineers do not need to understand the physical layer in depth, so there are fewer who do. People love to feel like they're part of a smaller "higher" class, for complex social reasons. This sometimes comes with a bit of a distaste towards those who are part of the "regular" class. But overall, I think you're taking this further than it really needs to be taken. The GP was just saying that not all network engineers are wizards in the technical details of lower layers, just like not all software engineers can write a compiler. Does that make them "worse" in some way? Well it makes them a worse fit for that few specific jobs that require that extra knowledge, but in general, I don't think so, and I doubt most people do either. That's why we have abstractions.
RandomThoughts3 2y ago
The OSI model exists for a reason. You don't think about the life of the electrons going through your processors when you code. Traceroute is a view at a certain level of abstraction. It also doesn't tell you if your packet was delivered using ethernet, wifi or a token ring. It just doesn't matter.
- mecsred 2y ago
  It just doesn't matter until it does. It's fine to work at a higher level of abstraction. But people who understand a lower level of abstraction can do things people will call "impossible" with fault injection exploits, rowhammer etc.
- EvanAnderson 2y ago
  The more you know about how something works the better equipped you are to handle things breaking. It's a safe bet that semiconductor physics and the gate-level construction of CPUs isn't necessary to be a good programmer, but not much further up that stack are things like understanding superscalar processor architecture, how caches work, how CPU protection levels work, etc. Knowing about those things, for sufficiently performance or security-intensive applications, can make a ton of difference. There's an analogy to networking there, too. You don't necessarily need to know how wave-division multiplexing, BGP, or DNS work to communicate over the Internet. For some categories of problems, though, a little bit of knowledge allows you to punch just a bit above your level.
- treve 2y ago
  The OSI model hasn't been accurate representation of ip networking since pretty much day 1. It was made specifically for a different protocol, but in the stack we use today some layers are better split up in 2, some protocols exists in multiple layers. It's a nice metaphor but I think it's time to drop it! https://computer.rip/2021-03-27-the-actual-osi-model.html https://computer.rip/2021-03-27-the-actual-osi-model.html
  - w7 2y ago
    Something that I've noticed that somehow ends up lost when people learn "the model" is the encapsulation aspects. I don't know if it's missing in people's course work or what, but I've had to use http://www.tcpipguide.com/free/diagrams/ipencap.png http://www.tcpipguide.com/free/diagrams/ipencap.png many a times to explain how stuff like VPNs work, correct statements like "firewalls don't have a routing table, firewalling is layer 4", explain things like MTU and payload size, or why certain traffic doesn't go beyond a broadcast segment normally. Personally I think this is one of the better visualizations.
FuriouslyAdrift 2y ago
A lot of ISP internal networks is going to be running over MPLS or encapsulation. You're not going to see any of those hops. The packet will just look like it teleported from the CPE to the Internet.
Hikikomori 2y ago
Not sure why this matters for anyone not working for the ISP that those packets go through, and people working there would know about it. And they're mostly completely passive devices like filters. Traceroute is what we have, and its mostly enough as you can say at which hop things went wrong, then its up the ISP to figure out what devices they need to look at.
- mcmcmc 2y ago
  It definitely helps to understand it when you are trying to explain to the ISP that it’s their problem. Knowing what you’re talking about will get your issue resolved much more quickly.
  - groby_b 2y ago
    That depends who's doing the talking. You're a large business with a dedicated connection and your own network engineering team? Yeah, totally matters. You're a single consumer, or a small business? Your dialog is limited to "Yes, I switched the router on and off. Yes, my local network works. No, I can reach the website over my phone, this is your problem". Repeat three times, then get the inevitable reply of "it'll be up again in the next 24 hours". Unless you're a network engineer, this really doesn't matter.
  - Hikikomori 2y ago
    How does it help the ISP if you're just guessing what they might have between two routers?
yusyusyus 2y ago
as a predominantly wave-purchaser than builder, it matters when it matters and it doesn’t when it doesn’t. we get by fairly well assuming l1 is working as designed when the light is on and not throwing errors.. at least on day-to-day ops. planning/sourcing is a bit of a different thing. one note to your point though, back when people were still regularly picking up OC3s or doing frame relay or whatever, it was certainly more day-to-day to understand these things, but cant really fault anyone since most of that junk has gone away and we’re left with the happy ethernet handoff. especially in small scale DC/enterprise. cloud, too, made us care less.
juancb 2y ago
And on the active networking component side of things it doesn't touch on MPLS which also doesn't modify the IP headers. You can enter a network in New York and get MPLS switched across the country via active network devices all the way to California and have it show up as a single hop on traceroute. The explanation is great for a toy network bu in today's Internet the vast majority of routes are going to be asymmetrical and that requires running traceroutes from both ends and interpreting the results to find the faulty hop. The author also doesn't cover equal cost multipath (ECMP) which is everywhere. With ECMP you have multiple ports that lead to the same. Next hop and packets are hashed based on some part of the fourtuple, sometimes five tuple including the input Port. In order to track down the faulty link, you need to pro each and every one of the ports which requires that you use a higher level protocol like UDP. Using icmp in this case will not show you an issue some percent of time, providing false negatives which makes it less useful.
johannes1234321 2y ago
Below the IP layer there is always the physical layer somewhere with all the complications. Traceroute also doesn't show VPNs or other transports. But it shows the TCP hops and the timing, thus gives me the tools to see who to call or which routing table or wore to analyse or fix. When doing that I got to lok at that connection.
graycat 2y ago
Right, there is also the issue of BGP, Border Gateway Protocol for the core of the Internet?
icehawk 2y ago
The smart man knows all of these things exist. The wise man understands when these things matter. DWDM protected waves, MPLS, etc, are all out of scope when someone does a traceroute, finds their traffic is going to Europe asks me about it, and then I see a local upstream has picked up Cogent as a transit provider, and Cogent is preferring their customer routes over routes from their Tier 1 settlement free peers. But this article is very much a primer, it's using the unix format of traceroute and talking about ICMP.
- myself248 2y ago
  You don't know how many times I've seen people claim that they had two redundant connections, when those connections clearly rode the same conduit across a bridge not half a mile down the road. No, I don't care what your traceroute says, the backhoe knows otherwise.
begueradj 2y ago
Which command does go beyond the "traceroute" limitations you mentioned ?

zamfi 2y ago

In traceroute’s defense, it is traceroute — for sure it doesn’t tell you anything about the devices that don’t operate at the IP level. Those devices either don’t affect the IP “route” abstraction (e.g., signal boosters) or do so in ways that end up plausibly visible in the next hop. There’s a reason the network layer abstraction is so strong, and an analogy to CPU ISAs here that have a similar strength. TCP, similarly, doesn’t tell you when packets are deduplicated/resent/reordered/etc. — that’s just not part of the presented abstraction. Want that? Use UDP.

PeterCorless 2y ago
"It doesn't show me the local digital loop carrier!" "Is the digital loop carrier doing any IP-level routing?" "No, but..."

latchkey 2y ago

> In my experience, this leads to two types of network engineers, separated by their understanding of these underlying realities. What's wrong with that? Certainly, someone with a complete picture is "better", but it is effectively two different types of problems. Do they need to be combined?

aftbit 2y ago
I think you're reading the connotation into this. In my opinion, there is nothing wrong with being a higher level engineer. There's plenty of work "above the fold" so to speak. For example, I think it behooves every software engineer to have a general grasp of how CPUs work, what speculative execution is, how CPU caching and invalidation works, etc, but the average webdev doesn't really need to know this, and might run into some abstraction breaking implications only a few times in their career, while debugging a tricky bug or performance regression. I imagine something similar is true for network engineers. Likely many can work for years at a time without worrying about fiber signal repeaters, other than that one weird packet loss issue that ends up getting traced back to a marginal optic in a cable vault somewhere. Of course, none of this applies to the compiler engineers or the people who build the physical network layer. They are in the "second type" of engineer that actually needs to understand this stuff in depth in order to do their jobs on a day to day basis.
- latchkey 2y ago
  > I think you're reading the connotation into this. I am reading the connotation into this and asking about it. 4 paragraphs of talking only about tech and then it diverged into a personal statement at the end.
  - aftbit 2y ago
    What I meant by that statement is that whatever you're asking about is actually coming from you. I don't think the simple statement in the comment actually contains any connotation. Just like the old programmer joke - there are 10 kinds of people in the world, those who understand binary and those who do not. We just like dividing people into classes because it helps us understand the world. I guess to engage with this a bit more, one reason you might find a negative connotation towards engineers who are ignorant of the underlying details is that it is the larger set. After all, most engineers do not need to understand the physical layer in depth, so there are fewer who do. People love to feel like they're part of a smaller "higher" class, for complex social reasons. This sometimes comes with a bit of a distaste towards those who are part of the "regular" class. But overall, I think you're taking this further than it really needs to be taken. The GP was just saying that not all network engineers are wizards in the technical details of lower layers, just like not all software engineers can write a compiler. Does that make them "worse" in some way? Well it makes them a worse fit for that few specific jobs that require that extra knowledge, but in general, I don't think so, and I doubt most people do either. That's why we have abstractions.

RandomThoughts3 2y ago

The OSI model exists for a reason. You don't think about the life of the electrons going through your processors when you code. Traceroute is a view at a certain level of abstraction. It also doesn't tell you if your packet was delivered using ethernet, wifi or a token ring. It just doesn't matter.

mecsred 2y ago
It just doesn't matter until it does. It's fine to work at a higher level of abstraction. But people who understand a lower level of abstraction can do things people will call "impossible" with fault injection exploits, rowhammer etc.
EvanAnderson 2y ago
The more you know about how something works the better equipped you are to handle things breaking. It's a safe bet that semiconductor physics and the gate-level construction of CPUs isn't necessary to be a good programmer, but not much further up that stack are things like understanding superscalar processor architecture, how caches work, how CPU protection levels work, etc. Knowing about those things, for sufficiently performance or security-intensive applications, can make a ton of difference. There's an analogy to networking there, too. You don't necessarily need to know how wave-division multiplexing, BGP, or DNS work to communicate over the Internet. For some categories of problems, though, a little bit of knowledge allows you to punch just a bit above your level.
treve 2y ago
The OSI model hasn't been accurate representation of ip networking since pretty much day 1. It was made specifically for a different protocol, but in the stack we use today some layers are better split up in 2, some protocols exists in multiple layers. It's a nice metaphor but I think it's time to drop it! https://computer.rip/2021-03-27-the-actual-osi-model.html https://computer.rip/2021-03-27-the-actual-osi-model.html
- w7 2y ago
  Something that I've noticed that somehow ends up lost when people learn "the model" is the encapsulation aspects. I don't know if it's missing in people's course work or what, but I've had to use http://www.tcpipguide.com/free/diagrams/ipencap.png http://www.tcpipguide.com/free/diagrams/ipencap.png many a times to explain how stuff like VPNs work, correct statements like "firewalls don't have a routing table, firewalling is layer 4", explain things like MTU and payload size, or why certain traffic doesn't go beyond a broadcast segment normally. Personally I think this is one of the better visualizations.

FuriouslyAdrift 2y ago

A lot of ISP internal networks is going to be running over MPLS or encapsulation. You're not going to see any of those hops. The packet will just look like it teleported from the CPE to the Internet.

Hikikomori 2y ago

Not sure why this matters for anyone not working for the ISP that those packets go through, and people working there would know about it. And they're mostly completely passive devices like filters. Traceroute is what we have, and its mostly enough as you can say at which hop things went wrong, then its up the ISP to figure out what devices they need to look at.

mcmcmc 2y ago
It definitely helps to understand it when you are trying to explain to the ISP that it’s their problem. Knowing what you’re talking about will get your issue resolved much more quickly.
- groby_b 2y ago
  That depends who's doing the talking. You're a large business with a dedicated connection and your own network engineering team? Yeah, totally matters. You're a single consumer, or a small business? Your dialog is limited to "Yes, I switched the router on and off. Yes, my local network works. No, I can reach the website over my phone, this is your problem". Repeat three times, then get the inevitable reply of "it'll be up again in the next 24 hours". Unless you're a network engineer, this really doesn't matter.
- Hikikomori 2y ago
  How does it help the ISP if you're just guessing what they might have between two routers?

yusyusyus 2y ago

as a predominantly wave-purchaser than builder, it matters when it matters and it doesn’t when it doesn’t. we get by fairly well assuming l1 is working as designed when the light is on and not throwing errors.. at least on day-to-day ops. planning/sourcing is a bit of a different thing. one note to your point though, back when people were still regularly picking up OC3s or doing frame relay or whatever, it was certainly more day-to-day to understand these things, but cant really fault anyone since most of that junk has gone away and we’re left with the happy ethernet handoff. especially in small scale DC/enterprise. cloud, too, made us care less.

juancb 2y ago

And on the active networking component side of things it doesn't touch on MPLS which also doesn't modify the IP headers. You can enter a network in New York and get MPLS switched across the country via active network devices all the way to California and have it show up as a single hop on traceroute. The explanation is great for a toy network bu in today's Internet the vast majority of routes are going to be asymmetrical and that requires running traceroutes from both ends and interpreting the results to find the faulty hop. The author also doesn't cover equal cost multipath (ECMP) which is everywhere. With ECMP you have multiple ports that lead to the same. Next hop and packets are hashed based on some part of the fourtuple, sometimes five tuple including the input Port. In order to track down the faulty link, you need to pro each and every one of the ports which requires that you use a higher level protocol like UDP. Using icmp in this case will not show you an issue some percent of time, providing false negatives which makes it less useful.

johannes1234321 2y ago

Below the IP layer there is always the physical layer somewhere with all the complications. Traceroute also doesn't show VPNs or other transports. But it shows the TCP hops and the timing, thus gives me the tools to see who to call or which routing table or wore to analyse or fix. When doing that I got to lok at that connection.

graycat 2y ago

Right, there is also the issue of BGP, Border Gateway Protocol for the core of the Internet?

icehawk 2y ago

The smart man knows all of these things exist. The wise man understands when these things matter. DWDM protected waves, MPLS, etc, are all out of scope when someone does a traceroute, finds their traffic is going to Europe asks me about it, and then I see a local upstream has picked up Cogent as a transit provider, and Cogent is preferring their customer routes over routes from their Tier 1 settlement free peers. But this article is very much a primer, it's using the unix format of traceroute and talking about ICMP.

myself248 2y ago
You don't know how many times I've seen people claim that they had two redundant connections, when those connections clearly rode the same conduit across a bridge not half a mile down the road. No, I don't care what your traceroute says, the backhoe knows otherwise.

begueradj 2y ago

Which command does go beyond the "traceroute" limitations you mentioned ?