11 ms·

Quick fix for an early Internet problem lives on a quarter-century later (2015)

95 points by uncle_stupid 10y ago

johansch 10y ago
I get the feeling that the core Internet protocols (in the 80s and early 90s) were built by people who were like 75% sysadmins and 25% developers/system architects. Typically sysadmins employed by US universities. (I don't want to cast any shade on their accomplishments! They made brilliant things for the time.) Then things got stuck because the Internet started hyper-scaling in the mid 90s. This is just an outside observation though; it would obviously be great to hear any insider perspectives from people who were there.
- stonogo 10y ago
  That divide did not exist at the time. Intimate knowledge of the systems involved was a requirement to do either task. It wasn't until later that people were able to choose the set of abstractions that enabled them to focus on a task the the point that other considerations could be missed. People wrote entire bespoke operating systems to serve as early arpanet nodes. The arrival of unix was bemoaned by many because it removed this requirement for mastery -- this gave rise to the era of copy-and-paste systems administration and works-on-my-machine development. We're still digging our way out of that hole, by implementing more layers of abstraction. Hence the current popularity of cloud computing: the ultimate copy-and-paste deployment, combined with works-on-my-machine development, where all machines can now be 'my machine'.
  - johansch 10y ago
    You were around in the 80s Internet scene?
    - teh_klev 10y ago
      Some of us are a lot older than than you might think :)
      - johansch 10y ago
        You seem to have misinterpreted my simple question as statement of disbelief? Why would you do that, since I explicitly asked for statements of people who where around at the time? confused (Perhaps I misread you - in that case: thanks to the grand-parent comment for sharing :) ).
- slv77 10y ago
  It sounds like you're suggesting that a lot of system admin/developer types that were left in a room without adult supervision and, oops, the Internet! The truth was that there was no lack of experienced architect/developer types at the time working on networking standards. The OSI model, for example, traces its roots back to 1977! The struggle was that the "adults in the room" were typically were older and held senior positions in large corporations and government agencies. The standards that they created were typically developed through a consensus protocol and by committee where the single biggest aim was to make sure that no company would gain or lose a competitive advantage based on the standard. These standards were always pitched as being just over the horizon and, if anything, did a lot of early damage by delaying rollout of simpler protocols. I personally won't ever get back the hours that I spent studying protocols that were never relevant and never would be.
  - johansch 10y ago
    Thank you for sharing!
- gist 10y ago
  Forgetting that security wasn't designed into it the fact that it was an honor system (email as well) was just plain naive on the part of the academics.
  - mywittyname 10y ago
    Or it was a compromise that wasn't worth it. We could make cars with bullet-proof glass that would make them extremely difficult to break into, but to most people, the cost is not worth the benefit, so it's not done. Given the state of encryption technology in the 80s, it's possible that any attempts to secure the protocol would have been rendered pointless by technological advances. Or, it would be more brittle than a non-secure competitor and would have fallen out of use.
- forgottenpass 10y ago
  I don't want to cast any shade on their accomplishments! Yeah, considering the gigantic backhanded compliment that follows, you do. You're just saying "sorry not sorry."
  - johansch 10y ago
    I beg to disagree. I do think that what was designed back then (TCP, DNS, etc) was elegant. It was a great foundation.. but because they were so elegant and good-enough, and because of the Internet hyperscaling in the 90s and onwards, there was never an opportunity to improve on it.
dmit 10y ago
Similarly, UTF-8 was designed on a diner placemat. https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt
- Asooka 10y ago
  UTF-8 was also designed under rather severe restrictions: you want to keep binary compatibility with 7-bit ASCII, i.e. 128 out of 256 possible byte values are taken and you want to encode the entirety of Unicode including any future additions. This means you need variable-length encoding. You also want to detect if the stream is well-formed and you want to detect if you're starting decoding in the middle of a character. UTF-8 is one of very few possible solutions that cover all requirements. BGP had a lot more to work with since you're designing an entire protocol that doesn't need any backwards compatibility with anything else.
  - barrkel 10y ago
    > You also want to detect if the stream is well-formed and you want to detect if you're starting decoding in the middle of a character These are positive characteristics of the UTF-8 design, but weren't necessary pre-requisites. Other encodings don't have the same redundancy (for integrity check) or freedom from boundary ambiguity.
- fisherjeff 10y ago
  I've also always enjoyed that El Torito[0] was named after the Mexican restaurant where it was designed: "Stevens and Merkin went out to lunch at the El Torito Grill. Stan Merkin had calamari fajitas and Curtis Stevens had steak fajitas, both with fresh tortillas. They wrote the basics of the specification on a napkin. The rest, as they say, is history."[1] [0] https://en.wikipedia.org/wiki/El_Torito_(CD-ROM_standard) https://en.wikipedia.org/wiki/El_Torito_(CD-ROM_standard) [1] https://web.archive.org/web/19991008045553/http://www.cdpage.com/Compact_Disc_Variations/danaboot.html https://web.archive.org/web/19991008045553/http://www.cdpage...
no_protocol 10y ago
> BGP won out because it was simple, solved the problem at hand and proved versatile enough to keep data flowing as the Internet doubled in size, again and again and again. So if you want to replace it, make something that is equally simple and solves the new problem(s) at hand without un-solving the original one. If you don't do that, people have no incentive to switch. It sounds like the (vaguely described) new alternative is more complex. If it was better for business, things would have gravitated toward it quickly.
- klodolph 10y ago
  It's unreasonable to demand that a new solution be "equally simple". History is full of simple solutions that aren't good enough that get replaced with more comprehensive yet complicated solutions. HTTPS isn't as simple as HTTP, C++ isn't as simple as C, democracy isn't as simple as monarchy, the courts aren't as simple as mob justice.
  - no_protocol 10y ago
    > HTTPS isn't as simple as HTTP, C++ isn't as simple as C, democracy isn't as simple as monarchy Right, none of those have led to obsolescence even with long periods of time to grow. Instead, think of replacements that have caused something to basically just disappear. I'm not convinced this is the best example, but I think it helps get my point across. This is from a user's perspective, not a manufacturer's. Maybe someone can share better examples from software if these aren't appropriate. When a new type of cable connector enters the market, it has to be at least as simple as the old one or do something more than the old one did. Serial or parallel plugs have screws to help hold the cable in place, newer plug types like FireWire or USB don't need the screws, so they are simpler. Now we are seeing new types of plugs that work "upside down" so they are even simpler for consumers to use. I guess a better example might be media distribution. From a user's perspective, the compact cassette was just as simple to use as the 8-track. Or BluRay replacing DVD replacing CD. Would high speed internet be as popular if it required a time consuming arcane ritual to establish a connection, compared to the simple bleep bloop of a dial-up modem? So if you want the users of BGP (network operators) to switch to something better, make it just as simple for them to use, while providing additional benefits. Yeah it isn't easy, but it's certainly been done in other arenas.
TranceMan 10y ago
BGP at 18 from the guy himself: https://m.youtube.com/watch?v=_Mn4kKVBdaM https://m.youtube.com/watch?v=_Mn4kKVBdaM
witty_username 10y ago
Why isn't the sensitive data sent encrypted?
- detaro 10y ago
  The issue is not that a third party can read contents, it is that a participant in the protocol (which could encrypt/decrypt) can do malicious things, so encryption wouldn't do anything. Newer proposals include signatures and other steps to solve this, but as the article discusses uptake is slow and it makes operations more complicated. The weakness of BGP against hijacking also means that it's quite simple and very flexible, which operators like.
  - witty_username 10y ago
    So, why would the US military care if sensitive data is being routed through Beijing?
    - detaro 10y ago
      (I misunderstood your question and thought you referred to BGP protocol data, which is why I mostly answered about BGP, but...) The attacker could have just thrown the data away, totally interrupting all these communications. Even if encrypted, traffic patterns could be interesting and harder to obtain otherwise.
detaro 10y ago
from 2015, discussion here back then: https://news.ycombinator.com/item?id=9636141 https://news.ycombinator.com/item?id=9636141
analogwzrd 10y ago
There's a motto in my department at work: Nothing is more permanent than a temporary solution.
- jobu 10y ago
  I've heard a similar quote: "Broken code gets fixed, but shitty code lasts forever."
- outworlder 10y ago
  Yes, I learned this pretty early in my career, from a (technically literate) client. "I've added a temporary solution for X" "These temporary solutions are usually permanent" Sure enough. The temporary workaround was eventually deployed. There's another one that I've learned, loosely translated to English it would be "The Symmetrical Workaround Theory". It states that workarounds usually come in pairs. If you fail to figure out where the other half of the workaround goes, a bug will take its place.
deleted 10y ago
[deleted]
Animats 10y ago
BGP was intended for a specific niche. Earlier, there was the ARPANET, operated by BBN out of Boston, and peripheral stuff connected to it. Early Internet thinking was that there was a "core" with its own routing protocol, and BGP was how the core communicated with small peripheral networks. BGP wasn't intended as the core routing protocol. Mutually mistrustful routing is hard. I wrote some stuff on this in the early 1980s. The best I came up with was a scheme where forgeries would be detected, and once the source of the forgeries had been kicked off the network, the system would heal itself. (Asymmetric cryptographic signing had been invented but wasn't yet used or trusted.)
vocatus_gate 10y ago
"Temporary is permanent." "Test is production." These two mantras have saved me and coworkers from making stupid "temporary" fixes countless times through the years.
vocatus_gate 10y ago
"Temporary is permanent." "Test is production." These two mantras have saved me and coworkers from making stupid "temporary" fixes countless times through the years.
cryptarch 10y ago
Related project: http://www.scion-architecture.net/ http://www.scion-architecture.net/ SCION is a BGP replacement that's in development at the ETH Zurich network security group; it promises isolation of routing hijack attacks (by means of so-called "isolation domains", planned to be grouped by jurisdiction AFAIK). It's even already deployed in a bunch of places (edit: https://www.scion-architecture.net/#deployment https://www.scion-architecture.net/#deployment). It also has cool extensions like SIBRA, which provides DDoS protection by means of bandwith reservation, and HORNET (which you might've heard of before), which is a high-speed anonymous communication network (which is sadly not yet available to the public in implemented/working form). Previous HN discussion on HORNET: https://news.ycombinator.com/item?id=9930929 https://news.ycombinator.com/item?id=9930929
d33 10y ago
A related, interesting read: https://security.stackexchange.com/q/56069/15648-what-security-mechanisms-are-used-in-bgp-and-why-do-they-fail https://security.stackexchange.com/q/56069/15648-what-securi...
swehner 10y ago
IPv6?

95 points by uncle_stupid 10y ago

I get the feeling that the core Internet protocols (in the 80s and early 90s) were built by people who were like 75% sysadmins and 25% developers/system architects. Typically sysadmins employed by US universities. (I don't want to cast any shade on their accomplishments! They made brilliant things for the time.) Then things got stuck because the Internet started hyper-scaling in the mid 90s. This is just an outside observation though; it would obviously be great to hear any insider perspectives from people who were there.

stonogo 10y ago
That divide did not exist at the time. Intimate knowledge of the systems involved was a requirement to do either task. It wasn't until later that people were able to choose the set of abstractions that enabled them to focus on a task the the point that other considerations could be missed. People wrote entire bespoke operating systems to serve as early arpanet nodes. The arrival of unix was bemoaned by many because it removed this requirement for mastery -- this gave rise to the era of copy-and-paste systems administration and works-on-my-machine development. We're still digging our way out of that hole, by implementing more layers of abstraction. Hence the current popularity of cloud computing: the ultimate copy-and-paste deployment, combined with works-on-my-machine development, where all machines can now be 'my machine'.
- johansch 10y ago
  You were around in the 80s Internet scene?
  - teh_klev 10y ago
    Some of us are a lot older than than you might think :)
    - johansch 10y ago
      You seem to have misinterpreted my simple question as statement of disbelief? Why would you do that, since I explicitly asked for statements of people who where around at the time? confused (Perhaps I misread you - in that case: thanks to the grand-parent comment for sharing :) ).
slv77 10y ago
It sounds like you're suggesting that a lot of system admin/developer types that were left in a room without adult supervision and, oops, the Internet! The truth was that there was no lack of experienced architect/developer types at the time working on networking standards. The OSI model, for example, traces its roots back to 1977! The struggle was that the "adults in the room" were typically were older and held senior positions in large corporations and government agencies. The standards that they created were typically developed through a consensus protocol and by committee where the single biggest aim was to make sure that no company would gain or lose a competitive advantage based on the standard. These standards were always pitched as being just over the horizon and, if anything, did a lot of early damage by delaying rollout of simpler protocols. I personally won't ever get back the hours that I spent studying protocols that were never relevant and never would be.
- johansch 10y ago
  Thank you for sharing!
gist 10y ago
Forgetting that security wasn't designed into it the fact that it was an honor system (email as well) was just plain naive on the part of the academics.
- mywittyname 10y ago
  Or it was a compromise that wasn't worth it. We could make cars with bullet-proof glass that would make them extremely difficult to break into, but to most people, the cost is not worth the benefit, so it's not done. Given the state of encryption technology in the 80s, it's possible that any attempts to secure the protocol would have been rendered pointless by technological advances. Or, it would be more brittle than a non-secure competitor and would have fallen out of use.
forgottenpass 10y ago
I don't want to cast any shade on their accomplishments! Yeah, considering the gigantic backhanded compliment that follows, you do. You're just saying "sorry not sorry."
- johansch 10y ago
  I beg to disagree. I do think that what was designed back then (TCP, DNS, etc) was elegant. It was a great foundation.. but because they were so elegant and good-enough, and because of the Internet hyperscaling in the 90s and onwards, there was never an opportunity to improve on it.

dmit 10y ago

Similarly, UTF-8 was designed on a diner placemat. https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt

Asooka 10y ago
UTF-8 was also designed under rather severe restrictions: you want to keep binary compatibility with 7-bit ASCII, i.e. 128 out of 256 possible byte values are taken and you want to encode the entirety of Unicode including any future additions. This means you need variable-length encoding. You also want to detect if the stream is well-formed and you want to detect if you're starting decoding in the middle of a character. UTF-8 is one of very few possible solutions that cover all requirements. BGP had a lot more to work with since you're designing an entire protocol that doesn't need any backwards compatibility with anything else.
- barrkel 10y ago
  > You also want to detect if the stream is well-formed and you want to detect if you're starting decoding in the middle of a character These are positive characteristics of the UTF-8 design, but weren't necessary pre-requisites. Other encodings don't have the same redundancy (for integrity check) or freedom from boundary ambiguity.
fisherjeff 10y ago
I've also always enjoyed that El Torito[0] was named after the Mexican restaurant where it was designed: "Stevens and Merkin went out to lunch at the El Torito Grill. Stan Merkin had calamari fajitas and Curtis Stevens had steak fajitas, both with fresh tortillas. They wrote the basics of the specification on a napkin. The rest, as they say, is history."[1] [0] https://en.wikipedia.org/wiki/El_Torito_(CD-ROM_standard) https://en.wikipedia.org/wiki/El_Torito_(CD-ROM_standard) [1] https://web.archive.org/web/19991008045553/http://www.cdpage.com/Compact_Disc_Variations/danaboot.html https://web.archive.org/web/19991008045553/http://www.cdpage...

no_protocol 10y ago

> BGP won out because it was simple, solved the problem at hand and proved versatile enough to keep data flowing as the Internet doubled in size, again and again and again. So if you want to replace it, make something that is equally simple and solves the new problem(s) at hand without un-solving the original one. If you don't do that, people have no incentive to switch. It sounds like the (vaguely described) new alternative is more complex. If it was better for business, things would have gravitated toward it quickly.

klodolph 10y ago
It's unreasonable to demand that a new solution be "equally simple". History is full of simple solutions that aren't good enough that get replaced with more comprehensive yet complicated solutions. HTTPS isn't as simple as HTTP, C++ isn't as simple as C, democracy isn't as simple as monarchy, the courts aren't as simple as mob justice.
- no_protocol 10y ago
  > HTTPS isn't as simple as HTTP, C++ isn't as simple as C, democracy isn't as simple as monarchy Right, none of those have led to obsolescence even with long periods of time to grow. Instead, think of replacements that have caused something to basically just disappear. I'm not convinced this is the best example, but I think it helps get my point across. This is from a user's perspective, not a manufacturer's. Maybe someone can share better examples from software if these aren't appropriate. When a new type of cable connector enters the market, it has to be at least as simple as the old one or do something more than the old one did. Serial or parallel plugs have screws to help hold the cable in place, newer plug types like FireWire or USB don't need the screws, so they are simpler. Now we are seeing new types of plugs that work "upside down" so they are even simpler for consumers to use. I guess a better example might be media distribution. From a user's perspective, the compact cassette was just as simple to use as the 8-track. Or BluRay replacing DVD replacing CD. Would high speed internet be as popular if it required a time consuming arcane ritual to establish a connection, compared to the simple bleep bloop of a dial-up modem? So if you want the users of BGP (network operators) to switch to something better, make it just as simple for them to use, while providing additional benefits. Yeah it isn't easy, but it's certainly been done in other arenas.

TranceMan 10y ago

BGP at 18 from the guy himself: https://m.youtube.com/watch?v=_Mn4kKVBdaM https://m.youtube.com/watch?v=_Mn4kKVBdaM

witty_username 10y ago

Why isn't the sensitive data sent encrypted?

detaro 10y ago
The issue is not that a third party can read contents, it is that a participant in the protocol (which could encrypt/decrypt) can do malicious things, so encryption wouldn't do anything. Newer proposals include signatures and other steps to solve this, but as the article discusses uptake is slow and it makes operations more complicated. The weakness of BGP against hijacking also means that it's quite simple and very flexible, which operators like.
- witty_username 10y ago
  So, why would the US military care if sensitive data is being routed through Beijing?
  - detaro 10y ago
    (I misunderstood your question and thought you referred to BGP protocol data, which is why I mostly answered about BGP, but...) The attacker could have just thrown the data away, totally interrupting all these communications. Even if encrypted, traffic patterns could be interesting and harder to obtain otherwise.

detaro 10y ago

from 2015, discussion here back then: https://news.ycombinator.com/item?id=9636141 https://news.ycombinator.com/item?id=9636141

analogwzrd 10y ago

There's a motto in my department at work: Nothing is more permanent than a temporary solution.

jobu 10y ago
I've heard a similar quote: "Broken code gets fixed, but shitty code lasts forever."
outworlder 10y ago
Yes, I learned this pretty early in my career, from a (technically literate) client. "I've added a temporary solution for X" "These temporary solutions are usually permanent" Sure enough. The temporary workaround was eventually deployed. There's another one that I've learned, loosely translated to English it would be "The Symmetrical Workaround Theory". It states that workarounds usually come in pairs. If you fail to figure out where the other half of the workaround goes, a bug will take its place.

deleted 10y ago

[deleted]

Animats 10y ago

BGP was intended for a specific niche. Earlier, there was the ARPANET, operated by BBN out of Boston, and peripheral stuff connected to it. Early Internet thinking was that there was a "core" with its own routing protocol, and BGP was how the core communicated with small peripheral networks. BGP wasn't intended as the core routing protocol. Mutually mistrustful routing is hard. I wrote some stuff on this in the early 1980s. The best I came up with was a scheme where forgeries would be detected, and once the source of the forgeries had been kicked off the network, the system would heal itself. (Asymmetric cryptographic signing had been invented but wasn't yet used or trusted.)

vocatus_gate 10y ago

"Temporary is permanent." "Test is production." These two mantras have saved me and coworkers from making stupid "temporary" fixes countless times through the years.

vocatus_gate 10y ago

"Temporary is permanent." "Test is production." These two mantras have saved me and coworkers from making stupid "temporary" fixes countless times through the years.

cryptarch 10y ago

Related project: http://www.scion-architecture.net/ http://www.scion-architecture.net/ SCION is a BGP replacement that's in development at the ETH Zurich network security group; it promises isolation of routing hijack attacks (by means of so-called "isolation domains", planned to be grouped by jurisdiction AFAIK). It's even already deployed in a bunch of places (edit: https://www.scion-architecture.net/#deployment https://www.scion-architecture.net/#deployment). It also has cool extensions like SIBRA, which provides DDoS protection by means of bandwith reservation, and HORNET (which you might've heard of before), which is a high-speed anonymous communication network (which is sadly not yet available to the public in implemented/working form). Previous HN discussion on HORNET: https://news.ycombinator.com/item?id=9930929 https://news.ycombinator.com/item?id=9930929

d33 10y ago

A related, interesting read: https://security.stackexchange.com/q/56069/15648-what-security-mechanisms-are-used-in-bgp-and-why-do-they-fail https://security.stackexchange.com/q/56069/15648-what-securi...

swehner 10y ago

IPv6?