5 ms·

The case of the 500-mile email (2002)

246 points by libeclipse 9y ago

sdoering 9y ago
Already really often times submitted. Last time was (I believe) 24 days ago: https://news.ycombinator.com/from?site=ibiblio.org https://news.ycombinator.com/from?site=ibiblio.org [Edit] Typo
- sjcsjc 9y ago
  I love this every time I see it. Newcomers may not have seen it so reposting doesn't seem a problem. And 24 days ago it was totally ignored in the new submission lottery, so again reposting doesn't seem a problem.
  - stefanpie 9y ago
    It was my first time reading about it; it was fantastic.
    - cyphar 9y ago
      If you haven't read it before, there's also "The Story of Mel" which is quite a bit older (1983) but is also a very fun read. http://www.catb.org/jargon/html/story-of-mel.html http://www.catb.org/jargon/html/story-of-mel.html
  - ceejayoz 9y ago
    Yeah, I'll never not upvote this one.
- Confiks 9y ago
  I actually looked up this story myself yesterday, because it was mentioned by the top comment of this thread: https://news.ycombinator.com/item?id=14672174 https://news.ycombinator.com/item?id=14672174 I wonder if the current OP posted it for the same reason.
  - libeclipse 9y ago
    Yep! Hadn't seen it before and bet a few others probably wouldn't have either.
    - dudus 9y ago
      That can be said about any hn story.
- okket 9y ago
  It is a rather popular story, indeed: https://hn.algolia.com/?query=The%20case%20of%20the%20500-mile%20email&sort=byDate&dateRange=all&type=story&storyText=false&prefix&page=0 https://hn.algolia.com/?query=The%20case%20of%20the%20500-mi...
jessaustin 9y ago
Statisticians make really good bug reports!
- DamonHD 9y ago
  Maybe in this case, with n=1, but we'd need a better sampling of impoverished departmental heads to support your thesis.
js2 9y ago
I've read this story dozens of times and this is the first time SunOS jumped out at me. Around 2001 or so there was a Solaris release (2.7 maybe?) that had a bug in a particular patch release where sleep() returned instantly. That would have exhibited all sorts of interesting symptoms such as this story.
- shakna 9y ago
  I had the joy of re-engineering a terrible in-house web server because of that bug. It had used the system sleep to pause execution between the fake threads in their awful program. Suddenly everything was hitting race problems, instead of about 10%. Not something I really understood why it was going so bad, so I just dumped the inhouse implementation of threads, and took up the Solaris one, which was a joy by comparison.
  - mikeash 9y ago
    Sleep() returns early if a signal is delivered to the program, so doesn't your code need to handle this case anyway? Of course, this is super rare, so incorrect code could survive for a long time on a correctly implemented system.
    - corprew 9y ago
      this is the point of that terrible 'threading' strategy, actually, if i guess what they are doing correctly.
    - geocar 9y ago
      You should always handle your errors, but people trust in exceptions and "what works" often enough that we have a culture that doesn't. http://geocar.sdf1.org/close.html http://geocar.sdf1.org/close.html
krylon 9y ago
Reading this kind of debugging story is kinda like watching a horror movie. It's very interesting and fun to read if you know the kind of problems you have to deal with are tame and furry in comparison. But if I actually had to troubleshoot a problem like this, it would give me nightmares for weeks.
nashashmi 9y ago
A lengthier discussion: https://news.ycombinator.com/item?id=9338708 https://news.ycombinator.com/item?id=9338708
alekratz 9y ago
> And I was fairly certain I hadn't enabled the "FAIL_MAIL_OVER_500_MILES" option. If this actually was a config option, I would bet anything the default would inexplicably be set to "true".
- komali2 9y ago
  It's like this weird setting in Ubuntu's network driver config file, "11n_disable." I won't pretend to know much about it but every time I install Ubuntu on a machine, wifi internet speeds are slow until I set the above to "8". To me it seems like literally a "go faster" switch.
  - rincebrain 9y ago
    Only 8, and not 1? Because, at least on my system, looking at iwlwifi... parm: 11n_disable:disable 11n functionality, bitmap: 1: full, 2: disable agg TX, 4: disable agg RX, 8 enable agg TX (uint) So 8 is only aggregated TX, while 1 would be aggregated TX and RX. (It does, however, also inexplicably default to 0 on my system.)
    - komali2 9y ago
      Oh neat, lemme try that out then.
      - rincebrain 9y ago
        I'd be curious to learn what it does for you, because my intuitive interpretation would say that 1 would be "disable all", but that doesn't explain why "enable agg TX" would be an improvement unless agg RX is worse than the baseline...which is probably a bug. e: Oh, apparently this is a pathological re-use of an existing module flag to force something on that they disabled due to instability in some cases, see also: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=205e2210daa975d92ace485a65a31ccc4077fe1a https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... So I'm sorry, but my assumption about =1 is not valid, as I assumed that all of the bits (1,2,4,8) were mutating the same thing, not "one bit does something VERY DIFFERENT".
danschumann 9y ago
"And I was fairly certain I hadn't enabled the "FAIL_MAIL_OVER_500_MILES" option." I laughed.
stevefeinstein 9y ago
TIL: units is a really useful utility command that I had forgotten even existed.
- late2part 9y ago
  You have: 100 millilightseconds unknown unit 'millilightseconds' You have: ^D Apparently I need to add more units to achieve this level of usefulness.
  - parraccourci 9y ago
    try "3 ms"
  - dmacedo 9y ago
    Something tells me you're on OS X, and like myself, you'll find you mostly want everything replaced with GNU binaries: $ brew install gnu-units $ gunits Also see: https://apple.stackexchange.com/questions/69223/how-to-replace-mac-os-x-utilities-with-gnu-core-utilities https://apple.stackexchange.com/questions/69223/how-to-repla...
  - emilfihlman 9y ago
    Well user@server:~$ units -V GNU Units version 2.11 with readline, with utf8, locale en_US Units data file is '/usr/share/units/definitions.units' Personal units data file is '/home/user/.units' (file does not exist) Copyright (C) 2014 Free Software Foundation, Inc. GNU Units comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GNU Units under the terms of the GNU General Public License. user@server:~$ units Currency exchange rates from www.timegenie.com on 2014-04-02 2866 units, 109 prefixes, 79 nonlinear units You have: 3millilightseconds You want: miles * 558.84719 / 0.0017893979 You have: user@server:~$ Works for me.
- shubhamjain 9y ago
  It does induce a geeky joy but it's ephemeral for the most part as Google is vastly superior in usefulness. I learned about units few years back, used it for a while, and I can't even remember when but I just started Googling the conversions instead of switching to the terminal window.
curtis 9y ago
The 500-mile email story seems to be the Hacker News equivalent to Reddit's SR-71 speed check story [1]. It keeps coming back again and again because everybody loves it. [1] https://www.reddit.com/r/SR71/comments/2dpmw7/the_sr71_speed_check_story/ https://www.reddit.com/r/SR71/comments/2dpmw7/the_sr71_speed... (and many others, too numerous to reference)
- thearn4 9y ago
  The SR-71 story never fails to put a smile on my face.
- okket 9y ago
  https://xkcd.com/1053/ https://xkcd.com/1053/
  - rconti 9y ago
    This new XKCD comic is quite amusing! I read back through several of them and they're so true to my life experiences!
- amarraja 9y ago
  Thanks, first I've read that and it certainly put a smile on my face. One of the first articles I remember reading on HN was about an SR-71 [1], and it seems it's by the same pilot because Walter is his crewmate. [1] http://sploid.gizmodo.com/5511236/the-thrill-of-flying-the-sr-71-blackbird http://sploid.gizmodo.com/5511236/the-thrill-of-flying-the-s...
- mercer 9y ago
  Has anyone made a collection of the most re-posted HN articles?
wmichelin 9y ago
I'm a simple man, I see the 500 mile email story, I upvote.
reaperducer 9y ago
Stories like this remind me of the days when the internet was young, and fun, and silly, and best experienced in its green-on-black usenet glory. (418 Error: I'm a teapot, for example.) The internet just doesn't seem fun anymore.
- chrissnell 9y ago
  It's not that the Internet has lost its fun--it's just too easy to find it these days. You appreciated everything a little more when you had to get on your bike and ride a mile across campus to the campus computing center just to read that silly USENET post on a 80x24 serial terminal. Now you can read it on your watch or your glasses or yell at your home stereo to read it to you. I used to wonder what it would be like to time-travel to ancient times and show a gun or a car to a Neanderthal. I think it would be just as crazy to show 2017 me to 1993 me.
- Piskvorrr 9y ago
  It was me who approved the merge request replacing our 418 with a boring 400 a few months ago; I'm sorry. (On the other hand, there's still a server out there returning a redirect 303 Sea Otter, so I guess it balances out)
throwaway-emc2 9y ago
[throwaway account] A couple of years ago, my company (SEA) was bought by a private equity-based company that was assembling a portfolio of similar companies. The IT group got taken over by a competent yet overworked group in their main office (PHX.) The remote IT group wanted to move everyone's computer to their windows VMs that were in their colo in PHX, which were managed (and presumably logged) out of a central management panel. From their point of view, this was a great solution. I was trying to get them to put a separate VM bank up locally, because of the speed of light delay (at least 15ms) that would be small but annoying. I had convinced the remote IT people that this would be a good idea, and also the [PHX] CTO apparently thought it was a good idea, but my local 'location boss,' who used to run our company decided that I was just doing it to increase my power. (which was untrue, because I would be going back to just ENG and not IT after the transition.) So, my boss cancels it and throws me under the bus to management at our new home office. It takes a couple of weeks, then the terminals that log into the remote hosts get installed. The speed of light is one of the constants of the universe, so the problem I mention immediately happens. My location boss gets annoyed by the slow performance on his computer and, remembering that I'd said something about slowness, storms into my office and starts demanding that I fix the computers. I try and calm him down and tell him that there's no way to fix this without moving the terminals & vms closer together. He calls up the remote IT folks and they agree with me. He, at his next trip down to HQ, starts slagging on me to anyone who'd listen. Various people at the new home office warn me about this, but there's really nothing I can do about it. After some hilarity, I'm transferred to a project that was designed to fail so there can be paperwork justifying laying me off. So that is another way the speed of light can have real-world consequences.
orangepenguin 9y ago
I thoroughly enjoyed this story, but wonder if the details have changed a bit over time, perhaps? If light can travel ~558 miles in 3 milliseconds, it seems that mail sent more than 250 miles would have failed (because RTT)? Also, I'm not sure how big a difference it makes, but signals propagate at less than the "speed of light" (vacuum) when the medium is copper.
yoav_hollander 9y ago
Fun story - never heard it before. BTW, I think information travels over "normal" networking gear (e.g. fiber optics) at about 70% of the speed of light (which is why high-frequency trading tends to move to over-the-air microwave networking to shave a few nanoseconds - see [1]). But I guess the story still works, at the resolution it is told. [1] https://en.wikipedia.org/wiki/High-frequency_trading https://en.wikipedia.org/wiki/High-frequency_trading
YesThatTom2 9y ago
I was a friend of Trey back then and I remember him explaining this and thinking "wow, that's some good debugging!" You'd be amazed at how many people tell Trey that this story is fake. Ugh. There were plenty of witnesses. Trey made news recently during the flood of negative United Airlines stories: http://www.newnownext.com/how-united-airlines-almost-ruined-a-gay-couples-honeymoon/04/2017/ http://www.newnownext.com/how-united-airlines-almost-ruined-...

246 points by libeclipse 9y ago

Already really often times submitted. Last time was (I believe) 24 days ago: https://news.ycombinator.com/from?site=ibiblio.org https://news.ycombinator.com/from?site=ibiblio.org [Edit] Typo

sjcsjc 9y ago
I love this every time I see it. Newcomers may not have seen it so reposting doesn't seem a problem. And 24 days ago it was totally ignored in the new submission lottery, so again reposting doesn't seem a problem.
- stefanpie 9y ago
  It was my first time reading about it; it was fantastic.
  - cyphar 9y ago
    If you haven't read it before, there's also "The Story of Mel" which is quite a bit older (1983) but is also a very fun read. http://www.catb.org/jargon/html/story-of-mel.html http://www.catb.org/jargon/html/story-of-mel.html
- ceejayoz 9y ago
  Yeah, I'll never not upvote this one.
Confiks 9y ago
I actually looked up this story myself yesterday, because it was mentioned by the top comment of this thread: https://news.ycombinator.com/item?id=14672174 https://news.ycombinator.com/item?id=14672174 I wonder if the current OP posted it for the same reason.
- libeclipse 9y ago
  Yep! Hadn't seen it before and bet a few others probably wouldn't have either.
  - dudus 9y ago
    That can be said about any hn story.
okket 9y ago
It is a rather popular story, indeed: https://hn.algolia.com/?query=The%20case%20of%20the%20500-mile%20email&sort=byDate&dateRange=all&type=story&storyText=false&prefix&page=0 https://hn.algolia.com/?query=The%20case%20of%20the%20500-mi...

jessaustin 9y ago

Statisticians make really good bug reports!

DamonHD 9y ago
Maybe in this case, with n=1, but we'd need a better sampling of impoverished departmental heads to support your thesis.

js2 9y ago

I've read this story dozens of times and this is the first time SunOS jumped out at me. Around 2001 or so there was a Solaris release (2.7 maybe?) that had a bug in a particular patch release where sleep() returned instantly. That would have exhibited all sorts of interesting symptoms such as this story.

shakna 9y ago
I had the joy of re-engineering a terrible in-house web server because of that bug. It had used the system sleep to pause execution between the fake threads in their awful program. Suddenly everything was hitting race problems, instead of about 10%. Not something I really understood why it was going so bad, so I just dumped the inhouse implementation of threads, and took up the Solaris one, which was a joy by comparison.
- mikeash 9y ago
  Sleep() returns early if a signal is delivered to the program, so doesn't your code need to handle this case anyway? Of course, this is super rare, so incorrect code could survive for a long time on a correctly implemented system.
  - corprew 9y ago
    this is the point of that terrible 'threading' strategy, actually, if i guess what they are doing correctly.
  - geocar 9y ago
    You should always handle your errors, but people trust in exceptions and "what works" often enough that we have a culture that doesn't. http://geocar.sdf1.org/close.html http://geocar.sdf1.org/close.html

krylon 9y ago

Reading this kind of debugging story is kinda like watching a horror movie. It's very interesting and fun to read if you know the kind of problems you have to deal with are tame and furry in comparison. But if I actually had to troubleshoot a problem like this, it would give me nightmares for weeks.

nashashmi 9y ago

A lengthier discussion: https://news.ycombinator.com/item?id=9338708 https://news.ycombinator.com/item?id=9338708

alekratz 9y ago

> And I was fairly certain I hadn't enabled the "FAIL_MAIL_OVER_500_MILES" option. If this actually was a config option, I would bet anything the default would inexplicably be set to "true".

komali2 9y ago
It's like this weird setting in Ubuntu's network driver config file, "11n_disable." I won't pretend to know much about it but every time I install Ubuntu on a machine, wifi internet speeds are slow until I set the above to "8". To me it seems like literally a "go faster" switch.
- rincebrain 9y ago
  Only 8, and not 1? Because, at least on my system, looking at iwlwifi... parm: 11n_disable:disable 11n functionality, bitmap: 1: full, 2: disable agg TX, 4: disable agg RX, 8 enable agg TX (uint) So 8 is only aggregated TX, while 1 would be aggregated TX and RX. (It does, however, also inexplicably default to 0 on my system.)
  - komali2 9y ago
    Oh neat, lemme try that out then.
    - rincebrain 9y ago
      I'd be curious to learn what it does for you, because my intuitive interpretation would say that 1 would be "disable all", but that doesn't explain why "enable agg TX" would be an improvement unless agg RX is worse than the baseline...which is probably a bug. e: Oh, apparently this is a pathological re-use of an existing module flag to force something on that they disabled due to instability in some cases, see also: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=205e2210daa975d92ace485a65a31ccc4077fe1a https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... So I'm sorry, but my assumption about =1 is not valid, as I assumed that all of the bits (1,2,4,8) were mutating the same thing, not "one bit does something VERY DIFFERENT".

danschumann 9y ago

"And I was fairly certain I hadn't enabled the "FAIL_MAIL_OVER_500_MILES" option." I laughed.

stevefeinstein 9y ago

TIL: units is a really useful utility command that I had forgotten even existed.

late2part 9y ago
You have: 100 millilightseconds unknown unit 'millilightseconds' You have: ^D Apparently I need to add more units to achieve this level of usefulness.
- parraccourci 9y ago
  try "3 ms"
- dmacedo 9y ago
  Something tells me you're on OS X, and like myself, you'll find you mostly want everything replaced with GNU binaries: $ brew install gnu-units $ gunits Also see: https://apple.stackexchange.com/questions/69223/how-to-replace-mac-os-x-utilities-with-gnu-core-utilities https://apple.stackexchange.com/questions/69223/how-to-repla...
- emilfihlman 9y ago
  Well user@server:~$ units -V GNU Units version 2.11 with readline, with utf8, locale en_US Units data file is '/usr/share/units/definitions.units' Personal units data file is '/home/user/.units' (file does not exist) Copyright (C) 2014 Free Software Foundation, Inc. GNU Units comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GNU Units under the terms of the GNU General Public License. user@server:~$ units Currency exchange rates from www.timegenie.com on 2014-04-02 2866 units, 109 prefixes, 79 nonlinear units You have: 3millilightseconds You want: miles * 558.84719 / 0.0017893979 You have: user@server:~$ Works for me.
shubhamjain 9y ago
It does induce a geeky joy but it's ephemeral for the most part as Google is vastly superior in usefulness. I learned about units few years back, used it for a while, and I can't even remember when but I just started Googling the conversions instead of switching to the terminal window.

curtis 9y ago

The 500-mile email story seems to be the Hacker News equivalent to Reddit's SR-71 speed check story [1]. It keeps coming back again and again because everybody loves it. [1] https://www.reddit.com/r/SR71/comments/2dpmw7/the_sr71_speed_check_story/ https://www.reddit.com/r/SR71/comments/2dpmw7/the_sr71_speed... (and many others, too numerous to reference)

thearn4 9y ago
The SR-71 story never fails to put a smile on my face.
okket 9y ago
https://xkcd.com/1053/ https://xkcd.com/1053/
- rconti 9y ago
  This new XKCD comic is quite amusing! I read back through several of them and they're so true to my life experiences!
amarraja 9y ago
Thanks, first I've read that and it certainly put a smile on my face. One of the first articles I remember reading on HN was about an SR-71 [1], and it seems it's by the same pilot because Walter is his crewmate. [1] http://sploid.gizmodo.com/5511236/the-thrill-of-flying-the-sr-71-blackbird http://sploid.gizmodo.com/5511236/the-thrill-of-flying-the-s...
mercer 9y ago
Has anyone made a collection of the most re-posted HN articles?

wmichelin 9y ago

I'm a simple man, I see the 500 mile email story, I upvote.

reaperducer 9y ago

Stories like this remind me of the days when the internet was young, and fun, and silly, and best experienced in its green-on-black usenet glory. (418 Error: I'm a teapot, for example.) The internet just doesn't seem fun anymore.

chrissnell 9y ago
It's not that the Internet has lost its fun--it's just too easy to find it these days. You appreciated everything a little more when you had to get on your bike and ride a mile across campus to the campus computing center just to read that silly USENET post on a 80x24 serial terminal. Now you can read it on your watch or your glasses or yell at your home stereo to read it to you. I used to wonder what it would be like to time-travel to ancient times and show a gun or a car to a Neanderthal. I think it would be just as crazy to show 2017 me to 1993 me.
Piskvorrr 9y ago
It was me who approved the merge request replacing our 418 with a boring 400 a few months ago; I'm sorry. (On the other hand, there's still a server out there returning a redirect 303 Sea Otter, so I guess it balances out)

throwaway-emc2 9y ago

[throwaway account] A couple of years ago, my company (SEA) was bought by a private equity-based company that was assembling a portfolio of similar companies. The IT group got taken over by a competent yet overworked group in their main office (PHX.) The remote IT group wanted to move everyone's computer to their windows VMs that were in their colo in PHX, which were managed (and presumably logged) out of a central management panel. From their point of view, this was a great solution. I was trying to get them to put a separate VM bank up locally, because of the speed of light delay (at least 15ms) that would be small but annoying. I had convinced the remote IT people that this would be a good idea, and also the [PHX] CTO apparently thought it was a good idea, but my local 'location boss,' who used to run our company decided that I was just doing it to increase my power. (which was untrue, because I would be going back to just ENG and not IT after the transition.) So, my boss cancels it and throws me under the bus to management at our new home office. It takes a couple of weeks, then the terminals that log into the remote hosts get installed. The speed of light is one of the constants of the universe, so the problem I mention immediately happens. My location boss gets annoyed by the slow performance on his computer and, remembering that I'd said something about slowness, storms into my office and starts demanding that I fix the computers. I try and calm him down and tell him that there's no way to fix this without moving the terminals & vms closer together. He calls up the remote IT folks and they agree with me. He, at his next trip down to HQ, starts slagging on me to anyone who'd listen. Various people at the new home office warn me about this, but there's really nothing I can do about it. After some hilarity, I'm transferred to a project that was designed to fail so there can be paperwork justifying laying me off. So that is another way the speed of light can have real-world consequences.

orangepenguin 9y ago

I thoroughly enjoyed this story, but wonder if the details have changed a bit over time, perhaps? If light can travel ~558 miles in 3 milliseconds, it seems that mail sent more than 250 miles would have failed (because RTT)? Also, I'm not sure how big a difference it makes, but signals propagate at less than the "speed of light" (vacuum) when the medium is copper.

yoav_hollander 9y ago

Fun story - never heard it before. BTW, I think information travels over "normal" networking gear (e.g. fiber optics) at about 70% of the speed of light (which is why high-frequency trading tends to move to over-the-air microwave networking to shave a few nanoseconds - see [1]). But I guess the story still works, at the resolution it is told. [1] https://en.wikipedia.org/wiki/High-frequency_trading https://en.wikipedia.org/wiki/High-frequency_trading

YesThatTom2 9y ago

I was a friend of Trey back then and I remember him explaining this and thinking "wow, that's some good debugging!" You'd be amazed at how many people tell Trey that this story is fake. Ugh. There were plenty of witnesses. Trey made news recently during the flood of negative United Airlines stories: http://www.newnownext.com/how-united-airlines-almost-ruined-a-gay-couples-honeymoon/04/2017/ http://www.newnownext.com/how-united-airlines-almost-ruined-...