14 ms·

The 500-mile email (2002)

915 points by folz 12y ago

IgorPartola 12y ago
If only every bug report that I received had been processed by a geostatistician... Usually I get a "hey, I can't get X to work". One of three responses from me usually fixes it: "Is your computer on?", "are you online?", and "try hitting refresh". I am actually surprised the sysadmin in this scenario thought it was a bad thing that the statistics department did their research and presented a well documented error.
- mcguire 12y ago
  Well, technically, the geostatistician (Did I spell that right?) was doing research that was orthogonal to the actual problem and its symptoms. In this case, the results were sufficiently odd that they sort of pointed in the right direction, but I've been sent off on wild goose chases by people skillfully applying their own particular set of skills before. On the other hand, there's the word document with nothing but a screen shot showing half of a useless error message.
- deleted 12y ago
  [deleted]
- vacri 12y ago
  Reminds me... when I post a support request to Google Apps, the issue description header says "in as much detail as possible"... but the field is limited to 1000 characters. When you're dealing with anything other than simple first-level support issues, a user simply can't put in a usefully descriptive amount of detail...
mborsuk 12y ago
Every time I read this I am reminded of units(1) util, which is super useful and I always forget about and revert to Google. But yeah, that connect timeout to 500 mi correlation is fun too.
ilaksh 12y ago
If you're a sysadmin and someone brings in a consultant who gets root access and upgrades the whole OS to a new operating system which then almost takes out email.. wouldn't that be a problem? If I were the sysadmin and that happened, I would need to have a meeting with some people. What's the point of being a sysadmin if he operating system is randomly going to be completely changed without someone telling you? I have a fair amount of built up rage. This seems like one of those situations where it is actually your responsibility to rip people a new one.
nathancahill 12y ago
1 year ago: https://news.ycombinator.com/item?id=8176458 https://news.ycombinator.com/item?id=8176458 2 years ago: https://news.ycombinator.com/item?id=6296201 https://news.ycombinator.com/item?id=6296201 3 years ago: https://news.ycombinator.com/item?id=3802458 https://news.ycombinator.com/item?id=3802458 4 years ago: https://news.ycombinator.com/item?id=3114391 https://news.ycombinator.com/item?id=3114391 5 years ago: https://news.ycombinator.com/item?id=1293652 https://news.ycombinator.com/item?id=1293652 6 years ago: https://news.ycombinator.com/item?id=385068 https://news.ycombinator.com/item?id=385068 7 years ago: https://news.ycombinator.com/item?id=123489 https://news.ycombinator.com/item?id=123489
- Twirrim 12y ago
  Maybe the submitter is one of the ten thousand https://xkcd.com/1053/ https://xkcd.com/1053/ Bet there are a few more that will find this submission too.
  - emilioolivares 12y ago
    Ha, I'm one of the ten thousand for both the XKCD and this post. Lucky me!
    - golergka 12y ago
      You mean, for the whole XKCD? Not just this issue, but the whole thing? I envy you so hard right now. (Unless you were going to be productive today).
      - qznc 12y ago
        Productivity? Overrated. http://beza1e1.tuxen.de/lore/ http://beza1e1.tuxen.de/lore/
  - yzzxy 12y ago
    This doesn't apply very well... HN is heavily archived... this comic is about being rude to people for not knowing about something, not justifying shoving the same cyclical content in people's faces repeatedly.
Twirrim 12y ago
This one comes up ever 3-4 years or so in sysadmin communities, and I read it every single time. because it's worth it. It's one of those things that I highly doubt would have occurred to me to have even checked, or given even a moments thought to, under normal circumstances.
- tootie 12y ago
  This and the story of Mel never get old. http://www.catb.org/jargon/html/story-of-mel.html http://www.catb.org/jargon/html/story-of-mel.html
- AceJohnny2 12y ago
  I was looking for another famous sysadmin story, where the guy who also happens to be a top Linux developer (so maybe Alan Cox?) rescues a deeply broken Linux system where even glibc is no longer accessible by manipulating inodes in a running process. Or something. Over the years, my Google-fu has failed me. Any clue? :)
  - dredmorbius 12y ago
    That's a classic. Best I can claim is zmodem transfers of uunecoded packages over a PLIP link as I tried to get ethernet support up on an old but fairly reliable box.
  - HarryHirsch 12y ago
    http://www.lug.wsu.edu/node/414 http://www.lug.wsu.edu/node/414 is what you are looking for.
    - AceJohnny2 12y ago
      That's pretty damn close (and impressive!), but I don't think that was it :) For one thing, I'm pretty certain the story involved recovering a deleted file's inode via some other running process's FDs.
tlrobinson 12y ago
I see this story every so often, and it's good one, but haven't thought to verify it. Has anyone else?
- ryan-c 12y ago
  This FAQ was posted in the comments of a previous posting: https://webcache.googleusercontent.com/search?q=cache:http://www.ibiblio.org/harris/500milemail-faq.html https://webcache.googleusercontent.com/search?q=cache:http:/...
andrewchambers 12y ago
I was so happy to discover that units command line program, then i realized that Google already does this, it just wasn't as fun.
- tim333 12y ago
  yeah and units on the mac terminal doesn't recognise "3 millilightseconds" whereas Google works for "0.003 light seconds to miles"
  - chris_b 12y ago
    works exactly as in the blog post in fedora.
    - lucb1e 12y ago
      And Debian. For Mac, try brew install gnu-units and then run gunits.
      - TreyHarris 12y ago
        As I wrote in the FAQ, I decorated my own units.dat (units.lib in some implementations) with lots of stuff because I like easily editing units. (Nowadays I use Emacs Calc, but I still add a bunch of my own units, like the binary prefixes like mebi, gibi, etc.) I suspect that a lot more can convert millilightseconds to miles out-of-the-box now at least in part because of the popularity of this story over the past 13 years.
- btilly 12y ago
  Google has units, but not detailed commentary on unit definitions. See https://futureboy.us/frinkdata/units.txt https://futureboy.us/frinkdata/units.txt for that.
tsaoutourpants 12y ago
Forgot to account for the difference between traditional speed of light (in a vacuum) and speed of light traveling through copper of fiber. :)
- sampo 12y ago
  And the time it takes to make a round trip.
Scarbutt 12y ago
Who though it was going to be a TTL issue before finishing reading the story? :)
- gabeio 12y ago
  Not I, I don't know why but I was thinking IP version issue but that makes no sense.
- mobiplayer 12y ago
  You probably mean something else (RTT?) but definitely not TTL, which is a completely different thing :)
  - motoboi 12y ago
    TTL is involved when dealing with routed networks. The farther the destiny, you normally get more hops on the way. If the starting TTL is low, you won't reach the destiny. So, TTL values cause problems like this, although the radius wouldn't be so precise. Damn statisticians!
    - mobiplayer 11y ago
      Hehe yeah, I know what's TTL and how it works :) hence why I was pointing out (as politely as I could) that the previous comment was wrong. But I tend to agree with your conclusion. Damn statisticians! :)
kazinator 12y ago
> And also being a good system administrator, I had written a sendmail.cf [...] Say what? Nobody writes a sendmail.cf from scratch, unless they are crazy. > ... that used the nice long self-documenting option and variable names available in Sendmail 8 rather than the cryptic punctuation-mark codes that had been used in Sendmail 5 Good system administrators stick to conservative, portable subsets of configuration and scripting languages, rather than bleeding edge stuff. When they deviate, they have a clear plan. They document their choice to use something new and shiny, and they keep it separated from the default system configuration. Since SunOS came with Sendmail 5, the upgraded Sendmail 8 should have been installed in some custom location with its own path so that it coexists with the stock Sendmail, and is not perturbed if the OS happens to upgrade that. A good syadmin would stick that in some /usr/local/bin type local directory, and not overwrite /usr/bin/sendmail. The consultant was not wrong to update the OS. People have reasons to do that. The consultant should have consulted with the sysadmin, of course. But even in that event, it might not have immediately occurred to the sysadmin what the implication would be to the sendmail setup.
- TreyHarris 12y ago
  Goodness, you're determined to find fault, aren't you? (For the record in re your comment later about my "basis to call [myself] a good system admin", those claims were a) jokey, and b) fairly well-substantiated by my reputation by that time, I should think. I was published by that point and had been on several conference committees along with many who'd be reading that mailing list; I hardly needed to peacock like you seem to think I was doing.) But I think your criticisms seem a little uninformed (or possibly over-informed by later practice to the point where you aren't considering this in the context of mid-1990's practice). Let's see... > > And also being a good system administrator, I had written a sendmail.cf [...] > Say what? Nobody writes a sendmail.cf from scratch, unless they are crazy. I didn't say "from scratch". I used the m4 macros to create a cf, like everyone did at the time. Using the default file would only work if you still used email programs that read raw mbox files, had no email lists, and needed no interesting aliasing or vacation script behavior. Oh, and ran in an environment where it was reasonable to assume someone's canonical email address could be found via the equivalent of "echo "${USER}@${HOST#.}". Very few production systems could get away with that; writing a sendmail.cf was standard practice. And with m4, you usually spoke of "writing" a file where today we'd call it "configuring" a file; either way it was taking boilerplate and replacing bits with things that were right for your situation. I assume you wouldn't have had an issue with my writing that I'd "configured" the sendmail.cf. That's all I did. > > ... that used the nice long self-documenting option and variable names available in Sendmail 8 rather than the cryptic punctuation-mark codes that had been used in Sendmail 5 > Good system administrators stick to conservative, portable subsets of configuration and scripting languages, rather than bleeding edge stuff. Hmm, you either weren't administering SunOS in the mid-90's or you're forgetting some details. SunOS still came with Sendmail 5 years* after best practice was to use Sendmail 8. Check out the O'Reilly Sendmail book of the time's pagecount: it was longer than the prior and the later versions because it had to document both. I'm not entirely certain SunOS (as opposed to Solaris) ever was upgraded to Sendmail 8 in the distribution; obviously the people using SunOS still so late were change-averse. "Bleeding edge" != "the version that all but the most conservative holdouts are using". Also, remember that this was the same period we were doing the rsh/rlogin conversion to SSH. Sendmail 5 still had known security issues that were fixed in Sendmail 8. We were used to replacing system components when what the OS vendor was shipping us was literally dangerous to run. And Sendmail 8's Sendmail 5 compatibility mode was simply there for testing; it was never intended to be used production long-term, so using a least-common-denominator sendmail.cf wouldn't have been "conservative and portable"; it would have been risky, bordering on malpractice. > Since SunOS came with Sendmail 5, the upgraded Sendmail 8 should have been installed in some custom location with its own path so that it coexists with the stock Sendmail, and is not perturbed if the OS happens to upgrade that. > A good syadmin would stick that in some /usr/local/bin type local directory, and not overwrite /usr/bin/sendmail. Again, either you didn't run this installation in the mid-90's or you're forgetting some details. /usr/lib/sendmail (notice the "lib"! Your referring to "/usr/bin/sendmail" suggests to me you definitely weren't running SunOS 4 or have forgotten details; sendmail was never in /usr/bin) couldn't be left alone, as other tools hardcoded that path. The actual executable was there, so symlinking couldn't be used to get around that.
bpp 12y ago
Another of the 10,000 here - this is such a delightful story. Also just discovered the "units" conversion program and disappointed that the default Mac library has only 586 units. And shockingly there don't seem to be compatible libraries out there.
- girvo 12y ago
  `brew install gnu-units` should do it :) Edit: You'll then want to run it with `gunits` rather than `units`
  - oaktowner 12y ago
    Now I know where the rapper got his name.
  - tim333 12y ago
    Yay! Works.
  - bpp 12y ago
    Awesome, thank you!
kissickas 12y ago
Why do I get: > unknown unit 'millilightseconds' Is this one of the embellishments that just makes the story more entertaining?
- anonfunction 12y ago
  I had the same thing happen to me. From the manpage I gathered that units uses the definitions defined in /usr/share/misc/units.lib, by running cat /usr/share/misc/units.lib | grep light I found I only had lightyear and it's shortcut ly defined. I added lightsecond, and since milli prefix is already defined it worked a treat. Here's the line you'll want to add: lightsecond lightyear / 365.25 / 24 / 60 / 60
- masklinn 12y ago
  More complete units library. Note how the original author's units has 1311 units and 63 prefixes, OSX only has 586 and 56.
- riking 12y ago
  If you're on a mac, try $ brew install gnu-units - it's probably using a very incomplete library of units.
- kaishiro 12y ago
  Not an embellishment at all. Via 'man units': "The conversion information is read from a units data file that is called 'definitions.units' and is usually located in the '/usr/share/units' directory." Via definitions.units (L. 223), you can see the milli- prefix: https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-definitions-units-L223 https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-... Via deifnitions.units (L. 1060), you can see the lightsecond unit: https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-definitions-units-L1060 https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-... Maybe check it for completeness? Edit: Spelling
  - anonfunction 12y ago
    Some distributions only support lightyear so adding this line to your units file (which you can find with man units) will give you support for *lightseconds: lightsecond lightyear / 365.25 / 24 / 60 / 60
th0ma5 12y ago
This always reminds me of the email around the world: http://phrack.org/issues/41/4.html http://phrack.org/issues/41/4.html
- fabulist 12y ago
  Thanks for a good read. Its strange to think about a time when there were a myriad of incompatible networks, and their different capabilities could be exploited.
carlesfe 12y ago
That was great out-of-the-box thinking, and I wonder if that could be used as one of these job interview questions: Q: "Your email server for some reason is only working for addresses within 500 miles of the server. What may go wrong?" And let the candidate think logically and reach some sane answer, even if not 100% accurate (i.e. check routers first, connectivity, DNS, timeouts...)
- Xorlev 12y ago
  Anyone who short circuits and brings up this story should get a +1.
- iopq 12y ago
  That's one of those interview questions that tests for someone reading hacker news and pretending that they figured it out all on their own...
  - rmc 12y ago
    "culture fit"
    - DaFranker 12y ago
      Translation: "Tunnel Vision" and "Lack of diversity"
elchief 12y ago
Can we get a nice "HN Classic" tag to put beside annual stories like this? I'm fine if stories like this pop up every year, actually.
copperx 12y ago
I didn't know about the units program. Is there any resource out there that lists these little *nix utility programs?
- deleted 12y ago
  [deleted]
- motters 12y ago
  That's the thing about unix-like systems. No matter how much you have learned there's always some command you don't know.
  - creshal 12y ago
    paste was my most recent "holy shit this saves so much time" discovery. I blame it on the not quite intuitive name.
    - nchelluri 12y ago
      What do you use it for?
      - pwg 12y ago
        One use is converting columns of numbers into math strings for bc. Example (contrived): $ seq 10 20 | paste -s -d + 10+11+12+13+14+15+16+17+18+19+20 $ seq 10 20 | paste -s -d + | bc 165 Or converting columns of strings into regex 'or' clauses for searching (contrived example again): $ cut -f 3 -d , something.csv | paste -s -d "|" a|b|c|d|e|f $ egrep "$(cut -f 3 -d , something.csv | paste -s -d "|")" another_file ... result lines appear here ...
        discardorama 12y ago
        I've used paste all my life, but I never knew you could do paste - - to convert stdin into 2 columns (or "paste - - - -" to get 4 columns!). TIL ...
t27 12y ago
A perfect answer to the YC application question - "Tell us something surprising or amusing that one of you has discovered" :)
danbruc 12y ago
Another email incident at Microsoft worth reading [1]. [1] http://blogs.technet.com/b/exchange/archive/2004/04/08/109626.aspx http://blogs.technet.com/b/exchange/archive/2004/04/08/10962...
- sp332 12y ago
  One listserve (can't remember which) made up a list for people who complained like this instead of following the unsubscribe instructions. The admins would remove complainers from the normal lists and add them all to one mailing list, where the only emails they got were each others' demands to be taken off the mailing list, with unsubscribe instructions added to the beginning and the end of every single email.
- Shaaan 12y ago
  Literally the exact same thing happened at Case Western this past weekend
- digi_owl 12y ago
  Ah yes, the age old "reply-all" email storm. The bit about the recipient processing bug is novel tough, ouch.
- JonnieCache 12y ago
  My favorite version of this tale: "Free Bananas in the Kitchen!" http://www.metafilter.com/78177/PLEASE-UNSUBSCRIBE-ME-FROM-THIS-LIST#2408665 http://www.metafilter.com/78177/PLEASE-UNSUBSCRIBE-ME-FROM-T... I remember it happened at NYU a couple of years ago and they turned it into a kind of ad-hoc social network/partyline. I wonder if anyone archived those emails? I suppose they deserve to remain "private."
- AtmaScout 12y ago
  Thanks for the link. I was surprised that it was written by Larry Osterman. I enjoy listening to his stories about Microsoft. Have you seen his Channel 9 videos [0]? I really enjoy the checking in videos with Erik Meijer. [0] http://channel9.msdn.com/tags/Larry+Osterman http://channel9.msdn.com/tags/Larry+Osterman
- 12y ago
fbnt 12y ago
Shouldn't this account for a round trip, and the speed through copper (~ 2/3rd of the speed of light)? That would lower the radius to much more than 500 miles.
- dantillberg 12y ago
  I had this thought when reading this before as well. I imagine that the "3 milliseconds" they determined from testing was a typical number, maybe the median/mean, and that the actual timeout varied considerably depending on CPU load at that particular moment. Add in a number of retries for the server to attempt sending each email, and the effective timeout might have been a few milliseconds more... or at least it must have been, because `(2 * 500 miles) / (2/3 speed of light)` works out to about 8 milliseconds (where the 2X is for the round trip, and 2/3 is a rough multiplier for the speed of light traveling in either copper or optical fiber).
  - ErrantX 12y ago
    The FAQ answers this question. Basically; it was a long time ago, and the point of the story isn't in the detail. :) http://www.ibiblio.org/harris/500milemail-faq.html http://www.ibiblio.org/harris/500milemail-faq.html
    - lloydde 12y ago
      And this is why we can't have nice stories. I felt for the author as I got deeper into the faq, and recognized this pattern of cynicism, then decided the author was so generous and thorough, not out of obligation (make the emails stop!), but because that is the type of detailed person he is -- and good at dinner parties too!
      - chatmasta 12y ago
        Yes, I found this to be one of the most refreshing technical anecdotes I've ever read. The tone and style actually put a smile on my face as I read. I enjoyed how the author guided us through the process of discovery, one which we all know so well, driven by an insatiable curiosity to go continually deeper down the rabbit hole until we find the bottom.
bastomi29 12y ago
https://gunungbromosunrisetour.wordpress.com/2015/01/08/dari-surabaya-ke-gunung-bromo/ https://gunungbromosunrisetour.wordpress.com/2015/01/08/dari...
Aissen 12y ago
Damn statisticians. They do know their job quite well.
- nchelluri 12y ago
  It was a seriously accurate bug report. If only all users were so thoughtful.
  - nashashmi 12y ago
    > If only all users were so thoughtful. But then it sent him off in a direction not worth going. He literally started to map out how far emails would go if they succeeded. The whole time the error was in the timeout instead.
    - Retra 12y ago
      It is pretty trivial to translate distance to time for something like this, so I wouldn't consider the wrong direction at all.
    - bdevine 12y ago
      The mapping was done for the purpose of reproducing the error though. I wouldn't call that a wasted effort.
bontoJR 12y ago
Absolytely a good reading. Sometimes this kind of readings can help in a complete different problem. Sometime happens you are dealing with another problem, then you remember this story, and you figure out what's wrong because there're some similarities. I remember to have fixed a problem with Postgresql remembering a story about Unicode and Postfix, different domain, but similar problem.
anonfunction 12y ago
Since I've seen a few comments about units not having lightseconds so here are a few ways to add the missing unit if you don't have it. 1) Add this line under the lightyear definition in /usr/share/misc/units.lib (or wherever `man units` says the standard units library is under the FILES section) lightsecond lightyear / 365.25 / 24 / 60 / 60 2) If you're on a mac and use homebrew just `brew install gnu-units` and then run `gunits`
- chernevik 12y ago
  I did #2, then: sudo mv units macunits sudo ln -s $(which gunits) units
  - lloydde 12y ago
    or use brew install option --with-default-names and put your homebrew at the start of your path.
- smoyer 12y ago
  That's the speed of light in a vacuum ... through fiber-optic cable the speed of light is about two-thirds that value.
vog 12y ago
Better link that contains more headers (showing the email's date, and linking to a FAQ): http://www.ibiblio.org/harris/500milemail.html http://www.ibiblio.org/harris/500milemail.html
laex 12y ago
I tried the 'units' program on OSX. It seems that it does not recognise the 'millilightseconds' unit.
- JulianMorrison 12y ago
  Try one L in mili?
  - laex 12y ago
    Didn't work.
    - laex 12y ago
      @girvo suggested installing gnu units, which does the trick. > brew install gnu-units > gunits
kowdermeister 12y ago
I'm wondering how many hits that email address got at the bottom of the page :)
- anonu 12y ago
  Was this just a clever way to let people know he was looking for a job?
ai_ja_nai 12y ago
old but gold

915 points by folz 12y ago

If only every bug report that I received had been processed by a geostatistician... Usually I get a "hey, I can't get X to work". One of three responses from me usually fixes it: "Is your computer on?", "are you online?", and "try hitting refresh". I am actually surprised the sysadmin in this scenario thought it was a bad thing that the statistics department did their research and presented a well documented error.

mcguire 12y ago
Well, technically, the geostatistician (Did I spell that right?) was doing research that was orthogonal to the actual problem and its symptoms. In this case, the results were sufficiently odd that they sort of pointed in the right direction, but I've been sent off on wild goose chases by people skillfully applying their own particular set of skills before. On the other hand, there's the word document with nothing but a screen shot showing half of a useless error message.
deleted 12y ago
[deleted]
vacri 12y ago
Reminds me... when I post a support request to Google Apps, the issue description header says "in as much detail as possible"... but the field is limited to 1000 characters. When you're dealing with anything other than simple first-level support issues, a user simply can't put in a usefully descriptive amount of detail...

mborsuk 12y ago

Every time I read this I am reminded of units(1) util, which is super useful and I always forget about and revert to Google. But yeah, that connect timeout to 500 mi correlation is fun too.

ilaksh 12y ago

If you're a sysadmin and someone brings in a consultant who gets root access and upgrades the whole OS to a new operating system which then almost takes out email.. wouldn't that be a problem? If I were the sysadmin and that happened, I would need to have a meeting with some people. What's the point of being a sysadmin if he operating system is randomly going to be completely changed without someone telling you? I have a fair amount of built up rage. This seems like one of those situations where it is actually your responsibility to rip people a new one.

nathancahill 12y ago

1 year ago: https://news.ycombinator.com/item?id=8176458 https://news.ycombinator.com/item?id=8176458 2 years ago: https://news.ycombinator.com/item?id=6296201 https://news.ycombinator.com/item?id=6296201 3 years ago: https://news.ycombinator.com/item?id=3802458 https://news.ycombinator.com/item?id=3802458 4 years ago: https://news.ycombinator.com/item?id=3114391 https://news.ycombinator.com/item?id=3114391 5 years ago: https://news.ycombinator.com/item?id=1293652 https://news.ycombinator.com/item?id=1293652 6 years ago: https://news.ycombinator.com/item?id=385068 https://news.ycombinator.com/item?id=385068 7 years ago: https://news.ycombinator.com/item?id=123489 https://news.ycombinator.com/item?id=123489

Twirrim 12y ago
Maybe the submitter is one of the ten thousand https://xkcd.com/1053/ https://xkcd.com/1053/ Bet there are a few more that will find this submission too.
- emilioolivares 12y ago
  Ha, I'm one of the ten thousand for both the XKCD and this post. Lucky me!
  - golergka 12y ago
    You mean, for the whole XKCD? Not just this issue, but the whole thing? I envy you so hard right now. (Unless you were going to be productive today).
    - qznc 12y ago
      Productivity? Overrated. http://beza1e1.tuxen.de/lore/ http://beza1e1.tuxen.de/lore/
- yzzxy 12y ago
  This doesn't apply very well... HN is heavily archived... this comic is about being rude to people for not knowing about something, not justifying shoving the same cyclical content in people's faces repeatedly.

Twirrim 12y ago

This one comes up ever 3-4 years or so in sysadmin communities, and I read it every single time. because it's worth it. It's one of those things that I highly doubt would have occurred to me to have even checked, or given even a moments thought to, under normal circumstances.

tootie 12y ago
This and the story of Mel never get old. http://www.catb.org/jargon/html/story-of-mel.html http://www.catb.org/jargon/html/story-of-mel.html
AceJohnny2 12y ago
I was looking for another famous sysadmin story, where the guy who also happens to be a top Linux developer (so maybe Alan Cox?) rescues a deeply broken Linux system where even glibc is no longer accessible by manipulating inodes in a running process. Or something. Over the years, my Google-fu has failed me. Any clue? :)
- dredmorbius 12y ago
  That's a classic. Best I can claim is zmodem transfers of uunecoded packages over a PLIP link as I tried to get ethernet support up on an old but fairly reliable box.
- HarryHirsch 12y ago
  http://www.lug.wsu.edu/node/414 http://www.lug.wsu.edu/node/414 is what you are looking for.
  - AceJohnny2 12y ago
    That's pretty damn close (and impressive!), but I don't think that was it :) For one thing, I'm pretty certain the story involved recovering a deleted file's inode via some other running process's FDs.

tlrobinson 12y ago

I see this story every so often, and it's good one, but haven't thought to verify it. Has anyone else?

ryan-c 12y ago
This FAQ was posted in the comments of a previous posting: https://webcache.googleusercontent.com/search?q=cache:http://www.ibiblio.org/harris/500milemail-faq.html https://webcache.googleusercontent.com/search?q=cache:http:/...

andrewchambers 12y ago

I was so happy to discover that units command line program, then i realized that Google already does this, it just wasn't as fun.

tim333 12y ago
yeah and units on the mac terminal doesn't recognise "3 millilightseconds" whereas Google works for "0.003 light seconds to miles"
- chris_b 12y ago
  works exactly as in the blog post in fedora.
  - lucb1e 12y ago
    And Debian. For Mac, try brew install gnu-units and then run gunits.
    - TreyHarris 12y ago
      As I wrote in the FAQ, I decorated my own units.dat (units.lib in some implementations) with lots of stuff because I like easily editing units. (Nowadays I use Emacs Calc, but I still add a bunch of my own units, like the binary prefixes like mebi, gibi, etc.) I suspect that a lot more can convert millilightseconds to miles out-of-the-box now at least in part because of the popularity of this story over the past 13 years.
btilly 12y ago
Google has units, but not detailed commentary on unit definitions. See https://futureboy.us/frinkdata/units.txt https://futureboy.us/frinkdata/units.txt for that.

tsaoutourpants 12y ago

Forgot to account for the difference between traditional speed of light (in a vacuum) and speed of light traveling through copper of fiber. :)

sampo 12y ago
And the time it takes to make a round trip.

Scarbutt 12y ago

Who though it was going to be a TTL issue before finishing reading the story? :)

gabeio 12y ago
Not I, I don't know why but I was thinking IP version issue but that makes no sense.
mobiplayer 12y ago
You probably mean something else (RTT?) but definitely not TTL, which is a completely different thing :)
- motoboi 12y ago
  TTL is involved when dealing with routed networks. The farther the destiny, you normally get more hops on the way. If the starting TTL is low, you won't reach the destiny. So, TTL values cause problems like this, although the radius wouldn't be so precise. Damn statisticians!
  - mobiplayer 11y ago
    Hehe yeah, I know what's TTL and how it works :) hence why I was pointing out (as politely as I could) that the previous comment was wrong. But I tend to agree with your conclusion. Damn statisticians! :)

kazinator 12y ago

> And also being a good system administrator, I had written a sendmail.cf [...] Say what? Nobody writes a sendmail.cf from scratch, unless they are crazy. > ... that used the nice long self-documenting option and variable names available in Sendmail 8 rather than the cryptic punctuation-mark codes that had been used in Sendmail 5 Good system administrators stick to conservative, portable subsets of configuration and scripting languages, rather than bleeding edge stuff. When they deviate, they have a clear plan. They document their choice to use something new and shiny, and they keep it separated from the default system configuration. Since SunOS came with Sendmail 5, the upgraded Sendmail 8 should have been installed in some custom location with its own path so that it coexists with the stock Sendmail, and is not perturbed if the OS happens to upgrade that. A good syadmin would stick that in some /usr/local/bin type local directory, and not overwrite /usr/bin/sendmail. The consultant was not wrong to update the OS. People have reasons to do that. The consultant should have consulted with the sysadmin, of course. But even in that event, it might not have immediately occurred to the sysadmin what the implication would be to the sendmail setup.

TreyHarris 12y ago
Goodness, you're determined to find fault, aren't you? (For the record in re your comment later about my "basis to call [myself] a good system admin", those claims were a) jokey, and b) fairly well-substantiated by my reputation by that time, I should think. I was published by that point and had been on several conference committees along with many who'd be reading that mailing list; I hardly needed to peacock like you seem to think I was doing.) But I think your criticisms seem a little uninformed (or possibly over-informed by later practice to the point where you aren't considering this in the context of mid-1990's practice). Let's see... > > And also being a good system administrator, I had written a sendmail.cf [...] > Say what? Nobody writes a sendmail.cf from scratch, unless they are crazy. I didn't say "from scratch". I used the m4 macros to create a cf, like everyone did at the time. Using the default file would only work if you still used email programs that read raw mbox files, had no email lists, and needed no interesting aliasing or vacation script behavior. Oh, and ran in an environment where it was reasonable to assume someone's canonical email address could be found via the equivalent of "echo "${USER}@${HOST#.}". Very few production systems could get away with that; writing a sendmail.cf was standard practice. And with m4, you usually spoke of "writing" a file where today we'd call it "configuring" a file; either way it was taking boilerplate and replacing bits with things that were right for your situation. I assume you wouldn't have had an issue with my writing that I'd "configured" the sendmail.cf. That's all I did. > > ... that used the nice long self-documenting option and variable names available in Sendmail 8 rather than the cryptic punctuation-mark codes that had been used in Sendmail 5 > Good system administrators stick to conservative, portable subsets of configuration and scripting languages, rather than bleeding edge stuff. Hmm, you either weren't administering SunOS in the mid-90's or you're forgetting some details. SunOS still came with Sendmail 5 years* after best practice was to use Sendmail 8. Check out the O'Reilly Sendmail book of the time's pagecount: it was longer than the prior and the later versions because it had to document both. I'm not entirely certain SunOS (as opposed to Solaris) ever was upgraded to Sendmail 8 in the distribution; obviously the people using SunOS still so late were change-averse. "Bleeding edge" != "the version that all but the most conservative holdouts are using". Also, remember that this was the same period we were doing the rsh/rlogin conversion to SSH. Sendmail 5 still had known security issues that were fixed in Sendmail 8. We were used to replacing system components when what the OS vendor was shipping us was literally dangerous to run. And Sendmail 8's Sendmail 5 compatibility mode was simply there for testing; it was never intended to be used production long-term, so using a least-common-denominator sendmail.cf wouldn't have been "conservative and portable"; it would have been risky, bordering on malpractice. > Since SunOS came with Sendmail 5, the upgraded Sendmail 8 should have been installed in some custom location with its own path so that it coexists with the stock Sendmail, and is not perturbed if the OS happens to upgrade that. > A good syadmin would stick that in some /usr/local/bin type local directory, and not overwrite /usr/bin/sendmail. Again, either you didn't run this installation in the mid-90's or you're forgetting some details. /usr/lib/sendmail (notice the "lib"! Your referring to "/usr/bin/sendmail" suggests to me you definitely weren't running SunOS 4 or have forgotten details; sendmail was never in /usr/bin) couldn't be left alone, as other tools hardcoded that path. The actual executable was there, so symlinking couldn't be used to get around that.

bpp 12y ago

Another of the 10,000 here - this is such a delightful story. Also just discovered the "units" conversion program and disappointed that the default Mac library has only 586 units. And shockingly there don't seem to be compatible libraries out there.

girvo 12y ago
`brew install gnu-units` should do it :) Edit: You'll then want to run it with `gunits` rather than `units`
- oaktowner 12y ago
  Now I know where the rapper got his name.
- tim333 12y ago
  Yay! Works.
- bpp 12y ago
  Awesome, thank you!

kissickas 12y ago

Why do I get: > unknown unit 'millilightseconds' Is this one of the embellishments that just makes the story more entertaining?

anonfunction 12y ago
I had the same thing happen to me. From the manpage I gathered that units uses the definitions defined in /usr/share/misc/units.lib, by running cat /usr/share/misc/units.lib | grep light I found I only had lightyear and it's shortcut ly defined. I added lightsecond, and since milli prefix is already defined it worked a treat. Here's the line you'll want to add: lightsecond lightyear / 365.25 / 24 / 60 / 60
masklinn 12y ago
More complete units library. Note how the original author's units has 1311 units and 63 prefixes, OSX only has 586 and 56.
riking 12y ago
If you're on a mac, try $ brew install gnu-units - it's probably using a very incomplete library of units.
kaishiro 12y ago
Not an embellishment at all. Via 'man units': "The conversion information is read from a units data file that is called 'definitions.units' and is usually located in the '/usr/share/units' directory." Via definitions.units (L. 223), you can see the milli- prefix: https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-definitions-units-L223 https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-... Via deifnitions.units (L. 1060), you can see the lightsecond unit: https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-definitions-units-L1060 https://gist.github.com/anonymous/f06769de95e0c7f9e658#file-... Maybe check it for completeness? Edit: Spelling
- anonfunction 12y ago
  Some distributions only support lightyear so adding this line to your units file (which you can find with man units) will give you support for *lightseconds: lightsecond lightyear / 365.25 / 24 / 60 / 60

th0ma5 12y ago

This always reminds me of the email around the world: http://phrack.org/issues/41/4.html http://phrack.org/issues/41/4.html

fabulist 12y ago
Thanks for a good read. Its strange to think about a time when there were a myriad of incompatible networks, and their different capabilities could be exploited.

carlesfe 12y ago

That was great out-of-the-box thinking, and I wonder if that could be used as one of these job interview questions: Q: "Your email server for some reason is only working for addresses within 500 miles of the server. What may go wrong?" And let the candidate think logically and reach some sane answer, even if not 100% accurate (i.e. check routers first, connectivity, DNS, timeouts...)

Xorlev 12y ago
Anyone who short circuits and brings up this story should get a +1.
iopq 12y ago
That's one of those interview questions that tests for someone reading hacker news and pretending that they figured it out all on their own...
- rmc 12y ago
  "culture fit"
  - DaFranker 12y ago
    Translation: "Tunnel Vision" and "Lack of diversity"

elchief 12y ago

Can we get a nice "HN Classic" tag to put beside annual stories like this? I'm fine if stories like this pop up every year, actually.

copperx 12y ago

I didn't know about the units program. Is there any resource out there that lists these little *nix utility programs?

deleted 12y ago
[deleted]
motters 12y ago
That's the thing about unix-like systems. No matter how much you have learned there's always some command you don't know.
- creshal 12y ago
  paste was my most recent "holy shit this saves so much time" discovery. I blame it on the not quite intuitive name.
  - nchelluri 12y ago
    What do you use it for?
    - pwg 12y ago
      One use is converting columns of numbers into math strings for bc. Example (contrived): $ seq 10 20 | paste -s -d + 10+11+12+13+14+15+16+17+18+19+20 $ seq 10 20 | paste -s -d + | bc 165 Or converting columns of strings into regex 'or' clauses for searching (contrived example again): $ cut -f 3 -d , something.csv | paste -s -d "|" a|b|c|d|e|f $ egrep "$(cut -f 3 -d , something.csv | paste -s -d "|")" another_file ... result lines appear here ...
      - discardorama 12y ago
        I've used paste all my life, but I never knew you could do paste - - to convert stdin into 2 columns (or "paste - - - -" to get 4 columns!). TIL ...

t27 12y ago

A perfect answer to the YC application question - "Tell us something surprising or amusing that one of you has discovered" :)

danbruc 12y ago

Another email incident at Microsoft worth reading [1]. [1] http://blogs.technet.com/b/exchange/archive/2004/04/08/109626.aspx http://blogs.technet.com/b/exchange/archive/2004/04/08/10962...

sp332 12y ago
One listserve (can't remember which) made up a list for people who complained like this instead of following the unsubscribe instructions. The admins would remove complainers from the normal lists and add them all to one mailing list, where the only emails they got were each others' demands to be taken off the mailing list, with unsubscribe instructions added to the beginning and the end of every single email.
Shaaan 12y ago
Literally the exact same thing happened at Case Western this past weekend
digi_owl 12y ago
Ah yes, the age old "reply-all" email storm. The bit about the recipient processing bug is novel tough, ouch.
JonnieCache 12y ago
My favorite version of this tale: "Free Bananas in the Kitchen!" http://www.metafilter.com/78177/PLEASE-UNSUBSCRIBE-ME-FROM-THIS-LIST#2408665 http://www.metafilter.com/78177/PLEASE-UNSUBSCRIBE-ME-FROM-T... I remember it happened at NYU a couple of years ago and they turned it into a kind of ad-hoc social network/partyline. I wonder if anyone archived those emails? I suppose they deserve to remain "private."
AtmaScout 12y ago
Thanks for the link. I was surprised that it was written by Larry Osterman. I enjoy listening to his stories about Microsoft. Have you seen his Channel 9 videos [0]? I really enjoy the checking in videos with Erik Meijer. [0] http://channel9.msdn.com/tags/Larry+Osterman http://channel9.msdn.com/tags/Larry+Osterman
12y ago

fbnt 12y ago

Shouldn't this account for a round trip, and the speed through copper (~ 2/3rd of the speed of light)? That would lower the radius to much more than 500 miles.

dantillberg 12y ago
I had this thought when reading this before as well. I imagine that the "3 milliseconds" they determined from testing was a typical number, maybe the median/mean, and that the actual timeout varied considerably depending on CPU load at that particular moment. Add in a number of retries for the server to attempt sending each email, and the effective timeout might have been a few milliseconds more... or at least it must have been, because `(2 * 500 miles) / (2/3 speed of light)` works out to about 8 milliseconds (where the 2X is for the round trip, and 2/3 is a rough multiplier for the speed of light traveling in either copper or optical fiber).
- ErrantX 12y ago
  The FAQ answers this question. Basically; it was a long time ago, and the point of the story isn't in the detail. :) http://www.ibiblio.org/harris/500milemail-faq.html http://www.ibiblio.org/harris/500milemail-faq.html
  - lloydde 12y ago
    And this is why we can't have nice stories. I felt for the author as I got deeper into the faq, and recognized this pattern of cynicism, then decided the author was so generous and thorough, not out of obligation (make the emails stop!), but because that is the type of detailed person he is -- and good at dinner parties too!
    - chatmasta 12y ago
      Yes, I found this to be one of the most refreshing technical anecdotes I've ever read. The tone and style actually put a smile on my face as I read. I enjoyed how the author guided us through the process of discovery, one which we all know so well, driven by an insatiable curiosity to go continually deeper down the rabbit hole until we find the bottom.

bastomi29 12y ago

https://gunungbromosunrisetour.wordpress.com/2015/01/08/dari-surabaya-ke-gunung-bromo/ https://gunungbromosunrisetour.wordpress.com/2015/01/08/dari...

Aissen 12y ago

Damn statisticians. They do know their job quite well.

nchelluri 12y ago
It was a seriously accurate bug report. If only all users were so thoughtful.
- nashashmi 12y ago
  > If only all users were so thoughtful. But then it sent him off in a direction not worth going. He literally started to map out how far emails would go if they succeeded. The whole time the error was in the timeout instead.
  - Retra 12y ago
    It is pretty trivial to translate distance to time for something like this, so I wouldn't consider the wrong direction at all.
  - bdevine 12y ago
    The mapping was done for the purpose of reproducing the error though. I wouldn't call that a wasted effort.

bontoJR 12y ago

Absolytely a good reading. Sometimes this kind of readings can help in a complete different problem. Sometime happens you are dealing with another problem, then you remember this story, and you figure out what's wrong because there're some similarities. I remember to have fixed a problem with Postgresql remembering a story about Unicode and Postfix, different domain, but similar problem.

anonfunction 12y ago

Since I've seen a few comments about units not having lightseconds so here are a few ways to add the missing unit if you don't have it. 1) Add this line under the lightyear definition in /usr/share/misc/units.lib (or wherever `man units` says the standard units library is under the FILES section) lightsecond lightyear / 365.25 / 24 / 60 / 60 2) If you're on a mac and use homebrew just `brew install gnu-units` and then run `gunits`

chernevik 12y ago
I did #2, then: sudo mv units macunits sudo ln -s $(which gunits) units
- lloydde 12y ago
  or use brew install option --with-default-names and put your homebrew at the start of your path.
smoyer 12y ago
That's the speed of light in a vacuum ... through fiber-optic cable the speed of light is about two-thirds that value.

vog 12y ago

Better link that contains more headers (showing the email's date, and linking to a FAQ): http://www.ibiblio.org/harris/500milemail.html http://www.ibiblio.org/harris/500milemail.html

laex 12y ago

I tried the 'units' program on OSX. It seems that it does not recognise the 'millilightseconds' unit.

JulianMorrison 12y ago
Try one L in mili?
- laex 12y ago
  Didn't work.
  - laex 12y ago
    @girvo suggested installing gnu units, which does the trick. > brew install gnu-units > gunits

kowdermeister 12y ago

I'm wondering how many hits that email address got at the bottom of the page :)

anonu 12y ago
Was this just a clever way to let people know he was looking for a job?

ai_ja_nai 12y ago

old but gold