9 ms·

UUCP must stay; Fetchmail sucks (2001)

108 points by segfaultbuserr 6y ago

tyingq 6y ago
More context: http://pld.cs.luc.edu/courses/412/spr20/mnotes/bazaar.html http://pld.cs.luc.edu/courses/412/spr20/mnotes/bazaar.html
0xbadcafebee 6y ago
Can confirm that fetchmail is garbage. Found bugs, reported them, no fix. Wrote my own tool instead. Such is life in open source.
- KirinDave 6y ago
  I wonder if many folks who only joined the world of software development after DVCSs (and indeed, just VCSs) became popularized can understand why Fetchmail was not trivial to fork and fix.
- aatharuv 6y ago
  The only time I ever interacted with ESR on fetchmail, I reported a bug for an issue I ran into (with Kerberos 4 IMAP), submitted a patch, and it was accepted without discussion in the next release. Admittedly, this was 19 years ago, and it was a relatively small fix.
- jjav 6y ago
  I've been using fetchmail every two minutes (daemon mode) for well over two decades and not once has it failed to do what it needs to do. So it's a bit of a stretch to call it garbage. Other than emacs and mutt, there's no other tool with such stable longevity in my toolbox.
- athms 6y ago
  Under ESR, fetchmail was garbage. However, a new team took over development in the early 2000s after ESR abandoned the program. Many parts have been re-written to address design criticisms.
dvfjsdhgfv 6y ago
This doesn't seem so much about UUCP vs fetchmail as about cathedral vs bazaar style of development. In hindsight, it's clear that both of these work, for better or worse, with each having its own disadvantages and benefits.
- mprovost 6y ago
  Some context for those who are unaware: fetchmail was written by the same person (Eric S Raymond/ESR) as the "Cathedral and the Bazaar" essay.
  - freedomben 6y ago
    If you have any interest in open source development (which these days we all should given that 95% of the lower parts of our stack are open source projects) it's well worth reading the essay/book. If you don't want to do that, at least read the wikipedia page: https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar
    - segfaultbuserr 6y ago
      I found this interesting post while reading some historical documents, so I submitted it to HN. It seems that most people have missed the point. I think the point is: while The Cathedral and the Bazaar was a good campaign pamphlet and a general overview of FOSS development, but ESR's attempt of using his fetchmail as an example was arguably the greatest flaw of the book ("to such an extent that it could damage the Open Source movement"). From a technical perspective, fetchmail had numerous problems and security vulnerabilities. ESR was also criticized for mismanaging the project by putting it in "maintenance mode" early and refusing to accept patches that could address those issues. After a group of new maintainers had taken over the project, one of the first things they did was to systematically eliminate countless buffer and integer overflows from the program.
    - tptacek 6y ago
      Not really. I read it when it came out and remembered liking it (we were all on kind of a buzz about open source and Linux at the time) and then again much more recently for a thread on HN, and I found it to be mostly a rehash of other, better material --- Brooks and Kernighan, in particular. That, plus the modern context that what he was writing about was Fetchmail, and Fetchmail as it turns out is hot garbage, to me makes the whole thing very skippable. It's also responsible for hatching one of the great canards of open source, "Linus's Law" (I'm sure Linus is thrilled with having his name attached to that), "many eyes make all bugs shallow", which has in the last 10 years become one of the great jokes of computer science.
rasengan 6y ago
I used to use UUCP to get my email when I was on a dialup, non permanent connection. I was able to host my own mail server. There were a lot of technologies back then helping the internet stay somewhat decentralized. Now I, like most, just use Gmail. What has become of our beloved free(digital)land?
- cmrdporcupine 6y ago
  Yep, I remember spooling usenet and email to my Atari ST over a 1200 baud modem over UUCP in 1990, 1991. Bang path and all. And you're right, the centralization is crazy, and opposite of what techno utopians predicted/promised at the time.
  - vidarh 6y ago
    My first access to the web around '93 was via CERNs e-mail to web gateway coupled with UUCP exchange from my Amiga to my local BBS which exchanged with its upstream 4-6 times a day.
    - cmrdporcupine 6y ago
      Yeah I used to dial into the local university's gopher "campus information server" -- which was just a dial-up to telnet to a gopher client -- hit the ^] telnet escape character, close the connection, and re-open to another host where I had a shell and go from there. If I recall right it was probably gnu.ai.mit.edu that gave out free shell accounts, or something else in that domain connected somehow to Stallman. There was no ISP in my area back then, not even dial-up. Again, around 1991.
  - toyg 6y ago
    Sadly, the upsides are just too good. I love decentralization, but I have to recognise that I've not lost a message since I moved to gmail in 2005; before then, losing entire mailbox files was not uncommon, as well as missing messages from unreliable mailservers, spam management, and so on. I'm sure I could have done better etc etc, but the bottom line is that with gmail-like services you don't have to. I enjoy doing sysadmin but not 24/7.
duckerude 6y ago
A similar rant: http://pyropus.ca/software/getmail/faq.html#faq-about-why http://pyropus.ca/software/getmail/faq.html#faq-about-why
thwarted 6y ago
Fetchmail grew to be a mail transport swiss army knife and encompassed half baked functionality of a domain-wide MTA rather focusing on single user async email consolidation and collection for a logical single account, which is largely what this rant is about. Half of the half-bakedness is due to lack of MTA protocol support for MUA interactions on the receiving side. pop3 was never meant as a spool for MTA to MTA email transfers, but was instead meant for an MUA to access. It's not fetchmail's fault that the envelope TO isn't available via pop3. That, at the time, email service providers bolted on entire domain delivery to a single mailbox, and then users of those services put multiple recipients behind that single mailbox. Fetchmail made this work as best it could. Also missing at the time was any kind of serious connecting-client-as-MTA or MTA-to-MTA authentication (rather, relying on DNS and TCP!), which might have made async, receiver controlled MTA initiation more workable. This was also due to common SMTP MTAs not having good (or at least easy/obvious to configure) store-for-later-pickup functionality where an MTA could query for email destined for itself rather than be always-on connected to the internet waiting for another MTA to initiate delivery. Fetchmail filled a hole that existed in email delivery at the time, but it was really a hole that should have never existed. The quality of the fetchmail code is another, albeit serious, issue in itself. These days we have a better understanding of the differences between MTAs, MSAs, and MUAs.
- ggm 6y ago
  These days we have a better understanding of the differences between MTAs, MSAs, and MUAs. I don't want to be "that guy" but we've known about this distinction since the 1980s. I know, because I was fringing in the discussions in standards ml during the x.400 years.
  - 0xbadcafebee 6y ago
    I actually learned the difference between MTAs, MUAs, etc because of fetchmail. Fetchmail kept randomly crashing on select spam messages and my bug reports went unheeded (and I wasn't what anyone would call a programmer) so I needed a new way to download my mail. So I wrote a crappy Perl script. All it does is use standard Perl modules to download the mail, and pipes it off to an MDA (postfix, maildrop, etc) to deal with them. After nearly two decades, no matter what system I plunk it on or mail provider I use, the same stupid script just keeps working.
jstimpfle 6y ago
> When mail is delivered to a POP3 maildrop, envelope information is destroyed. To combat this, you would need to tunnel the enveleope information in headers. Isn't that exactly what Envelope-to is supposed to be? Protocol-level data? It's not really part of the e-mail, but added when sending it. If you forward the mail to another machine, there will be a new envelope. I can see that it's all a gray-ish area with email (because email is a mess (which might be part of why it's been successful)) but I can't see a reason to complain about losing Envelope-to.
znpy 6y ago
Would anyone that's well-versed in UUCP, Fetchmail and related mail protocols and technologies care to explain whether this discussion from 2001 (20 years ago!) is still relevant today?
mapgrep 6y ago
I have personally found that as you get serious about keeping your own archive of email you need something like fetchmail, but not fetchmail. I used getmail for years and it is solid. duckerude posted this elsewhere in the thread, where the author of getmail rants a bit about fetchmail. http://pyropus.ca/software/getmail/faq.html#faq-about-why http://pyropus.ca/software/getmail/faq.html#faq-about-why Once I left Gmail I was interested in something faster and more efficient, because I was using the transfer tool for more than just backup, it became the main way I receive email on my machine. Then the mail gets indexed (for text search) by another program and displayed by a third. Anyway I found mbsync, which is just this amazing, efficient, very solid transfer program that's been around for many years just quietly doing its job. I almost wrote it off because its homepage is on sourceforge but if you poke around a bit you'll see it is actively developed, there's an active mailing list, etc. I love it. It's kind of interesting how this project I had never heard of just does its job well and works and the program by the semi-famous software pundit whose programming essay went viral really does not (from what I have heard).
- hpfr 6y ago
  I just got started with mbsync and am having several issues with Gmail and Outlook.com. These are old accounts and I do have well-behaved IMAP emails from smaller hosts, but it would be nice to manage everything with mbsync. Trashing emails in mu4e and then syncing results in them being deleted from my local [Gmail]/Trash while they somehow remain in the remote. I thought Outlook.com would be fine considering they use more standard folders rather than the tagging system which seems to result in duplicates, but I have the same issues there. This is with Expunge None, which I expect would leave emails alone until the server empties them from the Trash. The bizarre thing is that no one online seems to have these issues? The mbsync configs for Gmail just seem to work for people, or something. I do have auto-expunge turned off in Gmail, so I’m kind of at my wit’s end with it. It’s a shame because it is an amazing program. It synced 10,000 emails without a hitch. I only started getting issues when I tried to sync from my maildir to the IMAP store.
  - fiddlerwoaroof 6y ago
    I’ve only run mbsync unidirectionally as a backup utility (in addition to tarsnap on my Mailserver). I wonder if the bidirectional features just aren’t used that much? Also, Gmail’s tags don’t map nicely onto folders.
kazinator 6y ago
Fetchmail was only supposed to be something like "offline IMAP" for someone with only occasional network connectivity, and whose mail program is too braindead to download and cache messages for offline reading when connected. Of course, fetchmail wouldn't have the envelope info, because that occurred in an SMTP chat that is not part of the message landing in your inbox. Something that accesses delivered mails in IMAP or POP3 boxes cannot be a full-blown MTA. E.e. we can't program a mail exchanger machine to accept the mails on behalf of hundreds of users, by delivering them into a giant mailbox, and then continuing the delivery path by sucking them out with a fetchmail-like tool, and forwarding to more forwarding hosts. The whole complaint of this irate BSD person is that you can't do this kind of nonsensical thing. There is a sort of valid criticism in that fetchmail (or at least did originally, I don't know about now) worked by re-sending the messages to the user over a mail transport mechanism, rather than delivering it into a mail directory structure. E-mail clients that download and cache messages certainly do not do anything of that sort. However, that shields fetchmail from having to know how the user's mail software likes to have the messages stored. Arguably, sending the mail is the right interface. Though the envelope info isn't available, it's assumed that it was already processed by the remote mail server. The remote server did all he spam checks on the SMTP info, and whatnot, decided that the message is deliverable to the user, so there it is. Fetchmail assumes that if the mail is in the remote box, then it's yours, and can be effectively forwarded. The envelope sender is Fetchmail itself. If that is broken, so are mailing lists. When you receive a message from someone via a mailing list, you also don't have the envelope which the mailing list robot's server received. The envelope is that of the mailing list's own delivery; the mailing list robot is the MAIL FROM: sender. For that matter, you are not even the original intended recipient! The mail was originally to the list, and now it is to you. Not having an envelope isn't a show-stopping impediment to further forwarding and delivery. You just have to understand and accept that it's more of a forwarding style of delivery, like a mailing list robot. Overall, it's almost like complaining that a dog which fetches the mail from your door and brings it to the kitchen is not a trained postal worker, does not understand and handle addresses right, and could not be reliably used to service an apartment building with six tenants, all by itself, such that the mailman could just put everything in one bin and leave it to the dog.

More context: http://pld.cs.luc.edu/courses/412/spr20/mnotes/bazaar.html http://pld.cs.luc.edu/courses/412/spr20/mnotes/bazaar.html

0xbadcafebee 6y ago

Can confirm that fetchmail is garbage. Found bugs, reported them, no fix. Wrote my own tool instead. Such is life in open source.

KirinDave 6y ago
I wonder if many folks who only joined the world of software development after DVCSs (and indeed, just VCSs) became popularized can understand why Fetchmail was not trivial to fork and fix.
aatharuv 6y ago
The only time I ever interacted with ESR on fetchmail, I reported a bug for an issue I ran into (with Kerberos 4 IMAP), submitted a patch, and it was accepted without discussion in the next release. Admittedly, this was 19 years ago, and it was a relatively small fix.
jjav 6y ago
I've been using fetchmail every two minutes (daemon mode) for well over two decades and not once has it failed to do what it needs to do. So it's a bit of a stretch to call it garbage. Other than emacs and mutt, there's no other tool with such stable longevity in my toolbox.
athms 6y ago
Under ESR, fetchmail was garbage. However, a new team took over development in the early 2000s after ESR abandoned the program. Many parts have been re-written to address design criticisms.

dvfjsdhgfv 6y ago

This doesn't seem so much about UUCP vs fetchmail as about cathedral vs bazaar style of development. In hindsight, it's clear that both of these work, for better or worse, with each having its own disadvantages and benefits.

mprovost 6y ago
Some context for those who are unaware: fetchmail was written by the same person (Eric S Raymond/ESR) as the "Cathedral and the Bazaar" essay.
- freedomben 6y ago
  If you have any interest in open source development (which these days we all should given that 95% of the lower parts of our stack are open source projects) it's well worth reading the essay/book. If you don't want to do that, at least read the wikipedia page: https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar
  - segfaultbuserr 6y ago
    I found this interesting post while reading some historical documents, so I submitted it to HN. It seems that most people have missed the point. I think the point is: while The Cathedral and the Bazaar was a good campaign pamphlet and a general overview of FOSS development, but ESR's attempt of using his fetchmail as an example was arguably the greatest flaw of the book ("to such an extent that it could damage the Open Source movement"). From a technical perspective, fetchmail had numerous problems and security vulnerabilities. ESR was also criticized for mismanaging the project by putting it in "maintenance mode" early and refusing to accept patches that could address those issues. After a group of new maintainers had taken over the project, one of the first things they did was to systematically eliminate countless buffer and integer overflows from the program.
  - tptacek 6y ago
    Not really. I read it when it came out and remembered liking it (we were all on kind of a buzz about open source and Linux at the time) and then again much more recently for a thread on HN, and I found it to be mostly a rehash of other, better material --- Brooks and Kernighan, in particular. That, plus the modern context that what he was writing about was Fetchmail, and Fetchmail as it turns out is hot garbage, to me makes the whole thing very skippable. It's also responsible for hatching one of the great canards of open source, "Linus's Law" (I'm sure Linus is thrilled with having his name attached to that), "many eyes make all bugs shallow", which has in the last 10 years become one of the great jokes of computer science.

rasengan 6y ago

I used to use UUCP to get my email when I was on a dialup, non permanent connection. I was able to host my own mail server. There were a lot of technologies back then helping the internet stay somewhat decentralized. Now I, like most, just use Gmail. What has become of our beloved free(digital)land?

cmrdporcupine 6y ago
Yep, I remember spooling usenet and email to my Atari ST over a 1200 baud modem over UUCP in 1990, 1991. Bang path and all. And you're right, the centralization is crazy, and opposite of what techno utopians predicted/promised at the time.
- vidarh 6y ago
  My first access to the web around '93 was via CERNs e-mail to web gateway coupled with UUCP exchange from my Amiga to my local BBS which exchanged with its upstream 4-6 times a day.
  - cmrdporcupine 6y ago
    Yeah I used to dial into the local university's gopher "campus information server" -- which was just a dial-up to telnet to a gopher client -- hit the ^] telnet escape character, close the connection, and re-open to another host where I had a shell and go from there. If I recall right it was probably gnu.ai.mit.edu that gave out free shell accounts, or something else in that domain connected somehow to Stallman. There was no ISP in my area back then, not even dial-up. Again, around 1991.
- toyg 6y ago
  Sadly, the upsides are just too good. I love decentralization, but I have to recognise that I've not lost a message since I moved to gmail in 2005; before then, losing entire mailbox files was not uncommon, as well as missing messages from unreliable mailservers, spam management, and so on. I'm sure I could have done better etc etc, but the bottom line is that with gmail-like services you don't have to. I enjoy doing sysadmin but not 24/7.

duckerude 6y ago

A similar rant: http://pyropus.ca/software/getmail/faq.html#faq-about-why http://pyropus.ca/software/getmail/faq.html#faq-about-why

thwarted 6y ago

Fetchmail grew to be a mail transport swiss army knife and encompassed half baked functionality of a domain-wide MTA rather focusing on single user async email consolidation and collection for a logical single account, which is largely what this rant is about. Half of the half-bakedness is due to lack of MTA protocol support for MUA interactions on the receiving side. pop3 was never meant as a spool for MTA to MTA email transfers, but was instead meant for an MUA to access. It's not fetchmail's fault that the envelope TO isn't available via pop3. That, at the time, email service providers bolted on entire domain delivery to a single mailbox, and then users of those services put multiple recipients behind that single mailbox. Fetchmail made this work as best it could. Also missing at the time was any kind of serious connecting-client-as-MTA or MTA-to-MTA authentication (rather, relying on DNS and TCP!), which might have made async, receiver controlled MTA initiation more workable. This was also due to common SMTP MTAs not having good (or at least easy/obvious to configure) store-for-later-pickup functionality where an MTA could query for email destined for itself rather than be always-on connected to the internet waiting for another MTA to initiate delivery. Fetchmail filled a hole that existed in email delivery at the time, but it was really a hole that should have never existed. The quality of the fetchmail code is another, albeit serious, issue in itself. These days we have a better understanding of the differences between MTAs, MSAs, and MUAs.

ggm 6y ago
These days we have a better understanding of the differences between MTAs, MSAs, and MUAs. I don't want to be "that guy" but we've known about this distinction since the 1980s. I know, because I was fringing in the discussions in standards ml during the x.400 years.
- 0xbadcafebee 6y ago
  I actually learned the difference between MTAs, MUAs, etc because of fetchmail. Fetchmail kept randomly crashing on select spam messages and my bug reports went unheeded (and I wasn't what anyone would call a programmer) so I needed a new way to download my mail. So I wrote a crappy Perl script. All it does is use standard Perl modules to download the mail, and pipes it off to an MDA (postfix, maildrop, etc) to deal with them. After nearly two decades, no matter what system I plunk it on or mail provider I use, the same stupid script just keeps working.

jstimpfle 6y ago

> When mail is delivered to a POP3 maildrop, envelope information is destroyed. To combat this, you would need to tunnel the enveleope information in headers. Isn't that exactly what Envelope-to is supposed to be? Protocol-level data? It's not really part of the e-mail, but added when sending it. If you forward the mail to another machine, there will be a new envelope. I can see that it's all a gray-ish area with email (because email is a mess (which might be part of why it's been successful)) but I can't see a reason to complain about losing Envelope-to.

znpy 6y ago

Would anyone that's well-versed in UUCP, Fetchmail and related mail protocols and technologies care to explain whether this discussion from 2001 (20 years ago!) is still relevant today?

mapgrep 6y ago

I have personally found that as you get serious about keeping your own archive of email you need something like fetchmail, but not fetchmail. I used getmail for years and it is solid. duckerude posted this elsewhere in the thread, where the author of getmail rants a bit about fetchmail. http://pyropus.ca/software/getmail/faq.html#faq-about-why http://pyropus.ca/software/getmail/faq.html#faq-about-why Once I left Gmail I was interested in something faster and more efficient, because I was using the transfer tool for more than just backup, it became the main way I receive email on my machine. Then the mail gets indexed (for text search) by another program and displayed by a third. Anyway I found mbsync, which is just this amazing, efficient, very solid transfer program that's been around for many years just quietly doing its job. I almost wrote it off because its homepage is on sourceforge but if you poke around a bit you'll see it is actively developed, there's an active mailing list, etc. I love it. It's kind of interesting how this project I had never heard of just does its job well and works and the program by the semi-famous software pundit whose programming essay went viral really does not (from what I have heard).

hpfr 6y ago
I just got started with mbsync and am having several issues with Gmail and Outlook.com. These are old accounts and I do have well-behaved IMAP emails from smaller hosts, but it would be nice to manage everything with mbsync. Trashing emails in mu4e and then syncing results in them being deleted from my local [Gmail]/Trash while they somehow remain in the remote. I thought Outlook.com would be fine considering they use more standard folders rather than the tagging system which seems to result in duplicates, but I have the same issues there. This is with Expunge None, which I expect would leave emails alone until the server empties them from the Trash. The bizarre thing is that no one online seems to have these issues? The mbsync configs for Gmail just seem to work for people, or something. I do have auto-expunge turned off in Gmail, so I’m kind of at my wit’s end with it. It’s a shame because it is an amazing program. It synced 10,000 emails without a hitch. I only started getting issues when I tried to sync from my maildir to the IMAP store.
- fiddlerwoaroof 6y ago
  I’ve only run mbsync unidirectionally as a backup utility (in addition to tarsnap on my Mailserver). I wonder if the bidirectional features just aren’t used that much? Also, Gmail’s tags don’t map nicely onto folders.

kazinator 6y ago

Fetchmail was only supposed to be something like "offline IMAP" for someone with only occasional network connectivity, and whose mail program is too braindead to download and cache messages for offline reading when connected. Of course, fetchmail wouldn't have the envelope info, because that occurred in an SMTP chat that is not part of the message landing in your inbox. Something that accesses delivered mails in IMAP or POP3 boxes cannot be a full-blown MTA. E.e. we can't program a mail exchanger machine to accept the mails on behalf of hundreds of users, by delivering them into a giant mailbox, and then continuing the delivery path by sucking them out with a fetchmail-like tool, and forwarding to more forwarding hosts. The whole complaint of this irate BSD person is that you can't do this kind of nonsensical thing. There is a sort of valid criticism in that fetchmail (or at least did originally, I don't know about now) worked by re-sending the messages to the user over a mail transport mechanism, rather than delivering it into a mail directory structure. E-mail clients that download and cache messages certainly do not do anything of that sort. However, that shields fetchmail from having to know how the user's mail software likes to have the messages stored. Arguably, sending the mail is the right interface. Though the envelope info isn't available, it's assumed that it was already processed by the remote mail server. The remote server did all he spam checks on the SMTP info, and whatnot, decided that the message is deliverable to the user, so there it is. Fetchmail assumes that if the mail is in the remote box, then it's yours, and can be effectively forwarded. The envelope sender is Fetchmail itself. If that is broken, so are mailing lists. When you receive a message from someone via a mailing list, you also don't have the envelope which the mailing list robot's server received. The envelope is that of the mailing list's own delivery; the mailing list robot is the MAIL FROM: sender. For that matter, you are not even the original intended recipient! The mail was originally to the list, and now it is to you. Not having an envelope isn't a show-stopping impediment to further forwarding and delivery. You just have to understand and accept that it's more of a forwarding style of delivery, like a mailing list robot. Overall, it's almost like complaining that a dog which fetches the mail from your door and brings it to the kitchen is not a trained postal worker, does not understand and handle addresses right, and could not be reliably used to service an apartment building with six tenants, all by itself, such that the mailman could just put everything in one bin and leave it to the dog.