11 ms·

Some thoughts on security after ten years of Qmail 1.0

137 points by bello 9y ago

farnsworthy 9y ago
Nice summarizing article, with some programming concepts—explicit data flow, for example—that are even more generally applicable (though the topics of security and code volume/quality be linked).
jlgaddis 9y ago
Damn, it's been nearly 20 years since qmail 1.03 was released (June 1998)? It sure doesn't seem like that long! I recall setting up qmail "toasters" on FreeBSD to do virtual hosting. Maybe I was just too much of a "n00b" but I remember it being a big PITA to get all the services to play well together. There was this hip new outfit named Yahoo! that was using it for their new webmail service, though -- as opposed to sendmail, which pretty much every MTA on the Internet used at the time (and I was proficient enough with sendmail that I would edit my sendmail.cf by hand; pffft, who needs m4!?) -- so I assumed it was certainly capable of handling my volume of mail. (I wasn't running authoritative DNS servers at the time or I probably would've used djbdns over BIND as well.) qmail, unfortunately, never did become too popular (relatively speaking, of course) and that's really a shame, because, as the quote in the article says: > "We need invulnerable software systems, and we need them today, ..." While that was certainly true then, it's even more true now. On a side note, I'm surprised that the "qmail security guarantee" [0,1] wasn't mentioned in the article: > "In March 1997, I took the unusual step of publicly offering $500 to the first person to publish a verifiable security hole in the latest version of qmail: for example, a way for a user to exploit qmail to take over another account. My offer still stands. Nobody has found any security holes in qmail. I hereby increase the offer to $1000." [0]: https://cr.yp.to/qmail/guarantee.html https://cr.yp.to/qmail/guarantee.html [1]: https://cr.yp.to/qmail/qmailsec-20071101.pdf https://cr.yp.to/qmail/qmailsec-20071101.pdf (PDF)
- djsumdog 9y ago
  I remember qmail being the first MTA to really push Maildirs. I ran qmail personally back then on my Linux fom Scratch, but I also was a student lab admin and I think on our student e-mail server, we still ran sendmail at the time, on good old Redhat (back before it was split into RHEL and Fedora). Software like qmail and the dev file system at the time really rubbed a lot of people the wrong way because of the drastic design changes they push. I'm glad that particular dev file system died as it had a lot of weirdly named nodes and a devfs daemon that had to run to create symbolic links to all the known names.
  - zAy0LfpBZLC8mAC 9y ago
    Well, Maildir was invented by djb ;-)
    - arca_vorago 9y ago
      DJB is probably one of my favorite people in the tech world. Ever since I read about the court case he won against the US government while representing himself, he's been a sort of hero of mine.
      - oblio 9y ago
        That's quite impressive, but Wikipedia says that case was dismissed: https://en.wikipedia.org/wiki/Bernstein_v._United_States https://en.wikipedia.org/wiki/Bernstein_v._United_States
        tinus_hn 9y ago
        The rules he was challenging had changed so what he wanted to do was no longer against the rules. His case was that the rules did not allow him to do something that was allowed by the constitution. As that situation no longer existed the case was dismissed.
- SwellJoe 9y ago
  While qmail has faded in popularity as it has been sporadically maintained by a random bunch of folks over the years, there has been at least one other MTA written by someone with excellent security cred, and that has been continually maintained and has an excellent security record. We don't really need to mourn what could have been with qmail; we have Postfix, and it's really very good.
  - jlgaddis 9y ago
    Yep. With a few exceptions, Postfix is the MTA I've used pretty much everywhere for the last 10 years or so.
  - JdeBP 9y ago
    Making a world-readable, world-searchable, and world-writable drop directory because of a decision to have no set-UID and set-GID executables in Postfix, even appropriate ones; failing to learn the even then well-known lessons of the batch job (at), UUCP, and printing (lpr) subsystems when it comes to world-accessible input directories; was a fairly large blot. * https://cr.yp.to/maildisasters/postfix.19981221 https://cr.yp.to/maildisasters/postfix.19981221 * https://cr.yp.to/maildisasters/postfix.html https://cr.yp.to/maildisasters/postfix.html * https://groups.google.com/forum/#!msg/mailing.postfix.users/Bif9N7Nx8gM/BoJH5ibspvwJ https://groups.google.com/forum/#!msg/mailing.postfix.users/...
- geocar 9y ago
  > qmail, unfortunately, never did become too popular At one point, it was the second most popular MTA on the Internet. What pray tell would "too popular" look like? > I remember it being a big PITA to get all the services to play well together. When you were thinking about qmail correctly, it was an absolute pleasure to get everything to work together. Promise. Yet whilst the documentation was correct, it probably wasn't very good from the perspective of helping people think about it correctly. André Oppermann[1] (and perhaps Dave Sill[2]) did a much better job of this, so when they came available I would usually have pointed people there and see what kinds of questions they still had. [1]: http://www.nrg4u.com/ http://www.nrg4u.com/ [2]: http://www.lifewithqmail.org/lwq.html http://www.lifewithqmail.org/lwq.html
  - zakki 9y ago
    qmailtoaster.org maintained by Eric Broch remains updated regularly. The installation process is easy and you get current email server ‘requirements’ installed as well, i.e. spam filter, dkim, active sync, etc.
diafygi 9y ago
As we cast about trying to figure out ways to make software more secure or reliable, please remember that in other engineering fields (civil, chemical, mechanical, etc.) prioritizing safety and reliability is a _solved problem_. (1996) https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff > It is perfect, as perfect as human beings have achieved. Consider these stats: the last three versions of the program — each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors. > The process isn’t even rocket science. Its standard practice in almost every engineering discipline except software engineering. The problem is consequences. We had centuries of people dying in bridge collapses before we got our shit together and started prioritizing safety in civil engineering (i.e. engineers and managers going to prison if they don't). The same will be true for software. As more people get harmed by thrown together software (e.g. mass panic in Hawaii, state psychological exploitation on social media), we'll start regulating it like other engineering fields. As a former chemical engineer, I welcome this transition, but I realize it will likely also take centuries of hard lessons.
- pjmlp 9y ago
  That is my point of view as well, software needs to be liable just like in other fields of engineering, done by people that actually understand its consequences. I don't do a car inspection at a guy that just happens to know some stuff about mechanics.
- Joeri 9y ago
  The difference is that while you can’t make a bridge in your bedroom you can make an app. Should we forbid people from writing code without the proper certification? Should we close down the open internet and replace it with a regulated zone where only compliant software can be run? I agree that we need a higher standard of engineering in software, but I’m not clear on how to achieve it without draconian measures.
  - geocar 9y ago
    In the UK, you can build a bridge without certification as long as you have someone certified review the plans and the implementation before anyone else drives on it. I suspect this is similar for most civilised countries. A (perhaps short-term) idea would be to make software vendors liable, and do not permit them to sign away that liability.
pmoriarty 9y ago
My biggest takeaway from qmail has nothing to do with security, but rather that excessively restrictive licensing, highly opinionanted/unusual setup, and unwillingness to collaborate on its development squandered its potential. If it wasn't for all that, we might well all be using qmail-based mail servers today, as qmail was really ahead of its time in so many ways. It was kind of like the Amiga of mail servers, back in the day. It could have easily dominated the market, but it wound up a mere historical curiosity.
- b6 9y ago
  I think you're missing something if that's what you got out of qmail. The main idea I internalized was, if you find yourself making programming mistakes, take some time to write an API that does not allow you to make that kind of mistake and commit to using it everywhere. The licensing stuff, I'm not super clear on. If memory serves, DJB did not want to worry about bugs introduced by patches he'd never seen or approved, added by package maintainers for OSes he didn't use. As for the somewhat weird ecosystem (daemontools), I think it wasn't that it wasn't good enough, it was just that people always find reasons to be dissatisfied. I can't even keep track of the latest reinvention of whatever people are using instead of daemontools, but I bet it's a hundred times as complex, and much less reliable.
  - JdeBP 9y ago
    The licensing stuff was simple: There wasn't a licence. Some people creatively misinterpreted M. Bernstein's page, explaining what one could do under the law in the U.S. as it stood even without a copyright licence, as a licence. But that was a misinterpretation. * http://jdebp.eu./FGA/law-licence-free-softwares.html http://jdebp.eu./FGA/law-licence-free-softwares.html Daemontools ironically wasn't weird at all, as evidenced by the fact that over the past almost two decades the daemontools world got us changes to softwares, all of those do not daemonize options that have appeared in that time as well as things like removing mysql_safe and other Poor Man's Dæmon Supervisors, that made it a lot easier to use those softwares with other service managers. * http://jdebp.eu./Softwares/nosh/mariadb-and-mysql.html#Prompt http://jdebp.eu./Softwares/nosh/mariadb-and-mysql.html#Promp... Some of the things that people use instead of daemontools are not much less reliable. (-: * http://jdebp.eu./FGA/daemontools-family.html http://jdebp.eu./FGA/daemontools-family.html
pilif 9y ago
Something to keep in mind with regards to qmail is that it's extremely feature-poor and it never got features beyond its initial design goal. This makes it much easier to keep the bugs out, to the point that making software under such constraints is much more similar to traditional construction projects. I mean: Nobody ever tells you after you have built a bridge that they are now going to upgrade gravity to gravity 2.0 with 100% more pull. And nobody will ever tell you that your bridge will now get a shopping mall in the middle of it where people can purchase products of their favorite brands. Software starts to break down when it has to be taken above initial design constraints and when there is not enough time to rewrite subsystems (or all of it) but instead when you have to make the abstractions leaky and compromise. But back to qmail: qmail itself is so feature-poor that traditionally, nobody was and is actually running qmail. Instead everybody is running "qmail" which is qmail plus some patches. Sometimes home-grown, sometimes taken from third parties. But more often than not they are unmaintained and very far removed from the high quality standards of the underlying software. This is the downside. Yes. You have a bug-free core that totally meets its designers (limited) use-case, but in reality nobody is actually running that.
- geocar 9y ago
  In contrast, almost nobody runs "exim plus some home-grown patches" or "postfix plus some home-grown patches". Having the correct architecture, and being able to rewrite the subsystems piecemeal means it is possible for users to experiment with new features organically[1]. That's part of why some qmail installations have features not available in any other mail server even today. Or to put another way, software purity and homogeny isn't a "good thing", but a trade off: You get to share risk with everyone else who chose like you did, but you're also stuck with the same features and risk everyone else has. I'd choose "feature-poor download, but highest-feature in production" over "a-few-more-features download and limited-ability-to-upgrade" any day. [1]: If you're curious, some of the experiments I did are briefly mentioned here: https://news.ycombinator.com/item?id=16166530 https://news.ycombinator.com/item?id=16166530
  - vidarh 9y ago
    I built a webmail system on top of Qmail back in the day, and I loved it. We rewrote component by component as our needs changed, but the beauty of Qmail was that we could rewrite component by component by sticking to very simple contracts between them and/or start with the Qmail components themselves that for the most part are extremely simple. Our web frontend for example worked on top of a slightly modified Qmail POP3 server that relied on encoding some message state in the filename, so we need only scan the Maildir without opening the files to be able to e.g. get message read state, and various flags. We also added caching of metadata etc. The changes were tiny and self-contained, and added one extra non-standard command to the pop3 server to retrieve a message list with much more data. The design of Qmail let us start with just changing the pop3 server, and later some tweaks to local delivery, and know we could test those programs in isolation. The flexibility of Qmail even got us to use it as a messaging middleware later coupled with tinydns - all the routing and retry logic made it very convenient and extremely simple to troubleshoot.
viraptor 9y ago
I still don't get djb's distinction between untrusted and minimal privilege code. What he calls "not violating security requirements" is effectively a successful least privilege approach. Very few elements can become hacked without breaking security requirements. If you can't gain anything from hacking a piece of software, then why is it even executed? - it obviously didn't deal with anything the user wants. In his example, yes, you could change the DNS responses, but you still could not escalate to a higher lever where you can potentially modify stored user data. That is a success in practice.
- twhb 9y ago
  I think he intends “privilege” to refer only to filesystem and other OS-level privileges, not more generally to the capabilities of code, and I think he uses “untrusted” to mean minimally-trusted—more restricted than the OS can enforce. Taking the DNS Helper example, one could imagine a function-like DNS Helper which has the capability only to return a value. This would make libresolv just a bug, not a security hole, because the attacker would only pervert their own request.
- geocar 9y ago
  Something that can respond to a DNS request can put whatever it wants in the response. If there's a bug in that program, then whoever controls that bug can put whatever they want in the response. The only protection from this is to make the code that does this as small as possible so that us human beings can convince ourselves that it is correct and that the risk of a bug that someone can control is zero (or as close to zero as to make no odds). When Jim Reid wants to pat himself on the back because "at least they didn't get root on my nameserver box", he misses the point: gethostbyname()'s spec doesn't say "it may or may not return. if it returns it could return anything. don't trust it, don't even use it!" They say gethostbyname() return a structure describing the address of the named Internet host, so people expect that and depend on that. Something that "suddenly" violates that gets in the news[1]. Fortunately, nobody remembers what Jim said so the BBC doesn't ask him for a comment. Anyway. "Minimizing privilege" doesn't solve that problem because the DNS server needs the privilege to respond to DNS requests. It might be easier to think about a better example. Let's talk about zlib. A program that needs to decompress some text is not concerned with the contents of the compressed text, only the uncompressed text. Resource limits on our program exist to keep some things from getting out of control[2], but what about bugs? If we could run zlib's decompress() with the permission only to decompress text, then the worst-case impact would either spin the cpu or be equivalent to "getting out of control". What do we need to do that? • No creating file descriptors can be done with setrlimit() except for the dynamic linker is going to open a shittonne of files. We need to know what the minimum number of files are, and decompress can't ever change that without changing our program anyway. • No accessing files or the network could be done with a setuid wrapper and iptables. At least on Linux. Most programmers don't do this, and most sysadmins only do what they're told, so in practice this doesn't happen. • Sandboxing! Google published some clever user-level sandboxing that works on Linux to whitelist each syscall. This "verifier" could do it as long as it's smaller than decompress()! That sandboxing one is tricky: A tiny inflate routine takes around 500 lines of C done the normal way, but how big is our sandbox? Probably a lot bigger. • Ask the operating system for help! This is what DJB suggests. Ask for a disablefiles and a disablenetwork system call. OpenBSD is implementing this with their pledge[3] system call. There's not a portable and satisfying solution here yet, but you can see they all cluster around reducing the privileges of the untrusted program. Now, what's to prevent decompress from lying? What if someone can produce a content stream that causes a future decompress run from producing invalid results. Maybe something really sneaky[4]. What possible protection could we have? As you can see, in this case so long as decompress is supposed to produce "text", there's nothing we can do to make sure it produces the "correct text". That's why DJB doesn't want to focus on the "untrusted" aspect, and instead on trying to solve the problem that we have to solve anyway: How do we write software that is correct? [1]: http://news.bbc.co.uk/2/hi/technology/7496735.stm http://news.bbc.co.uk/2/hi/technology/7496735.stm [2]: https://swtch.com/r.gz https://swtch.com/r.gz [3]: https://man.openbsd.org/pledge.2 https://man.openbsd.org/pledge.2 [4]: https://cmaurice.fr/pdf/ndss17_maurice.pdf https://cmaurice.fr/pdf/ndss17_maurice.pdf
1110001110 9y ago
Interesting article, the only thing I fail to see how this is related to Meltdown and Spectre. Those are not simple 'bugs', it's multiple good features of modern processors combined to yield an attack vector. My opinion is that with any level of process problems like this will arise sooner or later just because the complexity is so high.
- dchest 9y ago
  They are caused by performance optimizations with disregard to security (likely not intentional, but caused by not considering security aspects of optimizations carefully).
joveian 9y ago
My favorite quote from that paper is "I have discovered that there are two types of command interfaces in the world of computing: good interfaces and user interfaces." As others have pointed out, one thing left out of the paper is not updating the software. qmail doesn't support SPF or other security extensions, which makes it useless these days without patches.

137 points by bello 9y ago

Nice summarizing article, with some programming concepts—explicit data flow, for example—that are even more generally applicable (though the topics of security and code volume/quality be linked).

jlgaddis 9y ago

Damn, it's been nearly 20 years since qmail 1.03 was released (June 1998)? It sure doesn't seem like that long! I recall setting up qmail "toasters" on FreeBSD to do virtual hosting. Maybe I was just too much of a "n00b" but I remember it being a big PITA to get all the services to play well together. There was this hip new outfit named Yahoo! that was using it for their new webmail service, though -- as opposed to sendmail, which pretty much every MTA on the Internet used at the time (and I was proficient enough with sendmail that I would edit my sendmail.cf by hand; pffft, who needs m4!?) -- so I assumed it was certainly capable of handling my volume of mail. (I wasn't running authoritative DNS servers at the time or I probably would've used djbdns over BIND as well.) qmail, unfortunately, never did become too popular (relatively speaking, of course) and that's really a shame, because, as the quote in the article says: > "We need invulnerable software systems, and we need them today, ..." While that was certainly true then, it's even more true now. On a side note, I'm surprised that the "qmail security guarantee" [0,1] wasn't mentioned in the article: > "In March 1997, I took the unusual step of publicly offering $500 to the first person to publish a verifiable security hole in the latest version of qmail: for example, a way for a user to exploit qmail to take over another account. My offer still stands. Nobody has found any security holes in qmail. I hereby increase the offer to $1000." [0]: https://cr.yp.to/qmail/guarantee.html https://cr.yp.to/qmail/guarantee.html [1]: https://cr.yp.to/qmail/qmailsec-20071101.pdf https://cr.yp.to/qmail/qmailsec-20071101.pdf (PDF)

djsumdog 9y ago
I remember qmail being the first MTA to really push Maildirs. I ran qmail personally back then on my Linux fom Scratch, but I also was a student lab admin and I think on our student e-mail server, we still ran sendmail at the time, on good old Redhat (back before it was split into RHEL and Fedora). Software like qmail and the dev file system at the time really rubbed a lot of people the wrong way because of the drastic design changes they push. I'm glad that particular dev file system died as it had a lot of weirdly named nodes and a devfs daemon that had to run to create symbolic links to all the known names.
- zAy0LfpBZLC8mAC 9y ago
  Well, Maildir was invented by djb ;-)
  - arca_vorago 9y ago
    DJB is probably one of my favorite people in the tech world. Ever since I read about the court case he won against the US government while representing himself, he's been a sort of hero of mine.
    - oblio 9y ago
      That's quite impressive, but Wikipedia says that case was dismissed: https://en.wikipedia.org/wiki/Bernstein_v._United_States https://en.wikipedia.org/wiki/Bernstein_v._United_States
      - tinus_hn 9y ago
        The rules he was challenging had changed so what he wanted to do was no longer against the rules. His case was that the rules did not allow him to do something that was allowed by the constitution. As that situation no longer existed the case was dismissed.
SwellJoe 9y ago
While qmail has faded in popularity as it has been sporadically maintained by a random bunch of folks over the years, there has been at least one other MTA written by someone with excellent security cred, and that has been continually maintained and has an excellent security record. We don't really need to mourn what could have been with qmail; we have Postfix, and it's really very good.
- jlgaddis 9y ago
  Yep. With a few exceptions, Postfix is the MTA I've used pretty much everywhere for the last 10 years or so.
- JdeBP 9y ago
  Making a world-readable, world-searchable, and world-writable drop directory because of a decision to have no set-UID and set-GID executables in Postfix, even appropriate ones; failing to learn the even then well-known lessons of the batch job (at), UUCP, and printing (lpr) subsystems when it comes to world-accessible input directories; was a fairly large blot. * https://cr.yp.to/maildisasters/postfix.19981221 https://cr.yp.to/maildisasters/postfix.19981221 * https://cr.yp.to/maildisasters/postfix.html https://cr.yp.to/maildisasters/postfix.html * https://groups.google.com/forum/#!msg/mailing.postfix.users/Bif9N7Nx8gM/BoJH5ibspvwJ https://groups.google.com/forum/#!msg/mailing.postfix.users/...
geocar 9y ago
> qmail, unfortunately, never did become too popular At one point, it was the second most popular MTA on the Internet. What pray tell would "too popular" look like? > I remember it being a big PITA to get all the services to play well together. When you were thinking about qmail correctly, it was an absolute pleasure to get everything to work together. Promise. Yet whilst the documentation was correct, it probably wasn't very good from the perspective of helping people think about it correctly. André Oppermann[1] (and perhaps Dave Sill[2]) did a much better job of this, so when they came available I would usually have pointed people there and see what kinds of questions they still had. [1]: http://www.nrg4u.com/ http://www.nrg4u.com/ [2]: http://www.lifewithqmail.org/lwq.html http://www.lifewithqmail.org/lwq.html
- zakki 9y ago
  qmailtoaster.org maintained by Eric Broch remains updated regularly. The installation process is easy and you get current email server ‘requirements’ installed as well, i.e. spam filter, dkim, active sync, etc.

diafygi 9y ago

As we cast about trying to figure out ways to make software more secure or reliable, please remember that in other engineering fields (civil, chemical, mechanical, etc.) prioritizing safety and reliability is a _solved problem_. (1996) https://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff > It is perfect, as perfect as human beings have achieved. Consider these stats: the last three versions of the program — each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors. > The process isn’t even rocket science. Its standard practice in almost every engineering discipline except software engineering. The problem is consequences. We had centuries of people dying in bridge collapses before we got our shit together and started prioritizing safety in civil engineering (i.e. engineers and managers going to prison if they don't). The same will be true for software. As more people get harmed by thrown together software (e.g. mass panic in Hawaii, state psychological exploitation on social media), we'll start regulating it like other engineering fields. As a former chemical engineer, I welcome this transition, but I realize it will likely also take centuries of hard lessons.

pjmlp 9y ago
That is my point of view as well, software needs to be liable just like in other fields of engineering, done by people that actually understand its consequences. I don't do a car inspection at a guy that just happens to know some stuff about mechanics.
Joeri 9y ago
The difference is that while you can’t make a bridge in your bedroom you can make an app. Should we forbid people from writing code without the proper certification? Should we close down the open internet and replace it with a regulated zone where only compliant software can be run? I agree that we need a higher standard of engineering in software, but I’m not clear on how to achieve it without draconian measures.
- geocar 9y ago
  In the UK, you can build a bridge without certification as long as you have someone certified review the plans and the implementation before anyone else drives on it. I suspect this is similar for most civilised countries. A (perhaps short-term) idea would be to make software vendors liable, and do not permit them to sign away that liability.

pmoriarty 9y ago

My biggest takeaway from qmail has nothing to do with security, but rather that excessively restrictive licensing, highly opinionanted/unusual setup, and unwillingness to collaborate on its development squandered its potential. If it wasn't for all that, we might well all be using qmail-based mail servers today, as qmail was really ahead of its time in so many ways. It was kind of like the Amiga of mail servers, back in the day. It could have easily dominated the market, but it wound up a mere historical curiosity.

b6 9y ago
I think you're missing something if that's what you got out of qmail. The main idea I internalized was, if you find yourself making programming mistakes, take some time to write an API that does not allow you to make that kind of mistake and commit to using it everywhere. The licensing stuff, I'm not super clear on. If memory serves, DJB did not want to worry about bugs introduced by patches he'd never seen or approved, added by package maintainers for OSes he didn't use. As for the somewhat weird ecosystem (daemontools), I think it wasn't that it wasn't good enough, it was just that people always find reasons to be dissatisfied. I can't even keep track of the latest reinvention of whatever people are using instead of daemontools, but I bet it's a hundred times as complex, and much less reliable.
- JdeBP 9y ago
  The licensing stuff was simple: There wasn't a licence. Some people creatively misinterpreted M. Bernstein's page, explaining what one could do under the law in the U.S. as it stood even without a copyright licence, as a licence. But that was a misinterpretation. * http://jdebp.eu./FGA/law-licence-free-softwares.html http://jdebp.eu./FGA/law-licence-free-softwares.html Daemontools ironically wasn't weird at all, as evidenced by the fact that over the past almost two decades the daemontools world got us changes to softwares, all of those do not daemonize options that have appeared in that time as well as things like removing mysql_safe and other Poor Man's Dæmon Supervisors, that made it a lot easier to use those softwares with other service managers. * http://jdebp.eu./Softwares/nosh/mariadb-and-mysql.html#Prompt http://jdebp.eu./Softwares/nosh/mariadb-and-mysql.html#Promp... Some of the things that people use instead of daemontools are not much less reliable. (-: * http://jdebp.eu./FGA/daemontools-family.html http://jdebp.eu./FGA/daemontools-family.html

pilif 9y ago

Something to keep in mind with regards to qmail is that it's extremely feature-poor and it never got features beyond its initial design goal. This makes it much easier to keep the bugs out, to the point that making software under such constraints is much more similar to traditional construction projects. I mean: Nobody ever tells you after you have built a bridge that they are now going to upgrade gravity to gravity 2.0 with 100% more pull. And nobody will ever tell you that your bridge will now get a shopping mall in the middle of it where people can purchase products of their favorite brands. Software starts to break down when it has to be taken above initial design constraints and when there is not enough time to rewrite subsystems (or all of it) but instead when you have to make the abstractions leaky and compromise. But back to qmail: qmail itself is so feature-poor that traditionally, nobody was and is actually running qmail. Instead everybody is running "qmail" which is qmail plus some patches. Sometimes home-grown, sometimes taken from third parties. But more often than not they are unmaintained and very far removed from the high quality standards of the underlying software. This is the downside. Yes. You have a bug-free core that totally meets its designers (limited) use-case, but in reality nobody is actually running that.

geocar 9y ago
In contrast, almost nobody runs "exim plus some home-grown patches" or "postfix plus some home-grown patches". Having the correct architecture, and being able to rewrite the subsystems piecemeal means it is possible for users to experiment with new features organically[1]. That's part of why some qmail installations have features not available in any other mail server even today. Or to put another way, software purity and homogeny isn't a "good thing", but a trade off: You get to share risk with everyone else who chose like you did, but you're also stuck with the same features and risk everyone else has. I'd choose "feature-poor download, but highest-feature in production" over "a-few-more-features download and limited-ability-to-upgrade" any day. [1]: If you're curious, some of the experiments I did are briefly mentioned here: https://news.ycombinator.com/item?id=16166530 https://news.ycombinator.com/item?id=16166530
- vidarh 9y ago
  I built a webmail system on top of Qmail back in the day, and I loved it. We rewrote component by component as our needs changed, but the beauty of Qmail was that we could rewrite component by component by sticking to very simple contracts between them and/or start with the Qmail components themselves that for the most part are extremely simple. Our web frontend for example worked on top of a slightly modified Qmail POP3 server that relied on encoding some message state in the filename, so we need only scan the Maildir without opening the files to be able to e.g. get message read state, and various flags. We also added caching of metadata etc. The changes were tiny and self-contained, and added one extra non-standard command to the pop3 server to retrieve a message list with much more data. The design of Qmail let us start with just changing the pop3 server, and later some tweaks to local delivery, and know we could test those programs in isolation. The flexibility of Qmail even got us to use it as a messaging middleware later coupled with tinydns - all the routing and retry logic made it very convenient and extremely simple to troubleshoot.

viraptor 9y ago

I still don't get djb's distinction between untrusted and minimal privilege code. What he calls "not violating security requirements" is effectively a successful least privilege approach. Very few elements can become hacked without breaking security requirements. If you can't gain anything from hacking a piece of software, then why is it even executed? - it obviously didn't deal with anything the user wants. In his example, yes, you could change the DNS responses, but you still could not escalate to a higher lever where you can potentially modify stored user data. That is a success in practice.

twhb 9y ago
I think he intends “privilege” to refer only to filesystem and other OS-level privileges, not more generally to the capabilities of code, and I think he uses “untrusted” to mean minimally-trusted—more restricted than the OS can enforce. Taking the DNS Helper example, one could imagine a function-like DNS Helper which has the capability only to return a value. This would make libresolv just a bug, not a security hole, because the attacker would only pervert their own request.
geocar 9y ago
Something that can respond to a DNS request can put whatever it wants in the response. If there's a bug in that program, then whoever controls that bug can put whatever they want in the response. The only protection from this is to make the code that does this as small as possible so that us human beings can convince ourselves that it is correct and that the risk of a bug that someone can control is zero (or as close to zero as to make no odds). When Jim Reid wants to pat himself on the back because "at least they didn't get root on my nameserver box", he misses the point: gethostbyname()'s spec doesn't say "it may or may not return. if it returns it could return anything. don't trust it, don't even use it!" They say gethostbyname() return a structure describing the address of the named Internet host, so people expect that and depend on that. Something that "suddenly" violates that gets in the news[1]. Fortunately, nobody remembers what Jim said so the BBC doesn't ask him for a comment. Anyway. "Minimizing privilege" doesn't solve that problem because the DNS server needs the privilege to respond to DNS requests. It might be easier to think about a better example. Let's talk about zlib. A program that needs to decompress some text is not concerned with the contents of the compressed text, only the uncompressed text. Resource limits on our program exist to keep some things from getting out of control[2], but what about bugs? If we could run zlib's decompress() with the permission only to decompress text, then the worst-case impact would either spin the cpu or be equivalent to "getting out of control". What do we need to do that? • No creating file descriptors can be done with setrlimit() except for the dynamic linker is going to open a shittonne of files. We need to know what the minimum number of files are, and decompress can't ever change that without changing our program anyway. • No accessing files or the network could be done with a setuid wrapper and iptables. At least on Linux. Most programmers don't do this, and most sysadmins only do what they're told, so in practice this doesn't happen. • Sandboxing! Google published some clever user-level sandboxing that works on Linux to whitelist each syscall. This "verifier" could do it as long as it's smaller than decompress()! That sandboxing one is tricky: A tiny inflate routine takes around 500 lines of C done the normal way, but how big is our sandbox? Probably a lot bigger. • Ask the operating system for help! This is what DJB suggests. Ask for a disablefiles and a disablenetwork system call. OpenBSD is implementing this with their pledge[3] system call. There's not a portable and satisfying solution here yet, but you can see they all cluster around reducing the privileges of the untrusted program. Now, what's to prevent decompress from lying? What if someone can produce a content stream that causes a future decompress run from producing invalid results. Maybe something really sneaky[4]. What possible protection could we have? As you can see, in this case so long as decompress is supposed to produce "text", there's nothing we can do to make sure it produces the "correct text". That's why DJB doesn't want to focus on the "untrusted" aspect, and instead on trying to solve the problem that we have to solve anyway: How do we write software that is correct? [1]: http://news.bbc.co.uk/2/hi/technology/7496735.stm http://news.bbc.co.uk/2/hi/technology/7496735.stm [2]: https://swtch.com/r.gz https://swtch.com/r.gz [3]: https://man.openbsd.org/pledge.2 https://man.openbsd.org/pledge.2 [4]: https://cmaurice.fr/pdf/ndss17_maurice.pdf https://cmaurice.fr/pdf/ndss17_maurice.pdf

1110001110 9y ago

Interesting article, the only thing I fail to see how this is related to Meltdown and Spectre. Those are not simple 'bugs', it's multiple good features of modern processors combined to yield an attack vector. My opinion is that with any level of process problems like this will arise sooner or later just because the complexity is so high.

dchest 9y ago
They are caused by performance optimizations with disregard to security (likely not intentional, but caused by not considering security aspects of optimizations carefully).

joveian 9y ago

My favorite quote from that paper is "I have discovered that there are two types of command interfaces in the world of computing: good interfaces and user interfaces." As others have pointed out, one thing left out of the paper is not updating the software. qmail doesn't support SPF or other security extensions, which makes it useless these days without patches.