12 ms·

A Hacker's Replacement for Gmail

464 points by dbpatterson 13y ago

LeafStorm 13y ago
While I don't necessarily trust an external company with all my emails, I also don't trust myself to maintain the myriad daemons involved in this setup without doing something subtly wrong that results in my server not sending/receiving all the mail it should -- or, worse, being used for spam. What would be useful is a pre-assembled virtual machine image or other form of appliance that allows you to deploy and test a mail server within about an hour or so, without having to duct-tape any of this together yourself.
- ics 13y ago
  > What would be useful is a pre-assembled virtual machine image or other form of appliance that allows you to deploy and test a mail server within about an hour or so, without having to duct-tape any of this together yourself. There at least a couple Linode stack scripts that will do this, but I haven't given any of them a try since last time I checked they were mostly/all Ubuntu based. Dovecot + postfix isn't all that difficult to set up, and there are literally tons of guides all over the place (Arch wiki, Linode, Dovecot's site, etc.) for dovecot, postfix, courier, and anything else you might want. Perhaps the most dumbfounding thing for a beginner is the certificate step...
- moxie 13y ago
  I've been hosting my own mail since 1996. It's actually one of the easier services to self-host: 1) SMTP was developed for unreliable environments. If you have problems with uptime, your incoming email will bounce around for 5 days before it gets dropped. So assuming you can get your SMTP server running one day out of five, you shouldn't be in danger of losing anything. 2) Contemporary daemons like postfix and dovecot have sane defaults, so even a naive default install should be mostly secure. They're also extremely low velocity, so once you set it up there's not a lot of ongoing maintenance.
  - chj 13y ago
    What do you use for spam filtering? Another problem is you will have to use a third party smtp server, otherwise your mail will be rejected by a lot of email providers.
    - spudlyo 13y ago
      SpamAssassin and friends still do a reasonable job for me, and have for years. I delete a handful every day, but like the parent I've had my email address since 1996, and I haven't been too careful about using it. If I had to do it over again today, I'd look at rspamd. Getting people to accept your mail can be a pain in the ass, yes, but it's not impossible. * Get a "clean" statically allocated IP address. * Control your forward and reverse DNS and make them match. * Set up SPF/DKIM. * Fill out the Yahoo Bulk Email sender form That last one probably isn't really necessary, depending on how much email you send to Yahoo.com addresses.
      - ciniglio 13y ago
        Do you have any tips on testing for deliverability? I have a hunch that if I try to send something, say to my gmail or one of my close friends' email, that the spam filter will allow it because it's a familiar name/domain, but I would expect that if I email a stranger's gmail, it might get flagged...
        SmokyBorbon 13y ago
        You can use http://www.mxtoolbox.com/ http://www.mxtoolbox.com/ to see if your domain or IP have been blacklisted by any ofthr public lists. Also, if you get blacklisted you should receive a bounceback email telling you who blacklisted you and where to go to get yourself removed.m
      - chalst 13y ago
        I've been running my own mail server since 2006 and have never had a problem with being rejected. I've run a fairly tight ship: 1. Only allow email locally (i.e., via webmail and ssh) - which many would regard as a real pain, but I and the few people who use my server have been OK with. IMAP is very useful with this policy, for batch move/send from outbox folders. 2. Set up TLS to only deliver via SSL 3. Advertise both these facts via SPF (I haven't bothered with DKIM)
- sciurus 13y ago
  Perhaps you'd like something more integrated? There are plenty of projects in this space that are entirely or mostly open source. http://www.kolab.org/ http://www.kolab.org/ http://www.sogo.nu/english.html http://www.sogo.nu/english.html http://www.zarafa.com/ http://www.zarafa.com/ http://www.open-xchange.com/home.html http://www.open-xchange.com/home.html https://www.zimbra.com/ https://www.zimbra.com/
  - jackbravo 13y ago
    Citadel is another integrated solution that is really easy to setup. Just one application, all written in C. The web interface is the only thing that I think could be better. http://citadel.org/ http://citadel.org/
    - lucaspiller 13y ago
      "Citadel offers versatile email services with very low administration needed. It provides its own implementations of these server protocols: IMAP, POP3, SMTP, ManageSieve, XMPP, Citadel." Wow, I'm impressed!
      - alan_cx 13y ago
        Call me behind the times, but I like the fact you can DL a virtual appliance, and have it working in 10 mins flat.
    - sciurus 13y ago
      I didn't mention Citadel because there hasn't been a release since November 2011; however on further investigation it looks like development is still active. http://code.citadel.org/?p=citadel.git;a=shortlog http://code.citadel.org/?p=citadel.git;a=shortlog
  - chalst 13y ago
    I've been looking at Kolab. They have a nice page listing their dependencies: http://www.kolab.org/community/upstream-communities http://www.kolab.org/community/upstream-communities All of these look fine - I already deploy all the server items - except Round Cube: does anyone know anything about them?
- captn3m0 13y ago
  Bitnami offers a ready-made mail solution with roundcube.[1] [1]: http://bitnami.com/stack/roundcube http://bitnami.com/stack/roundcube
- chj 13y ago
  Agreed. Setting up the mail server is such a nightmare. I'll gladly pay for a for-dummy-solution.
- 13throwaway 13y ago
  https://arkos.io/ https://arkos.io/
- seanp2k2 13y ago
  E-mail and the extensions and extras that make it go and make it nice /are/ duct-tape. I wouldn't advocate for /anyone/ running their own mailserver unless they're totally aware of what they're doing and on top of spam issues. When considering if running your own mailserver is practical for you, consider the total cost of ownership; you'll most likely be paying for a dedicated server (or vps, VM, or whatever the cool kids are using these days) which will require staying on top of all the "normal" things; monitoring, backups (and you'll probably want to test them), updates, what blacklists to use, ensuring that /your/ server doesn't end up blacklisted, etc. Unless you've been a sysadmin before and know what this all entails, I wouldn't really recommend it. If you're not part of some bigger org (a hosting co or bigger company), it'll also be harder to get your one-off no-name server removed from blacklists, since this looks just like something Joe Linkfarm would do. I could show you this one really simple trick that a mother found to whiten your server reputation, but... As far as creating a VM image that sets all this up for you, it's kinda the point that it doesn't exist; mailservers don't work well when run by people who don't really care about them. It's quite far away from a "set and forget" deal. This is why many hackers / sysadmins still use GMail; it's not the best at everything, but it's OK enough at most things that it's the least hassle to use. That all said, I do agree that it'd be great to have a service which offers both more control for power users (I'd love a mailfilter-style config w/regex support for GMail), better privacy (easy PGP integration, etc), tag support, threading, and normal IMAP support, but I'm not quite holding my breath. This seems like a decent problem for some start-up to solve :) It's not a "sexy" problem, but it's a real one.
- easy_rider 13y ago
  I was about to say, doesn't this scream for a container? i.e. docker.io
brongondwana 13y ago
I'm not sure why you can't do those things on FastMail. (disclaimer: I work for FastMail) Sure we have folders rather than tags, which means you can't add multiple of them to the same message. Probably the biggest lack is that you can't manage IMAP flags via the web interface. Otherwise, our search is now very powerful (since about March this year) and allows you to build filters that show messages from multiple folders in a single view.
- dbpatterson 13y ago
  OP here. Fastmail had 4 out of my required 5 features. I used it for a while. I'm still a paying customer (I paid for a bunch of years). Tags were a deal breaker. It's just not sufficient for how I want to organize things.
  - brongondwana 13y ago
    I'm going to raise this in our next meeting (Tuesday) and see if there's a way we can have IMAP flags exposed somehow. Along with fast cross-folder searching on flags, we could quite easily implement virtual folders per flag - which I think fills your use-case perfectly. The difficult parts are: 1) UI 2) limits. We have a hard limit of 128 "user flags" because it's in 4 x 32 bit fields in a fixed-width data format. Subtract a few for our internal tooling, and you probably only have 120 you can use for yourself. Would 120 be enough? One thing that many older clients did was had $Label1 => $Label5. That was often enough for people... so I suspect 120 is probably fine.
    - dbpatterson 13y ago
      I think if you'd had that before, I never would have left (and I think many other people would be happy with that).
    - deleted 13y ago
      [deleted]
- mrgreenfur 13y ago
  I love FastMail. Thank you for your work!
t0 13y ago
>handing an advertising company most of my personal and professional correspondance seems like a bad idea That's your main complaint? Google is an advertising company. People buying ads on Adsense don't have access to your personal information. This is simply not true.
- pbsdp 13y ago
  Mining user profiles for targeted AdSense ads is just the tiniest part of the problem. Massive government spying revelations based on tapping into centralized infrastructure at cooperative organizations that maintain broad cross-web and cross-mobile profiles of all users, along with their personal communications and data, and you don't see how big this problem really is?
mjn 13y ago
Nice stringing together of unixy tools to get this working. I had not heard of notmuch and its related ecosystem (afew, alot, etc.), so that's a useful discovery on my part.
- ricardobeat 13y ago
  It probably is not much popular.
bstx 13y ago
Another relatively painless way to set up a mail server on your own box is http://www.iredmail.org http://www.iredmail.org. ( + http://z-push.sourceforge.net/soswp/ http://z-push.sourceforge.net/soswp/ for ActiveSync push mail) So far I am fairly happy with it.
- jrn 13y ago
  Does anyone know, if I used z-push for a product, would I be liable for not buying a license from Microsoft, for Exchange ActiveSync? It seems they are fairly litigious with activesync patents.
  - andor 13y ago
    The protocol is patented [0]. If you want to offer your product in the US (and other places that care about software patents), you need a license. Also note that Z-Push is AGPLv3 software. Unless you can get it under another license from Zarafa, you have to release your product with an AGPL-compatible license. [0] http://www.microsoft.com/en-us/legal/intellectualproperty/IPLicensing/Programs/exchangeactivesyncprotocol.aspx http://www.microsoft.com/en-us/legal/intellectualproperty/IP...
    - jrn 13y ago
      thanks
exratione 13y ago
Setting up a server in any hosting environment at this point comes with the assumption that its contents can be read at any time by the operators and whoever they let in without you ever knowing about it. That's still a lot better than Gmail. Setting up your own mail server is not a terrible woe-inducing undertaking if you have a working recipe to follow and are comfortable with the Unix command line (e.g. http://www.exratione.com/2012/05/a-mailserver-on-ubuntu-1204-postfix-dovecot-mysql/ http://www.exratione.com/2012/05/a-mailserver-on-ubuntu-1204...). Organization and categorization are the sticking point features, given what I've seen of most open source webmail applications. But worth looking around. If you have a basic mail server image, you can keep trying out applications on top of it to see what works for you. Going beyond that to something with a whole lot more encryption and less of an ability for hosting providers to read your data would really require a product dedicated to that end: that is hard to get right.
- mbell 13y ago
  > Setting up a server in any hosting environment at this point comes with the assumption that its contents can be read at any time by the operators and whoever they let in without you ever knowing about it. How exactly is that any different than it was 6 months ago?
  - bigiain 13y ago
    The difference today is that not only should I assume that Google sysadmins could read my mail, but there's now pretty convincing arguments that Google are sending all my (as a non-US person) email to the NSA - who're storing it forever "just in case it turns out useful". I'm somewhat less concerned about rogue individual sysadmins curiously snooping on my mail, than I am about the NSA's comprehensive perpetual archive.
    - mbell 13y ago
      What? Have you been following this topic at all? Exactly where are these convincing arguments that google is sending all e-mail to the NSA? Links? Data? Do you have anything that supports this at all? There is the telephone stuff that was leaked, which has been well known since 2006 [0]. But, where is this miraculous evidence that google is handing everything over to the NSA? [0] http://yahoo.usatoday.com/news/washington/2006-05-10-nsa_x.htm http://yahoo.usatoday.com/news/washington/2006-05-10-nsa_x.h...
peterlongnguyen 13y ago
Just a general fyi, there are $10 credits for Digital Ocean if you decide to sign up. The one I used this month was OMGSSD10, but it may only work for this month.
- bigiain 13y ago
  I've got (and am very happy with) a DO VM, but in the context of securing mail - I'm not sure using a US based server (or a server owned by a US based company). (And, though I'm in Australia and have an English passport, I'm not sure UK or Australian servers are any better…)
  - daned 13y ago
    Know any hosting companies in the travel lounge of a Moscow airport?
anemitz 13y ago
Out of curiosity, what was the reason for not picking a more traditional Dovecot + Postfix setup?
- dbpatterson 13y ago
  Debian decided to use exim4 by default, so I figured that it would be better supported / documented. Which for the most part was true. There is no reason why postfix+dovecot couldn't work equally well - I was familiar with none, so I chose what seemed like the past of least resistance.
  - spudlyo 13y ago
    Exim has had five serious vulnerabilities since 2010; two root privilege escalations and three remote code execution bugs. No confidence.
    - Svip 13y ago
      Agreed. I used Exim4 for a while, but after too many issues, I eventually switched to Postfix, which have been run smooth ever since.
ishbits 13y ago
Many of us did similar in the 90s. I might go this route again but would use Postfix and Dovecot. I'd do this for my wife and kids as well - but if I get hit by a bus, email eventually not working is not something I should burden my wife with.
- pyre 13y ago
  Just prepare for that eventuality. Make an upgrade path in the event of such a thing, and include it in your will.
- kryten 13y ago
  Tip: don't use dovecot-lda. Let postfix handle it. Dovecot LDA is chroot hell. I worry about the same things which is why I'm actually migrating my email to outlook.con slowly (in spite of the NSA etc). She knows my passwords already.
tbrock 13y ago
It's definintely time for an open source alternative to GMail... but I think everyone knows this isn't it. These tools like exim, horde, dovecot, etc. have been around and worked for decades but wouldn't it be great to have fresh solutions that weren't so ancient and archaic?
- pfraze 13y ago
  I made a very extensive search for an open-source webmail solution that exported a REST api. Unfortunately, none of them do (that I could find). Most of them are from the XMLRPC days. I'd definitely contribute to a kickstarter for a good FOSS REST Email server.
  - pyre 13y ago
    | good FOSS REST Email server You should specify what you're talking about here. An all-in-one solution? An SMTP server? An IMAP server? A webmail client (e.g. SquirrelMail / RoundCube)?
    - tacticus 13y ago
      I imagine it would be similar to what i've been looking for. Central app that manages the storage, sending (server to server) and receiving (server to server) of email exposed over a rest api
- taeric 13y ago
  You say ancient and archaic, I see ridiculously well tested and battle worn. Seriously, what would make a "fresh" solution better than a long standing one, in this case?
  - AnthonyMouse 13y ago
    You don't have to reinvent the wheel. Presumably the "better" solution would be using most of the existing code under the hood. But wouldn't it be nice if setting up an email server consisted primarily of typing something like "apt-get install email-server" and then setting the domain name and adding a few accounts?
Zash 13y ago
Run your own server FTW
voidlogic 13y ago
Hmm... It sounds easier to just run an instance of Zimbra community edition in a VM. https://s3.amazonaws.com/uploads.blog.zimbra.com/wp-content/uploads/2012/09/conversation1.jpeg https://s3.amazonaws.com/uploads.blog.zimbra.com/wp-content/...
- h2ohno 13y ago
  This is what I do, but it requires more than a $5/month VPS to run it. https://www.zimbra.com/docs/os/8.0.2/single_server_install/SS_Install_8.0.2_OS.ZCS_System_Requirements.html https://www.zimbra.com/docs/os/8.0.2/single_server_install/S...
  - voidlogic 13y ago
    True, but $10/mo should cover your needs fine. https://www.digitalocean.com/pricing https://www.digitalocean.com/pricing
cdjk 13y ago
I've thought about doing this, but email is important enough I don't trust myself to provide as much uptime as a commercial email provider. You probably should add SPF records too, if you don't want your outgoing mail marked as spam.
- kryten 13y ago
  Reliability doesn't matter that much. SMTP is store and forward so if you Bork something, it'll still get to you eventually.
  - cdjk 13y ago
    Reliability for incoming mail, sure. But there's making sure outgoing mail is delivered and not spam filtered, making sure your ip isn't on any blacklists, managing and testing backups, keeping up to date on security issues, etc. All of which are doable, but that's not really how I want to spend my time.
cliveholloway 13y ago
I would have thought a client side encryption plugin that will seamlessly encrypt/decrypt all your Gmail sent between yourself and any other user running said plugin would be a simpler option. Adding common mail suppliers as it goes forward.
- sygma 13y ago
  You could use something like Mailvelope [0]. You still have to manually encrypt things, but the process is fairly straightforward. [0]: http://www.mailvelope.com/ http://www.mailvelope.com/
rst 13y ago
Was hoping to see more discussion of backups. There are a bunch of possible approaches (depending on level of desired security, what the VPS provider offers, and how much you trust them), but for a mail server, there ought to be something...
- db48x 13y ago
  Use OfflineIMAP to sync to a local disk.
  - dumaspere 13y ago
    Sync is not backup.
- ishbits 13y ago
  I currently use mbsync to backup my all mail folder on gmail. There is a way to set it to never delete on your local side, turning it more into a backup than a sync. I run that daily, and Crashplan takes care of the rest. Sure it doesn't save the folder structure and what not, but I'm ok with that.
daurnimator 13y ago
Does anyone have a gmail exporter? i.e. something that imports email from gmail WITH labels.
- mcrittenden 13y ago
  I believe gmvault.org should be fine for that.
- mjn 13y ago
  This is a proof-of-concept someone wrote, but I have no idea how tested it is: http://git.zx2c4.com/gmail-notmuch/ http://git.zx2c4.com/gmail-notmuch/ It syncs Gmail labels to notmuch tags. As I understand it: by default, Gmail treats labels as IMAP folders, which means a standard IMAP exporter will end up with multiple copies of messages with more than one label. The workaround is to only export the "All Mail" folder, and then add support for Gmail's custom attributes, namely retrieving the X-GM-LABELS [1] attribute for each message and then doing something with it (in this case, storing it as notmuch tags). [1] https://developers.google.com/gmail/imap_extensions#access_to_gmail_labels_x-gm-labels https://developers.google.com/gmail/imap_extensions#access_t...
- vu3rdd 13y ago
  You can use offlineimap to do that. Every label is an imap folder. offlineimap allows what they call as "folder filters", using which one can write a short python function parameterised over the folder name.
kefs 13y ago
While k9mail is a must, I suggest linking to the repo, which is usually lightyears ahead of public releases on the Play store. https://code.google.com/p/k9mail/downloads/list https://code.google.com/p/k9mail/downloads/list On a side note, they seem to have just hit v4 two days ago. Second side note, if you decide to use k9, be sure to turn off the signature under composition settings for each account you add.. it's turned on by default.
richdougherty 13y ago
It would be incredibly useful if there was a mail service that received email over SMTP, encrypted it straight away with a public key, then just dumped the encrypted email into a general-purpose online storage solution (e.g. an S3 bucket). That would IMO provide a good base for encrypted client-side apps to build on top of. Open source would better be able address the problem of writing a client once the money needed for hosting and storage is taken out of the equation.
- LiveTheDream 13y ago
  Hushmail and Lavabit sort of do this. They encrypt emails before storing them. However, as far as I can tell they handle the public/private keypair on their own, so the security only goes as far as you trust them.
- dmix 13y ago
  Countermail does this: https://countermail.com/ https://countermail.com/
  - richdougherty 13y ago
    I was imagining an even lower level service. I guess that CounterMail must store unencrypted email headers in order to serve IMAP, right? I was thinking something that just manages the problem of being online 24/7 to receive email. This (and possibly sending email) is the only thing that can't be done completely client side. A service like I was thinking of would just accept messages, immediately encrypt them (headers and all) with an RSA key and then dump them into some third party online storage system (maybe sending a notification over XMPP too). The rest of the email pipeline could then be run completely client-side. I'm talking about doing everything client side: spam filtering, sorting into folders or tagging, running email rules, indexing, and of course viewing and reading email. (A clever filesystem/database would need to be layered on top of the online storage system to manage the state of the pipeline and provide fast indexes.) For maximum ease of use the client-side app could even be a rich Gmail-like JavaScript app that stores private keys locally and but stores most data remotely (e.g. uses S3 directly via CORS). This would all allow hackers like us to build interesting (but still encrypted) email services without having to worry about infrastructure. That's something I don't want to manage anyway! Also the barrier to entry of writing a Gmail clone would be much lower because you could do at almost zero cost. Since you don't have to receive email or manage storage your service would mostly be static JavaScript containing the client-side logic. Most - even all - of the service could just be served on a CDN.
magic_haze 13y ago
To play the devil's advocate, what exactly is the practical use of all this if most of your family and friends are on Gmail (and couldn't be arsed to figure out pgp)? From what I can see, your emails will now be sent in the clear over the internet, instead of staying within google's servers. Either way, the government's going to get your data, but at least you're protected against... /more/ unscrupulous people snooping on your stuff?
- thex86 13y ago
  That's a great point. Reminds of the time I taught my friend to use PGP and sent him an encrypted email. Every single time, he would reply in plain-text, thus exposing my older conversation. When asked why, he told me it's too much of a pain to do it. So my being careful about my privacy doesn't help if other people don't play along.
  - eru 13y ago
    The OTR plugin for Pidgin does a better job of userfriendliness and gives more useful privacy guarantees (like plausible deniability).
    - shrikant 13y ago
      FYI, last I remember, Pidgin stored your passwords in plaintext in an ASCII file on disk, unless you jumped through some hoops to integrate it with your desktop environment's keyring. They even have a article up on their site explaining why it's necessary to do this.
      - eliasmacpherson 13y ago
        https://developer.pidgin.im/wiki/PlainTextPasswords https://developer.pidgin.im/wiki/PlainTextPasswords well that's just great.
      - claudius 13y ago
        You can disable storing passwords. If you don’t want to have to enter a ‘master password’ when Pidgin starts up, there is no way they can store the passwords more securely than plaintext. Get full disk encryption and be happy.
cwp 13y ago
I ran a set up similar to this for many years. It's not that hard, for those with a little unix experience. As moxie mentions, email is very forgiving—you have to break it badly and leave it broken for a long time before you start to lose messages. What eventually drove me to GMail was spam. I tried a bunch of different filters, and never found one with good-enough accuracy. Finally I decided that the independence and privacy wasn't worth the time I spent fiddling with filters and dealing with misclassified messages. As far as I can tell, Gmail is 100% accurate. Problem solved.
- cabalamat 13y ago
  > As far as I can tell, Gmail is 100% accurate. My experience is I've had a few false negatives and false positives. Gmail is still very good though, and any replacement, if it is to be widely used, needs to solve the spam problem. If each message had to be separately encrypted for each receiver, would this add much to a spammer's costs? I'm guessing it would, but not by enough to make spam uneconomic. A better solution might be to require that the first email someone sends someone else, unless they've been OK'd by a third party, contains bitcoins to the value of $0.01.
  - comex 13y ago
    Fun fact: Bitcoin's proof-of-work algorithm is based on Hashcash, which was an anti-spam measure along those lines: a message contains a partial hash brute-force that takes the sender about a second of CPU time to compute (and depends on the recipient). But it didn't take off, and is problematic when spammers have vast amounts of CPU resources in the form of botnets.
- bifrost 13y ago
  > As far as I can tell, Gmail is 100% accurate. I can confirm that that is not correct. Google regularly flags internal ticket email as spam for one of my larger customers (mail that is sent within GApps). I also see about a 10-20% false positive rate on our various mailings (account setups, billing notices, etc).
hcarvalhoalves 13y ago
Normal people definitely don't want to manage a mail server though. Life is too short to waste figuring out why you're banned on Spamhaus for the 93th time. GMail sucks, but a home-made contraption is not the alternative.
- werid 13y ago
  Why must this be pointed out every time? There's enough of us weirdos out there.
- siddboots 13y ago
  > Normal people definitely don't want to... Life is too short to... Ugh! All you are saying is that if you aren't looking for a home made contraption, then a home made contraption isn't what you are looking for.
- Aldipower 13y ago
  I am not normal.
- mjn 13y ago
  To be fair to the author, I don't think an article that starts with "A Hacker's..." and/or involves Emacs has a pretension to being a solution for normal people.
  - hcarvalhoalves 13y ago
    Yeah, I know. I'm just pointing out that GMail alternatives should exist that don't involve a huge PITA. I know how to configure mail servers just fine, but I still wouldn't do it for myself. That's what I mean by "normal".
_oxford 13y ago
I see a couple of problems here: 1. It's likely he's storing emails on the VPS. This puts us back at square one. A third party has a copy of your emails. And we know email does not garner the same privacy protections as postal mail. 2. You need a domain name. That system (DNS), as it is currently implemented (i.e., everyone setting their root zone to servers they do not control), is highly centralized -- few people maintain their own root zone, despite being easy to do. Domain names are susceptible to false allegations copyright and trademark infringement by private parties, not to mention easy censorship by the US gov't. When you lose your domain you lose email. (Though you shouldn't have to: email works fine with IP addresses in brackets.) So what's the solution: 1. Get a reachable IP (e.g., through ISP) or get a VPS. But if you get a VPS only use it to pierce NAT (how is left as exercise for reader - hint: supernode), not run a mail server. Don't store sensitive data like email on a VPS, or route sensitive data through it. 2. Use IP addresses not domain names. Alternatively, set up your own DNS that is available as a peer-to-peer service, or have your email contacts use a DNS server and root zone you collectively maintain: free domain names that you control. No one can censor your DNS (phonebook), except you.
ramblerman 13y ago
An interesting solution if you have something to hide I suppose. As has been stated time and again, most people don't. The danger lies in politicians, ceo's and other figures of authority who do and can be blackmailed. Rather than a few hackers setting up their own SMTP servers I think a more powerful solution lies in keeping focus on the actual problem, the out of control NSA program.
csense 13y ago
I always thought a big part of the reason people used gmail was for the snazzy web-based UI that was one of the first popular AJAX-based web applications. I eagerly read the article to see what alternative to this feature the author was suggesting, so I was surprised to see he's reading the emails with a standalone client...in fact, it's an emacs plugin!
- rorrr2 13y ago
  Exactly. It's hardly a solution. Viewing in-body images and attachments turns into a nightmare.
p4bl0 13y ago
When I saw the link's title, I immediately thought that it would be another webmail client hosted by someone, especially given the domain name. Because almost everytime I see a "hacker's X" or "X for hacker" title on HN I'm just afflicted by the content, so I got used to it. But here, what a pleasant surprise. The post is actually describing a real hacker's replacement for Gmail (which coincidentally is almost my setting, except I use mu [1] instead of notmuch). I'll keep it as a reference to send people asking for alternative email hosting. [1] http://www.djcbsoftware.nl/code/mu/ http://www.djcbsoftware.nl/code/mu/
alemhnan 13y ago
We could push a step further: "EMail Server as a service for common people". Somehow like http://instantserver.io/ http://instantserver.io/ . Or like Heroku: you create your "managed" mail server with a click.
- omegant 13y ago
  Please!
lazylizard 13y ago
linode has nice guides...e.g. https://library.linode.com/email/postfix/dovecot-mysql-debian-6-squeeze https://library.linode.com/email/postfix/dovecot-mysql-debia... though for postfix, having something like postfixadmin makes it nicer.. or perhaps something instant like http://www.xeams.com/ http://www.xeams.com/ will do..
lazylizard 13y ago
as for antispam..http://www.mxhero.com/ http://www.mxhero.com/ is auto and easy, and i think http://spamcheck.sourceforge.net/ http://spamcheck.sourceforge.net/ is very nice. u don't get as many controls but it sends quarantine digests, and runs much lighter than mxhero. http://www.scrolloutf1.com/ http://www.scrolloutf1.com/ looks nice too. at work we sell spamtitan, which i admit is really nice..tons of config with nice gui, good defaults, easy setup..but its non free...
drdaeman 13y ago
Why use notmuch when one can have standard (RFC5228) Sieve support?
spo81rty 13y ago
If you want a really good and really easy to setup mail server I would recommend SmarterMail. It is also free for 1 email user. I have used their product for about 10 years. Note that it is Windows server based. http://www.smartertools.com/smartermail/mail-server-software.aspx http://www.smartertools.com/smartermail/mail-server-software...
fredsted 13y ago
Why make it so hard? Why not just install virtualmin [0] on a Debian (or whatever Linux you prefer) server and get it over with. You also get web hosting, DB hosting, mailing lists, webmail and more as a bonus. And you don't have to worry about security updates. Just install and create your virtual host, and modify DNS for your domain. Couldn't be easier. Oh, and it's completely free. [0] http://www.virtualmin.com http://www.virtualmin.com
shunter 13y ago
I've started defining 'hacker' as someone who's willing to 'eat their own dogfood' as it were. Someone that is willing to spend time working on the nuts and bolts that lead to some kind of productivity rather than just being productive with the tool / service to begin with. I used to classify myself as a 'Hacker' and still do when it's something I want to learn more about. Most of the time, however, I'm more interested in just getting the benefits rather than tinkering with the internals. Sometimes, I'm a Hacker, sometimes I'm a consumer.
philjackson 13y ago
Interesting. I remain a step behind this one in that I'm using offlineimap to sync local maildirs from google's servers and then using mu and mu4e in emacs. Means I get to use the Gmail Android client which is actually very good.
nvarsj 13y ago
I had a similar setup a couple of years ago. The main problem I had was the maintenance required. If you have any machine publicly accessible you have to be on top of security updates and proper system hardening. I gave up after my exim4 Debian system got 0-day rooted. If doing it again I would avoid a Debian based distro. I'd probably use openbsd. And the less ports open the better.
snambi 13y ago
awesome... its time move away from proprietary, snooping services such as gmail. Hopefully setting up such a service should become easier ( may be less than 5 steps ) with better cloud VMs. Then even non-tech savvy people can have their emails away from snooping.
just2n 13y ago
I did this for a long time, but it's really annoying: 1. If your provider goes down, you lose mail. 2. If you are conversing with people who are using an insecure mailer, such as gmail, Yahoo, etc (which is probably > 99.9% of all e-mail users), your e-mail is still accessible to the NSA, or to some Fortune 100 advertising company. 3. It's only a matter of time before the "big dogs" in email abuse the position and decide who is and isn't allowed to send/receive email outside of their little oligarchy, either on their own or at the behest of governments. Like so much else that has been corrupted, we need to scratch the current architecture as too insecure, and build something truly secure for the future. This isn't in the interests of the Googles of the world, and it's actively in the worst interests of the NSA/FBI/CIA, so it's probably the right thing to do.
- gnosis 13y ago
  "Like so much else that has been corrupted, we need to scratch the current architecture as too insecure, and build something truly secure for the future." This is really the core technological issue that has enabled much of the recent mass spying: the Internet really was not designed with security, privacy, or anonymity in mind. Remember that it started as a research network that was used primarily among academics. Academia is generally a very open and trusting environment. The last thing the inventors of the Internet though of was trying to protect their computers, data, or privacy. We are living with the consequences of that mistake. The second major cause of this mess is the computer illiteracy and historical ignorance of much of the world's population. They don't understand how they are making themselves vulnerable by sharing so much information about themselves and trusting corporations that provide "free" web services to them. This is slowly changing for the better, as people become more technologically savvy and stories like the current spying scandals and various security breaches hit the news. As for the technological and security minded among us, particularly those who are just now starting to think about running their own mail servers, what took you all so long? None of the recent revelations should have been unexpected. The Internet needs a major privacy, security, and anonymity overhaul. If it's not rebuilt with those concerns at the core, they will all remain mostly illusory to the overwhelming majority of Internet users.
- magicalist 13y ago
mwcampbell 13y ago
I think more of us need to run mail servers. For ourselves, for our families, and possibly for others who are willing to pay. Email is far too centralized now, at a handful of companies, in a handful of data centers. So in that regard, running a mail server on a VPS at one of the popular providers is kind of missing the point. My local cable ISP doesn't allow incoming or outgoing connections to port 25, nor incoming connections to port 80. So at least for now, I can't run a mail server in my home. I've thought about switching to DSL, but then I would take a major hit in speed, in both directions. Luckily, I have another option. There's a hosting provider where I live (Wichita, Kansas) that offers KVM-based virtual machine hosting. So I'll get a VM there, and if the service is any good, I'll move there from Linode. The pricing isn't competitive with Linode, let alone DigitalOcean, and I doubt that the connectivity is as good, since the server will be in a building here in Wichita rather than a real data center. But I'm willing to try it, in order to support a local business and fight the centralization of the Internet.

464 points by dbpatterson 13y ago

While I don't necessarily trust an external company with all my emails, I also don't trust myself to maintain the myriad daemons involved in this setup without doing something subtly wrong that results in my server not sending/receiving all the mail it should -- or, worse, being used for spam. What would be useful is a pre-assembled virtual machine image or other form of appliance that allows you to deploy and test a mail server within about an hour or so, without having to duct-tape any of this together yourself.

ics 13y ago
> What would be useful is a pre-assembled virtual machine image or other form of appliance that allows you to deploy and test a mail server within about an hour or so, without having to duct-tape any of this together yourself. There at least a couple Linode stack scripts that will do this, but I haven't given any of them a try since last time I checked they were mostly/all Ubuntu based. Dovecot + postfix isn't all that difficult to set up, and there are literally tons of guides all over the place (Arch wiki, Linode, Dovecot's site, etc.) for dovecot, postfix, courier, and anything else you might want. Perhaps the most dumbfounding thing for a beginner is the certificate step...
moxie 13y ago
I've been hosting my own mail since 1996. It's actually one of the easier services to self-host: 1) SMTP was developed for unreliable environments. If you have problems with uptime, your incoming email will bounce around for 5 days before it gets dropped. So assuming you can get your SMTP server running one day out of five, you shouldn't be in danger of losing anything. 2) Contemporary daemons like postfix and dovecot have sane defaults, so even a naive default install should be mostly secure. They're also extremely low velocity, so once you set it up there's not a lot of ongoing maintenance.
- chj 13y ago
  What do you use for spam filtering? Another problem is you will have to use a third party smtp server, otherwise your mail will be rejected by a lot of email providers.
  - spudlyo 13y ago
    SpamAssassin and friends still do a reasonable job for me, and have for years. I delete a handful every day, but like the parent I've had my email address since 1996, and I haven't been too careful about using it. If I had to do it over again today, I'd look at rspamd. Getting people to accept your mail can be a pain in the ass, yes, but it's not impossible. * Get a "clean" statically allocated IP address. * Control your forward and reverse DNS and make them match. * Set up SPF/DKIM. * Fill out the Yahoo Bulk Email sender form That last one probably isn't really necessary, depending on how much email you send to Yahoo.com addresses.
    - ciniglio 13y ago
      Do you have any tips on testing for deliverability? I have a hunch that if I try to send something, say to my gmail or one of my close friends' email, that the spam filter will allow it because it's a familiar name/domain, but I would expect that if I email a stranger's gmail, it might get flagged...
      - SmokyBorbon 13y ago
        You can use http://www.mxtoolbox.com/ http://www.mxtoolbox.com/ to see if your domain or IP have been blacklisted by any ofthr public lists. Also, if you get blacklisted you should receive a bounceback email telling you who blacklisted you and where to go to get yourself removed.m
    - chalst 13y ago
      I've been running my own mail server since 2006 and have never had a problem with being rejected. I've run a fairly tight ship: 1. Only allow email locally (i.e., via webmail and ssh) - which many would regard as a real pain, but I and the few people who use my server have been OK with. IMAP is very useful with this policy, for batch move/send from outbox folders. 2. Set up TLS to only deliver via SSL 3. Advertise both these facts via SPF (I haven't bothered with DKIM)
sciurus 13y ago
Perhaps you'd like something more integrated? There are plenty of projects in this space that are entirely or mostly open source. http://www.kolab.org/ http://www.kolab.org/ http://www.sogo.nu/english.html http://www.sogo.nu/english.html http://www.zarafa.com/ http://www.zarafa.com/ http://www.open-xchange.com/home.html http://www.open-xchange.com/home.html https://www.zimbra.com/ https://www.zimbra.com/
- jackbravo 13y ago
  Citadel is another integrated solution that is really easy to setup. Just one application, all written in C. The web interface is the only thing that I think could be better. http://citadel.org/ http://citadel.org/
  - lucaspiller 13y ago
    "Citadel offers versatile email services with very low administration needed. It provides its own implementations of these server protocols: IMAP, POP3, SMTP, ManageSieve, XMPP, Citadel." Wow, I'm impressed!
    - alan_cx 13y ago
      Call me behind the times, but I like the fact you can DL a virtual appliance, and have it working in 10 mins flat.
  - sciurus 13y ago
    I didn't mention Citadel because there hasn't been a release since November 2011; however on further investigation it looks like development is still active. http://code.citadel.org/?p=citadel.git;a=shortlog http://code.citadel.org/?p=citadel.git;a=shortlog
- chalst 13y ago
  I've been looking at Kolab. They have a nice page listing their dependencies: http://www.kolab.org/community/upstream-communities http://www.kolab.org/community/upstream-communities All of these look fine - I already deploy all the server items - except Round Cube: does anyone know anything about them?
captn3m0 13y ago
Bitnami offers a ready-made mail solution with roundcube.[1] [1]: http://bitnami.com/stack/roundcube http://bitnami.com/stack/roundcube
chj 13y ago
Agreed. Setting up the mail server is such a nightmare. I'll gladly pay for a for-dummy-solution.
13throwaway 13y ago
https://arkos.io/ https://arkos.io/
seanp2k2 13y ago
E-mail and the extensions and extras that make it go and make it nice /are/ duct-tape. I wouldn't advocate for /anyone/ running their own mailserver unless they're totally aware of what they're doing and on top of spam issues. When considering if running your own mailserver is practical for you, consider the total cost of ownership; you'll most likely be paying for a dedicated server (or vps, VM, or whatever the cool kids are using these days) which will require staying on top of all the "normal" things; monitoring, backups (and you'll probably want to test them), updates, what blacklists to use, ensuring that /your/ server doesn't end up blacklisted, etc. Unless you've been a sysadmin before and know what this all entails, I wouldn't really recommend it. If you're not part of some bigger org (a hosting co or bigger company), it'll also be harder to get your one-off no-name server removed from blacklists, since this looks just like something Joe Linkfarm would do. I could show you this one really simple trick that a mother found to whiten your server reputation, but... As far as creating a VM image that sets all this up for you, it's kinda the point that it doesn't exist; mailservers don't work well when run by people who don't really care about them. It's quite far away from a "set and forget" deal. This is why many hackers / sysadmins still use GMail; it's not the best at everything, but it's OK enough at most things that it's the least hassle to use. That all said, I do agree that it'd be great to have a service which offers both more control for power users (I'd love a mailfilter-style config w/regex support for GMail), better privacy (easy PGP integration, etc), tag support, threading, and normal IMAP support, but I'm not quite holding my breath. This seems like a decent problem for some start-up to solve :) It's not a "sexy" problem, but it's a real one.
easy_rider 13y ago
I was about to say, doesn't this scream for a container? i.e. docker.io

brongondwana 13y ago

I'm not sure why you can't do those things on FastMail. (disclaimer: I work for FastMail) Sure we have folders rather than tags, which means you can't add multiple of them to the same message. Probably the biggest lack is that you can't manage IMAP flags via the web interface. Otherwise, our search is now very powerful (since about March this year) and allows you to build filters that show messages from multiple folders in a single view.

dbpatterson 13y ago
OP here. Fastmail had 4 out of my required 5 features. I used it for a while. I'm still a paying customer (I paid for a bunch of years). Tags were a deal breaker. It's just not sufficient for how I want to organize things.
- brongondwana 13y ago
  I'm going to raise this in our next meeting (Tuesday) and see if there's a way we can have IMAP flags exposed somehow. Along with fast cross-folder searching on flags, we could quite easily implement virtual folders per flag - which I think fills your use-case perfectly. The difficult parts are: 1) UI 2) limits. We have a hard limit of 128 "user flags" because it's in 4 x 32 bit fields in a fixed-width data format. Subtract a few for our internal tooling, and you probably only have 120 you can use for yourself. Would 120 be enough? One thing that many older clients did was had $Label1 => $Label5. That was often enough for people... so I suspect 120 is probably fine.
  - dbpatterson 13y ago
    I think if you'd had that before, I never would have left (and I think many other people would be happy with that).
  - deleted 13y ago
    [deleted]
mrgreenfur 13y ago
I love FastMail. Thank you for your work!

t0 13y ago

>handing an advertising company most of my personal and professional correspondance seems like a bad idea That's your main complaint? Google is an advertising company. People buying ads on Adsense don't have access to your personal information. This is simply not true.

pbsdp 13y ago
Mining user profiles for targeted AdSense ads is just the tiniest part of the problem. Massive government spying revelations based on tapping into centralized infrastructure at cooperative organizations that maintain broad cross-web and cross-mobile profiles of all users, along with their personal communications and data, and you don't see how big this problem really is?

mjn 13y ago

Nice stringing together of unixy tools to get this working. I had not heard of notmuch and its related ecosystem (afew, alot, etc.), so that's a useful discovery on my part.

ricardobeat 13y ago
It probably is not much popular.

bstx 13y ago

Another relatively painless way to set up a mail server on your own box is http://www.iredmail.org http://www.iredmail.org. ( + http://z-push.sourceforge.net/soswp/ http://z-push.sourceforge.net/soswp/ for ActiveSync push mail) So far I am fairly happy with it.

jrn 13y ago
Does anyone know, if I used z-push for a product, would I be liable for not buying a license from Microsoft, for Exchange ActiveSync? It seems they are fairly litigious with activesync patents.
- andor 13y ago
  The protocol is patented [0]. If you want to offer your product in the US (and other places that care about software patents), you need a license. Also note that Z-Push is AGPLv3 software. Unless you can get it under another license from Zarafa, you have to release your product with an AGPL-compatible license. [0] http://www.microsoft.com/en-us/legal/intellectualproperty/IPLicensing/Programs/exchangeactivesyncprotocol.aspx http://www.microsoft.com/en-us/legal/intellectualproperty/IP...
  - jrn 13y ago
    thanks

exratione 13y ago

Setting up a server in any hosting environment at this point comes with the assumption that its contents can be read at any time by the operators and whoever they let in without you ever knowing about it. That's still a lot better than Gmail. Setting up your own mail server is not a terrible woe-inducing undertaking if you have a working recipe to follow and are comfortable with the Unix command line (e.g. http://www.exratione.com/2012/05/a-mailserver-on-ubuntu-1204-postfix-dovecot-mysql/ http://www.exratione.com/2012/05/a-mailserver-on-ubuntu-1204...). Organization and categorization are the sticking point features, given what I've seen of most open source webmail applications. But worth looking around. If you have a basic mail server image, you can keep trying out applications on top of it to see what works for you. Going beyond that to something with a whole lot more encryption and less of an ability for hosting providers to read your data would really require a product dedicated to that end: that is hard to get right.

mbell 13y ago
> Setting up a server in any hosting environment at this point comes with the assumption that its contents can be read at any time by the operators and whoever they let in without you ever knowing about it. How exactly is that any different than it was 6 months ago?
- bigiain 13y ago
  The difference today is that not only should I assume that Google sysadmins could read my mail, but there's now pretty convincing arguments that Google are sending all my (as a non-US person) email to the NSA - who're storing it forever "just in case it turns out useful". I'm somewhat less concerned about rogue individual sysadmins curiously snooping on my mail, than I am about the NSA's comprehensive perpetual archive.
  - mbell 13y ago
    What? Have you been following this topic at all? Exactly where are these convincing arguments that google is sending all e-mail to the NSA? Links? Data? Do you have anything that supports this at all? There is the telephone stuff that was leaked, which has been well known since 2006 [0]. But, where is this miraculous evidence that google is handing everything over to the NSA? [0] http://yahoo.usatoday.com/news/washington/2006-05-10-nsa_x.htm http://yahoo.usatoday.com/news/washington/2006-05-10-nsa_x.h...

peterlongnguyen 13y ago

Just a general fyi, there are $10 credits for Digital Ocean if you decide to sign up. The one I used this month was OMGSSD10, but it may only work for this month.

bigiain 13y ago
I've got (and am very happy with) a DO VM, but in the context of securing mail - I'm not sure using a US based server (or a server owned by a US based company). (And, though I'm in Australia and have an English passport, I'm not sure UK or Australian servers are any better…)
- daned 13y ago
  Know any hosting companies in the travel lounge of a Moscow airport?

anemitz 13y ago

Out of curiosity, what was the reason for not picking a more traditional Dovecot + Postfix setup?

dbpatterson 13y ago
Debian decided to use exim4 by default, so I figured that it would be better supported / documented. Which for the most part was true. There is no reason why postfix+dovecot couldn't work equally well - I was familiar with none, so I chose what seemed like the past of least resistance.
- spudlyo 13y ago
  Exim has had five serious vulnerabilities since 2010; two root privilege escalations and three remote code execution bugs. No confidence.
  - Svip 13y ago
    Agreed. I used Exim4 for a while, but after too many issues, I eventually switched to Postfix, which have been run smooth ever since.

ishbits 13y ago

Many of us did similar in the 90s. I might go this route again but would use Postfix and Dovecot. I'd do this for my wife and kids as well - but if I get hit by a bus, email eventually not working is not something I should burden my wife with.

pyre 13y ago
Just prepare for that eventuality. Make an upgrade path in the event of such a thing, and include it in your will.
kryten 13y ago
Tip: don't use dovecot-lda. Let postfix handle it. Dovecot LDA is chroot hell. I worry about the same things which is why I'm actually migrating my email to outlook.con slowly (in spite of the NSA etc). She knows my passwords already.

tbrock 13y ago

It's definintely time for an open source alternative to GMail... but I think everyone knows this isn't it. These tools like exim, horde, dovecot, etc. have been around and worked for decades but wouldn't it be great to have fresh solutions that weren't so ancient and archaic?

pfraze 13y ago
I made a very extensive search for an open-source webmail solution that exported a REST api. Unfortunately, none of them do (that I could find). Most of them are from the XMLRPC days. I'd definitely contribute to a kickstarter for a good FOSS REST Email server.
- pyre 13y ago
  | good FOSS REST Email server You should specify what you're talking about here. An all-in-one solution? An SMTP server? An IMAP server? A webmail client (e.g. SquirrelMail / RoundCube)?
  - tacticus 13y ago
    I imagine it would be similar to what i've been looking for. Central app that manages the storage, sending (server to server) and receiving (server to server) of email exposed over a rest api
taeric 13y ago
You say ancient and archaic, I see ridiculously well tested and battle worn. Seriously, what would make a "fresh" solution better than a long standing one, in this case?
- AnthonyMouse 13y ago
  You don't have to reinvent the wheel. Presumably the "better" solution would be using most of the existing code under the hood. But wouldn't it be nice if setting up an email server consisted primarily of typing something like "apt-get install email-server" and then setting the domain name and adding a few accounts?

Zash 13y ago

Run your own server FTW

voidlogic 13y ago

Hmm... It sounds easier to just run an instance of Zimbra community edition in a VM. https://s3.amazonaws.com/uploads.blog.zimbra.com/wp-content/uploads/2012/09/conversation1.jpeg https://s3.amazonaws.com/uploads.blog.zimbra.com/wp-content/...

h2ohno 13y ago
This is what I do, but it requires more than a $5/month VPS to run it. https://www.zimbra.com/docs/os/8.0.2/single_server_install/SS_Install_8.0.2_OS.ZCS_System_Requirements.html https://www.zimbra.com/docs/os/8.0.2/single_server_install/S...
- voidlogic 13y ago
  True, but $10/mo should cover your needs fine. https://www.digitalocean.com/pricing https://www.digitalocean.com/pricing

cdjk 13y ago

I've thought about doing this, but email is important enough I don't trust myself to provide as much uptime as a commercial email provider. You probably should add SPF records too, if you don't want your outgoing mail marked as spam.

kryten 13y ago
Reliability doesn't matter that much. SMTP is store and forward so if you Bork something, it'll still get to you eventually.
- cdjk 13y ago
  Reliability for incoming mail, sure. But there's making sure outgoing mail is delivered and not spam filtered, making sure your ip isn't on any blacklists, managing and testing backups, keeping up to date on security issues, etc. All of which are doable, but that's not really how I want to spend my time.

cliveholloway 13y ago

I would have thought a client side encryption plugin that will seamlessly encrypt/decrypt all your Gmail sent between yourself and any other user running said plugin would be a simpler option. Adding common mail suppliers as it goes forward.

sygma 13y ago
You could use something like Mailvelope [0]. You still have to manually encrypt things, but the process is fairly straightforward. [0]: http://www.mailvelope.com/ http://www.mailvelope.com/

rst 13y ago

Was hoping to see more discussion of backups. There are a bunch of possible approaches (depending on level of desired security, what the VPS provider offers, and how much you trust them), but for a mail server, there ought to be something...

db48x 13y ago
Use OfflineIMAP to sync to a local disk.
- dumaspere 13y ago
  Sync is not backup.
ishbits 13y ago
I currently use mbsync to backup my all mail folder on gmail. There is a way to set it to never delete on your local side, turning it more into a backup than a sync. I run that daily, and Crashplan takes care of the rest. Sure it doesn't save the folder structure and what not, but I'm ok with that.

daurnimator 13y ago

Does anyone have a gmail exporter? i.e. something that imports email from gmail WITH labels.

mcrittenden 13y ago
I believe gmvault.org should be fine for that.
mjn 13y ago
This is a proof-of-concept someone wrote, but I have no idea how tested it is: http://git.zx2c4.com/gmail-notmuch/ http://git.zx2c4.com/gmail-notmuch/ It syncs Gmail labels to notmuch tags. As I understand it: by default, Gmail treats labels as IMAP folders, which means a standard IMAP exporter will end up with multiple copies of messages with more than one label. The workaround is to only export the "All Mail" folder, and then add support for Gmail's custom attributes, namely retrieving the X-GM-LABELS [1] attribute for each message and then doing something with it (in this case, storing it as notmuch tags). [1] https://developers.google.com/gmail/imap_extensions#access_to_gmail_labels_x-gm-labels https://developers.google.com/gmail/imap_extensions#access_t...
vu3rdd 13y ago
You can use offlineimap to do that. Every label is an imap folder. offlineimap allows what they call as "folder filters", using which one can write a short python function parameterised over the folder name.

kefs 13y ago

While k9mail is a must, I suggest linking to the repo, which is usually lightyears ahead of public releases on the Play store. https://code.google.com/p/k9mail/downloads/list https://code.google.com/p/k9mail/downloads/list On a side note, they seem to have just hit v4 two days ago. Second side note, if you decide to use k9, be sure to turn off the signature under composition settings for each account you add.. it's turned on by default.

richdougherty 13y ago

It would be incredibly useful if there was a mail service that received email over SMTP, encrypted it straight away with a public key, then just dumped the encrypted email into a general-purpose online storage solution (e.g. an S3 bucket). That would IMO provide a good base for encrypted client-side apps to build on top of. Open source would better be able address the problem of writing a client once the money needed for hosting and storage is taken out of the equation.

LiveTheDream 13y ago
Hushmail and Lavabit sort of do this. They encrypt emails before storing them. However, as far as I can tell they handle the public/private keypair on their own, so the security only goes as far as you trust them.
dmix 13y ago
Countermail does this: https://countermail.com/ https://countermail.com/
- richdougherty 13y ago
  I was imagining an even lower level service. I guess that CounterMail must store unencrypted email headers in order to serve IMAP, right? I was thinking something that just manages the problem of being online 24/7 to receive email. This (and possibly sending email) is the only thing that can't be done completely client side. A service like I was thinking of would just accept messages, immediately encrypt them (headers and all) with an RSA key and then dump them into some third party online storage system (maybe sending a notification over XMPP too). The rest of the email pipeline could then be run completely client-side. I'm talking about doing everything client side: spam filtering, sorting into folders or tagging, running email rules, indexing, and of course viewing and reading email. (A clever filesystem/database would need to be layered on top of the online storage system to manage the state of the pipeline and provide fast indexes.) For maximum ease of use the client-side app could even be a rich Gmail-like JavaScript app that stores private keys locally and but stores most data remotely (e.g. uses S3 directly via CORS). This would all allow hackers like us to build interesting (but still encrypted) email services without having to worry about infrastructure. That's something I don't want to manage anyway! Also the barrier to entry of writing a Gmail clone would be much lower because you could do at almost zero cost. Since you don't have to receive email or manage storage your service would mostly be static JavaScript containing the client-side logic. Most - even all - of the service could just be served on a CDN.

magic_haze 13y ago

To play the devil's advocate, what exactly is the practical use of all this if most of your family and friends are on Gmail (and couldn't be arsed to figure out pgp)? From what I can see, your emails will now be sent in the clear over the internet, instead of staying within google's servers. Either way, the government's going to get your data, but at least you're protected against... /more/ unscrupulous people snooping on your stuff?

thex86 13y ago
That's a great point. Reminds of the time I taught my friend to use PGP and sent him an encrypted email. Every single time, he would reply in plain-text, thus exposing my older conversation. When asked why, he told me it's too much of a pain to do it. So my being careful about my privacy doesn't help if other people don't play along.
- eru 13y ago
  The OTR plugin for Pidgin does a better job of userfriendliness and gives more useful privacy guarantees (like plausible deniability).
  - shrikant 13y ago
    FYI, last I remember, Pidgin stored your passwords in plaintext in an ASCII file on disk, unless you jumped through some hoops to integrate it with your desktop environment's keyring. They even have a article up on their site explaining why it's necessary to do this.
    - eliasmacpherson 13y ago
      https://developer.pidgin.im/wiki/PlainTextPasswords https://developer.pidgin.im/wiki/PlainTextPasswords well that's just great.
    - claudius 13y ago
      You can disable storing passwords. If you don’t want to have to enter a ‘master password’ when Pidgin starts up, there is no way they can store the passwords more securely than plaintext. Get full disk encryption and be happy.

cwp 13y ago

I ran a set up similar to this for many years. It's not that hard, for those with a little unix experience. As moxie mentions, email is very forgiving—you have to break it badly and leave it broken for a long time before you start to lose messages. What eventually drove me to GMail was spam. I tried a bunch of different filters, and never found one with good-enough accuracy. Finally I decided that the independence and privacy wasn't worth the time I spent fiddling with filters and dealing with misclassified messages. As far as I can tell, Gmail is 100% accurate. Problem solved.

cabalamat 13y ago
> As far as I can tell, Gmail is 100% accurate. My experience is I've had a few false negatives and false positives. Gmail is still very good though, and any replacement, if it is to be widely used, needs to solve the spam problem. If each message had to be separately encrypted for each receiver, would this add much to a spammer's costs? I'm guessing it would, but not by enough to make spam uneconomic. A better solution might be to require that the first email someone sends someone else, unless they've been OK'd by a third party, contains bitcoins to the value of $0.01.
- comex 13y ago
  Fun fact: Bitcoin's proof-of-work algorithm is based on Hashcash, which was an anti-spam measure along those lines: a message contains a partial hash brute-force that takes the sender about a second of CPU time to compute (and depends on the recipient). But it didn't take off, and is problematic when spammers have vast amounts of CPU resources in the form of botnets.
bifrost 13y ago
> As far as I can tell, Gmail is 100% accurate. I can confirm that that is not correct. Google regularly flags internal ticket email as spam for one of my larger customers (mail that is sent within GApps). I also see about a 10-20% false positive rate on our various mailings (account setups, billing notices, etc).

hcarvalhoalves 13y ago

Normal people definitely don't want to manage a mail server though. Life is too short to waste figuring out why you're banned on Spamhaus for the 93th time. GMail sucks, but a home-made contraption is not the alternative.

werid 13y ago
Why must this be pointed out every time? There's enough of us weirdos out there.
siddboots 13y ago
> Normal people definitely don't want to... Life is too short to... Ugh! All you are saying is that if you aren't looking for a home made contraption, then a home made contraption isn't what you are looking for.
Aldipower 13y ago
I am not normal.
mjn 13y ago
To be fair to the author, I don't think an article that starts with "A Hacker's..." and/or involves Emacs has a pretension to being a solution for normal people.
- hcarvalhoalves 13y ago
  Yeah, I know. I'm just pointing out that GMail alternatives should exist that don't involve a huge PITA. I know how to configure mail servers just fine, but I still wouldn't do it for myself. That's what I mean by "normal".

_oxford 13y ago

I see a couple of problems here: 1. It's likely he's storing emails on the VPS. This puts us back at square one. A third party has a copy of your emails. And we know email does not garner the same privacy protections as postal mail. 2. You need a domain name. That system (DNS), as it is currently implemented (i.e., everyone setting their root zone to servers they do not control), is highly centralized -- few people maintain their own root zone, despite being easy to do. Domain names are susceptible to false allegations copyright and trademark infringement by private parties, not to mention easy censorship by the US gov't. When you lose your domain you lose email. (Though you shouldn't have to: email works fine with IP addresses in brackets.) So what's the solution: 1. Get a reachable IP (e.g., through ISP) or get a VPS. But if you get a VPS only use it to pierce NAT (how is left as exercise for reader - hint: supernode), not run a mail server. Don't store sensitive data like email on a VPS, or route sensitive data through it. 2. Use IP addresses not domain names. Alternatively, set up your own DNS that is available as a peer-to-peer service, or have your email contacts use a DNS server and root zone you collectively maintain: free domain names that you control. No one can censor your DNS (phonebook), except you.

ramblerman 13y ago

An interesting solution if you have something to hide I suppose. As has been stated time and again, most people don't. The danger lies in politicians, ceo's and other figures of authority who do and can be blackmailed. Rather than a few hackers setting up their own SMTP servers I think a more powerful solution lies in keeping focus on the actual problem, the out of control NSA program.

csense 13y ago

I always thought a big part of the reason people used gmail was for the snazzy web-based UI that was one of the first popular AJAX-based web applications. I eagerly read the article to see what alternative to this feature the author was suggesting, so I was surprised to see he's reading the emails with a standalone client...in fact, it's an emacs plugin!

rorrr2 13y ago
Exactly. It's hardly a solution. Viewing in-body images and attachments turns into a nightmare.

p4bl0 13y ago

When I saw the link's title, I immediately thought that it would be another webmail client hosted by someone, especially given the domain name. Because almost everytime I see a "hacker's X" or "X for hacker" title on HN I'm just afflicted by the content, so I got used to it. But here, what a pleasant surprise. The post is actually describing a real hacker's replacement for Gmail (which coincidentally is almost my setting, except I use mu [1] instead of notmuch). I'll keep it as a reference to send people asking for alternative email hosting. [1] http://www.djcbsoftware.nl/code/mu/ http://www.djcbsoftware.nl/code/mu/

alemhnan 13y ago

We could push a step further: "EMail Server as a service for common people". Somehow like http://instantserver.io/ http://instantserver.io/ . Or like Heroku: you create your "managed" mail server with a click.

omegant 13y ago
Please!

lazylizard 13y ago

linode has nice guides...e.g. https://library.linode.com/email/postfix/dovecot-mysql-debian-6-squeeze https://library.linode.com/email/postfix/dovecot-mysql-debia... though for postfix, having something like postfixadmin makes it nicer.. or perhaps something instant like http://www.xeams.com/ http://www.xeams.com/ will do..

lazylizard 13y ago

as for antispam..http://www.mxhero.com/ http://www.mxhero.com/ is auto and easy, and i think http://spamcheck.sourceforge.net/ http://spamcheck.sourceforge.net/ is very nice. u don't get as many controls but it sends quarantine digests, and runs much lighter than mxhero. http://www.scrolloutf1.com/ http://www.scrolloutf1.com/ looks nice too. at work we sell spamtitan, which i admit is really nice..tons of config with nice gui, good defaults, easy setup..but its non free...

drdaeman 13y ago

Why use notmuch when one can have standard (RFC5228) Sieve support?

spo81rty 13y ago

If you want a really good and really easy to setup mail server I would recommend SmarterMail. It is also free for 1 email user. I have used their product for about 10 years. Note that it is Windows server based. http://www.smartertools.com/smartermail/mail-server-software.aspx http://www.smartertools.com/smartermail/mail-server-software...

fredsted 13y ago

Why make it so hard? Why not just install virtualmin [0] on a Debian (or whatever Linux you prefer) server and get it over with. You also get web hosting, DB hosting, mailing lists, webmail and more as a bonus. And you don't have to worry about security updates. Just install and create your virtual host, and modify DNS for your domain. Couldn't be easier. Oh, and it's completely free. [0] http://www.virtualmin.com http://www.virtualmin.com

shunter 13y ago

I've started defining 'hacker' as someone who's willing to 'eat their own dogfood' as it were. Someone that is willing to spend time working on the nuts and bolts that lead to some kind of productivity rather than just being productive with the tool / service to begin with. I used to classify myself as a 'Hacker' and still do when it's something I want to learn more about. Most of the time, however, I'm more interested in just getting the benefits rather than tinkering with the internals. Sometimes, I'm a Hacker, sometimes I'm a consumer.

philjackson 13y ago

Interesting. I remain a step behind this one in that I'm using offlineimap to sync local maildirs from google's servers and then using mu and mu4e in emacs. Means I get to use the Gmail Android client which is actually very good.

nvarsj 13y ago

I had a similar setup a couple of years ago. The main problem I had was the maintenance required. If you have any machine publicly accessible you have to be on top of security updates and proper system hardening. I gave up after my exim4 Debian system got 0-day rooted. If doing it again I would avoid a Debian based distro. I'd probably use openbsd. And the less ports open the better.

snambi 13y ago

awesome... its time move away from proprietary, snooping services such as gmail. Hopefully setting up such a service should become easier ( may be less than 5 steps ) with better cloud VMs. Then even non-tech savvy people can have their emails away from snooping.

just2n 13y ago

I did this for a long time, but it's really annoying: 1. If your provider goes down, you lose mail. 2. If you are conversing with people who are using an insecure mailer, such as gmail, Yahoo, etc (which is probably > 99.9% of all e-mail users), your e-mail is still accessible to the NSA, or to some Fortune 100 advertising company. 3. It's only a matter of time before the "big dogs" in email abuse the position and decide who is and isn't allowed to send/receive email outside of their little oligarchy, either on their own or at the behest of governments. Like so much else that has been corrupted, we need to scratch the current architecture as too insecure, and build something truly secure for the future. This isn't in the interests of the Googles of the world, and it's actively in the worst interests of the NSA/FBI/CIA, so it's probably the right thing to do.

gnosis 13y ago
"Like so much else that has been corrupted, we need to scratch the current architecture as too insecure, and build something truly secure for the future." This is really the core technological issue that has enabled much of the recent mass spying: the Internet really was not designed with security, privacy, or anonymity in mind. Remember that it started as a research network that was used primarily among academics. Academia is generally a very open and trusting environment. The last thing the inventors of the Internet though of was trying to protect their computers, data, or privacy. We are living with the consequences of that mistake. The second major cause of this mess is the computer illiteracy and historical ignorance of much of the world's population. They don't understand how they are making themselves vulnerable by sharing so much information about themselves and trusting corporations that provide "free" web services to them. This is slowly changing for the better, as people become more technologically savvy and stories like the current spying scandals and various security breaches hit the news. As for the technological and security minded among us, particularly those who are just now starting to think about running their own mail servers, what took you all so long? None of the recent revelations should have been unexpected. The Internet needs a major privacy, security, and anonymity overhaul. If it's not rebuilt with those concerns at the core, they will all remain mostly illusory to the overwhelming majority of Internet users.
magicalist 13y ago

mwcampbell 13y ago

I think more of us need to run mail servers. For ourselves, for our families, and possibly for others who are willing to pay. Email is far too centralized now, at a handful of companies, in a handful of data centers. So in that regard, running a mail server on a VPS at one of the popular providers is kind of missing the point. My local cable ISP doesn't allow incoming or outgoing connections to port 25, nor incoming connections to port 80. So at least for now, I can't run a mail server in my home. I've thought about switching to DSL, but then I would take a major hit in speed, in both directions. Luckily, I have another option. There's a hosting provider where I live (Wichita, Kansas) that offers KVM-based virtual machine hosting. So I'll get a VM there, and if the service is any good, I'll move there from Linode. The pricing isn't competitive with Linode, let alone DigitalOcean, and I doubt that the connectivity is as good, since the server will be in a building here in Wichita rather than a real data center. But I'm willing to try it, in order to support a local business and fight the centralization of the Internet.