7 ms·

Yeah, the gmail UI (which is forgettable) is about 5% of the product; the other 95% is: * spam detection * search * mail header hacks I don't think that can

Yeah, the gmail UI (which is forgettable) is about 5% of the product; the other 95% is: * spam detection * search * mail header hacks I don't think that can be approximated in "a few months". > 50 mail addresses coming to my inbox On the other hand, no product (open source or not) should optimize for the outliers (not to be confused with edge cases).

vincentkriek 13y ago
What do you mean with mail header hacks?
- justinmk 13y ago
  Mail headers and mail relays are notoriously inconsistent. http://www.jwz.org/doc/threading.html http://www.jwz.org/doc/threading.html
  - eitland 13y ago
    Thanks for posting the link!
tluyben2 13y ago
Like I put in the tail of my post; I don't think i'm an outlier anymore soon. The 50 email addresses; sure, but the amount of mail, maybe for now, but not for long. And certainly not (already!) in size; there are plenty of people who get/send all their video/images/large docs via email. Edit: also, the Google team was really good to me and my (free! well I pay more for storage obviously, but it's not a google apps one) mailbox; on several occasions they contacted me when my mailbox was not functioning and helped fix the issues with it. Compared to a few years ago, the gmail backend, for me has gotten a lot better; search used to just stall; now it's always fast and accurate. And when I deleted a significant amount of mails, my mailbox would be unreachable for at least 1 day (giving a Oops error code 14 or something like that); no it just works. Edit2: I rather would like the spam AI from Google as a webservice for all clients to use, or an open source alternative. Spamassassin is a nice initiative, but it never worked for anyone I know. A lot of my colleagues just ended up routing their company mail through gmail to get a better spam filter.
- 300bps 13y ago
  You are an outlier and will always be an outlier. I've had email since I got my first modem in 1985. I've had hundreds of email addresses between BBSes, online services and the Internet. I've happily deleted all the mail and given up the addresses for all but a handful. What might change at some point is A&E launching a Digital Hoarder reality show wherein people with such a storage proclivity can get the help they need.
  - tluyben2 13y ago
    That would be one massively boring show :) Maybe you are right. I don't think so though; people are lazy and if the default status is that you can keep everything you have anyway (considering the allotted yearly storage growth and the plummeting prices of storage in general) you won't bother deleting anything. You seem to come from the same time as I do ; I had my first modem around '85 and was very active on BBSes. I ran my BBSes, for most of the 80s, of 720 kb disks, so I do know what storage limitation was. It's just not there anymore. Gmail now easily handles all of this and so should some other solutions imho, especially the ones wanting to compete with gmail. Google said; 'never delete another mail' and i'm sure people got used to that by now for all cloud services. The mail address and email hoarding came when I started making serious companies. And it's not like they bother me or whatever. Maybe I will throw everything away some time, but for now; I dodged an almost million euro gov fine a few years ago by having a complete mail conversations + attachments with my employees between 2002 and 2004. I would be in a whole load of crap if I wouldn't have those anymore.
    - 300bps 13y ago
      You said the following two things which are mutually exclusive: I wrote an export script for the mail system I was using at the time and imported 10 years of mail into gmail. I don't think so though; people are lazy and if the default status is that you can keep everything you have anyway The default status is that stuff goes away. I believe you're arguing that "this time it's different" but I see no evidence for that. In fact, there is a post almost every day on HN about Gmail users looking for an alternative. I myself can't wait to get off Gmail. When I do that I'll happily explicitly delete my Gmail account and all the email in it. Because I am lazy and it's easier to do that than it is to write a script to transfer it to what I use next. You seem to come from the same time as I do ; I had my first modem around '85 and was very active on BBSes. I got my first modem in 1985, got a phone line as my Christmas present from my parents in 1986 and started my own BBS. Started a consulting company in 1993. Started a small ISP in 1996, sold it in 1998 and started working for corporate America. In the U.S., I find from a legal perspective that the less documentation you have the better. That's why every large U.S. company has an email retention policy that is very short. Anyone looking to sue you needs proof and often that proof comes from your own records as part of discovery since they don't have any proof. If as part of a standard process you delete everything you're not legally required to retain then you can't be forced into helping their case.
e12e 13y ago
> On the other hand, no product (open source or not) should optimize for the outliers (not to be confused with edge cases). I think (hope!) there's room for niche products. I would never try to make an email client "for everyone". All the ones I've tried are crap (because they're "for everyone"). If you'd said: define your market and optimize for that - then I'd agree. But to not aim for a niche if you're doing a small project sounds just crazy (and wrong).
- justinmk 13y ago
  > define your market and optimize for that Right. But if you're writing a gmail clone, just don't get sidetracked by the token comment about email archives from 1985.
morkbot 13y ago
Add to that the speed. Roundcube looks pretty good, even Horde's could make a cut GUI-wise (judging by screenshots from the latest versions), but they both lag in the speed department, comparing to GMail or Fastmail.
hrjet 13y ago
My hunch is that Google's spam detection is not necessarily better than open-source solutions. It's just that they may have a larger corpus of data to train the system on. If I host my own email server, my spam filter will only have my inputs to determine what is spam. This leads me to an idea. There probably can be a service to build a dynamic corpus of spam samples. If I mark a message as spam, my email server can send a sample to the service, and periodically, my email server will download "top 100" samples to feed its filter.
- kilburn 13y ago
  This isn't necessarily a good idea. The type of SPAM you receive is different than the type of SPAM I receive. Moreover, it is very possible that I do (not) consider some messages as SPAM whereas you do (not). It is true that it takes some time to train the filter, but after that initial period you will get better results if you only train it with your own SPAM. I've been hosting my own email on a small personal server for some ~7 years now. All my email addresses redirect here (I have 9 frequently used addresses, and a bunch of website-specific ones). I'm using spamassassin (plus amavisd-new) as a proxy content filter. These are my filter's numbers for the last three days: 2 BAD-HEADER-2 1 BAD-HEADER-8 47 BANNED 297 CLEAN 6 INFECTED 426 SPAM 14 SPAMMY Spammy mails are delivered to a special "Spam" folder (just like in GMail). All those were indeed SPAM messages. I've received no spam messages outside of those. The high number of CLEAN mails is because I'm subscribed to a few mailing lists and I receive notifications/tracking info/alerts from a number of services. TL;DR: SpamAssassin is hard to setup and train correctly (just like the rest of a server's mail-stack). However, once you get over this part, the results can be as good as gmail's.
  - justinmk 13y ago
    > I've received no spam messages outside of those. How protective are you of your email address? My email is posted all over the internet, but I haven't seen spam in gmail for at least 6 months--probably longer.
    - kilburn 13y ago
snogglethorpe 13y ago
... and on top of all that is the simple fact that gmail is backed by a very robust infrastructure and supported by company that's very unlikely to go out of business anytime soon. [You can make jokes about Google reader etc, but gmail is far too important to Google's public image to make that a reasonable concern.] Together with the very decent UI, great spam handling, and free space, this makes gmail a pretty good choice for a lot of people. Even with a good free version of the software, duplicating the entire package is a huge task ...
mbesto 13y ago
I would happily pay someone non-google to get those 95% things right and then add in mobile sync, gmail-like interface, and you've got a customer.
- sp332 13y ago
  What parts of gmail's webmail interface do you find indispensable?
  - massysett 13y ago
    For me, nothing; the huge thing about Gmail is it works flawlessly from my Android phone. No IMAP solution came close. These days I read most of my mail on the phone. Once I was in a long check-in line at the airport. Silly me, I figured I wouldn't check in at home because I had to check bags. At the airport I got on my phone, pulled up the confirmation email, checked in, and went to the bag drop line. With the clumsiness of IMAP and dealing with it on phones I would never have managed to do this if I was hosting my own mail, and even if I had managed to do it, the cost would have been the hours and the hassle of maintaining mail. It was at that moment that I realized I would never host my own mail again. Just not worth it.
vpeters25 13y ago
Search is my biggest issue too. I switched out of gmail to my self-hosted server using roundcube and searching through IMAP is excruciatingly slow compared to gmail, even after I moved the mailboxes to an SSD. I am not sure whether this is a limitation on the IMAP protocol, but I suspect a proper replacement to gmail might involve coding a new mailbox backend and protocol.
- brdrak 13y ago
  I may be wrong, but I believe IMAP, the protocol, just defines how IMAP clients/servers should issue and respond to search commands, but doesn't deal with mail indexing and things like that. Cyrus IMAP, which I use, provides Squatter, which substantially speeds up search, but the IMAP client is still the interface. Performance I get is around a sec for a simple query, a few sec for a more complex one against a folder with 20K messages. Last I checked, Roundcube had very limited search capabilities. Thunderbird is much better, but I'd prefer a terminal based search. Something like SQL. Does anyone know of a tool like that?

vincentkriek 13y ago

What do you mean with mail header hacks?

justinmk 13y ago
Mail headers and mail relays are notoriously inconsistent. http://www.jwz.org/doc/threading.html http://www.jwz.org/doc/threading.html
- eitland 13y ago
  Thanks for posting the link!

tluyben2 13y ago

Like I put in the tail of my post; I don't think i'm an outlier anymore soon. The 50 email addresses; sure, but the amount of mail, maybe for now, but not for long. And certainly not (already!) in size; there are plenty of people who get/send all their video/images/large docs via email. Edit: also, the Google team was really good to me and my (free! well I pay more for storage obviously, but it's not a google apps one) mailbox; on several occasions they contacted me when my mailbox was not functioning and helped fix the issues with it. Compared to a few years ago, the gmail backend, for me has gotten a lot better; search used to just stall; now it's always fast and accurate. And when I deleted a significant amount of mails, my mailbox would be unreachable for at least 1 day (giving a Oops error code 14 or something like that); no it just works. Edit2: I rather would like the spam AI from Google as a webservice for all clients to use, or an open source alternative. Spamassassin is a nice initiative, but it never worked for anyone I know. A lot of my colleagues just ended up routing their company mail through gmail to get a better spam filter.

300bps 13y ago
You are an outlier and will always be an outlier. I've had email since I got my first modem in 1985. I've had hundreds of email addresses between BBSes, online services and the Internet. I've happily deleted all the mail and given up the addresses for all but a handful. What might change at some point is A&E launching a Digital Hoarder reality show wherein people with such a storage proclivity can get the help they need.
- tluyben2 13y ago
  That would be one massively boring show :) Maybe you are right. I don't think so though; people are lazy and if the default status is that you can keep everything you have anyway (considering the allotted yearly storage growth and the plummeting prices of storage in general) you won't bother deleting anything. You seem to come from the same time as I do ; I had my first modem around '85 and was very active on BBSes. I ran my BBSes, for most of the 80s, of 720 kb disks, so I do know what storage limitation was. It's just not there anymore. Gmail now easily handles all of this and so should some other solutions imho, especially the ones wanting to compete with gmail. Google said; 'never delete another mail' and i'm sure people got used to that by now for all cloud services. The mail address and email hoarding came when I started making serious companies. And it's not like they bother me or whatever. Maybe I will throw everything away some time, but for now; I dodged an almost million euro gov fine a few years ago by having a complete mail conversations + attachments with my employees between 2002 and 2004. I would be in a whole load of crap if I wouldn't have those anymore.
  - 300bps 13y ago
    You said the following two things which are mutually exclusive: I wrote an export script for the mail system I was using at the time and imported 10 years of mail into gmail. I don't think so though; people are lazy and if the default status is that you can keep everything you have anyway The default status is that stuff goes away. I believe you're arguing that "this time it's different" but I see no evidence for that. In fact, there is a post almost every day on HN about Gmail users looking for an alternative. I myself can't wait to get off Gmail. When I do that I'll happily explicitly delete my Gmail account and all the email in it. Because I am lazy and it's easier to do that than it is to write a script to transfer it to what I use next. You seem to come from the same time as I do ; I had my first modem around '85 and was very active on BBSes. I got my first modem in 1985, got a phone line as my Christmas present from my parents in 1986 and started my own BBS. Started a consulting company in 1993. Started a small ISP in 1996, sold it in 1998 and started working for corporate America. In the U.S., I find from a legal perspective that the less documentation you have the better. That's why every large U.S. company has an email retention policy that is very short. Anyone looking to sue you needs proof and often that proof comes from your own records as part of discovery since they don't have any proof. If as part of a standard process you delete everything you're not legally required to retain then you can't be forced into helping their case.

e12e 13y ago

> On the other hand, no product (open source or not) should optimize for the outliers (not to be confused with edge cases). I think (hope!) there's room for niche products. I would never try to make an email client "for everyone". All the ones I've tried are crap (because they're "for everyone"). If you'd said: define your market and optimize for that - then I'd agree. But to not aim for a niche if you're doing a small project sounds just crazy (and wrong).

justinmk 13y ago
> define your market and optimize for that Right. But if you're writing a gmail clone, just don't get sidetracked by the token comment about email archives from 1985.

morkbot 13y ago

Add to that the speed. Roundcube looks pretty good, even Horde's could make a cut GUI-wise (judging by screenshots from the latest versions), but they both lag in the speed department, comparing to GMail or Fastmail.

hrjet 13y ago

My hunch is that Google's spam detection is not necessarily better than open-source solutions. It's just that they may have a larger corpus of data to train the system on. If I host my own email server, my spam filter will only have my inputs to determine what is spam. This leads me to an idea. There probably can be a service to build a dynamic corpus of spam samples. If I mark a message as spam, my email server can send a sample to the service, and periodically, my email server will download "top 100" samples to feed its filter.

kilburn 13y ago
This isn't necessarily a good idea. The type of SPAM you receive is different than the type of SPAM I receive. Moreover, it is very possible that I do (not) consider some messages as SPAM whereas you do (not). It is true that it takes some time to train the filter, but after that initial period you will get better results if you only train it with your own SPAM. I've been hosting my own email on a small personal server for some ~7 years now. All my email addresses redirect here (I have 9 frequently used addresses, and a bunch of website-specific ones). I'm using spamassassin (plus amavisd-new) as a proxy content filter. These are my filter's numbers for the last three days: 2 BAD-HEADER-2 1 BAD-HEADER-8 47 BANNED 297 CLEAN 6 INFECTED 426 SPAM 14 SPAMMY Spammy mails are delivered to a special "Spam" folder (just like in GMail). All those were indeed SPAM messages. I've received no spam messages outside of those. The high number of CLEAN mails is because I'm subscribed to a few mailing lists and I receive notifications/tracking info/alerts from a number of services. TL;DR: SpamAssassin is hard to setup and train correctly (just like the rest of a server's mail-stack). However, once you get over this part, the results can be as good as gmail's.
- justinmk 13y ago
  > I've received no spam messages outside of those. How protective are you of your email address? My email is posted all over the internet, but I haven't seen spam in gmail for at least 6 months--probably longer.
  - kilburn 13y ago

snogglethorpe 13y ago

... and on top of all that is the simple fact that gmail is backed by a very robust infrastructure and supported by company that's very unlikely to go out of business anytime soon. [You can make jokes about Google reader etc, but gmail is far too important to Google's public image to make that a reasonable concern.] Together with the very decent UI, great spam handling, and free space, this makes gmail a pretty good choice for a lot of people. Even with a good free version of the software, duplicating the entire package is a huge task ...

mbesto 13y ago

I would happily pay someone non-google to get those 95% things right and then add in mobile sync, gmail-like interface, and you've got a customer.

sp332 13y ago
What parts of gmail's webmail interface do you find indispensable?
- massysett 13y ago
  For me, nothing; the huge thing about Gmail is it works flawlessly from my Android phone. No IMAP solution came close. These days I read most of my mail on the phone. Once I was in a long check-in line at the airport. Silly me, I figured I wouldn't check in at home because I had to check bags. At the airport I got on my phone, pulled up the confirmation email, checked in, and went to the bag drop line. With the clumsiness of IMAP and dealing with it on phones I would never have managed to do this if I was hosting my own mail, and even if I had managed to do it, the cost would have been the hours and the hassle of maintaining mail. It was at that moment that I realized I would never host my own mail again. Just not worth it.

vpeters25 13y ago

Search is my biggest issue too. I switched out of gmail to my self-hosted server using roundcube and searching through IMAP is excruciatingly slow compared to gmail, even after I moved the mailboxes to an SSD. I am not sure whether this is a limitation on the IMAP protocol, but I suspect a proper replacement to gmail might involve coding a new mailbox backend and protocol.

brdrak 13y ago
I may be wrong, but I believe IMAP, the protocol, just defines how IMAP clients/servers should issue and respond to search commands, but doesn't deal with mail indexing and things like that. Cyrus IMAP, which I use, provides Squatter, which substantially speeds up search, but the IMAP client is still the interface. Performance I get is around a sec for a simple query, a few sec for a more complex one against a folder with 20K messages. Last I checked, Roundcube had very limited search capabilities. Thunderbird is much better, but I'd prefer a terminal based search. Something like SQL. Does anyone know of a tool like that?