7 ms·

More On Gmail’s Delivery Delays

64 points by daigoba66 13y ago

arbuge 13y ago
The date on this appears wrong - I think they mean yesterday, Sep 23. I must be one of the 1.5% affected - my gmail messages arrived 5 hours late.
- Gmo 13y ago
  Same here, gmail addresses of our company were affected
- BMarkmann 13y ago
  Based on the percentage of people I know who use GMail and/or Google Apps, I really feel like they're low-balling the effected population. Just my thoughts.
  - singlow 13y ago
    1.5% of messages may include 30% of users. Each affected user may have had some messages delivered quickly and others delayed. If 30% of users had 5% of their messages delayed, then about 1.5% of total messages would be delayed, not considering the difference in volume between users.
- thrownaway2424 13y ago
  That's not what it says. It says that 1.5% of messages arrived at least 2 hours late. If you got 40 messages yesterday then you probably got at least one that was 2 hours late.
  - arbuge 13y ago
    Ah. Better drink more coffee I guess...
ChikkaChiChi 13y ago
I have a hard time believing these statistics. Unless of course the 1.5% happened to be every single person who happened to have a Twitter, HN, Facebook, or Reddit account and was willing to post about it.
- tcoppi 13y ago
  Note that they didn't say 1.5% of users, they said 1.5% of messages were affected.
  - ChikkaChiChi 13y ago
    I'm totally aware of how they said what they said, I just don't necessarily believe it.
- Terretta 13y ago
  The missing statistic is what % of users are active. If 97% of users are inactive, and this affected 1.5% of users, then 50% of active users were affected.
- hamax 13y ago
  As a counterpoint, nobody I know experienced any problems with gmail yesterday. Including coworkers.
alanbyrne 13y ago
In a 300,000 user mail system I've seen mail queues grow by many thousands of messages a minute causing hundreds of gigabytes of email to be queued on the disks. Once the issue is resolved and the queue starts flowing again it can take a very long time for these messages to be sent successfully, especially when you need to scan each one for viruses and account for disk latency that doesn't usually occur on these servers.
jrochkind1 13y ago
A ~2.5 hour delay that effected ~1% of messages -- I've never had a mail provider so perfect that a delay of a couple hours didn't very occasionally happen. As far as their telling us, the failure didn't cascade causing their queues to fill up and other horrible things to happen. As far as we know, no mail got bounced, no mail got dropped on the floor and lost permanently, etc. A 2.6 second delay isn't even worth mentioning, how could one even tell the difference between a 2.6 second delay due to a problem on google's side, or routine problems in the email network outside of google that could easily lead to a second delay or whatever. Did people really get upset about this? Was it worse than it seems because of details they are ommitting? Are they preemptively advertising how well they dealt with this barely-a-problem-noticeable-to-users problem, because the message is really "see, look how damn stable gmail is, our worst problem ever is barely noticeable"? Or, what? Maybe those 1.5% of messages were all to the same users, so some people had no delay, and some people had ALL their mail delayed a couple hours? That would make it more noticeable. But in general, occasional (like once a year or less) couple-hour mail queue delays should probably be expected from any mail provider, no?
- swalkergibson 13y ago
  This. Absolutely this. Considering hosting your own service involves paying a competent system administrator at least $75K/year in addition to buying hardware and network and power, $5/user/month is more than reasonable for such an extremely high level of service. If some mission critical email was delayed for a little over an hour, pick up the phone. This is the very definition of a mountain out of a molehill.
- dangrossman 13y ago
  > But in general, occasional (like once a year or less) couple-hour mail queue delays should probably be expected from any mail provider, no? I've hosted a dozen or so mailboxes at Rackspace Mail (formerly Mailtrust) for several years now. I have experienced zero delayed delivery or service failures of any kind in all those years. For $2/mo/mailbox ($10/mo minimum total), you get a 100% uptime SLA, 25GB per mailbox, IMAP-push/POP3, good spam filtering and 24/7/365 support. That's 100% uptime or you get paid, and a real person will pick up the phone if you have a problem at 3AM on Christmas, for $2.
westmere 13y ago
The problem for us wasn't so much the email delay, it was that whenever a user would send a message from their iPhone, they'd get an error message "Server rejected mail." If gmail had accepted the message but delayed it on the server side, we probably wouldn't have noticed an issue at all.

64 points by daigoba66 13y ago

The date on this appears wrong - I think they mean yesterday, Sep 23. I must be one of the 1.5% affected - my gmail messages arrived 5 hours late.

Gmo 13y ago
Same here, gmail addresses of our company were affected
BMarkmann 13y ago
Based on the percentage of people I know who use GMail and/or Google Apps, I really feel like they're low-balling the effected population. Just my thoughts.
- singlow 13y ago
  1.5% of messages may include 30% of users. Each affected user may have had some messages delivered quickly and others delayed. If 30% of users had 5% of their messages delayed, then about 1.5% of total messages would be delayed, not considering the difference in volume between users.
thrownaway2424 13y ago
That's not what it says. It says that 1.5% of messages arrived at least 2 hours late. If you got 40 messages yesterday then you probably got at least one that was 2 hours late.
- arbuge 13y ago
  Ah. Better drink more coffee I guess...

ChikkaChiChi 13y ago

I have a hard time believing these statistics. Unless of course the 1.5% happened to be every single person who happened to have a Twitter, HN, Facebook, or Reddit account and was willing to post about it.

tcoppi 13y ago
Note that they didn't say 1.5% of users, they said 1.5% of messages were affected.
- ChikkaChiChi 13y ago
  I'm totally aware of how they said what they said, I just don't necessarily believe it.
Terretta 13y ago
The missing statistic is what % of users are active. If 97% of users are inactive, and this affected 1.5% of users, then 50% of active users were affected.
hamax 13y ago
As a counterpoint, nobody I know experienced any problems with gmail yesterday. Including coworkers.

alanbyrne 13y ago

In a 300,000 user mail system I've seen mail queues grow by many thousands of messages a minute causing hundreds of gigabytes of email to be queued on the disks. Once the issue is resolved and the queue starts flowing again it can take a very long time for these messages to be sent successfully, especially when you need to scan each one for viruses and account for disk latency that doesn't usually occur on these servers.

jrochkind1 13y ago

A ~2.5 hour delay that effected ~1% of messages -- I've never had a mail provider so perfect that a delay of a couple hours didn't very occasionally happen. As far as their telling us, the failure didn't cascade causing their queues to fill up and other horrible things to happen. As far as we know, no mail got bounced, no mail got dropped on the floor and lost permanently, etc. A 2.6 second delay isn't even worth mentioning, how could one even tell the difference between a 2.6 second delay due to a problem on google's side, or routine problems in the email network outside of google that could easily lead to a second delay or whatever. Did people really get upset about this? Was it worse than it seems because of details they are ommitting? Are they preemptively advertising how well they dealt with this barely-a-problem-noticeable-to-users problem, because the message is really "see, look how damn stable gmail is, our worst problem ever is barely noticeable"? Or, what? Maybe those 1.5% of messages were all to the same users, so some people had no delay, and some people had ALL their mail delayed a couple hours? That would make it more noticeable. But in general, occasional (like once a year or less) couple-hour mail queue delays should probably be expected from any mail provider, no?

swalkergibson 13y ago
This. Absolutely this. Considering hosting your own service involves paying a competent system administrator at least $75K/year in addition to buying hardware and network and power, $5/user/month is more than reasonable for such an extremely high level of service. If some mission critical email was delayed for a little over an hour, pick up the phone. This is the very definition of a mountain out of a molehill.
dangrossman 13y ago
> But in general, occasional (like once a year or less) couple-hour mail queue delays should probably be expected from any mail provider, no? I've hosted a dozen or so mailboxes at Rackspace Mail (formerly Mailtrust) for several years now. I have experienced zero delayed delivery or service failures of any kind in all those years. For $2/mo/mailbox ($10/mo minimum total), you get a 100% uptime SLA, 25GB per mailbox, IMAP-push/POP3, good spam filtering and 24/7/365 support. That's 100% uptime or you get paid, and a real person will pick up the phone if you have a problem at 3AM on Christmas, for $2.

westmere 13y ago

The problem for us wasn't so much the email delay, it was that whenever a user would send a message from their iPhone, they'd get an error message "Server rejected mail." If gmail had accepted the message but delayed it on the server side, we probably wouldn't have noticed an issue at all.