6 ms·

Gmail back soon for everyone

79 points by mjfern 16y ago

ck2 16y ago
How on earth do they backup all that to tape? Must be a massive amount of tape.
- amock 16y ago
  If you look at http://www.oracle.com/us/products/servers-storage/storage/tape-storage/index.html http://www.oracle.com/us/products/servers-storage/storage/ta... it talks about the reduced cost of ownership for a 20PB tape library, so there are tape products available for large volumes of data. The SL8500 supports 500PB of raw storage.
  - jedsmith 16y ago
    If everybody filled their mailbox, 500 PB would back up 66.1 million Gmail accounts. Since they don't, we're probably looking at everybody on Earth: 73.8 MB per mailbox, which sounds like a pretty good shot at an average. Looking at the SL8500, I think I'd intentionally lose data just so I could play with it to restore it.
    - PakG1 16y ago
      >> Looking at the SL8500, I think I'd intentionally lose data just so I could play with it to restore it. And this is why high-paid sysadmins are serious nasty hardasses when they're clocked in. :)
    - wlievens 16y ago
      Would 74 MB really cut it? I'm probably "above average" but mine uses 3 gb.
      - riffraff 16y ago
        74MB seems little to me too (I am thinking of how often people have sent me ppt of cute cats, word documents of a todo list and tiff scans of some thing they drew). It would be nice if grandparent has some reference for such a number to share
      - Peaker 16y ago
        Consider de-duplication.
        deleted 16y ago
        [deleted]
      - jedsmith 16y ago
        There's probably a lot of people that don't "get" Gmail and delete all of their messages. There's probably also a lot of people who sign up for an account and don't use it, ever, and they contribute to the numbers. Keep in mind ~80 MB is for everyone on the planet (all nearly 7 billion) in 500 PB. I'm betting the average is a lot lower than we'd think.
  - bdonlan 16y ago
    The hard part is backing up that kind of volume of data frequently enough that you always have a recent backup :)
- snewman 16y ago
  To quote from The Ringworld Engineers: "The Kzinti Patriarchy is not normally terrified by sheer magnitude." I can safely say that the same statement applies to Google.
- sacrilicious 16y ago
  Total conjecture, but 3 ways come to mind: De-duplication, compression, and hierarchical storage management. Keep less-frequently accessed parts of the data 'Nearline', and snapshot the lower-demand, older data slightly less often. Continuous data protection appliances were probably designed and tuned specifically for this problem set, and one could assume it was built into the original architecture so that an appropriate amount of the infrastructure already had backup space in mind and accounted for.
tdfx 16y ago
"If you were affected by this issue, it’s important to note that email sent to you between 6:00 PM PST on February 27 and 2:00 PM PST on February 28 was likely not delivered to your mailbox, and the senders would have received a notification that their messages weren’t delivered. " That's pretty scary to me. I would've hated to have gotten any e-mail alerts or payment due date reminders during that time.
- tomjen3 16y ago
  Since they got a warning that the email address was temporarily unavailable (at least that's what they should have received) any well configured mailserver should have resend the email.
  - kgo 16y ago
    Yeah, scary would be no bounce, no delivery. Bouncing is relatively graceful.
    - stretchwithme 16y ago
      even more graceful would be keeping track of bounces for 24 hours so you could at least tell someone what they missed if there's a cliche like this.
      - stretchwithme 16y ago
        glitch!
  - wtallis 16y ago
    From a bounce I got: ... From: Mail Delivery Subsystem <mailer-daemon@googlemail.com> To: [me]@gmail.com X-Failed-Recipients: [someone else]@gmail.com Subject: Delivery Status Notification (Failure) Message-ID: <20cf30433ea07a2745049d4fe20e@google.com> Date: Mon, 28 Feb 2011 04:12:14 +0000 Content-Type: text/plain; charset=ISO-8859-1 Delivery to the following recipient failed permanently: [someone else]@gmail.com Technical details of permanent failure: Account disabled ----- Original message ----- ... Not exactly the kind of thing that would make you think resending would get the message through.
there 16y ago
i love how they keep mentioning "0.02%" so you think it's some insignificant number, but it's actually something like 150,000 accounts.
- mhansen 16y ago
  0.02%
- mitjak 16y ago
  And how many of those are likely to be active and be expecting serious email instead of newsletters and spam?
  - kenjackson 16y ago
    I presume the same percentage as all gmail accounts. I saw nothing that indicated that these users were less active than anyone else. Unlike Hotmail, I think a very large percentage of Gmail accounts are pretty active.
  - foobarbazetc 16y ago
    In what world does that matter? I think the word "tape" tells you all you need to know about how serious this issue is.
    - mitjak 16y ago
      The original commenter pointed out that 0.02% is a more significant figure than the Gmail team makes it out to be, and that 150,000 is very significant. I'm saying the 150,000 figure isn't as significant as it looks, either.
- kgo 16y ago
  150,000 accounts out of SEVEN HUNDRED AND FIFTY FUCKING MILLION had trouble accessing email for LESS THAN A SINGLE DAY on A WEEKEND. No accounts or emails were deleted. I'd argue that is an insignificant number.
akent 16y ago
"We released a storage software update that introduced the unexpected bug, which caused 0.02% of Gmail users to temporarily lose access to their email. When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version." Some bug!!
Cariapa 16y ago
I can't remember having this sort of problem with yahoo mail. Then again they don't build their own filesystem.
- jasonkester 16y ago
  Yahoo is far worse. When they lose your mail, it's gone. Yahoo Mail lost two of my accounts over the years. The first time was early in the service, and they seem to have lost all their accounts at once. Suddenly you couldn't log in, and there was an apology message saying that you'd need to create a new account. And that you couldn't get the same email address back, which is why my @yahoo.com address had a 2 at the end of it for several years... Until they locked me out of it one day. As in, I checked my mail at night, then in the morning my password didn't work anymore. Since they have no human beings working for them, an entire month's worth of attempts to get access back resulted in nothing but a big pile of form letters and links to the Forgot Password page. So no, you're mistaken. Yahoo does have this problem. If you have an account with them I'd strongly advise you to back everything up. Then switch to Gmail.
  - iwwr 16y ago
    Yahoo also charges a monthly fee for access to pop3. Their web interface is slower and buggier than gmail's. There is free unlimited storage, but a smaller limit on file sizes. The virus scan on outgoing messages is usually slow; if gmail is doing virus scans, it's on the backend, it does not delay the user.
  - Cariapa 16y ago
    Oh, I'm on Gmail already, trust me. But thats because I like the faster interface, search capabilities, being able to check multiple mail accounts, gtalk etc. Yahoo is a great repository for my non-friends and family email (semi-spam). I guess I've been very lucky that never happened to my Yahoo account.
btilly 16y ago
It is worth noting that this is the first significant data loss issue that I am aware of involving gmail. As much as the incident sucks for those involved, that is a pretty good record overall.
bane 16y ago
This is great, last night I went to bed thinking "How could Google just walk away from this problem without fixing it, don't they have backups, what about corporate users?" And this morning "we have backups, we're restoring, oh and we're trying to fix what caused it in the first place"
l0c0b0x 16y ago
Would it be safe to assume then, that Google tests it's software upgrades with 0.02% of it's gmail account? (However many thousands clients are affected, it's probably better than affecting ALL their clients). I wonder how they choose their 'test' client base.

79 points by mjfern 16y ago

ck2 16y ago

How on earth do they backup all that to tape? Must be a massive amount of tape.

amock 16y ago
If you look at http://www.oracle.com/us/products/servers-storage/storage/tape-storage/index.html http://www.oracle.com/us/products/servers-storage/storage/ta... it talks about the reduced cost of ownership for a 20PB tape library, so there are tape products available for large volumes of data. The SL8500 supports 500PB of raw storage.
- jedsmith 16y ago
  If everybody filled their mailbox, 500 PB would back up 66.1 million Gmail accounts. Since they don't, we're probably looking at everybody on Earth: 73.8 MB per mailbox, which sounds like a pretty good shot at an average. Looking at the SL8500, I think I'd intentionally lose data just so I could play with it to restore it.
  - PakG1 16y ago
    >> Looking at the SL8500, I think I'd intentionally lose data just so I could play with it to restore it. And this is why high-paid sysadmins are serious nasty hardasses when they're clocked in. :)
  - wlievens 16y ago
    Would 74 MB really cut it? I'm probably "above average" but mine uses 3 gb.
    - riffraff 16y ago
      74MB seems little to me too (I am thinking of how often people have sent me ppt of cute cats, word documents of a todo list and tiff scans of some thing they drew). It would be nice if grandparent has some reference for such a number to share
    - Peaker 16y ago
      Consider de-duplication.
      - deleted 16y ago
        [deleted]
    - jedsmith 16y ago
      There's probably a lot of people that don't "get" Gmail and delete all of their messages. There's probably also a lot of people who sign up for an account and don't use it, ever, and they contribute to the numbers. Keep in mind ~80 MB is for everyone on the planet (all nearly 7 billion) in 500 PB. I'm betting the average is a lot lower than we'd think.
- bdonlan 16y ago
  The hard part is backing up that kind of volume of data frequently enough that you always have a recent backup :)
snewman 16y ago
To quote from The Ringworld Engineers: "The Kzinti Patriarchy is not normally terrified by sheer magnitude." I can safely say that the same statement applies to Google.
sacrilicious 16y ago
Total conjecture, but 3 ways come to mind: De-duplication, compression, and hierarchical storage management. Keep less-frequently accessed parts of the data 'Nearline', and snapshot the lower-demand, older data slightly less often. Continuous data protection appliances were probably designed and tuned specifically for this problem set, and one could assume it was built into the original architecture so that an appropriate amount of the infrastructure already had backup space in mind and accounted for.

tdfx 16y ago

"If you were affected by this issue, it’s important to note that email sent to you between 6:00 PM PST on February 27 and 2:00 PM PST on February 28 was likely not delivered to your mailbox, and the senders would have received a notification that their messages weren’t delivered. " That's pretty scary to me. I would've hated to have gotten any e-mail alerts or payment due date reminders during that time.

tomjen3 16y ago
Since they got a warning that the email address was temporarily unavailable (at least that's what they should have received) any well configured mailserver should have resend the email.
- kgo 16y ago
  Yeah, scary would be no bounce, no delivery. Bouncing is relatively graceful.
  - stretchwithme 16y ago
    even more graceful would be keeping track of bounces for 24 hours so you could at least tell someone what they missed if there's a cliche like this.
    - stretchwithme 16y ago
      glitch!
- wtallis 16y ago
  From a bounce I got: ... From: Mail Delivery Subsystem <mailer-daemon@googlemail.com> To: [me]@gmail.com X-Failed-Recipients: [someone else]@gmail.com Subject: Delivery Status Notification (Failure) Message-ID: <20cf30433ea07a2745049d4fe20e@google.com> Date: Mon, 28 Feb 2011 04:12:14 +0000 Content-Type: text/plain; charset=ISO-8859-1 Delivery to the following recipient failed permanently: [someone else]@gmail.com Technical details of permanent failure: Account disabled ----- Original message ----- ... Not exactly the kind of thing that would make you think resending would get the message through.

there 16y ago

i love how they keep mentioning "0.02%" so you think it's some insignificant number, but it's actually something like 150,000 accounts.

mhansen 16y ago
0.02%
mitjak 16y ago
And how many of those are likely to be active and be expecting serious email instead of newsletters and spam?
- kenjackson 16y ago
  I presume the same percentage as all gmail accounts. I saw nothing that indicated that these users were less active than anyone else. Unlike Hotmail, I think a very large percentage of Gmail accounts are pretty active.
- foobarbazetc 16y ago
  In what world does that matter? I think the word "tape" tells you all you need to know about how serious this issue is.
  - mitjak 16y ago
    The original commenter pointed out that 0.02% is a more significant figure than the Gmail team makes it out to be, and that 150,000 is very significant. I'm saying the 150,000 figure isn't as significant as it looks, either.
kgo 16y ago
150,000 accounts out of SEVEN HUNDRED AND FIFTY FUCKING MILLION had trouble accessing email for LESS THAN A SINGLE DAY on A WEEKEND. No accounts or emails were deleted. I'd argue that is an insignificant number.

akent 16y ago

"We released a storage software update that introduced the unexpected bug, which caused 0.02% of Gmail users to temporarily lose access to their email. When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version." Some bug!!

Cariapa 16y ago

I can't remember having this sort of problem with yahoo mail. Then again they don't build their own filesystem.

jasonkester 16y ago
Yahoo is far worse. When they lose your mail, it's gone. Yahoo Mail lost two of my accounts over the years. The first time was early in the service, and they seem to have lost all their accounts at once. Suddenly you couldn't log in, and there was an apology message saying that you'd need to create a new account. And that you couldn't get the same email address back, which is why my @yahoo.com address had a 2 at the end of it for several years... Until they locked me out of it one day. As in, I checked my mail at night, then in the morning my password didn't work anymore. Since they have no human beings working for them, an entire month's worth of attempts to get access back resulted in nothing but a big pile of form letters and links to the Forgot Password page. So no, you're mistaken. Yahoo does have this problem. If you have an account with them I'd strongly advise you to back everything up. Then switch to Gmail.
- iwwr 16y ago
  Yahoo also charges a monthly fee for access to pop3. Their web interface is slower and buggier than gmail's. There is free unlimited storage, but a smaller limit on file sizes. The virus scan on outgoing messages is usually slow; if gmail is doing virus scans, it's on the backend, it does not delay the user.
- Cariapa 16y ago
  Oh, I'm on Gmail already, trust me. But thats because I like the faster interface, search capabilities, being able to check multiple mail accounts, gtalk etc. Yahoo is a great repository for my non-friends and family email (semi-spam). I guess I've been very lucky that never happened to my Yahoo account.

btilly 16y ago

It is worth noting that this is the first significant data loss issue that I am aware of involving gmail. As much as the incident sucks for those involved, that is a pretty good record overall.

bane 16y ago

This is great, last night I went to bed thinking "How could Google just walk away from this problem without fixing it, don't they have backups, what about corporate users?" And this morning "we have backups, we're restoring, oh and we're trying to fix what caused it in the first place"

l0c0b0x 16y ago

Would it be safe to assume then, that Google tests it's software upgrades with 0.02% of it's gmail account? (However many thousands clients are affected, it's probably better than affecting ALL their clients). I wonder how they choose their 'test' client base.