9 ms·

Google Drive issue affecting a significant subset of users

68 points by pgrote 10y ago

pgrote 10y ago
At first, I thought my machine was hacked. I didn't know Google Drive was changing their name and the error message I received looked fake: http://i.imgur.com/zAJVpAF.png http://i.imgur.com/zAJVpAF.png Notice the grammar and punctuation issues not to mention the email address to nowhere.
- PuffinBlue 10y ago
  Hmm, that's interesting. Looks like an error message meant for internal Google users. uploader-eng@ would presumably be uploader-eng@google.com The error message also looks like it is also geared towards users with google.com accounts. The whole thing smacks of someone pushing something internal from development to production.
  - pgrote 10y ago
    Good point. I hope the details as to how this happened become public. I cannot imagine a release to production can happen without multiple approvals.
    - PuffinBlue 10y ago
      It's definitely quite surprising, especially with Google's SRE strength and multiple automated check systems that have been previously documented here. They often do detailed postmortems so hopefully this one might get that treatment - though given the accidental exposure of the 'not for public release' UI and client they may well not go into much depth with this one.
    - brazzledazzle 10y ago
      Aren't a bunch of approvals an anathema to CI/CD? I have a hard time imagining an old school change management system being tolerated there.
- d--b 10y ago
  Same here, I was a little freaked out.
- citrusui 10y ago
  A friend also received the same message: https://twitter.com/resir014/status/842430307538149376 https://twitter.com/resir014/status/842430307538149376 Regarding spelling and grammar, it checks out fine for me. When you're typing a lot of code, it's easier to speak in short sentences and redact useless info (in this case, the @google.com email domain.)
johansch 10y ago
From experience, I've learned to parse "some users" as "half a billion users" when it comes to Google issues like these. I really think they have some policy to say "some" unless it's literally 100.00000% of the user base. Cause PR.. I wish they would improve on their communication accuracy in this aspect. I really do think it would make people trust them more in the long run. (Just tell us a percentage estimate! Is it 1% or 70%? Don't have an exact count? Just give us your best guess! Anything else than this really an insult. We're all in this together.)
- pgrote 10y ago
  The status board stil says, "Some users of the Google Drive Sync client for Windows will receive the error message "Sorry, Backup and Sync needs to quit" and will be unable to use the client. Other Sync clients are not affected. Affected users may continue using the Google Drive web interface and mobile applications." What is odd ... I cannot find anyone who was upgraded to the new client that isn't getting the error message. I wonder how they define "some." It would be interesting to see what criteria was used for upgrading. I've seen reports of Windows 10 and Windows 7 users being upgraded, but I have a Windows 7 machine that is still running the old Google Drive and hasn't been upgraded.
  - M_Grey 10y ago
    >What is odd ... I cannot find anyone who was upgraded to the new client that isn't getting the error message. I wonder how they define "some."What is odd ... I cannot find anyone who was upgraded to the new client that isn't getting the error message. I wonder how they define "some." Well there's this one guy in Pacoima, and he's having no problems at all. "Some".
- pirocks 10y ago
  I'm not affected so its definitely not 100%.
  - pgrote 10y ago
    This is awesome. Do you have a .google.com email address? Can you post a screenshot of the menu options for the new new Google Backup and Sync? I am interested in learning what is changing.
jjkmk 10y ago
I fixed this issue by fully removing google drive and reinstalling it. It seems like google released an early beta / alpha version accidentally.
- RachelF 10y ago
  It is also affecting online users, not just the desktop client - "The affected users are able to access Google Drive, but are seeing error messages and/or other unexpected behavior."
- e40 10y ago
  You mean the desktop version?
pgrote 10y ago
Appears the issue is resolved by an updated client through a push or manual download: Google Drive service has already been restored for some users, and we expect a resolution for all users within the next 12 hours. Please note this time frame is an estimate and may change. We are deploying an updated client now. The Windows client will automatically update with this fixed version within the next 12 hours. Alternatively, affected users can download the updated version of the client from https://www.google.com/drive/download/ https://www.google.com/drive/download/. Affected Windows 10 users may be required to sign in to their Google account within the client.
- RachelF 10y ago
  Indeed. There is more at the Google Drive forum. Seems to be limited to the Windows client from the reports. It could be serious as there are reports of folders disappearing. https://productforums.google.com/forum/#!forum/drive https://productforums.google.com/forum/#!forum/drive
mustafabisic1 10y ago
But I love the way they're dealing with it. :D it simply feels human from such a big company. Kudos Google
jhancock 10y ago
Apologies for the long post...this is a bit of a cry for help...is there a support group for programmers/admins that have been bitten by the Drive cool aide? My organisation did a migration to Google Drive (from internal Windows server) in the last few months. I've had quite a few issues (I'm the administrator). I'm posting this (abridged) laundry list of problems here...would love to have some feedback or pointers to what's wrong or what I'm doing wrong. When I've managed to get through to Google Drive tech support I don't get much in the way of answers. For context, I'll confine this to just two central share accounts: fileshare@mydomain and records@mydomain. Each account has around 1.5 terabytes of files. Our org has 85 G-suite accounts (users). This is a small to mid sized business. * I'll mostly skip the part of how difficult, slow and expensive a process it was to get around 3 terabytes into Google Drive. rclone was a good friend to us. It took months as we had to constantly re-sync as we migrate....we are running a business while we migrate. I'll also mostly skip the problems we had with the Drive sync client...its default behavior is to sync everything. I had to globally disable usage of this tool. Guess what happens to your company network when 50 users install the sync tool and click "add to drive" for the company fileshare. Good luck educating users with a wide range of PC skills on proper sync usage. * Drive's file sharing operations are opaque and take an indeterminate amount of time to complete a share (could take days). If I need to restrict access to a large subset of files, I have no idea when the share is complete. I have found that sometimes it does not complete for all users and I'm left with some files not shared. I can't find a pattern. I have no tools to help me find what (under a large set of folders) didn't get shared. If a user or me as admin needs to unshare, this can take days. So there is no practical way to protect sensitive access files except put them in different root folders or different share account. * No practical way to get a backup. I'm trying spanning backup at the moment. Its been running against my fileshare account all week. The spanning backup console tells me I have about 100 days to wait to complete the initial backup. Yeah, 100 days to backup 1.5 terabytes. What's the use of unlimited storage if I can't protect it? In this case, I'm suspecting Google Drive, not Spanning as I've become familiar with how performance variant the Google Drive API is. * Google Drive allows an end user to easily do "destructive" things to the central file share. Last Friday I received a call from a staff. She knew something bad just happened and quickly reached out for support. She had accidentally used the move or delete operation in the Drive UI to cut out a huge subset of our fileshare. I say "move or delete" because she didn't know and as an admin I couldn't figure out what happened. I dug around both the fileshare account and her account and couldn't find the files. The Trash folder on both accounts was empty. This didn't make sense as the activity panel clearly showed her and other users deleting things. At first the Trash view returned a server error. I refreshed my browser and got it to show. Empty. I tracked through the activity list and found the operation of concern....it showed the folder had been removed. Fortunately, I was able to click on the folder in the activity panel and it took me to it. From what I could tell, the files were in the system, but not attached to anything...no parent folder, not marked as deleted or moved. I was able to move this unattached folder under a new temp folder at the root of the fileshare account. Then go to that folder and move it again back to its original place. Then we wait...for the removed share permissions to fall back into place. Good thing this happened late on Friday as we had the weekend to wait for the shares to come back. * The Google Drive API is slow and unstable. I'm a programmer. We spent quite a bit of time writing a system to push files from our records management system to our Google Drive records account. Currently, we are adding less than 1000 files a day; mostly mid sized jpg's (1 to 3mb). So well below our API limit of 1 billion requests a day. Our server using the drive API tracks errors. Currently, around 1/3 of the uploads fail and need to be retried up to 10 times. Some never make it and we have to handle it manually. Good thing we wrote this in Elixir/Erlang as fault tolerance on the API client is our friend. * The Google Drive API is "global". When you create a service account (Google's API method for server to server communication), you give access for the scope of the API but not the account. Our system needs read/write access to only the records account. No can do. You have to give your API keys full read/write access to every account. So yeah, your dev and test systems have full access to all production data. Better be a very careful programmer. If you mess something up in dev, you could mess up your production drive accounts. * Combine these API problems with the inability to do a backup (and restore as well as restore to a test account) in any reasonable amount of time...and you can throw away your discipline of separating dev from production and being able to run repeatable tests before deployment. I'm going to spend the rest of my day reviewing code with a programmer. The code is to refactor our folder structure in our records account. It shouldn't delete anything...just move lots of folders into a new structure. But if we have just one bug in our migration scripts, well who knows. So I'll be reviewing code today and on pins and needles Saturday while our migration script runs on our production drive.
19eightyfour 10y ago
It's unclear to me if Google Drive really has changed its name to backup and sync. However it occurs to me that Google Drive is a name that might elicit some kind of regret since it was a product named and created before self-driving cars became a big thing and Google got into it with their in my opinion kind of clumsily named waymo.

68 points by pgrote 10y ago

At first, I thought my machine was hacked. I didn't know Google Drive was changing their name and the error message I received looked fake: http://i.imgur.com/zAJVpAF.png http://i.imgur.com/zAJVpAF.png Notice the grammar and punctuation issues not to mention the email address to nowhere.

PuffinBlue 10y ago
Hmm, that's interesting. Looks like an error message meant for internal Google users. uploader-eng@ would presumably be uploader-eng@google.com The error message also looks like it is also geared towards users with google.com accounts. The whole thing smacks of someone pushing something internal from development to production.
- pgrote 10y ago
  Good point. I hope the details as to how this happened become public. I cannot imagine a release to production can happen without multiple approvals.
  - PuffinBlue 10y ago
    It's definitely quite surprising, especially with Google's SRE strength and multiple automated check systems that have been previously documented here. They often do detailed postmortems so hopefully this one might get that treatment - though given the accidental exposure of the 'not for public release' UI and client they may well not go into much depth with this one.
  - brazzledazzle 10y ago
    Aren't a bunch of approvals an anathema to CI/CD? I have a hard time imagining an old school change management system being tolerated there.
d--b 10y ago
Same here, I was a little freaked out.
citrusui 10y ago
A friend also received the same message: https://twitter.com/resir014/status/842430307538149376 https://twitter.com/resir014/status/842430307538149376 Regarding spelling and grammar, it checks out fine for me. When you're typing a lot of code, it's easier to speak in short sentences and redact useless info (in this case, the @google.com email domain.)

johansch 10y ago

From experience, I've learned to parse "some users" as "half a billion users" when it comes to Google issues like these. I really think they have some policy to say "some" unless it's literally 100.00000% of the user base. Cause PR.. I wish they would improve on their communication accuracy in this aspect. I really do think it would make people trust them more in the long run. (Just tell us a percentage estimate! Is it 1% or 70%? Don't have an exact count? Just give us your best guess! Anything else than this really an insult. We're all in this together.)

pgrote 10y ago
The status board stil says, "Some users of the Google Drive Sync client for Windows will receive the error message "Sorry, Backup and Sync needs to quit" and will be unable to use the client. Other Sync clients are not affected. Affected users may continue using the Google Drive web interface and mobile applications." What is odd ... I cannot find anyone who was upgraded to the new client that isn't getting the error message. I wonder how they define "some." It would be interesting to see what criteria was used for upgrading. I've seen reports of Windows 10 and Windows 7 users being upgraded, but I have a Windows 7 machine that is still running the old Google Drive and hasn't been upgraded.
- M_Grey 10y ago
  >What is odd ... I cannot find anyone who was upgraded to the new client that isn't getting the error message. I wonder how they define "some."What is odd ... I cannot find anyone who was upgraded to the new client that isn't getting the error message. I wonder how they define "some." Well there's this one guy in Pacoima, and he's having no problems at all. "Some".
pirocks 10y ago
I'm not affected so its definitely not 100%.
- pgrote 10y ago
  This is awesome. Do you have a .google.com email address? Can you post a screenshot of the menu options for the new new Google Backup and Sync? I am interested in learning what is changing.

jjkmk 10y ago

I fixed this issue by fully removing google drive and reinstalling it. It seems like google released an early beta / alpha version accidentally.

RachelF 10y ago
It is also affecting online users, not just the desktop client - "The affected users are able to access Google Drive, but are seeing error messages and/or other unexpected behavior."
e40 10y ago
You mean the desktop version?

pgrote 10y ago

Appears the issue is resolved by an updated client through a push or manual download: Google Drive service has already been restored for some users, and we expect a resolution for all users within the next 12 hours. Please note this time frame is an estimate and may change. We are deploying an updated client now. The Windows client will automatically update with this fixed version within the next 12 hours. Alternatively, affected users can download the updated version of the client from https://www.google.com/drive/download/ https://www.google.com/drive/download/. Affected Windows 10 users may be required to sign in to their Google account within the client.

RachelF 10y ago
Indeed. There is more at the Google Drive forum. Seems to be limited to the Windows client from the reports. It could be serious as there are reports of folders disappearing. https://productforums.google.com/forum/#!forum/drive https://productforums.google.com/forum/#!forum/drive

mustafabisic1 10y ago

But I love the way they're dealing with it. :D it simply feels human from such a big company. Kudos Google

jhancock 10y ago

Apologies for the long post...this is a bit of a cry for help...is there a support group for programmers/admins that have been bitten by the Drive cool aide? My organisation did a migration to Google Drive (from internal Windows server) in the last few months. I've had quite a few issues (I'm the administrator). I'm posting this (abridged) laundry list of problems here...would love to have some feedback or pointers to what's wrong or what I'm doing wrong. When I've managed to get through to Google Drive tech support I don't get much in the way of answers. For context, I'll confine this to just two central share accounts: fileshare@mydomain and records@mydomain. Each account has around 1.5 terabytes of files. Our org has 85 G-suite accounts (users). This is a small to mid sized business. * I'll mostly skip the part of how difficult, slow and expensive a process it was to get around 3 terabytes into Google Drive. rclone was a good friend to us. It took months as we had to constantly re-sync as we migrate....we are running a business while we migrate. I'll also mostly skip the problems we had with the Drive sync client...its default behavior is to sync everything. I had to globally disable usage of this tool. Guess what happens to your company network when 50 users install the sync tool and click "add to drive" for the company fileshare. Good luck educating users with a wide range of PC skills on proper sync usage. * Drive's file sharing operations are opaque and take an indeterminate amount of time to complete a share (could take days). If I need to restrict access to a large subset of files, I have no idea when the share is complete. I have found that sometimes it does not complete for all users and I'm left with some files not shared. I can't find a pattern. I have no tools to help me find what (under a large set of folders) didn't get shared. If a user or me as admin needs to unshare, this can take days. So there is no practical way to protect sensitive access files except put them in different root folders or different share account. * No practical way to get a backup. I'm trying spanning backup at the moment. Its been running against my fileshare account all week. The spanning backup console tells me I have about 100 days to wait to complete the initial backup. Yeah, 100 days to backup 1.5 terabytes. What's the use of unlimited storage if I can't protect it? In this case, I'm suspecting Google Drive, not Spanning as I've become familiar with how performance variant the Google Drive API is. * Google Drive allows an end user to easily do "destructive" things to the central file share. Last Friday I received a call from a staff. She knew something bad just happened and quickly reached out for support. She had accidentally used the move or delete operation in the Drive UI to cut out a huge subset of our fileshare. I say "move or delete" because she didn't know and as an admin I couldn't figure out what happened. I dug around both the fileshare account and her account and couldn't find the files. The Trash folder on both accounts was empty. This didn't make sense as the activity panel clearly showed her and other users deleting things. At first the Trash view returned a server error. I refreshed my browser and got it to show. Empty. I tracked through the activity list and found the operation of concern....it showed the folder had been removed. Fortunately, I was able to click on the folder in the activity panel and it took me to it. From what I could tell, the files were in the system, but not attached to anything...no parent folder, not marked as deleted or moved. I was able to move this unattached folder under a new temp folder at the root of the fileshare account. Then go to that folder and move it again back to its original place. Then we wait...for the removed share permissions to fall back into place. Good thing this happened late on Friday as we had the weekend to wait for the shares to come back. * The Google Drive API is slow and unstable. I'm a programmer. We spent quite a bit of time writing a system to push files from our records management system to our Google Drive records account. Currently, we are adding less than 1000 files a day; mostly mid sized jpg's (1 to 3mb). So well below our API limit of 1 billion requests a day. Our server using the drive API tracks errors. Currently, around 1/3 of the uploads fail and need to be retried up to 10 times. Some never make it and we have to handle it manually. Good thing we wrote this in Elixir/Erlang as fault tolerance on the API client is our friend. * The Google Drive API is "global". When you create a service account (Google's API method for server to server communication), you give access for the scope of the API but not the account. Our system needs read/write access to only the records account. No can do. You have to give your API keys full read/write access to every account. So yeah, your dev and test systems have full access to all production data. Better be a very careful programmer. If you mess something up in dev, you could mess up your production drive accounts. * Combine these API problems with the inability to do a backup (and restore as well as restore to a test account) in any reasonable amount of time...and you can throw away your discipline of separating dev from production and being able to run repeatable tests before deployment. I'm going to spend the rest of my day reviewing code with a programmer. The code is to refactor our folder structure in our records account. It shouldn't delete anything...just move lots of folders into a new structure. But if we have just one bug in our migration scripts, well who knows. So I'll be reviewing code today and on pins and needles Saturday while our migration script runs on our production drive.

19eightyfour 10y ago

It's unclear to me if Google Drive really has changed its name to backup and sync. However it occurs to me that Google Drive is a name that might elicit some kind of regret since it was a product named and created before self-driving cars became a big thing and Google got into it with their in my opinion kind of clumsily named waymo.