4 ms·

Disclaimer: SRE at Google, not on Drive. I didn't look into this and don't know what really happened, but I can guess. It's more likely a tradeoff. Either you

Disclaimer: SRE at Google, not on Drive. I didn't look into this and don't know what really happened, but I can guess. It's more likely a tradeoff. Either you set this limit and protect your service from an identified scalability limit with the current architecture, or you plan for a (possibly long and expensive) redesign to get rid of this bottleneck. My guess is that a group of SRE and devs identified the risk, listed their options, evaluated the impact on users (eg: which fraction of users have more than 5M files) and assumed that the change would be mostly unnoticed and that the error message would be enough to push users with more than 5M files in their drive account to do some cleanup. This group of people underestimated the impact and maybe chose to not involve product managers. It's also possible that the decision was rushed because of recent layoffs or any other random event which pushed engineers to act quickly. With the bad press, leadership got involved, decision to rollback was taken. In my team, we would have a retrospective doc to discuss the issue (not exactly a postmortem, as this process has specific requirements which would not be applicable to this case). I think this is a easy mistake to do even with very good intentions, and I can see myself doing it.

soundsgoodtome 4y ago
Is PM not consulted for changes like this? I feel like this is something that a competent PM would ask eng to pause on while they determined actual user impact and lines up messaging.
- martius 4y ago
  Short answer, yes, a PM should have been consulted and give their approval for the change. In the scenario I described above (again, it's a guess, I don't know what actually happened), it's possible that the PM was bypassed because the engineers for a reason they thought was good. For example: * they didn't even think about involving a PM because that's something you have never done, * the people who wrote/reviewed the change assumed the conversation already happened, * they were pressured to move fast to mitigate an imminent or existing problem (performance, scalability, ...), * the PM who should have made this call "left" the company and didn't (get a chance to) hand-off their responsibilities. I guess what I mean with these comments is that sometimes there are misses like this. Maybe it's a sign of a systematic failure and that internal processes should be improved, but I don't think it means that the company is fundamentally unable to handle these changes correctly or can't make the right technical or product decisions.
maccard 4y ago
> I think this is a easy mistake to do even with very good intentions, and I can see myself doing it. Hard disagree. You missed one very important part in your writeup - at no point did they communicate that they were imposing this limit, and that this limit appearead, undocumented, overnight. I was someone who was directly impacted by this change. We're a 40 person company who used (past tense) GDrive as a shared network drive, including for storing builds of our app. We pay $18/person, and as part of that, google workspace advertises 5TB per user pooled[0], and nowhere in the google docs does it mention that this limit will exist [1]. If I was aware of a limit, we would have cleaned up our old files, but instead we started getting spurious 403's - as far as we could tell we were well within our usage limits. It was only when https://issuetracker.google.com/issues/268606830?pli=1 https://issuetracker.google.com/issues/268606830?pli=1 this post hit HN, I realised what was wrong. [0] https://workspace.google.com/intl/en_us/pricing.html https://workspace.google.com/intl/en_us/pricing.html [1] https://support.google.com/a/users/search?q=Drive%20limits https://support.google.com/a/users/search?q=Drive%20limits
- ilyt 4y ago
  To add to the point: 5M files on 5TB storage would average 1MB per file So just storing 5TB of average web sized images would hit the limit, let alone smaller documents.
- scrollaway 4y ago
  > at no point did they communicate that they were imposing this limit, and that this limit appearead, undocumented, overnight. Not to defend google but I've seen plenty of engineers make such mistakes, and you probably have as well; it's just that it didn't then result in bad press. When you are an engineer working on a living product, and you identify some performance-related issue, changes you make to the product can easily be classified as bugfixes. For example, you identified an end point that should have a rate limit and didn't; you fixed it, it was a potential security issue, it didn't need communication to end users... as far as you knew, even if you misjudged. Strong XKCD1172 vibes here. https://xkcd.com/1172/ https://xkcd.com/1172/
  - 4y ago

soundsgoodtome 4y ago

Is PM not consulted for changes like this? I feel like this is something that a competent PM would ask eng to pause on while they determined actual user impact and lines up messaging.

martius 4y ago
Short answer, yes, a PM should have been consulted and give their approval for the change. In the scenario I described above (again, it's a guess, I don't know what actually happened), it's possible that the PM was bypassed because the engineers for a reason they thought was good. For example: * they didn't even think about involving a PM because that's something you have never done, * the people who wrote/reviewed the change assumed the conversation already happened, * they were pressured to move fast to mitigate an imminent or existing problem (performance, scalability, ...), * the PM who should have made this call "left" the company and didn't (get a chance to) hand-off their responsibilities. I guess what I mean with these comments is that sometimes there are misses like this. Maybe it's a sign of a systematic failure and that internal processes should be improved, but I don't think it means that the company is fundamentally unable to handle these changes correctly or can't make the right technical or product decisions.

maccard 4y ago

> I think this is a easy mistake to do even with very good intentions, and I can see myself doing it. Hard disagree. You missed one very important part in your writeup - at no point did they communicate that they were imposing this limit, and that this limit appearead, undocumented, overnight. I was someone who was directly impacted by this change. We're a 40 person company who used (past tense) GDrive as a shared network drive, including for storing builds of our app. We pay $18/person, and as part of that, google workspace advertises 5TB per user pooled[0], and nowhere in the google docs does it mention that this limit will exist [1]. If I was aware of a limit, we would have cleaned up our old files, but instead we started getting spurious 403's - as far as we could tell we were well within our usage limits. It was only when https://issuetracker.google.com/issues/268606830?pli=1 https://issuetracker.google.com/issues/268606830?pli=1 this post hit HN, I realised what was wrong. [0] https://workspace.google.com/intl/en_us/pricing.html https://workspace.google.com/intl/en_us/pricing.html [1] https://support.google.com/a/users/search?q=Drive%20limits https://support.google.com/a/users/search?q=Drive%20limits

ilyt 4y ago
To add to the point: 5M files on 5TB storage would average 1MB per file So just storing 5TB of average web sized images would hit the limit, let alone smaller documents.
scrollaway 4y ago
> at no point did they communicate that they were imposing this limit, and that this limit appearead, undocumented, overnight. Not to defend google but I've seen plenty of engineers make such mistakes, and you probably have as well; it's just that it didn't then result in bad press. When you are an engineer working on a living product, and you identify some performance-related issue, changes you make to the product can easily be classified as bugfixes. For example, you identified an end point that should have a rate limit and didn't; you fixed it, it was a potential security issue, it didn't need communication to end users... as far as you knew, even if you misjudged. Strong XKCD1172 vibes here. https://xkcd.com/1172/ https://xkcd.com/1172/
- 4y ago