5 ms·

Previously worked at Google in infrastructure but opinions are my own. I am always frustrated/embarrassed at Google’s messaging of these things, but it seems t

Previously worked at Google in infrastructure but opinions are my own. I am always frustrated/embarrassed at Google’s messaging of these things, but it seems to me like this tweet is trying to stir the pot for attention. The Pixel phone with unlimited uploads marketed at consumers is one thing, but this unlimited plan for universities assuredly contained terms that didn’t guarantee it forever and a large institution should be aware of that. I also don’t think unlimited is a synonym for eternally unlimited. It also seems like it would be a bad idea to store frequently accessed experimental data in Google Drive and based on follow up tweets the professor agrees and doesn’t use Google Drive for this… so it seems like they are mad about a hypothetical. I don’t think Google is always the most thoughtful with things like this but this seems like something to take up with their IT department and not a second Google Reader incident.

trunnell 5y ago
Why do you say it'd be a bad idea to store frequently accessed experimental data in Google Drive?
- g_p 5y ago
  Indeed - given much of Google drive marketing seems to be trying to encourage it's use for routine frequently accessed data, this is an interesting question. Much of the marketing for drive seems to still be (in sentiment at least) about dumping it all into Google drive and never having to worry about storage again. I assume they mean storing huge files in Google drive, but actually that's probably a significant use case for many users of cloud storage.
- tuckerman 5y ago
  I think the biggest issue is provenance and access control/logging—Drive doesn’t have a lot of great tools for that. Backups as well. That said, I’m struggling to imagine how you could even efficiently process on the order of 1 PiB of data from Drive. Having that in a data warehouse or some object storage close to your compute seems required to make any good use of it.
  - agentdrtran 5y ago
    Logging in Drive is pretty robust (backups are not). There's a few tools that can move that level of data, but google's now-first-party migrate product could likely handle it.
    - tuckerman 5y ago
      That logging will give you who/when but won’t have things like field level access logging or information about the analysis pipeline itself. With regards to data locality, I was more thinking about running a map reduce or similar ETL pipeline.
orware 5y ago
Just coming out of a higher education position I held for about 10 years...we were really only using GSuite for Education on the periphery of everything else we were doing (which was all Microsoft-based). So we were really only setting up the needed GSuite integrations to give some access for students/faculty/staff to use the GSuite apps if they wished, so the "unlimited" storage side of things was an extra bonus. While our actual students/staff have not abused the unlimited-ness at all, there's been a big trend in recent years for fake spam/bot type accounts to submit online applications (particularly for the inexpensive community college level here in California) which would allow for those fake people to get issued accounts within the college systems (typically giving them a student email account). In essence though, one bad account could in turn create multiple shared drives it seems, and then from there grant access to other external Gmail accounts and swap the ownership to them, while still leaving the unlimited storage capabilities in place (at least that's been my perception of what I had seen in our instance). Before I left recently, I had seen pages upon pages of shared drives in our GSuite Admin area of Shared Drives that were terabytes large (one as high as 100 TB or some ridiculously high number like that)...with very little mitigation options being provided by Google to help avoid that from occurring in the first place (aside from taking the extreme measure of locking things down after the fact...but cleaning up that existing mess seemed to not have a solution...supposedly there were going to be some new tools released early this year to help out, but not sure if those are available yet). Suffice to say...some bad apples are making it hard for all of the good people to take advantage of what was otherwise a wonderful offering for educational institutions (having administered many systems over the years...it's always nicer to not have to worry about license limits and other baloney like that since it simplifies things for what is usually an already overloaded IT staff). I could clearly see all of the bad accounts abusing our instance...if only there was an easy way to filter all accounts > 2 TB and then delete them I could have reduced our storage usage down considerably for Google. That multiplied by hundreds of institutions where this may have occurred during the pandemic and I can see how storage usage could have went up dramatically on Google's end...but I think that Admins in those institutions would be more than happy to clean things up if they had the tools/ability to do so.
jsiepkes 5y ago
> but this unlimited plan for universities assuredly contained terms that didn’t guarantee it forever Interested to hear on which products Google makes any promises regarding lifetime (ie. guaranteeing at least 10 years availability for example).

trunnell 5y ago

Why do you say it'd be a bad idea to store frequently accessed experimental data in Google Drive?

g_p 5y ago
Indeed - given much of Google drive marketing seems to be trying to encourage it's use for routine frequently accessed data, this is an interesting question. Much of the marketing for drive seems to still be (in sentiment at least) about dumping it all into Google drive and never having to worry about storage again. I assume they mean storing huge files in Google drive, but actually that's probably a significant use case for many users of cloud storage.
tuckerman 5y ago
I think the biggest issue is provenance and access control/logging—Drive doesn’t have a lot of great tools for that. Backups as well. That said, I’m struggling to imagine how you could even efficiently process on the order of 1 PiB of data from Drive. Having that in a data warehouse or some object storage close to your compute seems required to make any good use of it.
- agentdrtran 5y ago
  Logging in Drive is pretty robust (backups are not). There's a few tools that can move that level of data, but google's now-first-party migrate product could likely handle it.
  - tuckerman 5y ago
    That logging will give you who/when but won’t have things like field level access logging or information about the analysis pipeline itself. With regards to data locality, I was more thinking about running a map reduce or similar ETL pipeline.

orware 5y ago

Just coming out of a higher education position I held for about 10 years...we were really only using GSuite for Education on the periphery of everything else we were doing (which was all Microsoft-based). So we were really only setting up the needed GSuite integrations to give some access for students/faculty/staff to use the GSuite apps if they wished, so the "unlimited" storage side of things was an extra bonus. While our actual students/staff have not abused the unlimited-ness at all, there's been a big trend in recent years for fake spam/bot type accounts to submit online applications (particularly for the inexpensive community college level here in California) which would allow for those fake people to get issued accounts within the college systems (typically giving them a student email account). In essence though, one bad account could in turn create multiple shared drives it seems, and then from there grant access to other external Gmail accounts and swap the ownership to them, while still leaving the unlimited storage capabilities in place (at least that's been my perception of what I had seen in our instance). Before I left recently, I had seen pages upon pages of shared drives in our GSuite Admin area of Shared Drives that were terabytes large (one as high as 100 TB or some ridiculously high number like that)...with very little mitigation options being provided by Google to help avoid that from occurring in the first place (aside from taking the extreme measure of locking things down after the fact...but cleaning up that existing mess seemed to not have a solution...supposedly there were going to be some new tools released early this year to help out, but not sure if those are available yet). Suffice to say...some bad apples are making it hard for all of the good people to take advantage of what was otherwise a wonderful offering for educational institutions (having administered many systems over the years...it's always nicer to not have to worry about license limits and other baloney like that since it simplifies things for what is usually an already overloaded IT staff). I could clearly see all of the bad accounts abusing our instance...if only there was an easy way to filter all accounts > 2 TB and then delete them I could have reduced our storage usage down considerably for Google. That multiplied by hundreds of institutions where this may have occurred during the pandemic and I can see how storage usage could have went up dramatically on Google's end...but I think that Admins in those institutions would be more than happy to clean things up if they had the tools/ability to do so.

jsiepkes 5y ago

> but this unlimited plan for universities assuredly contained terms that didn’t guarantee it forever Interested to hear on which products Google makes any promises regarding lifetime (ie. guaranteeing at least 10 years availability for example).