4 ms·

On the Microsoft page[1] about this change they say > Storage of this “dark” data takes up space on servers and results in increased electricity consumption, g

by imgabe 3y ago

On the Microsoft page[1] about this change they say > Storage of this “dark” data takes up space on servers and results in increased electricity consumption, generating 4% of global greenhouse gas emissions in 2020 alone. Where "dark" data is data that is unused. However they link to this World Economic Forum Page [2] which says > In 2020, digitisation was purported to generate 4% of global greenhouse gas emissions. So the 4% figure is not the storage of only dark data, but the storage of ALL data. The WEF page says that half of all data is unused, but how much of the energy usage is due to the unused data? I would think the bulk of the energy would be used when the data is accessed, and data sitting on a drive not being accessed isn't really using any energy. This is really misleading. [1] https://www.microsoft.com/en-us/education/products/microsoft-365-storage-options https://www.microsoft.com/en-us/education/products/microsoft... [2] https://www.weforum.org/agenda/2022/10/dark-data-is-killing-the-planet-we-need-digital-decarbonisation/ https://www.weforum.org/agenda/2022/10/dark-data-is-killing-...

esskay 3y ago
I wonder why they've not instead pushed people towards a 'real' cold storage service like AWS Glacier. IIRC that uses a mixture of DVD/Bluray automation and for deep storage uses LTO tape drives. Solves most of the energy usage issues.
- philistine 3y ago
  Do you have sources on that? Discussions here in the past have indicated that Glacier uses unpowered drives, not tape or disks.
- deleted 3y ago
  [deleted]
akira2501 3y ago
_Purported_. By, this source[0]. Which doesn't cite the source and seems to impute "blockchain" into this total somehow, and was focused on 2020, a year in which corona virus changed a lot of digital trends temporarily. To the extent that the changes become permanent, they probably lead, working from home, to a far greater reduction in useless greenhouse emissions than the data ever could. It's not only misleading it's just detached from reality. [0] https://www.ey.com/en_ch/decarbonization/how-digitization-acts-as-a-driver-of-decarbonization https://www.ey.com/en_ch/decarbonization/how-digitization-ac...
bjornsing 3y ago
> So the 4% figure is not the storage of only dark data, but the storage of ALL data. Storage or ALL processing? SSDs typically need very little power to just store data. They need some power to read and write. But the lion’s share of energy use for “digitization” is compute in various forms.
- bayindirh 3y ago
  Disclaimer: Looking with my HPC admin hat. To be able to store that amount of data on SSDs, you need tons of enterprise mixed load SSDs, and they don't come cheap in any sense or form. To offset costs, you add tiering with mechanical disks, with varying capacities and readiness states. Spinning up cold storage takes time and energy, and if you need a three year old CSV file summarizing your results, that's a lot of energy and time. The storage pipeline storing rest of your research might go to "sleep" again until you make sure that you gonna need rest of the dataset and everything to re-run some of the experiments. The data access pattern maybe hitting some cold paths too often as a result, so Microsoft might be trying to confine research institutions to a warmer tier, and since that tiers are more expensive to operate, they might be slashing storage space. Otherwise the resident data might poison other user's access patterns. This is not to say Microsoft is doing right, I believe the opposite, but this is why they might be doing it.
robin_reala 3y ago
I believe that emissions figure includes the power consumed during the manufacturing and shipping of the storage devices too.
robertlagrant 3y ago
> > In 2020, digitisation was purported to generate 4% of global greenhouse gas emissions. > So the 4% figure is not the storage of only dark data, but the storage of ALL data. Even that sounds like a misread. "Digitisation" can refer to the sum total of moving a product or service or process to be digitised; that would also include the compute cost as well as the storage.
- bluescrn 3y ago
  And how does that 4% compare to previous/alternative methods of data acquisition, storage, and transportation? Producing, storing, and shipping masses of paper documents (and related printing equipment) isn't particularly efficient. And video/audio would need storing on physical disks/tapes, and so on.
tallanvor 3y ago
Isn't it more misleading to claim that this is just due to environmental reasons? The first link you provide actually says the first reason is due to security (you wouldn't believe how often student accounts are compromised and/or used for sharing pirated content). The second reason is the environment, and the 3rd, reason, even though they don't say it outright, is cost, and I'm guessing that's the real reason. But giving institutions 100TB of base storage with more for paid licenses isn't exactly stingy.
- michaelt 3y ago
  > The first link you provide actually says the first reason is due to security To me personally the security claims don't make much sense either. When I face disk space limitations I don't delete my spreadsheet of survey results that's full of private information. I just delete the original video of that public lecture that I've put onto youtube already anyway. Now, for professors there might be a case for shunting the files they haven't touched in a few years into secure offline storage in case it contains forgotten private information. But it sounds like this applies to student accounts, and there's already a process to eliminate forgotten years-old data: Graduation. IMHO it's far more likely this is a cost-saving measure.
- aragilar 3y ago
  Sorry, what? 100TB is stingy as! At my institution, there's approximately 50000 students and staff, if 100TB is all they get, that's only 2GB each (and that's not just for email, that's for all storage)!
  - tallanvor 3y ago
    Without knowing how much they're paying, how can we say if that's stingy or not? If a school gets 100TB storage for free, I'd hardly call that stingy.
    - iinnPP 3y ago
      It's not free. It requires students to sign a bunch of ToS "agreements" that are terrible. It locks students in to learning on their software or using their overpriced cloud offerings.
proactivesvcs 3y ago
From what I understand, some of the academic data must be stored for some years after publishing. This isn't dark data. It must be kept, so it is being used.

by imgabe 3y ago

esskay 3y ago

I wonder why they've not instead pushed people towards a 'real' cold storage service like AWS Glacier. IIRC that uses a mixture of DVD/Bluray automation and for deep storage uses LTO tape drives. Solves most of the energy usage issues.

philistine 3y ago
Do you have sources on that? Discussions here in the past have indicated that Glacier uses unpowered drives, not tape or disks.
deleted 3y ago
[deleted]

akira2501 3y ago

_Purported_. By, this source[0]. Which doesn't cite the source and seems to impute "blockchain" into this total somehow, and was focused on 2020, a year in which corona virus changed a lot of digital trends temporarily. To the extent that the changes become permanent, they probably lead, working from home, to a far greater reduction in useless greenhouse emissions than the data ever could. It's not only misleading it's just detached from reality. [0] https://www.ey.com/en_ch/decarbonization/how-digitization-acts-as-a-driver-of-decarbonization https://www.ey.com/en_ch/decarbonization/how-digitization-ac...

bjornsing 3y ago

> So the 4% figure is not the storage of only dark data, but the storage of ALL data. Storage or ALL processing? SSDs typically need very little power to just store data. They need some power to read and write. But the lion’s share of energy use for “digitization” is compute in various forms.

bayindirh 3y ago
Disclaimer: Looking with my HPC admin hat. To be able to store that amount of data on SSDs, you need tons of enterprise mixed load SSDs, and they don't come cheap in any sense or form. To offset costs, you add tiering with mechanical disks, with varying capacities and readiness states. Spinning up cold storage takes time and energy, and if you need a three year old CSV file summarizing your results, that's a lot of energy and time. The storage pipeline storing rest of your research might go to "sleep" again until you make sure that you gonna need rest of the dataset and everything to re-run some of the experiments. The data access pattern maybe hitting some cold paths too often as a result, so Microsoft might be trying to confine research institutions to a warmer tier, and since that tiers are more expensive to operate, they might be slashing storage space. Otherwise the resident data might poison other user's access patterns. This is not to say Microsoft is doing right, I believe the opposite, but this is why they might be doing it.

robin_reala 3y ago

I believe that emissions figure includes the power consumed during the manufacturing and shipping of the storage devices too.

robertlagrant 3y ago

> > In 2020, digitisation was purported to generate 4% of global greenhouse gas emissions. > So the 4% figure is not the storage of only dark data, but the storage of ALL data. Even that sounds like a misread. "Digitisation" can refer to the sum total of moving a product or service or process to be digitised; that would also include the compute cost as well as the storage.

bluescrn 3y ago
And how does that 4% compare to previous/alternative methods of data acquisition, storage, and transportation? Producing, storing, and shipping masses of paper documents (and related printing equipment) isn't particularly efficient. And video/audio would need storing on physical disks/tapes, and so on.

tallanvor 3y ago

Isn't it more misleading to claim that this is just due to environmental reasons? The first link you provide actually says the first reason is due to security (you wouldn't believe how often student accounts are compromised and/or used for sharing pirated content). The second reason is the environment, and the 3rd, reason, even though they don't say it outright, is cost, and I'm guessing that's the real reason. But giving institutions 100TB of base storage with more for paid licenses isn't exactly stingy.

michaelt 3y ago
> The first link you provide actually says the first reason is due to security To me personally the security claims don't make much sense either. When I face disk space limitations I don't delete my spreadsheet of survey results that's full of private information. I just delete the original video of that public lecture that I've put onto youtube already anyway. Now, for professors there might be a case for shunting the files they haven't touched in a few years into secure offline storage in case it contains forgotten private information. But it sounds like this applies to student accounts, and there's already a process to eliminate forgotten years-old data: Graduation. IMHO it's far more likely this is a cost-saving measure.
aragilar 3y ago
Sorry, what? 100TB is stingy as! At my institution, there's approximately 50000 students and staff, if 100TB is all they get, that's only 2GB each (and that's not just for email, that's for all storage)!
- tallanvor 3y ago
  Without knowing how much they're paying, how can we say if that's stingy or not? If a school gets 100TB storage for free, I'd hardly call that stingy.
  - iinnPP 3y ago
    It's not free. It requires students to sign a bunch of ToS "agreements" that are terrible. It locks students in to learning on their software or using their overpriced cloud offerings.

proactivesvcs 3y ago

From what I understand, some of the academic data must be stored for some years after publishing. This isn't dark data. It must be kept, so it is being used.