4 ms·

This is exactly the problem. Consider an order processing system that processes a million orders a day, and retains them for ninety days. What percentage of tho

This is exactly the problem. Consider an order processing system that processes a million orders a day, and retains them for ninety days. What percentage of those 89-day-old orders are actually needed at 89 days old? It could be quite low, maybe a couple thousand out of a million. But if those orders aren't there, shit hits the fan. PCI compliance audits fail. The ability for customers to reconcile their charges with their purchases breaks. In that 0.01% of cases where the order was fraudulent, placed by mistake, or just didn't have what the customer thought it had in it, not having that data makes the order processor read as, if not malicious, at least incompetent. The real question is, how much data do we need to store inefficiently, in a way that uses a lot of power and space?

AStonesThrow 2y ago
I was just pondering this today, in terms of how much data and objects I create in a Google or Microsoft account, for example, and then they create a burden of cost and maintenance for me down the road. Especially deleting old emails and photos. That's arduous and sometimes poignant as I flush away my personal life story. Cloud services make difficult and sometimes even byzantine processes for deleting stuff, and it's often impossible to operate en masse in order to clean up swaths of stuff quickly and efficiently. It's in their interest to retain everything at all costs, because more used storage can mean more profits. Cloud services also profit from unused storage, because if they're charging $20/year to 100,000 users who use 2% of their storage space, ka-ching! It irks me to this day that standard or even advanced filesystems don't include "expiration dates" or "purge dates" on file objects. Wouldn't it be logical, if an organization has a "data-rentention policy" that mandates destruction after X date, that the filesystem simply purges it automatically? Why does this always get delegated to handmade userland cron jobs? Moreover, to my knowledge, nobody is really interested in devising a way to comb through backup media in order to [selectively] destroy data that shouldn't exist anymore. Not even the read-write media! Google is now auto-deleting stuff like OTP SMS messages. I'd love it if auto-delete could be configurable account-wide, for more than just our web histories and Maps Timeline and stuff. Unfortunately, to "delete" cloud data means it still exists on backups anyway. But without deleting data in your cloud account, it becomes a juicier hacker target as it ages and accumulates personal stuff that shouldn't fall into the wrong hands. Likewise for any business, it behooves them to delete and destroy data that shouldn't be stolen. At least move it offline so that only physical access can restore it? I will say that modern encryption techniques can make it easy to "destroy" data, simply by destroying the encryption keys. You can quickly render entire SSDs unreadable in the firmware itself with such a command. Bonus: sometimes it's even done on purpose! But even deleting data presents a maintenance cost. So if 90% of an org's data is indeed crap, then 90% or more of your processing resources are going to be wasted on sifting through it at some later date. Imagine when your file formats and storage devices are obsolete, and some grunt needs to retrieve some record that's 30 years old, and 90% of your data was always crap. That grunt is hopefully paid by the hour. We really had this happen at a few of my jobs, where we had old reel-to-reel backup tapes and it was difficult enough to load the data into a modern SunOS machine. https://m.xkcd.com/1683/ https://m.xkcd.com/1683/
- nn3 2y ago
  I suspect for most cloud providers you deleting data is cheaper because the data is not charged by the byte. But then they like having data, maybe just to train their AI models or for bragging rights to their investors. For the expiration dates most modern file systems have the concept of arbitrary extended attributes per file. It's quite easy to add meta data like this yourself.
  - AStonesThrow 2y ago
    Unused services are always pure profit, and storage space is no exception. Providers can offer 100GB for like $24/year, because only like 2% of the subscribers will ever approach the limit, and so the extra space is never wasted but can be allocated to someone or something else. It's like gym memberships, ISP/telco service bundles, and amenities at your apartment complex. Anyone not using every possible service is wasting money, but it's impossible to purchase a bespoke service, so essentially everyone will waste money because they're chipping in money for services that someone else uses more than they do. Here at home I don't ever use the gym, the racquetball courts, the doggy-doo supplies, the laundry room, or a parking space, and yet my rent (everyone's rent) includes upkeep for all of those things. I'm subsidizing all my neighbors and all the wear-and-tear damage they put on those common amenities. Likewise, everyone who's paying $24/year for storage, or any business that purchases big multi-terabyte storage media, they're paying for unused storage space and giving profit. It's practically impossible to rightsize your storage media, and you never want it undersized, and you can't simply shrink them and reclaim the resources you invested, you just keep adding on new ones and replacing the malfunctioning ones. So nearly everyone always owns or rents more space than they can realistically utilize. Furthermore, you'll notice that I specified "automatic" destruction of data by expiration date. Of course it is trivial to tag any arbitrary file with arbitrary metadata, but the challenge is to create a filesystem that executes automatic data purges on schedule, rather than pushing it into a rickety old handmade cronjob in userland. I've never ever seen a filesystem with such a feature, nor does it seem that anyone's interested in doing so. And here I thought that computers were useful for automating business logic and easing the burden on human effort. And this is me, manually sifting through emails and photos in order to manually delete each one with 3 dialog boxes intervening. It takes hours, days, weeks.
makeitdouble 2y ago
> The real question is, how much data do we need to store inefficiently, in a way that uses a lot of power and space? This is indeed the critical question, and it's far from being trivial. One issue we all hit is moving the data from the higher tier storage to the cheaper and more efficient one, which requires sync and paying for the transfer most of the time, but also handling two separate access and authorization process, backup and recovery system for data that absolutely needs to be accessible for the few years of legal retention, and can/must completely disappear afterwards. In most orgs I've seen the cost of going through all that complexity is just not worth it, compared to "just" paying for the higher tier storage for the few years long lifetime of the data.
- newAccount2025 2y ago
  Seems like we should be solving this at the storage system layer, instead of per application?
numpad0 2y ago
probably a similar percentage of people claim on their car insurance. In that 0.01% of cases where the order was fraudulent, placed by mistake, how much data do we need to store inefficiently, in a way that uses a lot of power and space? I'm feeling, the real real question, as Sci-Fi as it gets, is, is the winning ticket data even data, OR is it more like a thumbnail of "the whole data" that is 98%+ worthless, than standalone piece of data? The winning ticket ID, e.g. a "0x-636e-7461-4e7b", only makes sense in the context as one among the entire cohort of contestants; I can make one up like I did, but I can't walkout with the payout unless the rest of the lottery didn't exist. Statistically, philosophically, technically, and all sorts of *-cally speaking, is the 2% data, the winning ticket datum, even data?
deleted 2y ago
[deleted]

AStonesThrow 2y ago

I was just pondering this today, in terms of how much data and objects I create in a Google or Microsoft account, for example, and then they create a burden of cost and maintenance for me down the road. Especially deleting old emails and photos. That's arduous and sometimes poignant as I flush away my personal life story. Cloud services make difficult and sometimes even byzantine processes for deleting stuff, and it's often impossible to operate en masse in order to clean up swaths of stuff quickly and efficiently. It's in their interest to retain everything at all costs, because more used storage can mean more profits. Cloud services also profit from unused storage, because if they're charging $20/year to 100,000 users who use 2% of their storage space, ka-ching! It irks me to this day that standard or even advanced filesystems don't include "expiration dates" or "purge dates" on file objects. Wouldn't it be logical, if an organization has a "data-rentention policy" that mandates destruction after X date, that the filesystem simply purges it automatically? Why does this always get delegated to handmade userland cron jobs? Moreover, to my knowledge, nobody is really interested in devising a way to comb through backup media in order to [selectively] destroy data that shouldn't exist anymore. Not even the read-write media! Google is now auto-deleting stuff like OTP SMS messages. I'd love it if auto-delete could be configurable account-wide, for more than just our web histories and Maps Timeline and stuff. Unfortunately, to "delete" cloud data means it still exists on backups anyway. But without deleting data in your cloud account, it becomes a juicier hacker target as it ages and accumulates personal stuff that shouldn't fall into the wrong hands. Likewise for any business, it behooves them to delete and destroy data that shouldn't be stolen. At least move it offline so that only physical access can restore it? I will say that modern encryption techniques can make it easy to "destroy" data, simply by destroying the encryption keys. You can quickly render entire SSDs unreadable in the firmware itself with such a command. Bonus: sometimes it's even done on purpose! But even deleting data presents a maintenance cost. So if 90% of an org's data is indeed crap, then 90% or more of your processing resources are going to be wasted on sifting through it at some later date. Imagine when your file formats and storage devices are obsolete, and some grunt needs to retrieve some record that's 30 years old, and 90% of your data was always crap. That grunt is hopefully paid by the hour. We really had this happen at a few of my jobs, where we had old reel-to-reel backup tapes and it was difficult enough to load the data into a modern SunOS machine. https://m.xkcd.com/1683/ https://m.xkcd.com/1683/

nn3 2y ago
I suspect for most cloud providers you deleting data is cheaper because the data is not charged by the byte. But then they like having data, maybe just to train their AI models or for bragging rights to their investors. For the expiration dates most modern file systems have the concept of arbitrary extended attributes per file. It's quite easy to add meta data like this yourself.
- AStonesThrow 2y ago
  Unused services are always pure profit, and storage space is no exception. Providers can offer 100GB for like $24/year, because only like 2% of the subscribers will ever approach the limit, and so the extra space is never wasted but can be allocated to someone or something else. It's like gym memberships, ISP/telco service bundles, and amenities at your apartment complex. Anyone not using every possible service is wasting money, but it's impossible to purchase a bespoke service, so essentially everyone will waste money because they're chipping in money for services that someone else uses more than they do. Here at home I don't ever use the gym, the racquetball courts, the doggy-doo supplies, the laundry room, or a parking space, and yet my rent (everyone's rent) includes upkeep for all of those things. I'm subsidizing all my neighbors and all the wear-and-tear damage they put on those common amenities. Likewise, everyone who's paying $24/year for storage, or any business that purchases big multi-terabyte storage media, they're paying for unused storage space and giving profit. It's practically impossible to rightsize your storage media, and you never want it undersized, and you can't simply shrink them and reclaim the resources you invested, you just keep adding on new ones and replacing the malfunctioning ones. So nearly everyone always owns or rents more space than they can realistically utilize. Furthermore, you'll notice that I specified "automatic" destruction of data by expiration date. Of course it is trivial to tag any arbitrary file with arbitrary metadata, but the challenge is to create a filesystem that executes automatic data purges on schedule, rather than pushing it into a rickety old handmade cronjob in userland. I've never ever seen a filesystem with such a feature, nor does it seem that anyone's interested in doing so. And here I thought that computers were useful for automating business logic and easing the burden on human effort. And this is me, manually sifting through emails and photos in order to manually delete each one with 3 dialog boxes intervening. It takes hours, days, weeks.

makeitdouble 2y ago

> The real question is, how much data do we need to store inefficiently, in a way that uses a lot of power and space? This is indeed the critical question, and it's far from being trivial. One issue we all hit is moving the data from the higher tier storage to the cheaper and more efficient one, which requires sync and paying for the transfer most of the time, but also handling two separate access and authorization process, backup and recovery system for data that absolutely needs to be accessible for the few years of legal retention, and can/must completely disappear afterwards. In most orgs I've seen the cost of going through all that complexity is just not worth it, compared to "just" paying for the higher tier storage for the few years long lifetime of the data.

newAccount2025 2y ago
Seems like we should be solving this at the storage system layer, instead of per application?

numpad0 2y ago

probably a similar percentage of people claim on their car insurance. In that 0.01% of cases where the order was fraudulent, placed by mistake, how much data do we need to store inefficiently, in a way that uses a lot of power and space? I'm feeling, the real real question, as Sci-Fi as it gets, is, is the winning ticket data even data, OR is it more like a thumbnail of "the whole data" that is 98%+ worthless, than standalone piece of data? The winning ticket ID, e.g. a "0x-636e-7461-4e7b", only makes sense in the context as one among the entire cohort of contestants; I can make one up like I did, but I can't walkout with the payout unless the rest of the lottery didn't exist. Statistically, philosophically, technically, and all sorts of *-cally speaking, is the 2% data, the winning ticket datum, even data?

deleted 2y ago

[deleted]