9 ms·

Data centers contain 90% crap data

227 points by billybuckwheat 2y ago

floppiplopp 2y ago
That's a pretty optimistic estimate, tbh.
boznz 2y ago
Don't forget emails.. I have everything I ever sent or received, and I have it backed up. I expect 90% of my inbox is the jpg signature logo they attach to the bottom of my clients email rather than hyperlink.
- hobs 2y ago
  That's not even crap data - that's archived data that might be useful someday (though de-dupe is probably a great idea and email sigs are definitely wasteful trash) - most of the crap data is things that would never have been useful under any circumstances. I have cleaned up dozens of product databases in cost management efforts and have found anywhere from 50-99% of data stored in product databases is crap, because they are not well managed and any single mistake can lead to a huge outsized impact on storage. What to log all those http requests for just a day? Might as well turn that on for all time...
- duxup 2y ago
  I ended up working on some software and I was deemed the email guy (it's a very small % of my job but it is the biggest pain). "I need an email when this happens.. and when this happens." The requests are endless and I'm convinced there are people who if they could would do their entire job from their inbox and get everything and anything an application can do via email. The insidious problem is that it never solves anything. "I didn't get the email!" is a constant refrain (yes they did, they always did). "Oh someone didn't do the thing so can you add this to the email too." and so on. It is such an abused tool.
  - Dylan16807 2y ago
    > The requests are endless and I'm convinced there are people who if they could would do their entire job from their inbox and get everything and anything an application can do via email. That sounds like a reasonable goal for a whole lot of job duties. And yes some entire office jobs. (Excluding some direct human communication but a lot of jobs already have too much of that in situations that could have been an email.) > "I didn't get the email!" is a constant refrain (yes they did, they always did). Well having to manually check wouldn't improve that, would it?
  - Cthulhu_ 2y ago
    I can see how that would work (email centric workflow), not unlike how some people now try to have a chat / Slack centric workflow.
    - saalweachter 2y ago
      Yeah, everyone knows the proper thing to do is orchestrate it all through Emacs.
  - mrweasel 2y ago
    We had a similar request for a sales team once: "If this fails we want an email".... Okay, but we never seen this fail, so you'll never get the email. As expected, they never got an email, because no failures. So instead they wanted an email every time the job ran (once every night), so they'd know that the job had failed, if they DIDN'T get the email. Only time that happened was because the email got trapped in a spamfilter I think. There must be thousands of copies of that email sitting in inboxes say: Job X ran successfully @ 04:30.
- jeffbee 2y ago
  Well "the cloud" will generally store exactly one logical copy of your static jpg, one of the reasons why clouds are pretty good for efficiency. There is really no sense at all in the article's claims that "we are destroying the environment" to do x y and z thing the author whines about. We are destroying the environment to drive a Dodge Ram to the Circle K to buy a 52oz Polar Pop. Information sector doesn't even show up on top ten lists of things that are destroying the environment.
- HPsquared 2y ago
  It's probably deduplicated on the server though, so the millions and millions of messages with that logo likely share the same piece of disk space. Probably one reason why free providers don't tend to offer End-to-End encryption. It prevents deduplication (and otherwise compressing redundant information).
  - justsomehnguy 2y ago
    Nope. Back in the day Exchange offered SIS but in 2010 they ditched it. It's plainly not effective any more. Even regarding the OP' "the jpg signature logo" - it's a part of multipart in the message, not a separate file. And one more thing - you can't just turn the dedup and be dandy, now you need to check against the hashes to determine if this chunk is unique or you already have it. And with TBs of data you need TBs of hashes to check. Until you have like 99% dedup efficiency, ie 99% of your incoming data is literally the same data you already have - it doesn't worth it. https://techcommunity.microsoft.com/blog/exchange/dude-wheres-my-single-instance/595267 https://techcommunity.microsoft.com/blog/exchange/dude-where...
    - Sprite_tm 2y ago
      Not sure about Microsoft solutions, but modern file systems like zfs and btrfs can do this on the filesystem level, no support from the mail server needed.
      - tored 2y ago
        NTFS and ReFS has both deduplication support (only on Windows Server).
    - LegionMammal978 2y ago
      Also, I can imagine such schemes getting ditched for the same reasons of "now people can detect whether we already have a copy of the file, which is a privacy hole!" that killed cross-origin caching in browsers. Or if everything in the year 20XX gets pushed into using E2E encryption, since that's pretty much antithetical to deduplication.
ctoth 2y ago
Surely a few more nines if the base rate is already Sturgeon.
matt-p 2y ago
Must be more, surely.
cactusplant7374 2y ago
Fortunately with LLM's we can generate data and not store it. If you think about it that means articles could always be relevant with additional details added as time goes on.
- tylerrobinson 2y ago
  No, the reality of this situation is that people need to save a snapshot of what the LLM said when they read it, just in case they need to substantiate it or blame someone for what they read at the time.
  - cactusplant7374 2y ago
    No, it requires a shift in thinking. There are no guarantees with information one reads as it is.
- irjustin 2y ago
  At the moment, I don't like this for fear of revisionist history. The equvilent would be if wikipedia didn't have to have a source reference. not saying that it's infallible but yeah we need some chain of audible at least 1 or 2 layers deep.
danpalmer 2y ago
> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is stored because we don't know what we will need later. Photos of kids? Maybe that one will be The One that we end up framing? Miscellaneous business records? Maybe those will be the ones we have to dig out for a tax audit? Web pages on government sites? Maybe there will suddenly be an interest in obscure pages on public health policy if a global pandemic happens. Complaining that data is mostly junk is not a particularly interesting conclusion without acknowledging this. Is there wastage? Yeah sure, but accuracy on what needs storing is directly traded off with time spent figuring that out, and often it's cheaper to store the data.
- mrexroad 2y ago
  I have ~2.5TB of photos in iCloud, via Apple Photos. Excluding the various sized previews, I doubt many originals have been accessed in quite a while. I also have about 1TB Lightroom library archived to a different service; representing countless hours of photo processing work spanning over a decade. Haven’t touched that one in years. Neither are crap. (Yes, both have other backups and, yes, I’ve probably forgotten to sync one of them).
  - snapplebobapple 2y ago
    Crap is in the eye of the beholder...
  - eloisius 2y ago
    Sometimes I wish there was a feasible system to audit and reduce redundant photos in my iCloud. I have an embarassing ton of pictures that I never look at just like everyone else. I wouldn’t mind a tool that said “here’s 10^x photos that are the most similar to all your other photos would you like to put them in the trash?”
somat 2y ago
Isn't this just a specific case of sturgeons law? https://en.wikipedia.org/wiki/Sturgeon%27s_law https://en.wikipedia.org/wiki/Sturgeon%27s_law
- BrenBarn 2y ago
  Yeah it seems like Sturgeon's law combined with a bit of Pareto principle.
- foobahify 2y ago
  [flagged]
RobotToaster 2y ago
I imagine there's a similar statistic for book shops and public libraries.
nameless912 2y ago
There's another dimension to this, that storage is so cheap that being wasteful with it isn't really disincentivized. I know for example at work of a portal that accepts uploads of large files from external clients that stores both the initial upload and every subsequent transformation of the file (of which there are 4-6) permanently. It's extremely useful for debugging, as one of the bits of metadata we shove on the zip archive is the git hash of the code that was running, so it's trivial to pull down any failed step and diagnose what happened. We are using 4-6 times as much storage as we need to, and these are often not small files (on the order of 100 MB - 5 GB, several dozen times a day) but fixing this overuse is so far down the priority list that I don't think it survived the great Jira purge of mid-2024.
- eadmund 2y ago
  > storage is so cheap that being wasteful with it isn't really disincentivized I think another way of phrasing that is usage is correctly incentivised. In the example you give, the value to debugging is more than the cost of storage — and even if that’s not the case it’s so low-priority that it might not even be on your list of priorities anymore! That literally means that it’s worth your limited, valuable time to do something else.
manyturtles 2y ago
About a decade and a half ago I worked on a large data migration project at a FAANG. Multi-exabyte scale, many clusters across many countries. Once everyone was moved the old storage platform wasn't completely empty, because the number of migrations was large and users were (naturally) more focused on ensuring their data was in place and available on the target platform rather than ensuring every last thing was deleted on the legacy platform. We weren't initially concerned about it because it would all get deleted when we turned down the old setup. As we were gearing up to declare victory and start turning down the several dozen legacy storage clusters someone mused that given some users were subject to litigation holds -- not allowed to delete any data -- that at least some of the leftover data on the old system might be subject to litigation hold, and we'd need to figure that out before we could delete it or incur legal risk. IIRC the leftover 'junk' data amounted to a few dozen petabytes spread across multiple clusters around the world, in different jurisdictions. We spent several months talking with the lawyers figuring that out. It was an interesting dance, because on the one hand we were quite confident that there was unlikely to be anything in the leftovers which was both meaningful and not migrated to the new platform, while on the other hand explaining that it wasn't practical to just "go and look" through a few dozen PB of data. I recall we ended up somewhere in between, coming up with ways to distinguish categories of data like caches and working data from various pipelines. It added over six months to the project, but was quite an interesting problem to work through that hadn't occurred to any of us earlier on, as we were thinking entirely in technical terms about infrastructure migration.
- isaacremuant 2y ago
  That does sound very interesting. Any insights on what would you do differently if you had to do it again? Any way to accelerate things now that you know the pain or do you think it's quite unavoidable and "legal Time"?
  - manyturtles 2y ago
    Doing it in parallel, which of course is only an option if you know about it. And templating that into a general approach and having legal folks sign off on that.
ein0p 2y ago
A gross underestimation, IMO. When I was in big data, fewer than 5% of data written was ever touched again, and only a single digit number of our large customers (out of tens of thousands) actually made real use of their "big data", and created most of the load. That's the trouble with "checkbox driven development" - 10 years ago you were required to have a "big data strategy" for anyone to take you seriously, even if your strategy boiled down to just ETL-ing a bunch of crap you're never going to need into the cloud and never touching it again. Now I'm in AI, and the same thing is happening to AI. It's great if you're selling shovels, so to speak, but not so great if you plan on selling them for an extended period of time. This, by the way, has implications on storage systems design. You want something that's cheap yet dense to encode, potentially at the slight expense of decode speed. Normally people really lose sleep about decode speed first and foremost, which, while important, does not minimize the overall resource bill.
jodrellblank 2y ago
It's a problem for us when it comes to the GDPR and rights to be forgotten; companies will say they store your data carefully, will say they have shown you everything they have, will say they delete it, but "the company" in aggregate has no idea there's a thousand SharePoint sites and ex-employees mailboxes and filestores, and copies of old filservers from before a migration, and test databases containing copies of real data from a half-abandoned project where new management fired the contractors and then never got around to hiring new ones.
- otterley 2y ago
  The right to be forgotten was never a practically enforceable right to begin with. It’s a right that burdens others with perpetual effort and cost, unlike most rights, which only require others not to do things. I’m not sure why Europeans are so obsessed with it.
qingcharles 2y ago
Photos not seen by humans again, but plenty of value for the AI overlords to examine. These things have value again. Didn't Facebook start to move most of their least-used data onto optical arrays a long time ago?
edwardoelliott6 2y ago
[dead]
tbrownaw 2y ago
Storage being cheap enough that it's not worth policing doesn't seem very consistent with it being expensive enough to include much energy use (what I assume the "destroying the environment" hyperbole is referencing).
chiggsy 2y ago
> Having to deal with senior managers has always been the most unsavory part of my job, because when it comes to technology, these managers exist on a whole other level of stupid vanity and narcissistic pursuit of their own selfish agendas. Whoever sent this dude made a mistake. People who don't share your worldview need to be persuaded, not insulted! Some dude stomps in, thinks all the snaps in the cloud are crap, things the big bosses are stupid for not instantly deleting the pictures they saved into the cloud.... and then what? Download Lisp? Thought we got over this, pal. WORSE IS BETTER. P.S. do not erase our porn. WORSE IS BETTER.
umutisik 2y ago
Make the cost of operating data centers reflect the damage to the environment and see how quickly people optimize. I don’t know how damaging storage is, but that’s the only way to make a difference.
- jeffbee 2y ago
  The cost to operate a data center is already, to a close approximation, the cost of its electricity. The data center operator already pays for every joule. Data centers used ~4% of USA electricity last year, about .15 quads, while the chemical industry used 10 quads, virtually all in the form of fossil fuel. If you are going to start reflecting the externality of energy consumption in the price of goods, information sector will probably not need to adjust anything, while the chemical sector will be fundamentally impaired.
- knowitnone 2y ago
  so charge more? I'm sure they are already charging as much as the market will bear. Governments can impose a tax which they already have.
zekenie 2y ago
Not my area of expertise but I don’t think storage is a big deal, relatively speaking. It’s all about the compute
ltbarcly3 2y ago
> We’re destroying our environment to store copies of copies of copies of stuff we have no intention of ever looking at again While I agree that most of the stuff in data centers is probably crap, but that's because most of everything people do is crap. That's not for me to decide though, people save things because they find value in them. Most of what has value to another person won't have value to you. Most of what people treasure in their life is thrown away after they die because nobody wants it, even their closest family members. Who gets to tell everyone the bad news, that objectively their memories are trash and they don't have a right to keep them anymore? Gerry Fuckin' McGovern? Secondly, we aren't destroying the environment for any of this. Data centers use like 5% or less of the overall electricity use. It's a lot, but we don't have to put datacenters in random locations, we can (and do) put them where electricity is cheap. That generally means that the 5% of electricity used for data centers, kwh for kwh isn't as impactful as an average kwh of end use. Large companies like Meta and Google claim to have zero net carbon by obtaining offsets. So in general we aren't "destroying the environment" to store copies of photos.
- cyberjerkXX 2y ago
  I couldn't agree more - the data stored in data centers have "some" value, otherwise people wouldn't pay to host it. This author anecdotally comments about his experience while pearl clutching about the environment, typical.
xyst 2y ago
> because when it comes to technology, these managers exist on a whole other level of stupid vanity and narcissistic pursuit of their own selfish agendas Regardless of what you think about the article. This rings so true at many Fortune 500 companies. The number of times I have seen teams work through pointless bullshit to push some meaningless objective for the company. Just so the middle manager (aka “Director of SVP of X product of Y branch”) can get a bullet point(s) in the quarterly “all hands”. Oh and those 10 developers/off shore people that were just hired? It was all to pump his/her “head count” number to get to the promotion to next grade/level. Then when that person gets promoted, those people get scattered throughout the firm or just let go. It’s truly just weaponized incompetence.
eadmund 2y ago
> We’re destroying our environment to create and store trillions of blurred images, half-baked videos, rip-off AI ‘songs’, rip-off AI animations, videos and images, emails with mega attachments, never-to-be-watched-again presentations, never-to-be-read-again reports, files and drawings from cancelled projects, drafts of drafts of drafts, out of date, inaccurate and plain wrong information, and gigabytes and gigabytes of poorly written, meandering content. Storing the files for Mr. McGovern’s website requires plastics, metals, power and physical space, yet I assume he believes that environmental effect is worthwhile. Who is he to decide for others that their choice to pay for the storage of data is not equally worthwhile to them? That’s the beauty of a price system: each of us gets to decide what we will buy, and what we will not buy. Now, perhaps his argument should be that the price of storing digital data does not adequately reflect the true cost. Perhaps there are unaccounted-for externalities. If so, then he should make that argument, perhaps arguing for a tax to align prices with costs. Someone else might argue that data is a liability as well as an asset. That’s another argument he could make. But haranguing folks for spending their money in ways he doesn’t like doesn’t seem likely to produce the outcome he appears to wish.
alright2565 2y ago
It's pretty simple: sorting through the data to determine which 90% is crap is more expensive and uses more of our scarce resources than just storing it all.
cyberjerkXX 2y ago
I wonder what this guy thinks about Internet Archive?
- maigret 2y ago
  This is the least of problems. The Wayback Machine is 100 petabytes. https://en.wikipedia.org/wiki/Wayback_Machine https://en.wikipedia.org/wiki/Wayback_Machine . "Every day, we create more than 350 million terabytes of data." https://www.rankred.com/largest-data-centers-in-the-world/ https://www.rankred.com/largest-data-centers-in-the-world/ , so 3500x the Wayback Machine every day.
SturgeonsLaw 2y ago
90% of everything is crap. I think there's a law about that or something
bnewbold 2y ago
I agree with the general sentiment here, but don't like the examples. 200 photos per person per year isn't very much! That is all fine. What really bloats things out is surveillance (video and online behavioral) and logging/tracking/tracing data. Some of this ends up cold, but a lot of it is also warm, for analytics. It bloats CPU/RAM/network, which is pretty resource intensive. The cost is justified because the margins of big tech companies are so wildly large. I'd argue those profits are mostly because of network effects and rentier behavior, not the actual value in the data being stored. If there was more competition pressure, these systems could be orders of magnitude more efficient without any significant different in value/quality/outcome, or really even productivity.
sali0 2y ago
When traveling, a funny thought I always have is watching other tourists take the same photos, from the same exact location, knowing it will be backed up to iCloud. I can't help but imagine how much disk space is taken up by duplicates of the exact same photo.
- pants2 2y ago
  That and videos of concerts / fireworks shows. For a fun example, every time I'm on the Las Vegas strip I see dozens of people taking videos of the Bellagio Water Show. There are 30 shows per night, if 50 people take videos in 4K 60fps (default on new iPhones), that's around 60 GB of data per show or ~600 TB per year of just videos of the Bellagio Fountain Show!
mproud 2y ago
There are a few arguments here, and yeah, I get it, some of the shit being stored is shit — or shitty versions of content already stored. But the argument that “these pages haven’t been reviewed in 20 years!” is exactly the reason for preservation! We want to be able to read, listen, and review content that is rare.
donatj 2y ago
Burn the libraries, they're full of books no one's read in years /sarcasm
wvenable 2y ago
I hate searching for vintage computer stuff only to find broken hyperlinks to lost pages that once contained useful information or software but is now gone forever.
- Borg3 2y ago
  Yeah, because we do NOT mirror usefull information. Only crap :) I remember years ago (20 years, even 10 years ago) it was easy to mirror stuff. These days, web pages are loaded w/ JS junk.. Is hard, so I just extract interesting text and put it into my storage.
_Algernon_ 2y ago
Surprised it isn't higher
smetj 2y ago
> Why were they created? Proof of work. Look at all this data I/we created. And the article didn't talk about logs and other operational data yet
bocytron 2y ago
It seems you're all missing the point here: it's not about storing useless data, it's about destroying the environment in the process. I understand you all want to keep everything, just in case, because it's cheap and you don't see the externalities. But there are externalities, and they are big.
- andybak 2y ago
  In which case we should be reading an article about how important it is to correctly price externalities. And that is not this article.
  - palata 2y ago
    Well that article clearly says "it's hurting the environment, and in my experience the vast majority of that data is useless". Which I believe is not uninteresting, given the amount of answers here where people say "Is that data useless? I don't know, I could imagine that it's not, I think it's a hard problem". Well here we have one person saying "I have experience in that, and I can tell you that most of it is useless". Just a data point, but that's still interesting.
- palata 2y ago
  This. We need to think about the data we need to store before we store it, only store the data that we need to store, and only store it for as long as needed. It reminds me of CIs. It's now so easy to throw 40 jobs on GitHub actions that people don't think about them. I have been in a startup where people would debug in CI: they wouldn't have e.g. Windows on their machine (maybe they should have, given that their product was supposed to run there) and were fixing compilation issues by sending patches and patches to the CI. Every single time it would trigger the 40 jobs. Sometimes you could see a patch sent every 5 min for 3 days (where reproducing the issue locally would actually take 3s and not 5min). They did not even bother disabling the 39 uninteresting jobs. For open source projects, it's just wasted energy, for private repos it was costing the company a lot. This was just malpractice. But nobody cared. The finance person would say "GitHub is expensive", the CEO that "well we need it" and the engineers that "I don't want that Windows crap on my computer", I suppose.
hannob 2y ago
I have seen takes about the environmental footprint of unused data before, and I have severe doubts this is a relevant issue. I mean, sure, there is some impact. Storage media has to be produced. But there's a reason storage is cheap, it's not a whole lot of resources going into it. And hard drives that are idle in some data center without being accessed don't consume a lot of electricity. There are very real and concerning problems with the environmental impact of IT. But they are primarily found in other areas. Energy consumption is mostly a function of "how much you compute with data", not "how much data you have". In other words: be concerned about so-called "AI", be concerned about Bitcoin. Don't worry about unused data too much.
nimchimpsky 2y ago
[dead]
tromp 2y ago
> the cost of storing data is less than the cost of figuring out what to do with the crap. Or the cost of figuring out that it's not worth saving...
otterley 2y ago
90% of libraries consist of books that are never opened. These books were all produced by destroying and processing trees, sometimes with toxic chemicals, and their information density is orders of magnitude lower than that of a hard disk or SSD. Same with photo processing, where 90% of photos taken are discarded, and the toxicity of the chemicals is even higher. So the question isn't simply whether storage is wasted; it's how much waste there is relative to the environmental impact. Granted, books and photographs don't need to be continuously fed energy to make the information available. However, the cost of storage is now so cheap that even with 90% waste, it's economically viable to keep it online. So the problem, if you can call it one, is that energy is too cheap, and externalities are not accounted for in the cost.
- foobahify 2y ago
  Yes. Rather than compare to paper, compare to yesterday's HDDs. Do we need more HDDs today than then?
- plasticeagle 2y ago
  > 90% of libraries consist of books that are never opened I'm reasonably certain that this statistic is completely made up. The best number I can find for the proportion of library books that are never borrowed was from a university library, and was 25%.
- mvc 2y ago
  > 90% of libraries consist of books that are never opened. Citation required. But don't bother because it's a meaningless statistic, or at least one designed to make it look like there's a lot more wastage in libraries than there actually is. The statistic could be true, and yet still be the case that the vast majority of library books are well utilized.
  - otterley 2y ago
    I should have been less specific than “library” as such. Libraries aren’t only institutional. Consider every book, magazine, and newspaper that sits today in everyone’s home, office, and in every institution in the world. The vast majority of them have been read once and then left on a shelf or in a box somewhere, taking up more or less valuable space.
hoseja 2y ago
"We’re destroying our environment to (...)" No we're not. I really dislike this "environmental" anti-technologist angle. A single steel plant in china has tenfold "environmental impact" than all photos stored on a platter everywhere. Would you prefer the photos are a cocktail of weird chemicals on a negative and printed on glossy photo paper? Digital data is the most ephemeral we are able to make it through vast effort.
Bluglionio 2y ago
[dead]
kdamica 2y ago
To repurpose an old saying: "90% of my data is crap. I just don't know which 90%"
mrb 2y ago
Some fun math: according to some estimates there is 175 zettabytes of data worldwide. Assuming 20-terabyte harddrives, this could be stored on 8.8 billion drives. Assuming 10 drives per rack unit, 42 RU per rack cabinet, 16 square feet per cabinet (including aisle space), that means you need about 330 million square feet of data center space to host this data. If it was hosted in a single square data center, it would be 3.5 miles long and wide. (I always like to picture the physical space something would occupy.) And energy wise, assuming 5 watts per drive, it would consume 44 gigawatt, so it could be powered by about two large hydro dams similar to the Three Gorges Dam (22 gigawatts capacity). I am assuming a PUE close to 1.0 for simplicity. Of course one would not be able to spin up all these drives at once, since a drive spinning up consumes about three times more power (15 watts). So you would definitely staggered spin up when booting the servers :-) If 90% of this data is "crap" and could be cut down, it would still be just a drop in the bucket compared to worldwide energy use.
- andyp-kw 2y ago
  I wonder how it compares to electric car usage, or street lights in the USA.
- vb-8448 2y ago
  Furthermore: a big amount of this data is stored on physical tapes and not hard drives. A LTO-9 tape is able to store up to 45TB of data and doesn't require any electricity when not mounted to be read.
the_real_cher 2y ago
> were destroying our environment citation needed
Jean-Papoulos 2y ago
The other day I had to go through 15 years old PowerPoint files to grab the originals of pictures made by a guy extremely proefficient at creating detailed artwork from PowerPoint forms that's now retired. Was can now render them to full-hd PNGs instead of the 256x256 BMP files we were using before. Storing "useless" data makes financial sense.
kkfx 2y ago
Not too related but... Did you ever tried to imaging a world where computing is back to interconnected desktops/homeservers? Well, if you see trends like {fog,edge}-computing or recent ideas to distributed LLM computing (BrianknowsAI DCI Network for instance) it's clear that giants well known they can't keep up the modern mainframe model preferring to makes users paying the iron and bandwidth while they handle the software stack. Now it's clear the new deal could be implemented only in homes/sheds with domestic p.v. and storage, smart cities keeps to fail since the ancient Fordlandia, see Neom, Songdo, Masdar, PlanIT Valley, Lavasa, Ordos, Santander city, Toronto Quayside (Google Sidewalk Labs), Amazon HQ2, Egypt new Cairo still nameless, Modi's Indian 100-smart city program, Arkadag, Innopolis, Nusantara, Proton City, ... and can't be powered with a smart-grid at such scale. So well, new well insulated buildings, with ventilation of course, with p.v. and storage, with room for a domestic rack(s), with FTTH. Anyone with such settlements could have his/her own "datacenter" at home, following the same trend for medical devices more and more cheaper and smaller. A LOM? Well a NanoKVM PCIe or an external JetKVM cost MUCH less than classic LOM and do much more. We have all the gear to makes such "datacenter at home" assemblies, anyone holding preferred crap and participating in distributed computing networks to pay at least a bit gear and bandwidth. It's not for all of course, some will be trapped in dense cities while some large owner dream an obviously not possible conversion from offices to apartments and datacenters like https://finance.yahoo.com/news/southern-californias-hottest-commercial-real-100007648.html https://finance.yahoo.com/news/southern-californias-hottest-... or https://www.euronews.com/next/2024/02/29/madrid-to-convert-underused-offices-into-flats https://www.euronews.com/next/2024/02/29/madrid-to-convert-u... and https://www.theguardian.com/society/2025/jan/05/office-to-homes-conversions-london-blocks-hold-fresh-allure-since-shift-to-home-working https://www.theguardian.com/society/2025/jan/05/office-to-ho... or https://czechdaily.cz/half-of-pragues-office-buildings-are-aging-a-shift-towards-residential-spaces/ https://czechdaily.cz/half-of-pragues-office-buildings-are-a... etc for all over the developed world. That's while we admit https://doi.org/10.1073/pnas.2304099120 https://doi.org/10.1073/pnas.2304099120 we need a full-remote DISTRIBUTED shift. Food, meds, general retail distributed by a single integrated logistic platform for maximum efficiency in a spread society, the IT evolution makes Distributism possible. Doing so erase the big amount of concentrated energy, dense network, heat handling and water problem of datacenter and also reduce much the crap, because anyone keep it's own personal and being not free keeping it they'll learn to be storage continuous.
srmarm 2y ago
Some fair points, but some 'crap data' has storage requirements orders of magnitudes from each others. One short video can equal a year worth of emails for someone. Similarly those many webpages that don't get viewed often probably require only a negligible amount of resources to keep online and might help someone who'd otherwise be faced with linkrot. Best to focus on the low hanging fruit.
imcritic 2y ago
WON'T SOMEBODY, PLEASE, THINK ABOUT THE ENVIRONMENT!?
geor9e 2y ago
Instagram and Facebook, at least, will compress a 4K video down to 360p if it doesn't get any views for a couple of months. So, there are some mechanisms to degrade the crap.
toader 2y ago
In this book 'World Wide Waste' he states 'Every time I download an email I contribute to global warming.' Is this true? Aren't some data centers, Google for example carbon neutral?
- BSDobelix 2y ago
  >>Google's carbon footprint jumped by 48% from 2019, amounting to 14.3 million tons of carbon dioxide emissions for 2023. According to the company's Environment Report 2024, 24% of its total emissions, or over 3.4 million tons, comes from market-based sources, i.e., purchased electricity published July 3, 2024: https://www.tomshardware.com/tech-industry/google-reveals-48-increase-in-greenhouse-gas-emissions-from-2019-largely-driven-by-data-center-energy-demands https://www.tomshardware.com/tech-industry/google-reveals-48...
knowitnone 2y ago
One man's crap is another man's crap. My data, which may be 100% crap to you, is obviously not (all) crap to me. I often unerase USB drives, HDDs, and SD cards find pictures, movies, etc. Obviously, they don't have much value to me so lots of it is crap but sometimes there's gold hidden in the crap if you dig.
acd 2y ago
Thats why human brains forget
iamleppert 2y ago
I recently worked on a project that proposed using ancient cave systems, full of priceless stalagmites as data centers. The first step in the operation? Raze the cave and cover everything in a thick layer of cement. Use of the underground water table was recommended to cool the AI machines. It was all very dystopian. It was even suggested that the national park system be sold off to build these mega data centers underground.
teleforce 2y ago
> The Cloud made the crap data problem infinitely worse. This article is mainly focusing about the unused data by website and enterprise databases, only toward the end of the article it barely touched upon "the elephant in the room" of data in cloud. Now everywhere in the world data centers are being built at breakneck speed to cater for the AI data modeling, training and serving. Most of the AI based data are being kept in datalake in the form of raw data that will probably never see the light of that day i.e never being processed. Bill Inmon warned us against this potential data swamps in data center due to the increasing popularity of the datalake [1]. Hopefully open table format like Apache Iceberg can rectify this unused raw data epidemic but time will tell [2]. [1] Lakehouses Prevent Data Swamps, Bill Inmon Says https://www.datanami.com/2021/06/01/lakehouses-prevent-data-swamps-bill-inmon-says/ https://www.datanami.com/2021/06/01/lakehouses-prevent-data-... [2] What Are Apache Iceberg Tables and How Are They Useful? https://www.snowflake.com/guides/what-are-apache-iceberg-tables/ https://www.snowflake.com/guides/what-are-apache-iceberg-tab...
precommunicator 2y ago
We recently discovered we store 500MiB of email tokens (expired). Then a copy of them in history table. Then this data is replicated. Then there are total of 4 backups of it. Backups that are done every 2 hours for half of each day. And we store those backup for up to half a year. I don't even want to add that up...
- 392 2y ago
  If you believe adding it up linearly is valid, and you care about the results, you should be using more efficient storage software.
  - precommunicator 2y ago
    It's "nice go have" i.e. cheaper to just pay for more storage than do it properly
ashoeafoot 2y ago
So if its 90% dead cold data, the temptation must be there for "ocassional hit"data, to switch from ssd/ hdd in a server to some slow access magnetic tape with a delayed reply..
elmolino89 2y ago
Without rules, metadata, systems to organize files even the most valuable data can be stored as a steaming pile of poo. In my case I was diving into a compressed tar containing compressed tar-s, which again may have giant uncompressed data files inside. The whole pile of crap had premature EOF so for sure some non trivial amount of data was lost. And on the top of it what was preserved should have a prefix: "unobtanium_". Storing several tar.gz several terabytes in size on some puny domestic use NAS made extraction of the files a crying game. In another site I found a mix of some old version windows disk images with data. With more crap inside. In the end: storage may be cheap. Storing piles of disorganized crap is very costly if you want to find something

That's a pretty optimistic estimate, tbh.

Don't forget emails.. I have everything I ever sent or received, and I have it backed up. I expect 90% of my inbox is the jpg signature logo they attach to the bottom of my clients email rather than hyperlink.

hobs 2y ago
That's not even crap data - that's archived data that might be useful someday (though de-dupe is probably a great idea and email sigs are definitely wasteful trash) - most of the crap data is things that would never have been useful under any circumstances. I have cleaned up dozens of product databases in cost management efforts and have found anywhere from 50-99% of data stored in product databases is crap, because they are not well managed and any single mistake can lead to a huge outsized impact on storage. What to log all those http requests for just a day? Might as well turn that on for all time...
duxup 2y ago
I ended up working on some software and I was deemed the email guy (it's a very small % of my job but it is the biggest pain). "I need an email when this happens.. and when this happens." The requests are endless and I'm convinced there are people who if they could would do their entire job from their inbox and get everything and anything an application can do via email. The insidious problem is that it never solves anything. "I didn't get the email!" is a constant refrain (yes they did, they always did). "Oh someone didn't do the thing so can you add this to the email too." and so on. It is such an abused tool.
- Dylan16807 2y ago
  > The requests are endless and I'm convinced there are people who if they could would do their entire job from their inbox and get everything and anything an application can do via email. That sounds like a reasonable goal for a whole lot of job duties. And yes some entire office jobs. (Excluding some direct human communication but a lot of jobs already have too much of that in situations that could have been an email.) > "I didn't get the email!" is a constant refrain (yes they did, they always did). Well having to manually check wouldn't improve that, would it?
- Cthulhu_ 2y ago
  I can see how that would work (email centric workflow), not unlike how some people now try to have a chat / Slack centric workflow.
  - saalweachter 2y ago
    Yeah, everyone knows the proper thing to do is orchestrate it all through Emacs.
- mrweasel 2y ago
  We had a similar request for a sales team once: "If this fails we want an email".... Okay, but we never seen this fail, so you'll never get the email. As expected, they never got an email, because no failures. So instead they wanted an email every time the job ran (once every night), so they'd know that the job had failed, if they DIDN'T get the email. Only time that happened was because the email got trapped in a spamfilter I think. There must be thousands of copies of that email sitting in inboxes say: Job X ran successfully @ 04:30.
jeffbee 2y ago
Well "the cloud" will generally store exactly one logical copy of your static jpg, one of the reasons why clouds are pretty good for efficiency. There is really no sense at all in the article's claims that "we are destroying the environment" to do x y and z thing the author whines about. We are destroying the environment to drive a Dodge Ram to the Circle K to buy a 52oz Polar Pop. Information sector doesn't even show up on top ten lists of things that are destroying the environment.
HPsquared 2y ago
It's probably deduplicated on the server though, so the millions and millions of messages with that logo likely share the same piece of disk space. Probably one reason why free providers don't tend to offer End-to-End encryption. It prevents deduplication (and otherwise compressing redundant information).
- justsomehnguy 2y ago
  Nope. Back in the day Exchange offered SIS but in 2010 they ditched it. It's plainly not effective any more. Even regarding the OP' "the jpg signature logo" - it's a part of multipart in the message, not a separate file. And one more thing - you can't just turn the dedup and be dandy, now you need to check against the hashes to determine if this chunk is unique or you already have it. And with TBs of data you need TBs of hashes to check. Until you have like 99% dedup efficiency, ie 99% of your incoming data is literally the same data you already have - it doesn't worth it. https://techcommunity.microsoft.com/blog/exchange/dude-wheres-my-single-instance/595267 https://techcommunity.microsoft.com/blog/exchange/dude-where...
  - Sprite_tm 2y ago
    Not sure about Microsoft solutions, but modern file systems like zfs and btrfs can do this on the filesystem level, no support from the mail server needed.
    - tored 2y ago
      NTFS and ReFS has both deduplication support (only on Windows Server).
  - LegionMammal978 2y ago
    Also, I can imagine such schemes getting ditched for the same reasons of "now people can detect whether we already have a copy of the file, which is a privacy hole!" that killed cross-origin caching in browsers. Or if everything in the year 20XX gets pushed into using E2E encryption, since that's pretty much antithetical to deduplication.

ctoth 2y ago

Surely a few more nines if the base rate is already Sturgeon.

matt-p 2y ago

Must be more, surely.

cactusplant7374 2y ago

Fortunately with LLM's we can generate data and not store it. If you think about it that means articles could always be relevant with additional details added as time goes on.

tylerrobinson 2y ago
No, the reality of this situation is that people need to save a snapshot of what the LLM said when they read it, just in case they need to substantiate it or blame someone for what they read at the time.
- cactusplant7374 2y ago
  No, it requires a shift in thinking. There are no guarantees with information one reads as it is.
irjustin 2y ago
At the moment, I don't like this for fear of revisionist history. The equvilent would be if wikipedia didn't have to have a source reference. not saying that it's infallible but yeah we need some chain of audible at least 1 or 2 layers deep.

danpalmer 2y ago

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a similar percentage of people claim on their car insurance. If only the rest realised they had "crap insurance" and were paying for nothing, they could save so much money! This is obviously sarcasm, but I think it's important to remember that much of the data is stored because we don't know what we will need later. Photos of kids? Maybe that one will be The One that we end up framing? Miscellaneous business records? Maybe those will be the ones we have to dig out for a tax audit? Web pages on government sites? Maybe there will suddenly be an interest in obscure pages on public health policy if a global pandemic happens. Complaining that data is mostly junk is not a particularly interesting conclusion without acknowledging this. Is there wastage? Yeah sure, but accuracy on what needs storing is directly traded off with time spent figuring that out, and often it's cheaper to store the data.

mrexroad 2y ago
I have ~2.5TB of photos in iCloud, via Apple Photos. Excluding the various sized previews, I doubt many originals have been accessed in quite a while. I also have about 1TB Lightroom library archived to a different service; representing countless hours of photo processing work spanning over a decade. Haven’t touched that one in years. Neither are crap. (Yes, both have other backups and, yes, I’ve probably forgotten to sync one of them).
- snapplebobapple 2y ago
  Crap is in the eye of the beholder...
- eloisius 2y ago
  Sometimes I wish there was a feasible system to audit and reduce redundant photos in my iCloud. I have an embarassing ton of pictures that I never look at just like everyone else. I wouldn’t mind a tool that said “here’s 10^x photos that are the most similar to all your other photos would you like to put them in the trash?”

somat 2y ago

Isn't this just a specific case of sturgeons law? https://en.wikipedia.org/wiki/Sturgeon%27s_law https://en.wikipedia.org/wiki/Sturgeon%27s_law

BrenBarn 2y ago
Yeah it seems like Sturgeon's law combined with a bit of Pareto principle.
foobahify 2y ago
[flagged]

RobotToaster 2y ago

I imagine there's a similar statistic for book shops and public libraries.

nameless912 2y ago

There's another dimension to this, that storage is so cheap that being wasteful with it isn't really disincentivized. I know for example at work of a portal that accepts uploads of large files from external clients that stores both the initial upload and every subsequent transformation of the file (of which there are 4-6) permanently. It's extremely useful for debugging, as one of the bits of metadata we shove on the zip archive is the git hash of the code that was running, so it's trivial to pull down any failed step and diagnose what happened. We are using 4-6 times as much storage as we need to, and these are often not small files (on the order of 100 MB - 5 GB, several dozen times a day) but fixing this overuse is so far down the priority list that I don't think it survived the great Jira purge of mid-2024.

eadmund 2y ago
> storage is so cheap that being wasteful with it isn't really disincentivized I think another way of phrasing that is usage is correctly incentivised. In the example you give, the value to debugging is more than the cost of storage — and even if that’s not the case it’s so low-priority that it might not even be on your list of priorities anymore! That literally means that it’s worth your limited, valuable time to do something else.

manyturtles 2y ago

About a decade and a half ago I worked on a large data migration project at a FAANG. Multi-exabyte scale, many clusters across many countries. Once everyone was moved the old storage platform wasn't completely empty, because the number of migrations was large and users were (naturally) more focused on ensuring their data was in place and available on the target platform rather than ensuring every last thing was deleted on the legacy platform. We weren't initially concerned about it because it would all get deleted when we turned down the old setup. As we were gearing up to declare victory and start turning down the several dozen legacy storage clusters someone mused that given some users were subject to litigation holds -- not allowed to delete any data -- that at least some of the leftover data on the old system might be subject to litigation hold, and we'd need to figure that out before we could delete it or incur legal risk. IIRC the leftover 'junk' data amounted to a few dozen petabytes spread across multiple clusters around the world, in different jurisdictions. We spent several months talking with the lawyers figuring that out. It was an interesting dance, because on the one hand we were quite confident that there was unlikely to be anything in the leftovers which was both meaningful and not migrated to the new platform, while on the other hand explaining that it wasn't practical to just "go and look" through a few dozen PB of data. I recall we ended up somewhere in between, coming up with ways to distinguish categories of data like caches and working data from various pipelines. It added over six months to the project, but was quite an interesting problem to work through that hadn't occurred to any of us earlier on, as we were thinking entirely in technical terms about infrastructure migration.

isaacremuant 2y ago
That does sound very interesting. Any insights on what would you do differently if you had to do it again? Any way to accelerate things now that you know the pain or do you think it's quite unavoidable and "legal Time"?
- manyturtles 2y ago
  Doing it in parallel, which of course is only an option if you know about it. And templating that into a general approach and having legal folks sign off on that.

ein0p 2y ago

A gross underestimation, IMO. When I was in big data, fewer than 5% of data written was ever touched again, and only a single digit number of our large customers (out of tens of thousands) actually made real use of their "big data", and created most of the load. That's the trouble with "checkbox driven development" - 10 years ago you were required to have a "big data strategy" for anyone to take you seriously, even if your strategy boiled down to just ETL-ing a bunch of crap you're never going to need into the cloud and never touching it again. Now I'm in AI, and the same thing is happening to AI. It's great if you're selling shovels, so to speak, but not so great if you plan on selling them for an extended period of time. This, by the way, has implications on storage systems design. You want something that's cheap yet dense to encode, potentially at the slight expense of decode speed. Normally people really lose sleep about decode speed first and foremost, which, while important, does not minimize the overall resource bill.

jodrellblank 2y ago

It's a problem for us when it comes to the GDPR and rights to be forgotten; companies will say they store your data carefully, will say they have shown you everything they have, will say they delete it, but "the company" in aggregate has no idea there's a thousand SharePoint sites and ex-employees mailboxes and filestores, and copies of old filservers from before a migration, and test databases containing copies of real data from a half-abandoned project where new management fired the contractors and then never got around to hiring new ones.

otterley 2y ago
The right to be forgotten was never a practically enforceable right to begin with. It’s a right that burdens others with perpetual effort and cost, unlike most rights, which only require others not to do things. I’m not sure why Europeans are so obsessed with it.

qingcharles 2y ago

Photos not seen by humans again, but plenty of value for the AI overlords to examine. These things have value again. Didn't Facebook start to move most of their least-used data onto optical arrays a long time ago?

edwardoelliott6 2y ago

[dead]

tbrownaw 2y ago

Storage being cheap enough that it's not worth policing doesn't seem very consistent with it being expensive enough to include much energy use (what I assume the "destroying the environment" hyperbole is referencing).

chiggsy 2y ago

> Having to deal with senior managers has always been the most unsavory part of my job, because when it comes to technology, these managers exist on a whole other level of stupid vanity and narcissistic pursuit of their own selfish agendas. Whoever sent this dude made a mistake. People who don't share your worldview need to be persuaded, not insulted! Some dude stomps in, thinks all the snaps in the cloud are crap, things the big bosses are stupid for not instantly deleting the pictures they saved into the cloud.... and then what? Download Lisp? Thought we got over this, pal. WORSE IS BETTER. P.S. do not erase our porn. WORSE IS BETTER.

umutisik 2y ago

Make the cost of operating data centers reflect the damage to the environment and see how quickly people optimize. I don’t know how damaging storage is, but that’s the only way to make a difference.

jeffbee 2y ago
The cost to operate a data center is already, to a close approximation, the cost of its electricity. The data center operator already pays for every joule. Data centers used ~4% of USA electricity last year, about .15 quads, while the chemical industry used 10 quads, virtually all in the form of fossil fuel. If you are going to start reflecting the externality of energy consumption in the price of goods, information sector will probably not need to adjust anything, while the chemical sector will be fundamentally impaired.
knowitnone 2y ago
so charge more? I'm sure they are already charging as much as the market will bear. Governments can impose a tax which they already have.

zekenie 2y ago

Not my area of expertise but I don’t think storage is a big deal, relatively speaking. It’s all about the compute

ltbarcly3 2y ago

> We’re destroying our environment to store copies of copies of copies of stuff we have no intention of ever looking at again While I agree that most of the stuff in data centers is probably crap, but that's because most of everything people do is crap. That's not for me to decide though, people save things because they find value in them. Most of what has value to another person won't have value to you. Most of what people treasure in their life is thrown away after they die because nobody wants it, even their closest family members. Who gets to tell everyone the bad news, that objectively their memories are trash and they don't have a right to keep them anymore? Gerry Fuckin' McGovern? Secondly, we aren't destroying the environment for any of this. Data centers use like 5% or less of the overall electricity use. It's a lot, but we don't have to put datacenters in random locations, we can (and do) put them where electricity is cheap. That generally means that the 5% of electricity used for data centers, kwh for kwh isn't as impactful as an average kwh of end use. Large companies like Meta and Google claim to have zero net carbon by obtaining offsets. So in general we aren't "destroying the environment" to store copies of photos.

cyberjerkXX 2y ago
I couldn't agree more - the data stored in data centers have "some" value, otherwise people wouldn't pay to host it. This author anecdotally comments about his experience while pearl clutching about the environment, typical.

xyst 2y ago

> because when it comes to technology, these managers exist on a whole other level of stupid vanity and narcissistic pursuit of their own selfish agendas Regardless of what you think about the article. This rings so true at many Fortune 500 companies. The number of times I have seen teams work through pointless bullshit to push some meaningless objective for the company. Just so the middle manager (aka “Director of SVP of X product of Y branch”) can get a bullet point(s) in the quarterly “all hands”. Oh and those 10 developers/off shore people that were just hired? It was all to pump his/her “head count” number to get to the promotion to next grade/level. Then when that person gets promoted, those people get scattered throughout the firm or just let go. It’s truly just weaponized incompetence.

eadmund 2y ago

> We’re destroying our environment to create and store trillions of blurred images, half-baked videos, rip-off AI ‘songs’, rip-off AI animations, videos and images, emails with mega attachments, never-to-be-watched-again presentations, never-to-be-read-again reports, files and drawings from cancelled projects, drafts of drafts of drafts, out of date, inaccurate and plain wrong information, and gigabytes and gigabytes of poorly written, meandering content. Storing the files for Mr. McGovern’s website requires plastics, metals, power and physical space, yet I assume he believes that environmental effect is worthwhile. Who is he to decide for others that their choice to pay for the storage of data is not equally worthwhile to them? That’s the beauty of a price system: each of us gets to decide what we will buy, and what we will not buy. Now, perhaps his argument should be that the price of storing digital data does not adequately reflect the true cost. Perhaps there are unaccounted-for externalities. If so, then he should make that argument, perhaps arguing for a tax to align prices with costs. Someone else might argue that data is a liability as well as an asset. That’s another argument he could make. But haranguing folks for spending their money in ways he doesn’t like doesn’t seem likely to produce the outcome he appears to wish.

alright2565 2y ago

It's pretty simple: sorting through the data to determine which 90% is crap is more expensive and uses more of our scarce resources than just storing it all.

cyberjerkXX 2y ago

I wonder what this guy thinks about Internet Archive?

maigret 2y ago
This is the least of problems. The Wayback Machine is 100 petabytes. https://en.wikipedia.org/wiki/Wayback_Machine https://en.wikipedia.org/wiki/Wayback_Machine . "Every day, we create more than 350 million terabytes of data." https://www.rankred.com/largest-data-centers-in-the-world/ https://www.rankred.com/largest-data-centers-in-the-world/ , so 3500x the Wayback Machine every day.

SturgeonsLaw 2y ago

90% of everything is crap. I think there's a law about that or something

bnewbold 2y ago

I agree with the general sentiment here, but don't like the examples. 200 photos per person per year isn't very much! That is all fine. What really bloats things out is surveillance (video and online behavioral) and logging/tracking/tracing data. Some of this ends up cold, but a lot of it is also warm, for analytics. It bloats CPU/RAM/network, which is pretty resource intensive. The cost is justified because the margins of big tech companies are so wildly large. I'd argue those profits are mostly because of network effects and rentier behavior, not the actual value in the data being stored. If there was more competition pressure, these systems could be orders of magnitude more efficient without any significant different in value/quality/outcome, or really even productivity.

sali0 2y ago

When traveling, a funny thought I always have is watching other tourists take the same photos, from the same exact location, knowing it will be backed up to iCloud. I can't help but imagine how much disk space is taken up by duplicates of the exact same photo.

pants2 2y ago
That and videos of concerts / fireworks shows. For a fun example, every time I'm on the Las Vegas strip I see dozens of people taking videos of the Bellagio Water Show. There are 30 shows per night, if 50 people take videos in 4K 60fps (default on new iPhones), that's around 60 GB of data per show or ~600 TB per year of just videos of the Bellagio Fountain Show!

mproud 2y ago

There are a few arguments here, and yeah, I get it, some of the shit being stored is shit — or shitty versions of content already stored. But the argument that “these pages haven’t been reviewed in 20 years!” is exactly the reason for preservation! We want to be able to read, listen, and review content that is rare.

donatj 2y ago

Burn the libraries, they're full of books no one's read in years /sarcasm

wvenable 2y ago

I hate searching for vintage computer stuff only to find broken hyperlinks to lost pages that once contained useful information or software but is now gone forever.

Borg3 2y ago
Yeah, because we do NOT mirror usefull information. Only crap :) I remember years ago (20 years, even 10 years ago) it was easy to mirror stuff. These days, web pages are loaded w/ JS junk.. Is hard, so I just extract interesting text and put it into my storage.

_Algernon_ 2y ago

Surprised it isn't higher

smetj 2y ago

> Why were they created? Proof of work. Look at all this data I/we created. And the article didn't talk about logs and other operational data yet

bocytron 2y ago

It seems you're all missing the point here: it's not about storing useless data, it's about destroying the environment in the process. I understand you all want to keep everything, just in case, because it's cheap and you don't see the externalities. But there are externalities, and they are big.

andybak 2y ago
In which case we should be reading an article about how important it is to correctly price externalities. And that is not this article.
- palata 2y ago
  Well that article clearly says "it's hurting the environment, and in my experience the vast majority of that data is useless". Which I believe is not uninteresting, given the amount of answers here where people say "Is that data useless? I don't know, I could imagine that it's not, I think it's a hard problem". Well here we have one person saying "I have experience in that, and I can tell you that most of it is useless". Just a data point, but that's still interesting.
palata 2y ago
This. We need to think about the data we need to store before we store it, only store the data that we need to store, and only store it for as long as needed. It reminds me of CIs. It's now so easy to throw 40 jobs on GitHub actions that people don't think about them. I have been in a startup where people would debug in CI: they wouldn't have e.g. Windows on their machine (maybe they should have, given that their product was supposed to run there) and were fixing compilation issues by sending patches and patches to the CI. Every single time it would trigger the 40 jobs. Sometimes you could see a patch sent every 5 min for 3 days (where reproducing the issue locally would actually take 3s and not 5min). They did not even bother disabling the 39 uninteresting jobs. For open source projects, it's just wasted energy, for private repos it was costing the company a lot. This was just malpractice. But nobody cared. The finance person would say "GitHub is expensive", the CEO that "well we need it" and the engineers that "I don't want that Windows crap on my computer", I suppose.

hannob 2y ago

I have seen takes about the environmental footprint of unused data before, and I have severe doubts this is a relevant issue. I mean, sure, there is some impact. Storage media has to be produced. But there's a reason storage is cheap, it's not a whole lot of resources going into it. And hard drives that are idle in some data center without being accessed don't consume a lot of electricity. There are very real and concerning problems with the environmental impact of IT. But they are primarily found in other areas. Energy consumption is mostly a function of "how much you compute with data", not "how much data you have". In other words: be concerned about so-called "AI", be concerned about Bitcoin. Don't worry about unused data too much.

nimchimpsky 2y ago

[dead]

tromp 2y ago

> the cost of storing data is less than the cost of figuring out what to do with the crap. Or the cost of figuring out that it's not worth saving...

otterley 2y ago

90% of libraries consist of books that are never opened. These books were all produced by destroying and processing trees, sometimes with toxic chemicals, and their information density is orders of magnitude lower than that of a hard disk or SSD. Same with photo processing, where 90% of photos taken are discarded, and the toxicity of the chemicals is even higher. So the question isn't simply whether storage is wasted; it's how much waste there is relative to the environmental impact. Granted, books and photographs don't need to be continuously fed energy to make the information available. However, the cost of storage is now so cheap that even with 90% waste, it's economically viable to keep it online. So the problem, if you can call it one, is that energy is too cheap, and externalities are not accounted for in the cost.

foobahify 2y ago
Yes. Rather than compare to paper, compare to yesterday's HDDs. Do we need more HDDs today than then?
plasticeagle 2y ago
> 90% of libraries consist of books that are never opened I'm reasonably certain that this statistic is completely made up. The best number I can find for the proportion of library books that are never borrowed was from a university library, and was 25%.
mvc 2y ago
> 90% of libraries consist of books that are never opened. Citation required. But don't bother because it's a meaningless statistic, or at least one designed to make it look like there's a lot more wastage in libraries than there actually is. The statistic could be true, and yet still be the case that the vast majority of library books are well utilized.
- otterley 2y ago
  I should have been less specific than “library” as such. Libraries aren’t only institutional. Consider every book, magazine, and newspaper that sits today in everyone’s home, office, and in every institution in the world. The vast majority of them have been read once and then left on a shelf or in a box somewhere, taking up more or less valuable space.

hoseja 2y ago

"We’re destroying our environment to (...)" No we're not. I really dislike this "environmental" anti-technologist angle. A single steel plant in china has tenfold "environmental impact" than all photos stored on a platter everywhere. Would you prefer the photos are a cocktail of weird chemicals on a negative and printed on glossy photo paper? Digital data is the most ephemeral we are able to make it through vast effort.

Bluglionio 2y ago

[dead]

kdamica 2y ago

To repurpose an old saying: "90% of my data is crap. I just don't know which 90%"

mrb 2y ago

Some fun math: according to some estimates there is 175 zettabytes of data worldwide. Assuming 20-terabyte harddrives, this could be stored on 8.8 billion drives. Assuming 10 drives per rack unit, 42 RU per rack cabinet, 16 square feet per cabinet (including aisle space), that means you need about 330 million square feet of data center space to host this data. If it was hosted in a single square data center, it would be 3.5 miles long and wide. (I always like to picture the physical space something would occupy.) And energy wise, assuming 5 watts per drive, it would consume 44 gigawatt, so it could be powered by about two large hydro dams similar to the Three Gorges Dam (22 gigawatts capacity). I am assuming a PUE close to 1.0 for simplicity. Of course one would not be able to spin up all these drives at once, since a drive spinning up consumes about three times more power (15 watts). So you would definitely staggered spin up when booting the servers :-) If 90% of this data is "crap" and could be cut down, it would still be just a drop in the bucket compared to worldwide energy use.

andyp-kw 2y ago
I wonder how it compares to electric car usage, or street lights in the USA.
vb-8448 2y ago
Furthermore: a big amount of this data is stored on physical tapes and not hard drives. A LTO-9 tape is able to store up to 45TB of data and doesn't require any electricity when not mounted to be read.

the_real_cher 2y ago

> were destroying our environment citation needed

Jean-Papoulos 2y ago

The other day I had to go through 15 years old PowerPoint files to grab the originals of pictures made by a guy extremely proefficient at creating detailed artwork from PowerPoint forms that's now retired. Was can now render them to full-hd PNGs instead of the 256x256 BMP files we were using before. Storing "useless" data makes financial sense.

kkfx 2y ago

Not too related but... Did you ever tried to imaging a world where computing is back to interconnected desktops/homeservers? Well, if you see trends like {fog,edge}-computing or recent ideas to distributed LLM computing (BrianknowsAI DCI Network for instance) it's clear that giants well known they can't keep up the modern mainframe model preferring to makes users paying the iron and bandwidth while they handle the software stack. Now it's clear the new deal could be implemented only in homes/sheds with domestic p.v. and storage, smart cities keeps to fail since the ancient Fordlandia, see Neom, Songdo, Masdar, PlanIT Valley, Lavasa, Ordos, Santander city, Toronto Quayside (Google Sidewalk Labs), Amazon HQ2, Egypt new Cairo still nameless, Modi's Indian 100-smart city program, Arkadag, Innopolis, Nusantara, Proton City, ... and can't be powered with a smart-grid at such scale. So well, new well insulated buildings, with ventilation of course, with p.v. and storage, with room for a domestic rack(s), with FTTH. Anyone with such settlements could have his/her own "datacenter" at home, following the same trend for medical devices more and more cheaper and smaller. A LOM? Well a NanoKVM PCIe or an external JetKVM cost MUCH less than classic LOM and do much more. We have all the gear to makes such "datacenter at home" assemblies, anyone holding preferred crap and participating in distributed computing networks to pay at least a bit gear and bandwidth. It's not for all of course, some will be trapped in dense cities while some large owner dream an obviously not possible conversion from offices to apartments and datacenters like https://finance.yahoo.com/news/southern-californias-hottest-commercial-real-100007648.html https://finance.yahoo.com/news/southern-californias-hottest-... or https://www.euronews.com/next/2024/02/29/madrid-to-convert-underused-offices-into-flats https://www.euronews.com/next/2024/02/29/madrid-to-convert-u... and https://www.theguardian.com/society/2025/jan/05/office-to-homes-conversions-london-blocks-hold-fresh-allure-since-shift-to-home-working https://www.theguardian.com/society/2025/jan/05/office-to-ho... or https://czechdaily.cz/half-of-pragues-office-buildings-are-aging-a-shift-towards-residential-spaces/ https://czechdaily.cz/half-of-pragues-office-buildings-are-a... etc for all over the developed world. That's while we admit https://doi.org/10.1073/pnas.2304099120 https://doi.org/10.1073/pnas.2304099120 we need a full-remote DISTRIBUTED shift. Food, meds, general retail distributed by a single integrated logistic platform for maximum efficiency in a spread society, the IT evolution makes Distributism possible. Doing so erase the big amount of concentrated energy, dense network, heat handling and water problem of datacenter and also reduce much the crap, because anyone keep it's own personal and being not free keeping it they'll learn to be storage continuous.

srmarm 2y ago

Some fair points, but some 'crap data' has storage requirements orders of magnitudes from each others. One short video can equal a year worth of emails for someone. Similarly those many webpages that don't get viewed often probably require only a negligible amount of resources to keep online and might help someone who'd otherwise be faced with linkrot. Best to focus on the low hanging fruit.

imcritic 2y ago

WON'T SOMEBODY, PLEASE, THINK ABOUT THE ENVIRONMENT!?

geor9e 2y ago

Instagram and Facebook, at least, will compress a 4K video down to 360p if it doesn't get any views for a couple of months. So, there are some mechanisms to degrade the crap.

toader 2y ago

In this book 'World Wide Waste' he states 'Every time I download an email I contribute to global warming.' Is this true? Aren't some data centers, Google for example carbon neutral?

BSDobelix 2y ago
>>Google's carbon footprint jumped by 48% from 2019, amounting to 14.3 million tons of carbon dioxide emissions for 2023. According to the company's Environment Report 2024, 24% of its total emissions, or over 3.4 million tons, comes from market-based sources, i.e., purchased electricity published July 3, 2024: https://www.tomshardware.com/tech-industry/google-reveals-48-increase-in-greenhouse-gas-emissions-from-2019-largely-driven-by-data-center-energy-demands https://www.tomshardware.com/tech-industry/google-reveals-48...

knowitnone 2y ago

One man's crap is another man's crap. My data, which may be 100% crap to you, is obviously not (all) crap to me. I often unerase USB drives, HDDs, and SD cards find pictures, movies, etc. Obviously, they don't have much value to me so lots of it is crap but sometimes there's gold hidden in the crap if you dig.

acd 2y ago

Thats why human brains forget

iamleppert 2y ago

I recently worked on a project that proposed using ancient cave systems, full of priceless stalagmites as data centers. The first step in the operation? Raze the cave and cover everything in a thick layer of cement. Use of the underground water table was recommended to cool the AI machines. It was all very dystopian. It was even suggested that the national park system be sold off to build these mega data centers underground.

teleforce 2y ago

> The Cloud made the crap data problem infinitely worse. This article is mainly focusing about the unused data by website and enterprise databases, only toward the end of the article it barely touched upon "the elephant in the room" of data in cloud. Now everywhere in the world data centers are being built at breakneck speed to cater for the AI data modeling, training and serving. Most of the AI based data are being kept in datalake in the form of raw data that will probably never see the light of that day i.e never being processed. Bill Inmon warned us against this potential data swamps in data center due to the increasing popularity of the datalake [1]. Hopefully open table format like Apache Iceberg can rectify this unused raw data epidemic but time will tell [2]. [1] Lakehouses Prevent Data Swamps, Bill Inmon Says https://www.datanami.com/2021/06/01/lakehouses-prevent-data-swamps-bill-inmon-says/ https://www.datanami.com/2021/06/01/lakehouses-prevent-data-... [2] What Are Apache Iceberg Tables and How Are They Useful? https://www.snowflake.com/guides/what-are-apache-iceberg-tables/ https://www.snowflake.com/guides/what-are-apache-iceberg-tab...

precommunicator 2y ago

We recently discovered we store 500MiB of email tokens (expired). Then a copy of them in history table. Then this data is replicated. Then there are total of 4 backups of it. Backups that are done every 2 hours for half of each day. And we store those backup for up to half a year. I don't even want to add that up...

392 2y ago
If you believe adding it up linearly is valid, and you care about the results, you should be using more efficient storage software.
- precommunicator 2y ago
  It's "nice go have" i.e. cheaper to just pay for more storage than do it properly

ashoeafoot 2y ago

So if its 90% dead cold data, the temptation must be there for "ocassional hit"data, to switch from ssd/ hdd in a server to some slow access magnetic tape with a delayed reply..

elmolino89 2y ago

Without rules, metadata, systems to organize files even the most valuable data can be stored as a steaming pile of poo. In my case I was diving into a compressed tar containing compressed tar-s, which again may have giant uncompressed data files inside. The whole pile of crap had premature EOF so for sure some non trivial amount of data was lost. And on the top of it what was preserved should have a prefix: "unobtanium_". Storing several tar.gz several terabytes in size on some puny domestic use NAS made extraction of the files a crying game. In another site I found a mix of some old version windows disk images with data. With more crap inside. In the end: storage may be cheap. Storing piles of disorganized crap is very costly if you want to find something