5 ms·

> One organization I knew of had 1,500 terabytes of data, with less than 2% ever having been accessed after it was first stored. On a related note, probably a

• mrexroad 2y ago
  I have ~2.5TB of photos in iCloud, via Apple Photos. Excluding the various sized previews, I doubt many originals have been accessed in quite a while. I also have about 1TB Lightroom library archived to a different service; representing countless hours of photo processing work spanning over a decade. Haven’t touched that one in years. Neither are crap. (Yes, both have other backups and, yes, I’ve probably forgotten to sync one of them).
  • snapplebobapple 2y ago
    Crap is in the eye of the beholder...
  • eloisius 2y ago
    Sometimes I wish there was a feasible system to audit and reduce redundant photos in my iCloud. I have an embarassing ton of pictures that I never look at just like everyone else. I wouldn’t mind a tool that said “here’s 10^x photos that are the most similar to all your other photos would you like to put them in the trash?”
    • makeitdouble 2y ago
      Thing is, the tool needs you to pick the one instance that survives the purge. Which means looking through 10^x of mostly similar photos to pick one. Removing duplicates isn't that complex already, and these tools exist so anyone can try it. It's just a truely grueling process.
      • secabeen 2y ago
        Yeah, there have been a number of companies that say they'll offer AI Photo Culling, but I have yet to find one that actually does it.
      • mattkrause 2y ago
        Can you recommend any particular tools to do this?
    • mrexroad 2y ago
      If you’re using Apple Photos, the feature is there. It will detect both exact copies as well as (nearly) identical visual duplicates. Look under utilities.
      • eloisius 2y ago
        Maybe it's different on the desktop, but on the iPhone it only detects the same photo that may have been saved at different resolutions or things like that. That's useful, but I really want something that can say use an embedding or something to group all 10 photos I shot of friends around the campfire and help me select one keeper and delete the other 9.
        • mrexroad 2y ago
          The feature works the same on iPhone. Looking through my duplicates just now, the detected duplicates that had a dozen or so photos to merge, are either high speed sports photography bursts or astrophotography—the rest were just mostly two photo duplicates. I suspect the threshold for determining a the photo as duplicate is too high for successive* photos to be considered such, as with your campfire example. With that said, I’m surprised Apple hasn’t implemented a feature to group/bundle successive photos beneath what is determined to be “best.” [*] Note: successive photos being separate shutter taps/actuations (potentially several seconds apart), where bursts are continuous (generally as fast as hardware allows, ms apart).
  • prawn 2y ago
    I have about the same in iCloud and then dozens of drives of video footage. Beyond the "keep it in case you need it" aspect, another angle is that it's often cheaper to just keep everything than spend all the time it would take to do a thorough cull. Video is much slower to review, but also chews up far more space, so maybe that balances out.
• ivraatiems 2y ago
  This is exactly the problem. Consider an order processing system that processes a million orders a day, and retains them for ninety days. What percentage of those 89-day-old orders are actually needed at 89 days old? It could be quite low, maybe a couple thousand out of a million. But if those orders aren't there, shit hits the fan. PCI compliance audits fail. The ability for customers to reconcile their charges with their purchases breaks. In that 0.01% of cases where the order was fraudulent, placed by mistake, or just didn't have what the customer thought it had in it, not having that data makes the order processor read as, if not malicious, at least incompetent. The real question is, how much data do we need to store inefficiently, in a way that uses a lot of power and space?
  • AStonesThrow 2y ago
    I was just pondering this today, in terms of how much data and objects I create in a Google or Microsoft account, for example, and then they create a burden of cost and maintenance for me down the road. Especially deleting old emails and photos. That's arduous and sometimes poignant as I flush away my personal life story. Cloud services make difficult and sometimes even byzantine processes for deleting stuff, and it's often impossible to operate en masse in order to clean up swaths of stuff quickly and efficiently. It's in their interest to retain everything at all costs, because more used storage can mean more profits. Cloud services also profit from unused storage, because if they're charging $20/year to 100,000 users who use 2% of their storage space, ka-ching! It irks me to this day that standard or even advanced filesystems don't include "expiration dates" or "purge dates" on file objects. Wouldn't it be logical, if an organization has a "data-rentention policy" that mandates destruction after X date, that the filesystem simply purges it automatically? Why does this always get delegated to handmade userland cron jobs? Moreover, to my knowledge, nobody is really interested in devising a way to comb through backup media in order to [selectively] destroy data that shouldn't exist anymore. Not even the read-write media! Google is now auto-deleting stuff like OTP SMS messages. I'd love it if auto-delete could be configurable account-wide, for more than just our web histories and Maps Timeline and stuff. Unfortunately, to "delete" cloud data means it still exists on backups anyway. But without deleting data in your cloud account, it becomes a juicier hacker target as it ages and accumulates personal stuff that shouldn't fall into the wrong hands. Likewise for any business, it behooves them to delete and destroy data that shouldn't be stolen. At least move it offline so that only physical access can restore it? I will say that modern encryption techniques can make it easy to "destroy" data, simply by destroying the encryption keys. You can quickly render entire SSDs unreadable in the firmware itself with such a command. Bonus: sometimes it's even done on purpose! But even deleting data presents a maintenance cost. So if 90% of an org's data is indeed crap, then 90% or more of your processing resources are going to be wasted on sifting through it at some later date. Imagine when your file formats and storage devices are obsolete, and some grunt needs to retrieve some record that's 30 years old, and 90% of your data was always crap. That grunt is hopefully paid by the hour. We really had this happen at a few of my jobs, where we had old reel-to-reel backup tapes and it was difficult enough to load the data into a modern SunOS machine. https://m.xkcd.com/1683/ https://m.xkcd.com/1683/
• jongjong 2y ago
  True, when it comes to data, there is a strong case to be a compulsive hoarder. It's cheap to store; often free and data is often marginally more useful and interesting than most generic mass-produced old junk which sits in garage gathering dust.
  • InsideOutSanta 2y ago
    Technology for sorting through unordered piles of data also gets better over time. When I started storing digital photographs, face detection wasn't a thing, but it is now, so those tens of thousands of photos from the 90s suddenly became more useful to me.
• mlinhares 2y ago
  If I knew the data i was going to need in the future i'd be in the future predicting business. But there's an important piece there that is about data that should not have been stored in the first place. All the big data bullshit made people believe every single piece of data you could possibly store from a user is data that you should store and this is both a huge waste of resources and a huge liability, because now a data leak with useless to you PII could completely bankrupt your business.
• 2muchcoffeeman 2y ago
  Photos of kids? Maybe that one will be The One that we end up framing? Look through the pictures with your kids (and with your kids) and pick out the best ones. Delete the rest.
  • philips 2y ago
    The photo apps really don't have a good "sorting through the shoe box" UX. It would be great if I could in Google Photos, for example: 1. Bulk favorite photos in Google Photos for long term archiving 2. Set a retention policy of months/years for photos based on metadata. 3. Have a UX to quickly sort the week or months photos via a swipe left/right UX.
  • stephen_g 2y ago
    I go back and look at old photos all the time (like when I think of a trip, and look back to it, and then just keep scrolling for a while). The pictures bring back more memories and that's to me a great joy - and on the other hand I find no value in deleting the photos that are not "the best" (I'm talking about good enough photos, I delete blurry, useless or duplicate ones immediately) - what would it gain? A few dollars less storage cost over the next decade? The satisfaction of digital cleaning?
  • foobahify 2y ago
    [flagged]
  • Double_a_92 2y ago
    The problem with that is that what you might find important or beautiful changes with time. That random photo of your messy living room might be trash right now, but beloved in 20 years when you want to remember your old home.
• userbinator 2y ago
  "Better to have and not need, than need and not have." Storage is cheap, very cheap.
• palata 2y ago
  I think you miss the point of the article, which precisely says that we store a ton of data we don't need to store in the first place. Photos of kids are obviously not part of that "crap" (even if we have too many of them and it would be worth triaging for our own sake). The question is: what makes for most of that data? Is that all business records, or is that storing stuff because we can? I've worked in multiple startups, all were tracking users as much as they could. Adding tools that collect data is easy, and storing that data is cheap. "We may need it later". Never needed any of that crap, and it was invading our users' privacy. In any case, I think the article raises a good point: it's so cheap to store crap that we don't even think about it. And it's bad for the environment. Just like it's so cheap to take a plane that we don't think about it, even when taking the train takes almost the same time.
• InsideOutSanta 2y ago
  The article touches on this issue by pointing out that: >The Cloud is what happens when the cost of storing data is less than the cost of figuring out what to do with the crap But I think that's wrong. The actual issue is that you often can't figure out "what to do with the crap" because the difference between useful data and crap data is determined at the point in time when you need it, not when you store it. I'm relatively careful with deleting data, but even so, there were countless instances where I thought something was no longer needed and deleted it, only to figure out that, a month later, I needed it after all. Also, the article has a few anecdotes like this: >Scottish Enterprise had 753 pages on its website, with 47 pages getting 80% of visits But that is completely orthogonal to the question of what is "crap." The fact that some data is rarely needed does not mean it's crap. If anything, the fact that it is rarely needed might make it more valuable because it might make it less likely that the same data can be found elsewhere.
  • ghaff 2y ago
    I do cull photos and docs but it takes effort and you will make mistakes. On the other hand, intelligent culling makes it easier to find things. It’s a trade off especially with digital where there’s not that much cost associated with holding onto things relative to the physical world. My whole house is basically in storage because of a kitchen fire and I’m planning to be pretty selective about what I move back in.
    • InsideOutSanta 2y ago
      >intelligent culling makes it easier to find things That's true, but things get easier to find over time. I have thousands of digital photographs from the 1990s that I burned on CDs and then copied to a NAS later. Today, for the first time, they're actually searchable because they're now indexed in Immich. So they're sorted by the person in them and searchable by their contents. If I had culled the photos back then, I would have lost many photos that have become searchable (and thus valuable) only recently.
      • ghaff 2y ago
• Pooge 2y ago
  Are you going to be able to find the data you're looking for if judges demand it? Data that ends up in that sea of crap is very often poorly labeled. Data that you cannot find again is useless. If you take more than 3 minutes to find a picture you wanted to show me, it doesn't deserve to be showed anymore.
• mft_ 2y ago
  > …I think it's important to remember that much of the data is stored because we don't know what we will need later. This is the basic argument that every hoarder uses to justify their hoarding.
• 1vuio0pswjnm7 2y ago
  "Photos of kids? Maybe that one will be The One that we end up framing? Miscellaneous business records? Maybe those will be the ones we have to dig out for a tax audit? Web pages on government sites? Maybe there will suddenly be an interest in obscure pages on public health policy if a global pandemic happens." Perhaps I am missing something, but these examples all sound like candidates for _offline_ storage, for which no third party custodian or data center is required. The price of large capacity NVMe SSDs continues to fall. The amount of energy, the resource requirements, not to mention the environmental and community impact, of running a car insurance company are miniscule in comparison to that of running a data center.
  • wongarsu 2y ago
    Offline storage is only easy on the very small and the very large scale. On the very small scale you can get archival-grade CD/DVD/Blurays. On the very large scale you dedicate climate-controlled rooms to an automatic tape library. In between is a vast gulf where your only good options are disks that you have to occasionally spin up to check integrity, have the disk trigger a data refresh if it's an SSD, and replace any disks that failed and rebuild their data from redundancy (raid or whatever you prefer). HDDs die even when off [1] and flash storage in SSDs is only rated to hold data for three years without power. Sure, you can roll this yourself, but it can easily go wrong. Easier to either keep the disks spinning or pay someone with a tape archive (e.g. AWS deep glacier) 1 https://www.tomshardware.com/pc-components/storage/twenty-percent-of-hard-drives-used-for-long-term-music-storage-in-the-90s-have-failed https://www.tomshardware.com/pc-components/storage/twenty-pe...
• jocaal 2y ago
  For your car note, insurance is only worth it if the thing being insured can ruin you if something were to happen to it. In the case of cars, you can potentially get ruined from the value of the other party's car. But if you live somewhere where most people drive normal cars, it might be worth it not having insurance. Our culture of insure everything, from your iphone to house is a market failure. If your house were to burn down, the value of the asset is still mostly concentrated in the land.