14 ms·
Digital hoarders: Our terabytes are put to use for the betterment of mankind
- sneak 7y agoJust picked up two more 36 bay sata 4Us for the garage on eBay - only $500 each! Shoot me an email if you have bulk old/used sata drives for cheap. I’m buying.
- leonroy 7y agoAre multiple used smaller drives really more cost effective than fewer new larger drives once you take into account cost of power, HBA, backplane and lifetime of used drives? I'm a data hoarder too (proof): https://i.imgur.com/j9jk9OP.jpg https://i.imgur.com/j9jk9OP.jpg But always been curious about cost so let's do some sums... With a used 3TB setup. Assume 12x3TB = 36TB storage. A reasonable 2U server would be needed to support that which would include a backplane and HBA. Power consumtpion would be 12x 5W (HD) + 10W (HBA) + 65W (server). Total is 135W. Power cost in UK for one year at 24/7 is approx £176. Assuming used 3TB drives give us two years of reliable operation and cost £30 each to buy (going price on ebay a few months back) total cost to buy would be £360 of just the drives (not including the server). Add cost of electricity for two years operation and we get a total of £712. Average the cost per year (£712/2) and we come to £356 per annum for the 3TB drive setup. Now with new 12TB drives. 3x12TB consumes 3x 5W of power + 30W for the server (no backplane or HBA needed). Total is 45W. Power cost in UK for one year at 24/7 is approx £60. Cost of drives would be 3x£300 = £900. Assume we get 5 years of life out of these new drives, plus electricity costs of £60 - over 5 years total would be £1200. Averaged the cost per year (£1200/5) and we come to £240 per annum for the 12TB drive setup. (I have not counted the additional cost of the 2U server for the 3TB scenario since that would complicate the math since servers tend to last longer than drives.) So I'm wondering what the appeal of large servers with lots of small drives is when back of the envelope math suggests it costs more in the long run. Unless of course it's because you like those blinkin' lights :) https://i.imgur.com/4ypumBQ.jpg https://i.imgur.com/4ypumBQ.jpg
- sokoloff 7y agoWhat level of redundancy are you calculating? I’m not comfortable with less than RAID6/Z2 on large drive arrays, so 3 12TBs is only 12TB usable. 12 3s I’d probably do 2 6-drive volumes, each with 2 drive redundancy, so 2x4x3 or 24 TB usable. That’s enough to flip the math.
- m463 7y agoThe calculations might depend on whether the drives are powered on all the time.
- omgwtfbyobbq 7y agoThat's my sense too. Unless someone has to have access to that data relatively quickly all the time I don't see why the system couldn't be left off. Even if the data is accessed frequently, lets say once every day or few, it could still be left off most of the time if someone is OK with waiting a few minutes for everything to power up.
- stock_toaster 7y ago> Are multiple used smaller drives really more cost effective than fewer new larger drives once you take into account cost of power, HBA, backplane and lifetime of used drives? Resilver times for drive replacement is generally much faster for smaller drives.
- sneak 7y agoOh, I got one of them to be a disk duplication machine for the next Defcon. Only one of them will be in full file server service, and I intend to do an 8-20 disk array of 12TB disks. That will still leave me with over a dozen slots for a secondary array of cheap refurb disks of any kind I can come across. Power is cheap here; my current hobby bottleneck is drive cost. Ideally I’d be able to buy a few dozen extremely cheap disks that have been pulled from service that someone wants to dump as a lot.
- throw32098 7y agoI do not like the shaming tone of this article. Petabyte server today is a small box that fits anywhere. Meanwhile some people have hundreds of shoes or clothes, but nobody calls them "hoarders". Anyway, I have offline storage, because my internet connectivity is slow. I do not like waiting, so all my content is downloaded overnight with a few cron scripts.
- mft_ 7y agoPeople with lots of clothing and shoes and other things absolutely might be criticised for their behaviour, and may be called hoarders. (I have direct personal experience of such people.) There is definitely a mindset —at its most extreme, a pathology— associated with hoarding behaviour. Digital hoarding may mostly avoid the negative life consequences of hoarding —as you say, you can store a lot of data in a relatively small physical space— but this doesn’t change the nature of the psychology behind the behaviour. (Although I’d wonder how many digital hoarders really have a small neat modern NAS as you imply, versus a room full of old racks?) Interestingly, the rationalisation given in the headline (“betterment of mankind”) isn’t so very far away (although maybe more grandiose) than the rationalisations that physical hoarders offer to justify their behaviour, around the perceived intrinsic value of their belongings.
- pessimizer 7y ago> this doesn’t change the nature of the psychology behind the behaviour. The "psychology" behind the behavior isn't the problem with hoarders, it's the actual hoarding. The reason we try to change hoarders' thinking is because of the physical consequences of their hoarding - we define this as illness because of its distressful results, if it had no consequences it wouldn't be illness. edit: I think you're accidentally using circular logic to rationalize a bit of magical thinking (person who obsessively collects garbage is ill, therefore person who obsessively collects anything is ill.) It's not the obsession that's the problem, it's the garbage and the distress.
- Mediterraneo10 7y agoI know that some of the members of the organized data-hoarder community are spending huge amounts of money on disks month after month, and it really bothers their spouses. So, even if there is no physical mess being created in their homes, their obsession is still affecting their lives and other people around them.
- hashhar 7y agoI "hoard" because there is a lot of media I own that I can't find elsewhere. I had stopped buying physical media and ripping them to my disks in the last 2 years and moved full time to streaming - Spotify for discovery, Apple Music for maintaining the library (both are cheap here in India). But recently I saw that a lot of music I owned was missing from both services. That made me decide to go back to my old ways. Also, having spent the effort in cataloging and ripping my collection has made me intimately familiar with my collection and has been an activity I can partake in to relax/wind-down.
- Mediterraneo10 7y agoI also remain a keen member of filesharing communities because I want to actually own my music and films, and I know it could disappear from streaming sites. But another advantage of having my own collection is that it lets me enjoy all this media at my country cottage, where the only internet access is 4G that gets throttled to hell after just a few gigabytes per month. I can torrent Blu-ray images in the city with my fiber connection, and then keep a backup hard drive at my country home. That has proven a lifesaver during this long virus lockdown when I am stuck in the countryside and can’t travel back and forth.
- timrichard 7y agoI'm in a similar position. The nice thing about the NAS is that it doesn't have a "leaving soon" section. Another good thing about using it for most video consumption is that disruption of internet service becomes a temporary issue upstream, rather than game over. Quality downgrade because of a drop in bandwidth doesn't happen either. Similar to you, I'm converting CD to Flac one cherished album at a time, taking a bit of time to enjoy it and tweak the artwork, etc.
- hashhar 7y agoHow do you handle listening to songs on mobile devices across the internet? I'm currently using Resilio Sync (trying to hack around for selective sync support in Syncthing) meanwhile to get albums I want onto my phone. I haven't yet looked into things like Plex, Kodi or Emby.
- shantara 7y agoI don't understand the negativity the article associates with "hoarding" data. It is the nature of the internet that any content or website you see can disappear at any moment without a prior warning or an explanation. From an author deleting their earlier work they were later dissatisfied with, to copyright claims, to people losing interest in their hobby project, to site owner forgetting to extend their domain, to website becoming impossible to find in search engines. It always pays to be properly paranoid and have backups in case the worst happens. Storage is cheap, so what's exactly the downside? My personal archive may not bring "the betterment of mankind", but it saves me from disappointment when looking up older content that has since disappeared, and also allows me to help other people find what they're looking for.
- zouhair 7y agoWhen you think about it is quite insane that you can find rock tablets from thousands of years back with text on them but we can lose all the data on the internet just like that.
- vorpalhex 7y agoWe really should go back to having people copy things on stone tablets, or at least encode them on sapphire discs..
- freehunter 7y agoIs that insane? It’s not like every stone tablet or parchment or scroll survived, many were lost. Some were lost immediately, some after a few years, some after a few centuries. Not every word on the Internet survives for long but not every word written on stone tablets lasted very long either.
- timrichard 7y agoOne that caught me three or four times was Udemy, where I'd paid for a course heavily discounted on sale meaning to study it at some point later. Much later, I'd go to the course page and find that the author had withdrawn it and Udemy were offering me a refund of the discounted price. All within their rights, but it feels annoying.
- joyj2nd 7y agoI hoard because shows, youtube videos (youtube-dl) and websites disappear. I use some GTD methods to organize my data (e.g. folders a,b, c...z) I use recoll to find data https://www.lesbonscomptes.com/recoll/ https://www.lesbonscomptes.com/recoll/ I would like to use an online backup service but I would like to have my data encrypted. I tried https://www.hetzner.com/storage/storage-box https://www.hetzner.com/storage/storage-box But data throughput to an encrypted storage was abysmal. I tried several methods, even included running an encrypted filesystem in a 10TB file. I tried https://www.cryfs.org/ https://www.cryfs.org/ but nothing worked good and reliable. I looked into https://www.infomaniak.com/en/kdrive/ https://www.infomaniak.com/en/kdrive/ but the encryption method seems like snake oil. Any suggestions?
- mschwaig 7y agoOn Linux ZFS has has an encryption feature developed by datto.com. They developed this feature for their own hosted encrypted backup service. If you have you data encryped in ZFS like that locally, you can use zfs send to send them encrypted snapshots of your data while only ever having to transmit the diffs. I am in the process of setting up something like this with my own backup server right now. I'm not 100 % confident yet that I'll get it working though. Have you looked into tarsnap.com?
- nnt38 7y agoI use rclone for that purpose. https://rclone.org/crypt/ https://rclone.org/crypt/
- tga 7y agoYou can keep a local Borg repository (https://www.borgbackup.org https://www.borgbackup.org) and copy it anywhere in the cloud with something like https://rclone.org https://rclone.org. To quote the website, BorgBackup gives you: - Space efficient storage of backups. - Secure, authenticated encryption. The encryption is performed on your machine, and you don't need any special features from the provider(s) beyond simply storing a lot of small chunks.
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- JKCalhoun 7y agoYes, hoarder here. Started with scanning old family photos, progressed from there. Was psyched to find youtube-dl after I had discovered many of the YouTube videos I had playlisted years ago, are now missing. Perhaps not very interesting things to most: research footage testing some aspect of the X-20 Dynasoar space plane, etc. Many things gone missing I don't even know what they were since YouTube doesn't indicate what the missing video was. So, yeah, hoard it is. "Japanology"? Got all I could find. "How It's Made?" Yep. Live footage of the band "The Embarrassment" sleeps on my drives as well....
- netjiro 7y ago2nd youtube-dl Feels insane to store youtube vids, but quite often they are simply gone for one reason or other after a few years.
- germinalphrase 7y agoMy only regret about digitizing my music over the years is that I didn’t care about the quality as much when I was younger. I’m older now - with better equipment - and would love if everything was FLAC rather than mixed .mp3 Years ago, I was a member on a private file trading site that only exchanged old and unusual art films. The kind of stuff that is entirely non-viable commercially and will never be found on a streaming device. It shut down - but I miss having that small corner of the Internet.
- jccalhoun 7y agoI'm definitely a digital hoarder but not on the level of the people in the subreddit. It started when I was in grad school and a web site I wanted to reference went down before I was able to archive it and it wasn't in archive.org and youtube wasn't a thing. So I started downloading copies of anything I thought relevant to my research. I know I have some rare stuff that is gone from the internet. One of these days I'll get around to posting them on archive.org. Until them I practice 3-2-1 backup to make sure I don't lose anything.