6 ms·

Internet Archive is having its own legal troubles at the moment, might no longer be the archival safe-haven we'd hope...

• bombcar 3y ago
  Storage is cheap, if there’s something you’re interested in it’s time to become a data hoarder.
  • sekh60 3y ago
    Datahoarder here, the drives are relatively cheap, but the overall cost to do it right is expensive. I have a somewhat durable setup (about 370TiB raw, replica count 3, 5 node Ceph cluster). And just that before backups I work total cost to be about $1000CAD per usable TiB. I don't shuck drives though, that could probably bring it down to 800.
    • MaKey 3y ago
      So you've invested six figures into a private data hoarding setup? Not bad.
      • sekh60 3y ago
        I guess it is around six figures by now. Better not let my wife realize that, lol.
    • edude03 3y ago
      3 replicas as in 3 complete copies of the data? Wouldn’t something would erasure coding be cheaper?
      • tremon 3y ago
        Erasure coding is only used for local duplication. If you have cross-site or cross-machine copies, you really don't want the speed of the rebuild to be dominated by available network bandwidth.
        • bombcar 3y ago
          For many datahoarding things “when I drive over to my friend’s house and retrieve the drives” is often enough for replicating restore speed.
        • edude03 3y ago
          I guess it depends. I run minio with erasure coding but I also have 2x100gb interconnects between nodes and even with NVMEs I’m not noticing any performance issues with rebuilds. That said with most disks I could see how this could be a problem
          • sekh60 3y ago
            I wish I had 100Gbps interconnects! My network is only 10Gbps.
      • sekh60 3y ago
        There are three copies of the data as part of ceph's replication. Note this is not replication for backup purposes, erase a file and all objects composing the file are gone. Ceph replication is for comparing checksums and that for guaranteeing integrity.
    • bombcar 3y ago
      There is archival quality data hoarding like you’re doing, and then there’s “this YouTube channel is interesting and I don’t want to lose it, yt-dlp to a USB drive”. Not being able to do the former doesn’t mean you shouldn’t do the latter, especially if the risk is “real”. (My system is relatively cheap as it only consists of a ZFS mirror array and some offline drives; a major fire might take me out but the stuff that should survive that is offsite.)
      • sekh60 3y ago
        For sure, usb sticks have their purpose! I keep secondary copies on other more temporary media of home videos and pictures, the actually important stuff. Our neighbourhood just recently got fiber to the home, so I will be going from 30Mbps upload to 940Mbps upload in a couple weeks. Then I can get some proper offsite backups going.
    • detourdog 3y ago
      Can you share depth and breadth of your archive?
      • sekh60 3y ago
        My archive is composed of cloud images and volumes for virtual machines for an OpenStack homelab. Then around 90TiB of movies, shows, YouTube videos, etc. Remuxes when available. About one TiB of books. Do I use it? My wife watches media on jellyfin, but I just have a compulsion to gather data and hoard it. I restrict myself to a 36U server rack at least, so it doesn't take up much physical space.
        • detourdog 3y ago
          I have no judgement, I was just curious.
          • sekh60 3y ago
            I didn't interpret you as judging. It is somewhat pathological I admit. The cluster is a bit more than 87% full, at 90% it slow to a crawl. I just asked my wife this morning if I could buy another hard drive (I buy 20TB seagate exos), and she suggested we delete some stuff. I find deletion of files when I do it psychologically stressful. So it is something I should get addressed, but I suppose we should be thankful I don't have dead cats lying around, lol.
            • detourdog 3y ago
              I say you have saved it this long keep going. That drive seems to cost around $350. I use SSD drives for working storage and I use platters for archival. I believe the platters have a better recovery methods. Do you have any opinions? Thanks for more information.
              • sekh60 3y ago
                I use enterprise SSDs for cloud images and VM volumes, I have about 7TiB useable of that. I don't know how data recovery from an SSD is, I think for a lot of issues you would have to replace the logic board and hope some sorcery works to figure out the ordering of data with wear leveling algorithms. Rust one can often recover data with something like ddrescue. I don't worry too much, with replica count of 3 for both flash and rust I can lose two drives with the same 4 MiB block and be fine, if the failing drives are within a chassis I can lose two whole nodes before I have data loss. I wish I had access to a tape library though, just it is hard to justify the upfront cost. My cluster has organically grown over the last ten years or so, making a huge near $8000 single purchase for a new tape drive, plus however much automatic tape rotation costs (I presume I need some sort of tape library for that? I have never used tape before) would be hard to justify.
    • hgsgm 3y ago
      $1000/TiB ?! Why 20x overhead over raw storage cost? May as well use
      • sekh60 3y ago
        That is all in, including motherboards, RAM, chassis, CPUs, etc. then replica count three as opposed to erasure coding. That replica count alone is 3x the cost. And TiB, not TB, and Canadian prices, with shipping from the US for Supermicro parts. Each node is about 6-7000 CAD before drives and I use 6 enterprise rust and two enterprise SSDs per node. Cost would be lower of I built nodes with more drive capacity, but I want to keep nodes to 2RU to have more nodes in 36RU.
    • tough 3y ago
      shuck was repurposing external drives popping them out and using the internal drive?
      • sekh60 3y ago
        That is correct.
  • hylaride 3y ago
    Storage is cheap, but moving it is expensive.