6 ms·

https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_say_you_wanted_to_back_up_the_internet/ https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_sa..

https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_say_you_wanted_to_back_up_the_internet/ https://www.reddit.com/r/DataHoarder/comments/h02jl4/lets_sa... I found this reddit thread from /r/DataHoarder about backing up the internet archive particularly interesting, given the circumstances

numpad0 2y ago
50 PB * $0.014/GB = $0.7M. $0.014/GB is from[1], bare drive cost without chassis, power, or redundancy. 1: https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/
- Aachen 2y ago
  How long does an average hard drive last? You'd have to spend that 700k every that many years (plus the extra bits you mentioned). Quite an operation actually
  - everforward 2y ago
    I actually find that fairly tame. For a point of comparison, Wikipedia gets ~$150M in revenue a year, an "asset rise" (I presume this is what non-profits call profit?) of ~$15M a year, and is sitting on about a quarter billion in the bank. Not that they want to, but I think Wikipedia could fund this using their current donations if they wanted. Hell, I almost wonder if one of the big storage providers would do it for free if they could do it in their staging environment so they get real traffic. It would be less good than real backups, but extra copies are still extra copies even if they're unreliable.
    - Aachen 2y ago
      You're right, I guess it is tame and achievable so far as organisations go. I was imagining trying to get some friends together to have a decent percentage of the IA backed up, but that seems out of reach based on this napkin math. Not that that is necessarily demotivating, but it's going to depend on a lot of people intuitively seeing the value and keeping up their share
      - everforward 2y ago
        Yeah, as a sort of pet project I don’t think backing up the whole thing is possible. You might be able to back up a significant portion of the unique data in IA if you limited it to text files. I think they probably have the highest information to file size ratio. It’s also probably the most likely to already be back up, though. Interesting issue; you might also get somewhere by cutting the 50TB up into 10GB torrents (or 100GB or whatever, something reasonable for a consumer hard drive) and maybe adding a script that checks the torrent swarm stats to recommend a torrent to download. Something where I run it, tell it I want to let it use 600GB, and it hands me torrent files for the least seeded 600GB. Maybe a super basic web UI so people can see how well backed up it is? Unsure if people would sign on or not; I probably would. I’ve got 10 or so TB of NFS I’m not using I could chuck at it. I would guess there are other data hoarders out there who would do the same, but only if it were somewhat easy. I’m probably not going to volunteer to do an hour of rtorrent cleanup a week to make sure I’m backing up the right things.
        rolandog 2y ago
        I think a part of the scope of this project may have already been solved by The FreeNet Project (now HyphaNet) [0]. [0]: https://www.hyphanet.org/ https://www.hyphanet.org/
    - Intralexical 2y ago
      They should probably consider it, really. A good portion of the text on Wikipedia relies on Wayback Machine links to remain verifiable. If they lose that, I guess the editors might have to comb every page for information which would need to be either resourced or deleted.
  - codemac 2y ago
    > How long does an average hard drive last? This is a great question, and a state of the art kind of thing. HDDs are sold with a lifetime drive read/write amount and power cycle warranty, along with usually some environmental operating envelope. read/write relates to the quality/space of the platter, power cycle is usually the actuator & read/write head being reseated/wearing out. Environment is the same as all other devices in a DC. Most folks replace drives when they die (reads/writes stall or return garbage), or when the warranty runs out. Some will pay for a warranty exception, and some will just use the drive outside of warranty. Depending on how you use the drive, what environment it's in, etc changes how much you can push things. I'd say anywhere from 4-8 years, depending on how it's used. In many cases it can be cheaper to have a worse environment for your fleet (thus using less power on hvac) and replace devices more frequently.
    - bigiain 2y ago
      There's also the fact that hard drive capacities keep increasing and increasing significantly faster that the power required, and sooner or later for very long term storage it'd become cheaper to migrate all your data from those 5 year old 4TB drives to more modern 16TB ones. That's assuming you want hot access to the data and don't plan on spinning them down as soon as you've written to them, like you'd do for a cold backup of the whole IA. I remember for a long time (I'm talking 20-ish years back here), every hard drive I bought had double or more the capacity of every drive I'd ever bought previously combined. My first ever 40MB (yes, megabyte) drive got upgraded to an 80MB one, that got updated to a 250MB one, then a 750MB, and then a whopping 2GB drive (how would I _ever_ fill that up???) - and so on. That's slowed down some, but I'm currently starting to think about upgrading my 8TB drives (Raid1 pair) with 20TB drives when the prices start to drop a bit more.
      - codemac 2y ago
        Just don't forget that IO matters. Drives do 140-220MB/s depending on the LBA distance of the readhead, and that's not really changing. 160MB/s is very common. So your 8TB drives, assuming 1MiB writes with a 20ms latency and 160MB/s, you can rewrite the drive ~155 times/year. At 20T this drops to ~62 times/yr.
  - bigiain 2y ago
    If this is a backup, you don't need it to be powered up and available 24x7. So the question becomes more like "how long does an average hard drive last while powered down and still reliably be able to power back up and be read?". I'm fairly sure that is a lot longer than the single digit years that'd be the probably answer to your question. I wonder if there are useful guidelines for long term storage of powered down hard drives? My gut feel is the major failure modes would be electrolytic capacitor failure, bearings sticking as the lubrication ages, and obseleting of the interfaces. I wonder how hard it'd be to find hardware that'd read my Mac SCSI hard drives from 25 years ago?
    - fn-mote 2y ago
      > I wonder how hard it'd be to find hardware that'd read my Mac SCSI hard drives from 25 years ago? Easy… that original Mac is sitting in my basement and it worked like a charm last time it was powered on 4 years ago.
  - tivert 2y ago
    > How long does an average hard drive last? You'd have to spend that 700k every that many years (plus the extra bits you mentioned). Quite an operation actually You'd have to spend a lot more, because with that many drives, you need redundancy now.
    - Aachen 2y ago
      True, that would be an up front cost. At the same time, the IA is still live. This initial expense can be softened by building up redundancy over some years rather than trying to do everything at once
      - tivert 2y ago
        > True, that would be an up front cost. At the same time, the IA is still live. This initial expense can be softened by building up redundancy over some years rather than trying to do everything at once I think with that many drives, you'd be losing them constantly, and I suppose you wouldn't know which ones until later (assuming you're doing an offline backup, if you aren't you have to factor in power costs).
  - jolmg 2y ago
    For archival, if you use tape, it comes out cheaper (~225k) and ought to last longer (~30 years).
  - viraptor 2y ago
    Backblaze keeps good stats. https://www.backblaze.com/cloud-storage/resources/hard-drive-test-data https://www.backblaze.com/cloud-storage/resources/hard-drive... 1.71% a year failure rate if you care for the hardware as much as they do.
    - hsbauauvhabzb 2y ago
      But that number would increase year on year, a 10 year old drive is far more likely to fail than a 1 year old drive
      - viraptor 2y ago
        Internet archive is going for long enough that I'd expect it to stabilise by now. If you replace enough of the drivers, you get a good mix.
  - nephanth 2y ago
    For that purpose you might want to use magnetic tape like they use in long term archival services They are cheaper per Gio, and last significantly longer
- ks2048 2y ago
  Interesting to compare their stated drive $/GB to their B2 offering: $6/TB/mo for "pay-as-you-go", hard-drive price: $0.014/GB B2 price (12*6/1024): $0.070/GB/year
  - alwayslikethis 2y ago
    Electricity, bandwidth, and generally running a business is not free. Also for these pay-as-you-go setups you'd need a considerable amount of free space available on demand. That said, it's not an especially cheap option. Hetzner has storage boxes for EUR 2.5/TB/mo (in fixed 5 and 10TB boxes though)
    - ks2048 2y ago
      Yeah, I wasn't trying to point out that it's a bad price. I think it's pretty good: same price for two years with all the maintenance.
- PostOnce 2y ago
  IA stores lots of redundant stuff in 5 file formats and none of them are particularly well-compressed, I think. There are (big) savings to be had, but maybe figuring that out (software dev and compute time) isn't worth it?
nikisweeting 2y ago
It's been tried several times, but it's hard because it's such a massive quantity of data. The IPFS backup never really got off the ground. They have their own backups which I think is good enough for now unless someone plans on donating a few hundred million.
- vincentpants 2y ago
  Oh no! I didn't know their IPFS initiative didn't pan out. What happened to it? I am surprised how hard it is to google. I remember interviewing for a role on that team at the archive to help move it to filecoin. Was so happy to hear that the effort was underway to decentralize their datastore. We need this more than ever.
  - nikisweeting 2y ago
    There are people still working on trying to make it happen but it's just a collosal amount of data and filesystems are notoriously hard, so it's very slow going. From my own personal experience doing distributed archiving with no relation to Archive.org, Filecoin/IPFS's UX isn't quite there yet. They still don't let you serve data to the network from a normal filesystem, you have to let their system ingest all of your stuff so you end up double-storing data or you have to give into everything being stored as inscrutable binary blobs. That's why I still haven't integrated ArchiveBox with IPFS/Filecoin/Storj, let my data live in a normal filesystem dammit!
    - Aachen 2y ago
      > They still don't let you serve data to the network from a normal filesystem, you have to let their system ingest all of your stuff so you end up double-storing data or you have to give into everything being stored as inscrutable binary blobs. I don't understand this part. What data would you have to give them? Why can't it just live next to your stuff on your OS' filesystem?
      - nightpool 2y ago
creer 2y ago
Backup / duplication is not an easy project for sure. But meanwhile for now IA is a single organization operating under one legal system. And one technical setup, would be relevant today. That's a major weakness.
EamonnMR 2y ago
Suppose we each backed up sites we cared about rather than trying to mirror the whole thing...

numpad0 2y ago

50 PB * $0.014/GB = $0.7M. $0.014/GB is from[1], bare drive cost without chassis, power, or redundancy. 1: https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/

Aachen 2y ago
How long does an average hard drive last? You'd have to spend that 700k every that many years (plus the extra bits you mentioned). Quite an operation actually
- everforward 2y ago
  I actually find that fairly tame. For a point of comparison, Wikipedia gets ~$150M in revenue a year, an "asset rise" (I presume this is what non-profits call profit?) of ~$15M a year, and is sitting on about a quarter billion in the bank. Not that they want to, but I think Wikipedia could fund this using their current donations if they wanted. Hell, I almost wonder if one of the big storage providers would do it for free if they could do it in their staging environment so they get real traffic. It would be less good than real backups, but extra copies are still extra copies even if they're unreliable.
  - Aachen 2y ago
    You're right, I guess it is tame and achievable so far as organisations go. I was imagining trying to get some friends together to have a decent percentage of the IA backed up, but that seems out of reach based on this napkin math. Not that that is necessarily demotivating, but it's going to depend on a lot of people intuitively seeing the value and keeping up their share
    - everforward 2y ago
      Yeah, as a sort of pet project I don’t think backing up the whole thing is possible. You might be able to back up a significant portion of the unique data in IA if you limited it to text files. I think they probably have the highest information to file size ratio. It’s also probably the most likely to already be back up, though. Interesting issue; you might also get somewhere by cutting the 50TB up into 10GB torrents (or 100GB or whatever, something reasonable for a consumer hard drive) and maybe adding a script that checks the torrent swarm stats to recommend a torrent to download. Something where I run it, tell it I want to let it use 600GB, and it hands me torrent files for the least seeded 600GB. Maybe a super basic web UI so people can see how well backed up it is? Unsure if people would sign on or not; I probably would. I’ve got 10 or so TB of NFS I’m not using I could chuck at it. I would guess there are other data hoarders out there who would do the same, but only if it were somewhat easy. I’m probably not going to volunteer to do an hour of rtorrent cleanup a week to make sure I’m backing up the right things.
      - rolandog 2y ago
        I think a part of the scope of this project may have already been solved by The FreeNet Project (now HyphaNet) [0]. [0]: https://www.hyphanet.org/ https://www.hyphanet.org/
  - Intralexical 2y ago
    They should probably consider it, really. A good portion of the text on Wikipedia relies on Wayback Machine links to remain verifiable. If they lose that, I guess the editors might have to comb every page for information which would need to be either resourced or deleted.
- codemac 2y ago
  > How long does an average hard drive last? This is a great question, and a state of the art kind of thing. HDDs are sold with a lifetime drive read/write amount and power cycle warranty, along with usually some environmental operating envelope. read/write relates to the quality/space of the platter, power cycle is usually the actuator & read/write head being reseated/wearing out. Environment is the same as all other devices in a DC. Most folks replace drives when they die (reads/writes stall or return garbage), or when the warranty runs out. Some will pay for a warranty exception, and some will just use the drive outside of warranty. Depending on how you use the drive, what environment it's in, etc changes how much you can push things. I'd say anywhere from 4-8 years, depending on how it's used. In many cases it can be cheaper to have a worse environment for your fleet (thus using less power on hvac) and replace devices more frequently.
  - bigiain 2y ago
    There's also the fact that hard drive capacities keep increasing and increasing significantly faster that the power required, and sooner or later for very long term storage it'd become cheaper to migrate all your data from those 5 year old 4TB drives to more modern 16TB ones. That's assuming you want hot access to the data and don't plan on spinning them down as soon as you've written to them, like you'd do for a cold backup of the whole IA. I remember for a long time (I'm talking 20-ish years back here), every hard drive I bought had double or more the capacity of every drive I'd ever bought previously combined. My first ever 40MB (yes, megabyte) drive got upgraded to an 80MB one, that got updated to a 250MB one, then a 750MB, and then a whopping 2GB drive (how would I _ever_ fill that up???) - and so on. That's slowed down some, but I'm currently starting to think about upgrading my 8TB drives (Raid1 pair) with 20TB drives when the prices start to drop a bit more.
    - codemac 2y ago
      Just don't forget that IO matters. Drives do 140-220MB/s depending on the LBA distance of the readhead, and that's not really changing. 160MB/s is very common. So your 8TB drives, assuming 1MiB writes with a 20ms latency and 160MB/s, you can rewrite the drive ~155 times/year. At 20T this drops to ~62 times/yr.
- bigiain 2y ago
  If this is a backup, you don't need it to be powered up and available 24x7. So the question becomes more like "how long does an average hard drive last while powered down and still reliably be able to power back up and be read?". I'm fairly sure that is a lot longer than the single digit years that'd be the probably answer to your question. I wonder if there are useful guidelines for long term storage of powered down hard drives? My gut feel is the major failure modes would be electrolytic capacitor failure, bearings sticking as the lubrication ages, and obseleting of the interfaces. I wonder how hard it'd be to find hardware that'd read my Mac SCSI hard drives from 25 years ago?
  - fn-mote 2y ago
    > I wonder how hard it'd be to find hardware that'd read my Mac SCSI hard drives from 25 years ago? Easy… that original Mac is sitting in my basement and it worked like a charm last time it was powered on 4 years ago.
- tivert 2y ago
  > How long does an average hard drive last? You'd have to spend that 700k every that many years (plus the extra bits you mentioned). Quite an operation actually You'd have to spend a lot more, because with that many drives, you need redundancy now.
  - Aachen 2y ago
    True, that would be an up front cost. At the same time, the IA is still live. This initial expense can be softened by building up redundancy over some years rather than trying to do everything at once
    - tivert 2y ago
      > True, that would be an up front cost. At the same time, the IA is still live. This initial expense can be softened by building up redundancy over some years rather than trying to do everything at once I think with that many drives, you'd be losing them constantly, and I suppose you wouldn't know which ones until later (assuming you're doing an offline backup, if you aren't you have to factor in power costs).
- jolmg 2y ago
  For archival, if you use tape, it comes out cheaper (~225k) and ought to last longer (~30 years).
- viraptor 2y ago
  Backblaze keeps good stats. https://www.backblaze.com/cloud-storage/resources/hard-drive-test-data https://www.backblaze.com/cloud-storage/resources/hard-drive... 1.71% a year failure rate if you care for the hardware as much as they do.
  - hsbauauvhabzb 2y ago
    But that number would increase year on year, a 10 year old drive is far more likely to fail than a 1 year old drive
    - viraptor 2y ago
      Internet archive is going for long enough that I'd expect it to stabilise by now. If you replace enough of the drivers, you get a good mix.
- nephanth 2y ago
  For that purpose you might want to use magnetic tape like they use in long term archival services They are cheaper per Gio, and last significantly longer
ks2048 2y ago
Interesting to compare their stated drive $/GB to their B2 offering: $6/TB/mo for "pay-as-you-go", hard-drive price: $0.014/GB B2 price (12*6/1024): $0.070/GB/year
- alwayslikethis 2y ago
  Electricity, bandwidth, and generally running a business is not free. Also for these pay-as-you-go setups you'd need a considerable amount of free space available on demand. That said, it's not an especially cheap option. Hetzner has storage boxes for EUR 2.5/TB/mo (in fixed 5 and 10TB boxes though)
  - ks2048 2y ago
    Yeah, I wasn't trying to point out that it's a bad price. I think it's pretty good: same price for two years with all the maintenance.
PostOnce 2y ago
IA stores lots of redundant stuff in 5 file formats and none of them are particularly well-compressed, I think. There are (big) savings to be had, but maybe figuring that out (software dev and compute time) isn't worth it?

nikisweeting 2y ago

It's been tried several times, but it's hard because it's such a massive quantity of data. The IPFS backup never really got off the ground. They have their own backups which I think is good enough for now unless someone plans on donating a few hundred million.

vincentpants 2y ago
Oh no! I didn't know their IPFS initiative didn't pan out. What happened to it? I am surprised how hard it is to google. I remember interviewing for a role on that team at the archive to help move it to filecoin. Was so happy to hear that the effort was underway to decentralize their datastore. We need this more than ever.
- nikisweeting 2y ago
  There are people still working on trying to make it happen but it's just a collosal amount of data and filesystems are notoriously hard, so it's very slow going. From my own personal experience doing distributed archiving with no relation to Archive.org, Filecoin/IPFS's UX isn't quite there yet. They still don't let you serve data to the network from a normal filesystem, you have to let their system ingest all of your stuff so you end up double-storing data or you have to give into everything being stored as inscrutable binary blobs. That's why I still haven't integrated ArchiveBox with IPFS/Filecoin/Storj, let my data live in a normal filesystem dammit!
  - Aachen 2y ago
    > They still don't let you serve data to the network from a normal filesystem, you have to let their system ingest all of your stuff so you end up double-storing data or you have to give into everything being stored as inscrutable binary blobs. I don't understand this part. What data would you have to give them? Why can't it just live next to your stuff on your OS' filesystem?
    - nightpool 2y ago

creer 2y ago

Backup / duplication is not an easy project for sure. But meanwhile for now IA is a single organization operating under one legal system. And one technical setup, would be relevant today. That's a major weakness.

EamonnMR 2y ago

Suppose we each backed up sites we cared about rather than trying to mirror the whole thing...