7 ms·

I've always wondered this - How does Archive.org work in terms of storage? Internet is massive and caching every single site periodically for years on, isn't th

I've always wondered this - How does Archive.org work in terms of storage? Internet is massive and caching every single site periodically for years on, isn't that unreasonably huge amount of data? Edit: I just checked Wikipedia, it says they're using about 15 PB of storage. Edit 2: 15 PB cost => 15,000 TB x $30/TB = $450,000. Ofcourse, back of the napkin cost (no maintenance, power, etc). That's not too bad actually.

albertgoeswoof 8y ago
Blockchain
- antoineMoPa 8y ago
  I see this is downvoted. Actually, if a blockchain used proof of actual useful storage or computation, that'd be great. (instead of wasting energy by computing "useless" hashes)
  - QuercusMax 8y ago
    It's probably downvoted because replying 'blockchain' to a question about how Archive.org stores their data is not even a useful answer. There may well be fascinating ways one can use blockchain for archives of this sort, but the grandparent comment was essentially useless noise.
  - contravariant 8y ago
    I suppose that would be kind of interesting. Although if it's conceivable that someone could just store all the useful information then you can't just request a random piece of information as a proof of 'work', so you'd need something clever. Anyway the comment was probably downvoted because it consisted of a single buzzword.
  - X6S1x6Okd1st 8y ago
    Proof of storage is used in siacoin, but not to secure the network. I haven't seen a scheme where proof of storage can be used to secure the network. In terms of making sure the work that is done is useful it's fairly hard to do that because if you let someone dictate what work should be done then you potentially allow them to do the work upfront and a 51% attack becomes pretty easy.
    - nostrademons 8y ago
      "you potentially allow them to do the work upfront and a 51% attack becomes pretty easy." I do wonder if there are potential consensus algorithms that rely upon game theory, the threat of being booted off the network, and the need for some investment before they're "trusted" by other peers. The basic idea is that yes, you could 51%-attack the network with a lot less than 51% computing power, but it would never be profitable because if you're discovered you're booted off into your own little partition where nobody plays with you, and the value you get from being part of the network in the future is greater than the value you would get from rewriting history into something only you believe. 51% attacks are possible now - roughly 75% of Bitcoin hashing power is located in China, so if the Chinese government decides that they want to shut it down, they send the military after the largest Bitcoin miners, say "Hash these transactions or else", and take control of the chain. They haven't found it profitable to do so yet, though - what would they gain from it?
      - X6S1x6Okd1st 8y ago
        The thing is that reputation isn't built into the protocol. Everyone is basically anonymous strangers.
    - dlubarov 8y ago
      When space-intensive computations are used for consensus, it's usually called proof of space. There are a few projects doing it already. https://en.wikipedia.org/wiki/Proof-of-space https://en.wikipedia.org/wiki/Proof-of-space
      - X6S1x6Okd1st 8y ago
        Proof of space and proof of storage are two different things.
- jasonmunro 8y ago
  Blockchain actually could be used to provide data integrity in a distributed environment with known bad actors (presuming that they fall under the 50% threshold), but that is not applicable in this case since archive data is not stored in a distributive fashion by clients (as far as I know). Having worked on a production application that uses blockchain, my take away was it cleverly provides distributed ledger transparency, but at a price that makes it very inefficient as a data store.
db48x 8y ago
A lot more than 15PB; it was over 20PB last year. And that figure doesn't count the duplicate copies they keep of everything.
- ktross 8y ago
  According to their website, it's over 35PB now. "We preserve 750 million Web pages per week! We’ve saved 35 petabytes (that’s 35,000,000,000,000,000 bytes) of data."
- 8bitsrule 8y ago
  According to this 9-28-2018 article, "Today, the Wayback Machine houses some 388B web pages, and its parent ... The Internet Archive’s collection, which spans not just the web, but books, audio 78rpm records, videos, images, and software, amounts to more than 40 petabytes, or 40 million gigabytes, of data. The Wayback Machine makes up about 63% of that." https://thehustle.co/inside-wayback-machine-internet-archive https://thehustle.co/inside-wayback-machine-internet-archive It's essentially the only cure for the serious link-rot problems in Wikipedia's references.
sekh60 8y ago
That doesn't account for redundancy and backups. I have asked some storage cost questions on /r/sysadmin and /r/datahoarder in the past about cost per TB and I tend to see numbers ranging from $300 to $1000 a TB of useable space.
- dlubarov 8y ago
  That must be for SSDs, which are ~$150/tb vs ~$20/tb for HDDs. I expect IA must use HDDs for the bulk of their data.
  - nomel 8y ago
    Or even tape. For offline redundancy it’s still cheaper.
    - deleted 8y ago
      [deleted]
    - deleted 8y ago
      [deleted]
  - lordlimecat 8y ago
    SSDs are about $400/TB for raw storage. Prices are substantially different for consumer and enterprise spaces, but 4TB disks (Micron, Samsung, etc) tend to be about $1500 and 8TB around $3200. If you factor in redundancy, you're looking at around $1000-1500 per TB of usable storage. This also isn't a matter of "just go with cheaper consumer disks". If you care at all about data integrity you won't use consumer-grade SSDs.
- lordlimecat 8y ago
  3-way mirrored spinning storage would cost roughly $100/TB. 12TB disks are about $350/ piece these days. There are a whole bunch of possible considerations and alternate scenarios, but this is probably the worst-case. That does not factor in compression / dedup, which can gain you a substantial amount of savings depending on your work profile; I bet if Archive.org wanted to, they could slash their storage using block-level dedup. Of course that creates potential recoverability headaches, so it's possible they don't.
pishpash 8y ago
If the internet grows exponentially, and sites decay in finite time, then that's about the same as caching the current internet, i.e. Google.
bnewbold 8y ago
The Archive currently has about 46 Petabytes of content ("bytes archived"), and over 120 PB of raw disk capacity; the difference is due to data replication, "currently filling" storage, non-storage infrastructure, etc. We save a lot on web content storage by de-duplicating "revists" when the page hasn't changed. This works out to save a whole lot for content like jQuery served from a common CDN URL; it doesn't work well when there is a page counter or any trivial changing content on a page. If you are interested in the storage back-end, it's actually pretty simple: HTTP requests/responses are concatenated and compressed in WARC files (sort of like .tar.gz) that get stored on regular old ext4 filesystems. An index of "what URL captures are in what WARC files on what servers" is continuously generated in the form of, basically, a giant sorted (and shareded) .tsv file; replay requests on web.archive.org look up the URL and timestamp and get a reference to a machine, file, and file offset, and make an HTTP 1.1 range request for the content in question. There are a bunch of other details, like checking robots.txt status, but the core design is super simple, cheap, and (relatively) easy to operate at scale. Apart from web crawl content (including, these days, "heavy" video content which is difficult to de-dupe), we have a large amount of live recorded TV, scanned books (raw photos), etc. (I currently work at IA)
- jvz 8y ago
  Have you evaluated compression algorithms that support custom dictionaries, like zstd? You could generate a compression dictionary for each domain, or just for those above a certain size.
- Nition 8y ago
  Would be nice if you could store a diff, so a changing counter would only have to store the changed counter after the initial save.
  - murukesh_s 8y ago
    Wondering the same. How if used git protocol itself? Not sure how efficient is git, but if it is then it's a relatively easier change

albertgoeswoof 8y ago

Blockchain

antoineMoPa 8y ago
I see this is downvoted. Actually, if a blockchain used proof of actual useful storage or computation, that'd be great. (instead of wasting energy by computing "useless" hashes)
- QuercusMax 8y ago
  It's probably downvoted because replying 'blockchain' to a question about how Archive.org stores their data is not even a useful answer. There may well be fascinating ways one can use blockchain for archives of this sort, but the grandparent comment was essentially useless noise.
- contravariant 8y ago
  I suppose that would be kind of interesting. Although if it's conceivable that someone could just store all the useful information then you can't just request a random piece of information as a proof of 'work', so you'd need something clever. Anyway the comment was probably downvoted because it consisted of a single buzzword.
- X6S1x6Okd1st 8y ago
  Proof of storage is used in siacoin, but not to secure the network. I haven't seen a scheme where proof of storage can be used to secure the network. In terms of making sure the work that is done is useful it's fairly hard to do that because if you let someone dictate what work should be done then you potentially allow them to do the work upfront and a 51% attack becomes pretty easy.
  - nostrademons 8y ago
    "you potentially allow them to do the work upfront and a 51% attack becomes pretty easy." I do wonder if there are potential consensus algorithms that rely upon game theory, the threat of being booted off the network, and the need for some investment before they're "trusted" by other peers. The basic idea is that yes, you could 51%-attack the network with a lot less than 51% computing power, but it would never be profitable because if you're discovered you're booted off into your own little partition where nobody plays with you, and the value you get from being part of the network in the future is greater than the value you would get from rewriting history into something only you believe. 51% attacks are possible now - roughly 75% of Bitcoin hashing power is located in China, so if the Chinese government decides that they want to shut it down, they send the military after the largest Bitcoin miners, say "Hash these transactions or else", and take control of the chain. They haven't found it profitable to do so yet, though - what would they gain from it?
    - X6S1x6Okd1st 8y ago
      The thing is that reputation isn't built into the protocol. Everyone is basically anonymous strangers.
  - dlubarov 8y ago
    When space-intensive computations are used for consensus, it's usually called proof of space. There are a few projects doing it already. https://en.wikipedia.org/wiki/Proof-of-space https://en.wikipedia.org/wiki/Proof-of-space
    - X6S1x6Okd1st 8y ago
      Proof of space and proof of storage are two different things.
jasonmunro 8y ago
Blockchain actually could be used to provide data integrity in a distributed environment with known bad actors (presuming that they fall under the 50% threshold), but that is not applicable in this case since archive data is not stored in a distributive fashion by clients (as far as I know). Having worked on a production application that uses blockchain, my take away was it cleverly provides distributed ledger transparency, but at a price that makes it very inefficient as a data store.

db48x 8y ago

A lot more than 15PB; it was over 20PB last year. And that figure doesn't count the duplicate copies they keep of everything.

ktross 8y ago
According to their website, it's over 35PB now. "We preserve 750 million Web pages per week! We’ve saved 35 petabytes (that’s 35,000,000,000,000,000 bytes) of data."
8bitsrule 8y ago
According to this 9-28-2018 article, "Today, the Wayback Machine houses some 388B web pages, and its parent ... The Internet Archive’s collection, which spans not just the web, but books, audio 78rpm records, videos, images, and software, amounts to more than 40 petabytes, or 40 million gigabytes, of data. The Wayback Machine makes up about 63% of that." https://thehustle.co/inside-wayback-machine-internet-archive https://thehustle.co/inside-wayback-machine-internet-archive It's essentially the only cure for the serious link-rot problems in Wikipedia's references.

sekh60 8y ago

That doesn't account for redundancy and backups. I have asked some storage cost questions on /r/sysadmin and /r/datahoarder in the past about cost per TB and I tend to see numbers ranging from $300 to $1000 a TB of useable space.

dlubarov 8y ago
That must be for SSDs, which are ~$150/tb vs ~$20/tb for HDDs. I expect IA must use HDDs for the bulk of their data.
- nomel 8y ago
  Or even tape. For offline redundancy it’s still cheaper.
  - deleted 8y ago
    [deleted]
  - deleted 8y ago
    [deleted]
- lordlimecat 8y ago
  SSDs are about $400/TB for raw storage. Prices are substantially different for consumer and enterprise spaces, but 4TB disks (Micron, Samsung, etc) tend to be about $1500 and 8TB around $3200. If you factor in redundancy, you're looking at around $1000-1500 per TB of usable storage. This also isn't a matter of "just go with cheaper consumer disks". If you care at all about data integrity you won't use consumer-grade SSDs.
lordlimecat 8y ago
3-way mirrored spinning storage would cost roughly $100/TB. 12TB disks are about $350/ piece these days. There are a whole bunch of possible considerations and alternate scenarios, but this is probably the worst-case. That does not factor in compression / dedup, which can gain you a substantial amount of savings depending on your work profile; I bet if Archive.org wanted to, they could slash their storage using block-level dedup. Of course that creates potential recoverability headaches, so it's possible they don't.

pishpash 8y ago

If the internet grows exponentially, and sites decay in finite time, then that's about the same as caching the current internet, i.e. Google.

bnewbold 8y ago

The Archive currently has about 46 Petabytes of content ("bytes archived"), and over 120 PB of raw disk capacity; the difference is due to data replication, "currently filling" storage, non-storage infrastructure, etc. We save a lot on web content storage by de-duplicating "revists" when the page hasn't changed. This works out to save a whole lot for content like jQuery served from a common CDN URL; it doesn't work well when there is a page counter or any trivial changing content on a page. If you are interested in the storage back-end, it's actually pretty simple: HTTP requests/responses are concatenated and compressed in WARC files (sort of like .tar.gz) that get stored on regular old ext4 filesystems. An index of "what URL captures are in what WARC files on what servers" is continuously generated in the form of, basically, a giant sorted (and shareded) .tsv file; replay requests on web.archive.org look up the URL and timestamp and get a reference to a machine, file, and file offset, and make an HTTP 1.1 range request for the content in question. There are a bunch of other details, like checking robots.txt status, but the core design is super simple, cheap, and (relatively) easy to operate at scale. Apart from web crawl content (including, these days, "heavy" video content which is difficult to de-dupe), we have a large amount of live recorded TV, scanned books (raw photos), etc. (I currently work at IA)

jvz 8y ago
Have you evaluated compression algorithms that support custom dictionaries, like zstd? You could generate a compression dictionary for each domain, or just for those above a certain size.
Nition 8y ago
Would be nice if you could store a diff, so a changing counter would only have to store the changed counter after the initial save.
- murukesh_s 8y ago
  Wondering the same. How if used git protocol itself? Not sure how efficient is git, but if it is then it's a relatively easier change