6 ms·

Internet Archive's Storage

296 points by zdw 9mo ago

tylerchilds 9mo ago
Why’s Wendy’s Terracotta moved?
- tylerchilds 9mo ago
  Every time I’ve seen that front pew in that first photo, she’s there too, holding this: https://en.wikipedia.org/wiki/Executive_Order_9066 https://en.wikipedia.org/wiki/Executive_Order_9066
ranger_danger 9mo ago
I was hoping an article about IA's storage would go into detail about how their storage currently works, what kind of devices they use, how much they store, how quickly they add new data, the costs etc., but this seems to only talk about quite old stats.
- jonas21 9mo ago
  It does have these details for the current generation hardware. And if you want more, click on the link at the top: https://hackernoon.com/the-long-now-of-the-web-inside-the-internet-archives-fight-against-forgetting https://hackernoon.com/the-long-now-of-the-web-inside-the-in...
  - reaperducer 9mo ago
    Yeah, this is just blogspam. Some guy re-hashing the Hackernoon article, interspersed with his own comments. I wouldn't be surprised if it's AI. It's time to come up with a term for blog posts that are just AI-augmented re-hashes of other people's writing. Maybe blogslop.
    - dexdal 9mo ago
      That pattern shows up when publishing has near-zero cost and review has no gate. The fix is procedural: define what counts as original contribution and require a quick verification pass before posting. Without an input filter and a stop rule, you get infinite rephrases that drown out the scarce primary work.
    - schainks 9mo ago
      What a slog post.
    - tolerance 9mo ago
      You and I must be different kinds of readers. I’m under the impression that this style of writing is what people wish they got when they asked AI to summarize a lengthy web page. It’s criticism and commentary. I can’t see how you missed out on the passages that add to and even correct or argue against statements made in the Hackernoon article. In a way I can’t tell how one can believe that “re-hashing [an article], interspersed with [the blogger’s] own comments” isn’t a common blogging practice. If not then the internet made a mistake by allowing the likes of John Gruber to earn a living this way. And trust that I enjoy a good knee-jerk “slop” charge myself. To me this doesn’t qualify a bit.
    - jonah-archive 9mo ago
      This comment has, I think, made me more sad than anything I've ever read on HN before. David is one of the most thoughtful, critical, and valuable voices on the topic of digital archival, and has been for quite some time. The idea of someone dismissing his review of a much more slop-adjacent article as such is incredibly depressing.
- metadat 9mo ago
  The Internet Archive's Infrastructure https://news.ycombinator.com/item?id=46613324 https://news.ycombinator.com/item?id=46613324 - 8 days ago, 124 comments
arjie 9mo ago
This is very cool. One thing I am curious about is the software side of things and the details of the hardware. What is the filesystem and RAID (or lack of) layer to deal with this optimally? Looking into it a little: * power budget dominates everything: I have access to a lot of rack hardware from old connections, but I don't want to put the army of old stuff in my cabinet because it will blow my power budget for not that much performance in comparison to my 9755. What disks does the IA use? Any specific variety or like Backblaze a large variety? * magnetic is bloody slow: I'm not the Internet Archive so I'm just going to have a couple of machines with a few hundred TiB. I'm planning on making them all a big zfs so I can deduplicate but it seems like if I get a single disk failure I'm doomed to a massive rebuild I'm sure I can work it out with a modern LLM, but maybe someone here has experience with actually running massive storage and the use-case where tomorrow's data is almost the same as today's - as is the case with the Internet Archive where tomorrow's copy of wiki.roshangeorge.dev will look, even at the block level, like yesterday's copy. The last time I built with multi-petabyte datasets we were still using Hadoop on HDFS, haha!
- Datagenerator 9mo ago
  You might want to look into using cephadm to setup CEPH. Use Erasure coding as data pool for very efficient data storage and protection (8+2). From that export large RBD to be used as zpool with dedup. Scales to Petabytes and has lots of failure protection options.
  - arjie 9mo ago
    Thank you for that recommendation. I think I'm probably too small time for the moment for Ceph considering that. I don't have multi-petabytes (yet or perhaps ever).
- genewitch 9mo ago
  a couple hundred TB arranged how? and for what purpose, generally? archival, warm, hot? for the first two, depending on throughput desired, you can do with spinning rust. you pick your exposure, single platter or not, speed or not, and interface. And no fancy raid hardware needed. I've had decent luck with 3+1 warm and 4+1 archival. if you don't need quick seeks but want streaming data to be nice, make sure your largest file fits on a single drive, and do two parity disks for archive, a single for warm. md + lvm; ext4 fs, too. my very biased opinion based on tried everything and am out of ideas, and i am tired, and that stuff just works. I am not quick to the point but you need to split your storage up. use 18+ SMR disks, shingled magnetic recording hard drives, for larger stuff that you don't need to transfer very fast. 4k video for consumption on a 4k televsion fits here. Use faster, more reliable disks for data used a lot, &c Hot or fast seeks & transfers is different, but i didn't get the idea that's what you were after. Hadoop ought be used for hot data, imo. People may argue that zfs of xfs or jfs or ffs is better than ext4, but are they gunna jump in and fix it for free when something goes wrong for whatever reason? sorry, this is confusing. Unsure how to fix that. i have files on this style system that have been in continuous readable condition since the mid 1990s. There's been some bumps as i tried every [sic] other system and method. TL;dr to scale my 1/10th size up, i personally would just get a bigger box to put the disks in, and add an additional /volumeN/ mountpoint for each additional array i added. it goes without saying that under that directory i would CIFS/NFS share subdirectories that fit that array's specifications. again, i am just tired of all of this, i'm also all socialed out so, apologies.
badlibrarian 9mo ago
No climate control. No backup power. And it's secured by a wireless camera sitting in a potted plant. Bless them, but wow.
- chimeracoder 9mo ago
  > In the unlikely, for San Francisco, event that the day is too hot, less-urgent tasks can be delayed, or some of the racks can have their clock rate reduced, disks put into sleep mode, or even be powered down. Redundancy means that the data will be available elsewhere. So it sounds like they have data in other locations as well, hopefully.
  - electroly 9mo ago
    There's a mention on Wikipedia [1] that the Internet Archive maintains international mirror sites in Egypt and the Netherlands, in addition to several domestic sites within North America. [1] https://en.wikipedia.org/wiki/Internet_Archive#Operations https://en.wikipedia.org/wiki/Internet_Archive#Operations
  - badlibrarian 9mo ago
    During the recent power outages in San Francisco, the site repeatedly went down. When a troubled individual set the power pole on fire outside their building, the site went down. Happy to give them the benefit of the doubt on data redundancy, but they publicly celebrate that Brewster himself has to bike down and flip switches to get the site back online. They don't even have employee redundancy.
- krackers 9mo ago
  And a site that's in a notorious earth-quake prone zone. I can only hope that with all the AI craze one of the bigcorp made a deal to take a copy of all data in exchange for providing it as backup if necessary
- dang 9mo ago
  "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." "Don't be snarky." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
mrexroad 9mo ago
> This "waste heat" system is a closed loop of efficiency. The 60+ kilowatts of heat energy produced by a storage cluster is not a byproduct to be eliminated but a resource to be harvested. Are there any other data centers harvesting waste heat for benefit?
- bilegeek 9mo ago
  Several swimming pools do to great effect: https://datacentremagazine.com/data-centres/excess-data-centre-heat-used-to-warm-olympic-swimming-pools https://datacentremagazine.com/data-centres/excess-data-cent... https://www.bbc.com/news/technology-64939558 https://www.bbc.com/news/technology-64939558 https://arstechnica.com/information-technology/2023/03/free-data-center-heat-is-allegedly-saving-a-struggling-public-pool-24k-a-year/ https://arstechnica.com/information-technology/2023/03/free-...
- deleted 9mo ago
  [deleted]
- stanac 9mo ago
  Yandex had a data center in Finland,, not sure if it's still operational. It was heating 1500 homes with 4 MW. https://www.euroheat.org/dhc/knowledge-hub/datacentre-supplies-local-heating-in-maentsaelae-finland https://www.euroheat.org/dhc/knowledge-hub/datacentre-suppli...
- miduil 9mo ago
  Yes, plenty - sometimes data centers are built together with apartment or office complexes for this particular purpose. Unfortunately that already pinpoints the core limitation, due to the low-temperature of the data centers. The higher the temperature difference is, the more affective heating becomes - with air cooled systems it requires preparation to ensure that can be used for heating. Also data centers need physical space, and often - you need heating where there is not a lot of space (cities), and for "district heating" you need higher temperatures usually.
- cloud-oak 9mo ago
  The EU mandates that all large data centres built/commissioned from July this year will make use of waste heat: https://www.twobirds.com/en/insights/2024/germany/rechenzentren-und-abwaerme-ein-ueberblick-ueber-die-gesetzlichen-vorgaben-zur-abwaermenutzung https://www.twobirds.com/en/insights/2024/germany/rechenzent...
dr_dshiv 9mo ago
> Li correctly points out that the Archive's budget, in the range of $25-30M/year, is vastly lower than any comparable website: By owning its hardware, using the PetaBox high-density architecture, avoiding air conditioning costs, and using open-source software, the Archive achieves a storage cost efficiency that is orders of magnitude better than commercial cloud rates. That’s impressive. Wikipedia spends $185m per year and the Seattle public library spends $102m. Maybe not comparable exactly, but $30m per year seems inexpensive for the memory of the world…
- AdamN 9mo ago
  I think the culture is one of 'we are doing this for all humankind' and when you get just a few smart people bought in on that level of commitment and they're trying to be lean (and also for sure underpaying themselves compared to what they might make at Big Tech) then you can get impressive results.
  - exe34 9mo ago
    aren't we told all the time though, that a board of directors beholden to shareholders and a god given edict to make numbers go up are the only way to do things efficiently, to be lean and productive? are you telling me that when people find there's a need for something to happen, they make it happen? for the good of mankind? no billionaires?
    - votepaunchy 9mo ago
      [flagged]
      - delusional 9mo ago
        > But if all we needed was to hold hands and sing kumbaya then Africa would be Wakanda. Are you of the impression that the problems African nations are facing is that they're holding hands and singing too much? Are the Africans just lazy?
      - 9mo ago
arcade79 9mo ago
While reading this kind of articles, I'm always surprised by how small the storage described is. Given that Microsoft released their paper on LRCs in 2012, Google patented a bunch in 2010, facebook talked about their stuff around the 2010-2014 era too. CEPH started getting good erasure codes around 2016-2020. Has any of the big ones released articles on their storage systems in the last 5-10 years?
- theMMaI 9mo ago
  All the big ones have talked about their storage systems, but have been reluctant publishing papers like they used to do, so it appears to be more of a marketing focused effort than trying to share the technical details with the world.
- smueller1234 9mo ago
  IIRC, the most recent and most technical public content we (Google) have published on Colossus are these: https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system https://cloud.google.com/blog/products/storage-data-transfer... https://cloud.google.com/blog/products/storage-data-transfer/how-colossus-optimizes-data-placement-for-performance https://cloud.google.com/blog/products/storage-data-transfer... Facebook's published content on Tectonic is quite good and I think it's well more recent than 2010-14. (Current Google employee, just pointing to public content, hope that's helpful.)
  - arcade79 9mo ago
    Nice, the L4 cache seem to be a newish addition. Love the detail about two filesystems with >10 exabytes of storage.
JavohirXR 9mo ago
I saw the word "delve" and already knew it was redacted or written by ai
1vuio0pswjnm7 9mo ago
List of references https://en.wikipedia.org/wiki/Wayback_Machine https://en.wikipedia.org/wiki/Wayback_Machine https://blog.archive.org/2025/09/02/looking-back-on-preserving-the-internet-from-1996/ https://blog.archive.org/2025/09/02/looking-back-on-preservi... https://archive.org/web/petabox.php https://archive.org/web/petabox.php https://en.wikipedia.org/wiki/PetaBox https://en.wikipedia.org/wiki/PetaBox https://ipfs.tech/ https://ipfs.tech/ https://github.com/internetarchive/dweb-archive https://github.com/internetarchive/dweb-archive https://en.wikipedia.org/wiki/Internet_Archive https://en.wikipedia.org/wiki/Internet_Archive https://www.eweek.com/storage/making-web-memories-with-the-petabox/ https://www.eweek.com/storage/making-web-memories-with-the-p... https://internetarchive.archiveteam.org/index.php/PetaBox https://internetarchive.archiveteam.org/index.php/PetaBox https://blog.archive.org/2010/07/27/the-fourth-generation-petabox/ https://blog.archive.org/2010/07/27/the-fourth-generation-pe... https://hackaday.com/2025/11/18/internet-archive-hits-one-trillion-web-pages/ https://hackaday.com/2025/11/18/internet-archive-hits-one-tr... https://www.computerworld.com/article/1562759/the-internet-archive-s-wayback-machine-gets-a-new-data-center.html https://www.computerworld.com/article/1562759/the-internet-a... https://www.datacenterknowledge.com/business/internet-archive-to-live-in-sun-blackbox https://www.datacenterknowledge.com/business/internet-archiv... https://www.rootsimple.com/2023/08/inside-the-internet-archive-a-meat-world-tour/ https://www.rootsimple.com/2023/08/inside-the-internet-archi... https://richmondsunsetnews.com/2017/03/11/internet-archive-preserves-data-from-world-wide-web/ https://richmondsunsetnews.com/2017/03/11/internet-archive-p... https://en.wikipedia.org/wiki/Heritrix https://en.wikipedia.org/wiki/Heritrix https://support.archive-it.org/hc/en-us/articles/115001081186-Archive-It-Crawling-Technology https://support.archive-it.org/hc/en-us/articles/11500108118... https://digitalcommons.odu.edu/cgi/viewcontent.cgi?article=1154&context=computerscience_fac_pubs https://digitalcommons.odu.edu/cgi/viewcontent.cgi?article=1... https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/ https://iipc.github.io/warc-specifications/specifications/wa... https://usehall.com/agents/heritrix-bot https://usehall.com/agents/heritrix-bot https://library.imaging.org/admin/apis/public/api/ist/website/downloadArticle/archiving/14/1/art00004 https://library.imaging.org/admin/apis/public/api/ist/websit... https://blog.archive.org/2025/03/ https://blog.archive.org/2025/03/ https://archive.org/details/alexacrawls https://archive.org/details/alexacrawls https://en.wikipedia.org/wiki/Alexa_Internet https://en.wikipedia.org/wiki/Alexa_Internet https://projects.propublica.org/nonprofits/organizations/943242767 https://projects.propublica.org/nonprofits/organizations/943... https://werd.io/update-on-the-20242025-end-of-term-web-archive/ https://werd.io/update-on-the-20242025-end-of-term-web-archi... https://www.historyascode.com/tools-data/archive-it/ https://www.historyascode.com/tools-data/archive-it/ https://digitization.archive.org/pricing/ https://digitization.archive.org/pricing/ https://www.sfgate.com/tech/article/bay-area-warehouse-internet-archive-19858332.php https://www.sfgate.com/tech/article/bay-area-warehouse-inter... https://vault-webservices.zendesk.com/hc/en-us/articles/22896482572180-Vault-Pricing-Model https://vault-webservices.zendesk.com/hc/en-us/articles/2289... https://en.wikipedia.org/wiki/Hachette_v._Internet_Archive https://en.wikipedia.org/wiki/Hachette_v._Internet_Archive https://copyrightalliance.org/copyright-cases/hachette-book-group-internet-archive/ https://copyrightalliance.org/copyright-cases/hachette-book-... https://law.justia.com/cases/federal/appellate-courts/ca2/23-1260/23-1260-2024-09-04.html https://law.justia.com/cases/federal/appellate-courts/ca2/23... https://www.library.upenn.edu/news/hachette-v-internet-archive https://www.library.upenn.edu/news/hachette-v-internet-archi... https://www.lutzker.com/ip_bit_pieces/internet-archives-open-library-and-copyright-law-the-final-chapter/ https://www.lutzker.com/ip_bit_pieces/internet-archives-open... https://blog.archive.org/2023/08/17/what-the-hachette-v-internet-archive-decision-means-for-our-library/ https://blog.archive.org/2023/08/17/what-the-hachette-v-inte... https://www.musicbusinessworldwide.com/labels-settle-copyright-lawsuit-against-internet-archive-over-streaming-of-vintage-vinyl-records/ https://www.musicbusinessworldwide.com/labels-settle-copyrig... https://consequence.net/2025/09/internet-archive-labels-settle-copyright-lawsuit/ https://consequence.net/2025/09/internet-archive-labels-sett... https://blog.archive.org/2025/09/15/an-update-on-the-great-78s-lawsuit/ https://blog.archive.org/2025/09/15/an-update-on-the-great-7... https://giga.law/daily-news/2025/9/15/music-publishers-internet-archive-settle-lawsuit-over-old-recordings https://giga.law/daily-news/2025/9/15/music-publishers-inter... https://www.webpronews.com/internet-archive-settles-copyright-suit-with-sony-universal-over-vintage-records/ https://www.webpronews.com/internet-archive-settles-copyrigh... https://blog.archive.org/2025/07/ https://blog.archive.org/2025/07/ https://blog.archive.org/2018/07/21/decentralized-web-faq/ https://blog.archive.org/2018/07/21/decentralized-web-faq/ https://blog.archive.org/2016/06/23/decentalized-web-server-possible-approach-with-cost-and-performance-estimates/ https://blog.archive.org/2016/06/23/decentalized-web-server-... https://blog.archive.org/2025/02/06/update-on-the-2024-2025-end-of-term-web-archive/ https://blog.archive.org/2025/02/06/update-on-the-2024-2025-... https://www.reddit.com/r/DataHoarder/comments/1ijkdjl/progress_update_from_the_end_of_term_web_archive/ https://www.reddit.com/r/DataHoarder/comments/1ijkdjl/progre... From https://news.ycombinator.com/item?id=46613324 https://news.ycombinator.com/item?id=46613324

296 points by zdw 9mo ago

tylerchilds 9mo ago

Why’s Wendy’s Terracotta moved?

tylerchilds 9mo ago
Every time I’ve seen that front pew in that first photo, she’s there too, holding this: https://en.wikipedia.org/wiki/Executive_Order_9066 https://en.wikipedia.org/wiki/Executive_Order_9066

ranger_danger 9mo ago

I was hoping an article about IA's storage would go into detail about how their storage currently works, what kind of devices they use, how much they store, how quickly they add new data, the costs etc., but this seems to only talk about quite old stats.

jonas21 9mo ago
It does have these details for the current generation hardware. And if you want more, click on the link at the top: https://hackernoon.com/the-long-now-of-the-web-inside-the-internet-archives-fight-against-forgetting https://hackernoon.com/the-long-now-of-the-web-inside-the-in...
- reaperducer 9mo ago
  Yeah, this is just blogspam. Some guy re-hashing the Hackernoon article, interspersed with his own comments. I wouldn't be surprised if it's AI. It's time to come up with a term for blog posts that are just AI-augmented re-hashes of other people's writing. Maybe blogslop.
  - dexdal 9mo ago
    That pattern shows up when publishing has near-zero cost and review has no gate. The fix is procedural: define what counts as original contribution and require a quick verification pass before posting. Without an input filter and a stop rule, you get infinite rephrases that drown out the scarce primary work.
  - schainks 9mo ago
    What a slog post.
  - tolerance 9mo ago
    You and I must be different kinds of readers. I’m under the impression that this style of writing is what people wish they got when they asked AI to summarize a lengthy web page. It’s criticism and commentary. I can’t see how you missed out on the passages that add to and even correct or argue against statements made in the Hackernoon article. In a way I can’t tell how one can believe that “re-hashing [an article], interspersed with [the blogger’s] own comments” isn’t a common blogging practice. If not then the internet made a mistake by allowing the likes of John Gruber to earn a living this way. And trust that I enjoy a good knee-jerk “slop” charge myself. To me this doesn’t qualify a bit.
  - jonah-archive 9mo ago
    This comment has, I think, made me more sad than anything I've ever read on HN before. David is one of the most thoughtful, critical, and valuable voices on the topic of digital archival, and has been for quite some time. The idea of someone dismissing his review of a much more slop-adjacent article as such is incredibly depressing.
metadat 9mo ago
The Internet Archive's Infrastructure https://news.ycombinator.com/item?id=46613324 https://news.ycombinator.com/item?id=46613324 - 8 days ago, 124 comments

arjie 9mo ago

This is very cool. One thing I am curious about is the software side of things and the details of the hardware. What is the filesystem and RAID (or lack of) layer to deal with this optimally? Looking into it a little: * power budget dominates everything: I have access to a lot of rack hardware from old connections, but I don't want to put the army of old stuff in my cabinet because it will blow my power budget for not that much performance in comparison to my 9755. What disks does the IA use? Any specific variety or like Backblaze a large variety? * magnetic is bloody slow: I'm not the Internet Archive so I'm just going to have a couple of machines with a few hundred TiB. I'm planning on making them all a big zfs so I can deduplicate but it seems like if I get a single disk failure I'm doomed to a massive rebuild I'm sure I can work it out with a modern LLM, but maybe someone here has experience with actually running massive storage and the use-case where tomorrow's data is almost the same as today's - as is the case with the Internet Archive where tomorrow's copy of wiki.roshangeorge.dev will look, even at the block level, like yesterday's copy. The last time I built with multi-petabyte datasets we were still using Hadoop on HDFS, haha!

Datagenerator 9mo ago
You might want to look into using cephadm to setup CEPH. Use Erasure coding as data pool for very efficient data storage and protection (8+2). From that export large RBD to be used as zpool with dedup. Scales to Petabytes and has lots of failure protection options.
- arjie 9mo ago
  Thank you for that recommendation. I think I'm probably too small time for the moment for Ceph considering that. I don't have multi-petabytes (yet or perhaps ever).
genewitch 9mo ago
a couple hundred TB arranged how? and for what purpose, generally? archival, warm, hot? for the first two, depending on throughput desired, you can do with spinning rust. you pick your exposure, single platter or not, speed or not, and interface. And no fancy raid hardware needed. I've had decent luck with 3+1 warm and 4+1 archival. if you don't need quick seeks but want streaming data to be nice, make sure your largest file fits on a single drive, and do two parity disks for archive, a single for warm. md + lvm; ext4 fs, too. my very biased opinion based on tried everything and am out of ideas, and i am tired, and that stuff just works. I am not quick to the point but you need to split your storage up. use 18+ SMR disks, shingled magnetic recording hard drives, for larger stuff that you don't need to transfer very fast. 4k video for consumption on a 4k televsion fits here. Use faster, more reliable disks for data used a lot, &c Hot or fast seeks & transfers is different, but i didn't get the idea that's what you were after. Hadoop ought be used for hot data, imo. People may argue that zfs of xfs or jfs or ffs is better than ext4, but are they gunna jump in and fix it for free when something goes wrong for whatever reason? sorry, this is confusing. Unsure how to fix that. i have files on this style system that have been in continuous readable condition since the mid 1990s. There's been some bumps as i tried every [sic] other system and method. TL;dr to scale my 1/10th size up, i personally would just get a bigger box to put the disks in, and add an additional /volumeN/ mountpoint for each additional array i added. it goes without saying that under that directory i would CIFS/NFS share subdirectories that fit that array's specifications. again, i am just tired of all of this, i'm also all socialed out so, apologies.

badlibrarian 9mo ago

No climate control. No backup power. And it's secured by a wireless camera sitting in a potted plant. Bless them, but wow.

chimeracoder 9mo ago
> In the unlikely, for San Francisco, event that the day is too hot, less-urgent tasks can be delayed, or some of the racks can have their clock rate reduced, disks put into sleep mode, or even be powered down. Redundancy means that the data will be available elsewhere. So it sounds like they have data in other locations as well, hopefully.
- electroly 9mo ago
  There's a mention on Wikipedia [1] that the Internet Archive maintains international mirror sites in Egypt and the Netherlands, in addition to several domestic sites within North America. [1] https://en.wikipedia.org/wiki/Internet_Archive#Operations https://en.wikipedia.org/wiki/Internet_Archive#Operations
- badlibrarian 9mo ago
  During the recent power outages in San Francisco, the site repeatedly went down. When a troubled individual set the power pole on fire outside their building, the site went down. Happy to give them the benefit of the doubt on data redundancy, but they publicly celebrate that Brewster himself has to bike down and flip switches to get the site back online. They don't even have employee redundancy.
krackers 9mo ago
And a site that's in a notorious earth-quake prone zone. I can only hope that with all the AI craze one of the bigcorp made a deal to take a copy of all data in exchange for providing it as backup if necessary
dang 9mo ago
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." "Don't be snarky." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

mrexroad 9mo ago

> This "waste heat" system is a closed loop of efficiency. The 60+ kilowatts of heat energy produced by a storage cluster is not a byproduct to be eliminated but a resource to be harvested. Are there any other data centers harvesting waste heat for benefit?

bilegeek 9mo ago
Several swimming pools do to great effect: https://datacentremagazine.com/data-centres/excess-data-centre-heat-used-to-warm-olympic-swimming-pools https://datacentremagazine.com/data-centres/excess-data-cent... https://www.bbc.com/news/technology-64939558 https://www.bbc.com/news/technology-64939558 https://arstechnica.com/information-technology/2023/03/free-data-center-heat-is-allegedly-saving-a-struggling-public-pool-24k-a-year/ https://arstechnica.com/information-technology/2023/03/free-...
deleted 9mo ago
[deleted]
stanac 9mo ago
Yandex had a data center in Finland,, not sure if it's still operational. It was heating 1500 homes with 4 MW. https://www.euroheat.org/dhc/knowledge-hub/datacentre-supplies-local-heating-in-maentsaelae-finland https://www.euroheat.org/dhc/knowledge-hub/datacentre-suppli...
miduil 9mo ago
Yes, plenty - sometimes data centers are built together with apartment or office complexes for this particular purpose. Unfortunately that already pinpoints the core limitation, due to the low-temperature of the data centers. The higher the temperature difference is, the more affective heating becomes - with air cooled systems it requires preparation to ensure that can be used for heating. Also data centers need physical space, and often - you need heating where there is not a lot of space (cities), and for "district heating" you need higher temperatures usually.
cloud-oak 9mo ago
The EU mandates that all large data centres built/commissioned from July this year will make use of waste heat: https://www.twobirds.com/en/insights/2024/germany/rechenzentren-und-abwaerme-ein-ueberblick-ueber-die-gesetzlichen-vorgaben-zur-abwaermenutzung https://www.twobirds.com/en/insights/2024/germany/rechenzent...

dr_dshiv 9mo ago

> Li correctly points out that the Archive's budget, in the range of $25-30M/year, is vastly lower than any comparable website: By owning its hardware, using the PetaBox high-density architecture, avoiding air conditioning costs, and using open-source software, the Archive achieves a storage cost efficiency that is orders of magnitude better than commercial cloud rates. That’s impressive. Wikipedia spends $185m per year and the Seattle public library spends $102m. Maybe not comparable exactly, but $30m per year seems inexpensive for the memory of the world…

AdamN 9mo ago
I think the culture is one of 'we are doing this for all humankind' and when you get just a few smart people bought in on that level of commitment and they're trying to be lean (and also for sure underpaying themselves compared to what they might make at Big Tech) then you can get impressive results.
- exe34 9mo ago
  aren't we told all the time though, that a board of directors beholden to shareholders and a god given edict to make numbers go up are the only way to do things efficiently, to be lean and productive? are you telling me that when people find there's a need for something to happen, they make it happen? for the good of mankind? no billionaires?
  - votepaunchy 9mo ago
    [flagged]
    - delusional 9mo ago
      > But if all we needed was to hold hands and sing kumbaya then Africa would be Wakanda. Are you of the impression that the problems African nations are facing is that they're holding hands and singing too much? Are the Africans just lazy?
    - 9mo ago

arcade79 9mo ago

While reading this kind of articles, I'm always surprised by how small the storage described is. Given that Microsoft released their paper on LRCs in 2012, Google patented a bunch in 2010, facebook talked about their stuff around the 2010-2014 era too. CEPH started getting good erasure codes around 2016-2020. Has any of the big ones released articles on their storage systems in the last 5-10 years?

theMMaI 9mo ago
All the big ones have talked about their storage systems, but have been reluctant publishing papers like they used to do, so it appears to be more of a marketing focused effort than trying to share the technical details with the world.
smueller1234 9mo ago
IIRC, the most recent and most technical public content we (Google) have published on Colossus are these: https://cloud.google.com/blog/products/storage-data-transfer/a-peek-behind-colossus-googles-file-system https://cloud.google.com/blog/products/storage-data-transfer... https://cloud.google.com/blog/products/storage-data-transfer/how-colossus-optimizes-data-placement-for-performance https://cloud.google.com/blog/products/storage-data-transfer... Facebook's published content on Tectonic is quite good and I think it's well more recent than 2010-14. (Current Google employee, just pointing to public content, hope that's helpful.)
- arcade79 9mo ago
  Nice, the L4 cache seem to be a newish addition. Love the detail about two filesystems with >10 exabytes of storage.

JavohirXR 9mo ago

I saw the word "delve" and already knew it was redacted or written by ai

1vuio0pswjnm7 9mo ago

List of references https://en.wikipedia.org/wiki/Wayback_Machine https://en.wikipedia.org/wiki/Wayback_Machine https://blog.archive.org/2025/09/02/looking-back-on-preserving-the-internet-from-1996/ https://blog.archive.org/2025/09/02/looking-back-on-preservi... https://archive.org/web/petabox.php https://archive.org/web/petabox.php https://en.wikipedia.org/wiki/PetaBox https://en.wikipedia.org/wiki/PetaBox https://ipfs.tech/ https://ipfs.tech/ https://github.com/internetarchive/dweb-archive https://github.com/internetarchive/dweb-archive https://en.wikipedia.org/wiki/Internet_Archive https://en.wikipedia.org/wiki/Internet_Archive https://www.eweek.com/storage/making-web-memories-with-the-petabox/ https://www.eweek.com/storage/making-web-memories-with-the-p... https://internetarchive.archiveteam.org/index.php/PetaBox https://internetarchive.archiveteam.org/index.php/PetaBox https://blog.archive.org/2010/07/27/the-fourth-generation-petabox/ https://blog.archive.org/2010/07/27/the-fourth-generation-pe... https://hackaday.com/2025/11/18/internet-archive-hits-one-trillion-web-pages/ https://hackaday.com/2025/11/18/internet-archive-hits-one-tr... https://www.computerworld.com/article/1562759/the-internet-archive-s-wayback-machine-gets-a-new-data-center.html https://www.computerworld.com/article/1562759/the-internet-a... https://www.datacenterknowledge.com/business/internet-archive-to-live-in-sun-blackbox https://www.datacenterknowledge.com/business/internet-archiv... https://www.rootsimple.com/2023/08/inside-the-internet-archive-a-meat-world-tour/ https://www.rootsimple.com/2023/08/inside-the-internet-archi... https://richmondsunsetnews.com/2017/03/11/internet-archive-preserves-data-from-world-wide-web/ https://richmondsunsetnews.com/2017/03/11/internet-archive-p... https://en.wikipedia.org/wiki/Heritrix https://en.wikipedia.org/wiki/Heritrix https://support.archive-it.org/hc/en-us/articles/115001081186-Archive-It-Crawling-Technology https://support.archive-it.org/hc/en-us/articles/11500108118... https://digitalcommons.odu.edu/cgi/viewcontent.cgi?article=1154&context=computerscience_fac_pubs https://digitalcommons.odu.edu/cgi/viewcontent.cgi?article=1... https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/ https://iipc.github.io/warc-specifications/specifications/wa... https://usehall.com/agents/heritrix-bot https://usehall.com/agents/heritrix-bot https://library.imaging.org/admin/apis/public/api/ist/website/downloadArticle/archiving/14/1/art00004 https://library.imaging.org/admin/apis/public/api/ist/websit... https://blog.archive.org/2025/03/ https://blog.archive.org/2025/03/ https://archive.org/details/alexacrawls https://archive.org/details/alexacrawls https://en.wikipedia.org/wiki/Alexa_Internet https://en.wikipedia.org/wiki/Alexa_Internet https://projects.propublica.org/nonprofits/organizations/943242767 https://projects.propublica.org/nonprofits/organizations/943... https://werd.io/update-on-the-20242025-end-of-term-web-archive/ https://werd.io/update-on-the-20242025-end-of-term-web-archi... https://www.historyascode.com/tools-data/archive-it/ https://www.historyascode.com/tools-data/archive-it/ https://digitization.archive.org/pricing/ https://digitization.archive.org/pricing/ https://www.sfgate.com/tech/article/bay-area-warehouse-internet-archive-19858332.php https://www.sfgate.com/tech/article/bay-area-warehouse-inter... https://vault-webservices.zendesk.com/hc/en-us/articles/22896482572180-Vault-Pricing-Model https://vault-webservices.zendesk.com/hc/en-us/articles/2289... https://en.wikipedia.org/wiki/Hachette_v._Internet_Archive https://en.wikipedia.org/wiki/Hachette_v._Internet_Archive https://copyrightalliance.org/copyright-cases/hachette-book-group-internet-archive/ https://copyrightalliance.org/copyright-cases/hachette-book-... https://law.justia.com/cases/federal/appellate-courts/ca2/23-1260/23-1260-2024-09-04.html https://law.justia.com/cases/federal/appellate-courts/ca2/23... https://www.library.upenn.edu/news/hachette-v-internet-archive https://www.library.upenn.edu/news/hachette-v-internet-archi... https://www.lutzker.com/ip_bit_pieces/internet-archives-open-library-and-copyright-law-the-final-chapter/ https://www.lutzker.com/ip_bit_pieces/internet-archives-open... https://blog.archive.org/2023/08/17/what-the-hachette-v-internet-archive-decision-means-for-our-library/ https://blog.archive.org/2023/08/17/what-the-hachette-v-inte... https://www.musicbusinessworldwide.com/labels-settle-copyright-lawsuit-against-internet-archive-over-streaming-of-vintage-vinyl-records/ https://www.musicbusinessworldwide.com/labels-settle-copyrig... https://consequence.net/2025/09/internet-archive-labels-settle-copyright-lawsuit/ https://consequence.net/2025/09/internet-archive-labels-sett... https://blog.archive.org/2025/09/15/an-update-on-the-great-78s-lawsuit/ https://blog.archive.org/2025/09/15/an-update-on-the-great-7... https://giga.law/daily-news/2025/9/15/music-publishers-internet-archive-settle-lawsuit-over-old-recordings https://giga.law/daily-news/2025/9/15/music-publishers-inter... https://www.webpronews.com/internet-archive-settles-copyright-suit-with-sony-universal-over-vintage-records/ https://www.webpronews.com/internet-archive-settles-copyrigh... https://blog.archive.org/2025/07/ https://blog.archive.org/2025/07/ https://blog.archive.org/2018/07/21/decentralized-web-faq/ https://blog.archive.org/2018/07/21/decentralized-web-faq/ https://blog.archive.org/2016/06/23/decentalized-web-server-possible-approach-with-cost-and-performance-estimates/ https://blog.archive.org/2016/06/23/decentalized-web-server-... https://blog.archive.org/2025/02/06/update-on-the-2024-2025-end-of-term-web-archive/ https://blog.archive.org/2025/02/06/update-on-the-2024-2025-... https://www.reddit.com/r/DataHoarder/comments/1ijkdjl/progress_update_from_the_end_of_term_web_archive/ https://www.reddit.com/r/DataHoarder/comments/1ijkdjl/progre... From https://news.ycombinator.com/item?id=46613324 https://news.ycombinator.com/item?id=46613324