6 ms·
1 Trillion Web Pages Archived
- yupyupyups 1y agohttps://hoarding.support/ https://hoarding.support/
- typpilol 1y agoI thought this was going to be a technical article but there was nothing in it
- ehsanu1 1y agoSeeing some stats would be fun. I wonder what the amount of data is here. And the distribution would be interesting too, especially since some pages are archived at multiple points in time, and pages have been getting heavier these days.
- ChrisArchitect 1y agoRelated blog post inviting stories: https://blog.archive.org/2025/09/23/celebrating-1-trillion-webpages-archived-share-your-wayback-story/ https://blog.archive.org/2025/09/23/celebrating-1-trillion-w...
- i-chuks 1y ago[dead]
- arjie 1y agoSomething I wish we could have is some kind of peer mirror of archive.org. The main IA web application gets angry pretty quickly if you're trying to click through a few different dates. If there were some kind of way to slowly mirror (torrent-style) and offer pages as a peer from archive.org that would be neat. It would be cool to show up as an alternative source for the data and the archive.org app could fetch it out of there on a user's choice and validate the checksum if required. In the end, I've ended up just keeping my own ArchiveBox and it's an all right experience. In the end, it's only useful for things I know I wanted to archive. For almost everything I go to the IA - which has so much.
- zapataband2 1y agoIs there such thing as "versioned" torrents? Assuming you have the right PGP key you could mix bittorrent and packaging systems to get an update-able distribution
- hsbauauvhabzb 1y agoA Torrent would probably suffocate under the small file distribution. I’m not sure how the romset torrents work but I thought they were versioned. But torrent is probably the wrong tech. I’m sure there would be many players willing to host a few TB or more each, which could be fronted via something so it’s transparent to the user. But a better option might be a subscription model, anything else will be slammed by crawlers.
- pabs3 1y agoCouple of BEPs related to updating torrents: https://www.bittorrent.org/beps/bep_0039.html https://www.bittorrent.org/beps/bep_0039.html https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0046.html
- throawayonthe 1y agotrere is the bittorrent v2 standard: https://blog.libtorrent.org/2020/09/bittorrent-v2/ https://blog.libtorrent.org/2020/09/bittorrent-v2/ but unfortunately most foss torrent clients do not support it, partly because at release libtorrent 2.0.x had poor io performance in some cases so torrent clients reverted to the 1.2.x branch
- pronoiac 1y agoI think SciOp is doing something in that area, with a catalog site and webseeds. https://sciop.net/ https://sciop.net/
- renegat0x0 1y ago- I can confirm that the web archive can be really slow - I think I have seen that AI scrapers create bottleneck in the bandwidth - To some digital archives you need to create scientific accounts (I think Common Crawl works like that) - Data quite easily can be very big. The goal is to store many things. We not only store Internet, but with additional dimension of time - Since there is a lot of data, it is difficult to navigate it, search it, so it easily can become unusable - For example that is why I created my own meta data link, I needed some information about domains Link: https://github.com/rumca-js/Internet-Places-Database https://github.com/rumca-js/Internet-Places-Database
- pabs3 1y agoIf anyone wants to help feed in more stuff, ArchiveTeam is a related volunteer group that sends data to IA: https://archiveteam.org/ https://archiveteam.org/
- londons_explore 1y agoPresumably there needs to be some human to decide something is worth archiving to stop someone just using it as a free way to store all their holiday snaps?
- wiredpancake 1y ago[dead]
- pabs3 1y agoArchiveTeam members are the ones with access to start crawls of websites, everyone can request they start a crawl, usually they ask for a reason for the crawl, and most reasons mean a crawl will happen.
- jonah-archive 1y agoHi, I run the datacenter/infrastructure team at the Internet Archive! We would love to see you at our various events this fall but if paying for the ticket is difficult for you, please email me (in bio) and we'll get you in (if possible).
- awesomeMilou 1y agoWhat events are we talking about here?
- jackling 1y agoProbably these: https://blog.archive.org/events/ https://blog.archive.org/events/
- NetOpWibby 1y agoI would love to work for IA but openings are rare
- moralestapia 1y agoHey, Q., so what's the size of the internet archive?
- deleted 1y ago[deleted]
- 1y ago
- zghst 1y agoA great milestone for internet history!
- itsme0000 1y agoYeah but their view and download metrics are flat out wrong all the time. If they weren’t a nonprofit they’d be sued for that. But still great company a place for obsolete AWS equipment to retire.
- psychoslave 1y agoWhat do you mean?
- itsme0000 1y agoI run a collection on AI. The view/download numbers are very likely the result of random botting and make no logical sense in terms of rationally what you’d expect to see. I’ll see an item downloaded 10000x normal numbers for one day etc. As for the AWS stuff. Look at the ties between these organizations, pretty clear Amazon is basically self-dealing via a non-profit to write stuff off or have some other scheme.
- FooBarWidget 1y agoI'm kinda surprised IA hasn't long been shutdown by copyright chasers. And for single page archives I tend to use archive.is nowadays. For as long as I can remember, IA has been unusably slow. But still kudos to them for the effort.
- fragmede 1y agoI very much don't get all of the show "king of the hill" being up on there.
- groos 1y agoIt wasn't shut down but definitely hobbled after they lost the lawsuit and were forced to pull copyrighted content from their site that they used to allow signed-in users to check out an hour at a time. My visits to the site dropped 10x after this.
- i_have_to_speak 1y agoIs there an index of all these pages?
- lofaszvanitt 1y agoWould be nice to have visit statistics per domain. So people who host their live sites could determine who visits and what on archive.org under their domain vs their live site :).
- lyu07282 1y agoI was hoping this would include a talk by Jason Scott/@textfiles his talks are always so much fun
- textfiles 1y agoBack at you.
- timmy777 1y agoHow do you prevent government (and other people who can access the data) from rewriting history? Do you hash them in some sort of block chain? The inability to rewrite history will be a fantastic gift to the world.
- not--felix 1y agoI wonder if openai has archived more pages by now
- msephton 1y ago1 trillion web pages archived is quite an achievement. But...there's no way to search them? You have to know what url your want to pull from the archive, which reduces the usefulness of the service. I'd like to search through all those trillion pages for, say, the name of an artist, or for a filename, or for image content.
- qwertytyyuu 1y agoThat would be hell to index
- citbl 1y agoIf it was a commercial problem, e.g. from Google, it would be solved. The reality is that many things don't exist simply because someone isn't paid to do it.
- Keyframe 1y agoHow much AI companies have benefited by leeching off of IA and Common Crawl, it's a shame there's no at least some money flowing back in.
- Exuma 1y agoI imagine it would be no different than current indexing strategies with a temporal aspect baked in... it would act almost like a different site, and maybe roll up the results after the fact by domain
- emporas 1y agoI use GPT web search, and I ask it usually to find textbooks from IA. It works really well for textbooks, but not sure about web pages.
- bluebarbet 1y agoConsider the privacy implications of that. It would effectively create a parallel web where `robots.txt` counts for nothing and where it becomes - retroactively - impossible to delete one's site. Yes, there's ultimately no way to prevent it happening, given that the data is public. But to make the existing IA searchable is IMO just a terrible idea.
- ks2048 1y agoI wonder if Internet Archive and Common Crawl have worked together? How does their scope or infrastructure compare? I know they serve different purposes, but both are essentially doing similar things.
- pabs3 1y agoI think IA ingests crawl WARCs from CC, as well as other groups like ArchiveTeam.
- BiraIgnacio 1y agoCongratulations!
- totaldude87 1y agoSo instead of scrapping all webpages, one just has to pay Archive and get all the data?
- strickinato 1y agoThe artist who is playing at the in person celebration event this week (Sam Reider) is great! That's exciting
- londons_explore 1y agoThe internet archive should be striking deals with AI companies.... We'll load a truck with a copy of our complete archive if you give us a substantial donation to keep the archive going for a few more years. If you don't agree to this deal, you can still access the archive, but it's gonna be at sluggish download speeds and take you years to get all the content.
- Lapra 1y agoThis would destroy the goodwill that they've built up as a public good. People generally don't mind you archiving their content, but if you're selling access to that data, they aren't going to stand for it.
- vivzkestrel 1y agokinda unrelated and stupid question: if we archived the version of every page on the internet every second for 10 years, would there be 1 decillion pages at the end of a decade?
- elleleleledlsl 1y ago[dead]
- philippz 1y agoWe should probably copy this whole thing to IPFS and put it on chain