4 ms·
Does "all" mean all the URLs publicly known, or did they exhaustively iterate the entire URL namespace?
by do_not_redeem 1y ago
Does "all" mean all the URLs publicly known, or did they exhaustively iterate the entire URL namespace?
- jedberg 1y agoThey iterated the entire URL namespace by having volunteers run a client so they didn't get IP banned.
- barbazoo 1y agoBeautiful. I wish I had seen this and could have helped.
- brokensegue 1y agothey are still archiving other url shorteners https://tracker.archiveteam.org:1338/ https://tracker.archiveteam.org:1338/ you can participate in that
- Imustaskforhelp 1y agoare we sure that the whole entire URL namespace has been mapped? How would that even function, I mean, did they loop through every single permutation and see the result, or what exactly/ how would that work?
- toomuchtodo 1y agoThe pipeline code is available for review of the mechanics of http requests made if you follow the ArchiveTeam wiki links.
- jedberg 1y ago> did they loop through every single permutation and see the result, or what exactly/ how would that work? In short, yes. Since no one can make new links, it's a pre-defined space to search. They just requested every possible key, and recorded the answer, and then uploaded it to a shared database.
- deleted 1y ago[deleted]
- ccgreg 1y agoThe goo.gl URLs that are publicly known are already in the Internet Archive and Common Crawl crawls.