4 ms·
What kind of resources would it take to make a search index / crawl the content of the books and papers in Anna's Archive? This would be the most important proj
by carlosjobim 3y ago
What kind of resources would it take to make a search index / crawl the content of the books and papers in Anna's Archive? This would be the most important project for information technology of this decade. They have torrent files of a few hundred gigabytes. Could crawling be done piece by piece on these files?
- criddell 3y agoI'm not exactly sure what you have in mind, but even a laptop should be able to index a terabyte in a day or two.