Most of what I make available are just sample data for testing the search engine, it's a tiny fraction of the entire crawl set for purely logistical reasons. I

Most of what I make available are just sample data for testing the search engine, it's a tiny fraction of the entire crawl set for purely logistical reasons. I don't have the bandwidth to upload datasets on the order of terabytes. Though I've begun distributing dumps of the domain and link database if anyone is interested: https://downloads.marginalia.nu/exports/ https://downloads.marginalia.nu/exports/ Might be persuaded to share the the document metadata databases as well in some form, as that's only order of 90 GB. Would contain titles and summaries and a few other flags for each URL. > We as a community should produce scrapping open source standards, and implement them. People interested in open source and interoperability of the web should always provide that kind of data. It should not be hidden behind redirects, and javascript code. Well there's already the WARC format for that.