4 ms·

Search engines scrape data. Why not simply mandate that they have to make that data available to all?

renegat0x0 3y ago
I do not believe that Google would share anything if it was not required by law. I heard there are other public community driven solutions, not necessarily search engines. For example marginalia search engine provides some domain data, but I think it was distributed as some sort of binary info. I have developed myself an engine to scrap data, which is using beautiful soup. I export my data to a github repository, where it is available. All domains receive a page rating, based on their content. I believe that all pages should provide scrap-able title, description and a valid status code. We as a community should produce scrapping open source standards, and implement them. People interested in open source and interoperability of the web should always provide that kind of data. It should not be hidden behind redirects, and javascript code.
- marginalia_nu 3y ago
  Most of what I make available are just sample data for testing the search engine, it's a tiny fraction of the entire crawl set for purely logistical reasons. I don't have the bandwidth to upload datasets on the order of terabytes. Though I've begun distributing dumps of the domain and link database if anyone is interested: https://downloads.marginalia.nu/exports/ https://downloads.marginalia.nu/exports/ Might be persuaded to share the the document metadata databases as well in some form, as that's only order of 90 GB. Would contain titles and summaries and a few other flags for each URL. > We as a community should produce scrapping open source standards, and implement them. People interested in open source and interoperability of the web should always provide that kind of data. It should not be hidden behind redirects, and javascript code. Well there's already the WARC format for that.

I do not believe that Google would share anything if it was not required by law. I heard there are other public community driven solutions, not necessarily search engines. For example marginalia search engine provides some domain data, but I think it was distributed as some sort of binary info. I have developed myself an engine to scrap data, which is using beautiful soup. I export my data to a github repository, where it is available. All domains receive a page rating, based on their content. I believe that all pages should provide scrap-able title, description and a valid status code. We as a community should produce scrapping open source standards, and implement them. People interested in open source and interoperability of the web should always provide that kind of data. It should not be hidden behind redirects, and javascript code.

marginalia_nu 3y ago
Most of what I make available are just sample data for testing the search engine, it's a tiny fraction of the entire crawl set for purely logistical reasons. I don't have the bandwidth to upload datasets on the order of terabytes. Though I've begun distributing dumps of the domain and link database if anyone is interested: https://downloads.marginalia.nu/exports/ https://downloads.marginalia.nu/exports/ Might be persuaded to share the the document metadata databases as well in some form, as that's only order of 90 GB. Would contain titles and summaries and a few other flags for each URL. > We as a community should produce scrapping open source standards, and implement them. People interested in open source and interoperability of the web should always provide that kind of data. It should not be hidden behind redirects, and javascript code. Well there's already the WARC format for that.