5 ms·
We should mutualize scraping efforts, creating a sort of Wikipedia of scraped data. I bet a ton of people and cool applications would benefit from it.
by joelthelion 2y ago
We should mutualize scraping efforts, creating a sort of Wikipedia of scraped data. I bet a ton of people and cool applications would benefit from it.
- sakisv 2y agoHaha all we have to do is agree on the format, right?
- Spivak 2y agoWe already did. The format supports attaching related content, the scraped info, with the archive itself. So you get your data along with the means to generate it yourself if you want something different. https://en.m.wikipedia.org/wiki/WARC_(file_format) https://en.m.wikipedia.org/wiki/WARC_(file_format)
- joelthelion 2y agoHonestly I don't think that matters a lot. Even if all sites were scraped in a different format, the collection would still be insanely useful. The most important part is being able to consistently scrape every day or so for a long time. That isn't easy.