17 ms·
Haha all we have to do is agree on the format, right?
by sakisv 2y ago
Haha all we have to do is agree on the format, right?
- Spivak 2y agoWe already did. The format supports attaching related content, the scraped info, with the archive itself. So you get your data along with the means to generate it yourself if you want something different. https://en.m.wikipedia.org/wiki/WARC_(file_format) https://en.m.wikipedia.org/wiki/WARC_(file_format)
- joelthelion 2y agoHonestly I don't think that matters a lot. Even if all sites were scraped in a different format, the collection would still be insanely useful. The most important part is being able to consistently scrape every day or so for a long time. That isn't easy.