4 ms·

Anecdotal evidence, but I produced a medium-size crawl in the past (~20TB compressed). I used distributed resources with off-the-shelf libraries (i.e. warcprox

Anecdotal evidence, but I produced a medium-size crawl in the past (~20TB compressed). I used distributed resources with off-the-shelf libraries (i.e. warcprox etc.) and managed to get corrupted data in some cases where neither the length-delimited (i.e. offset + payload length) nor the newline-delimited (triple newlines between records) logics were valid any longer. Took me some time to build a repair tool for that. Sqlite has an amazing set of well-understood and documented guarantees on top of performance, there's a host of potential validation tools to choose from and you can even use transactions etc. So that alone seems like a great idea. What's more, you can potentially skip CDX files if you have sqlite databases (or build your own meta sqlite database for the others quickly).

rengler33 4y ago
Is there a forum or somewhere web crawlers hang out online? I'd love the learn about more sophisticated projects like this.
- uniqueuid 4y ago
  In github issues of said projects, and at scientific web archival conferences. Although I'd absolutely welcome some sort of channel!
  - smcnally 4y ago
    Topics include Issues and activities across projects. These topics are quite active, e.g. https://github.com/topics/crawling https://github.com/topics/crawling https://github.com/topics/web-scraping https://github.com/topics/web-scraping https://github.com/topics/web-archiving https://github.com/topics/web-archiving
ikreymer 4y ago
A bit late to this thread, but I think WARC is a reasonable format for raw HTTP traffic. We should definitely have better tools to ensure WARC files produced are valid, and that's one of the things we build at Webrecorder. Unless you're crawling really text heavy content, most of the WARC data is binary content that doesn't really need to be in a db. However, sqlite or database as replacement for CDX is an appealing option, where WARC files can remain static data at rest and derived data (offests, full-text search, can be put into a db. We are experimenting with a new format, WACZ, which bundles WARC files into a ZIP, while adding CDXJ and exploring sqlite as an option for full-text search. I agree that it's better to build on solid, existing formats that can be validated, especially when large amounts of data are concerned!

rengler33 4y ago

Is there a forum or somewhere web crawlers hang out online? I'd love the learn about more sophisticated projects like this.

uniqueuid 4y ago
In github issues of said projects, and at scientific web archival conferences. Although I'd absolutely welcome some sort of channel!
- smcnally 4y ago
  Topics include Issues and activities across projects. These topics are quite active, e.g. https://github.com/topics/crawling https://github.com/topics/crawling https://github.com/topics/web-scraping https://github.com/topics/web-scraping https://github.com/topics/web-archiving https://github.com/topics/web-archiving

ikreymer 4y ago

A bit late to this thread, but I think WARC is a reasonable format for raw HTTP traffic. We should definitely have better tools to ensure WARC files produced are valid, and that's one of the things we build at Webrecorder. Unless you're crawling really text heavy content, most of the WARC data is binary content that doesn't really need to be in a db. However, sqlite or database as replacement for CDX is an appealing option, where WARC files can remain static data at rest and derived data (offests, full-text search, can be put into a db. We are experimenting with a new format, WACZ, which bundles WARC files into a ZIP, while adding CDXJ and exploring sqlite as an option for full-text search. I agree that it's better to build on solid, existing formats that can be validated, especially when large amounts of data are concerned!