4 ms·
> The main reason WARC looks like it does is to be as recoverable as possible if the crawler hard-crashes. I disagree on the recovery part. SQLite being ACID c
by kburman 3y ago
> The main reason WARC looks like it does is to be as recoverable as possible if the crawler hard-crashes.
I disagree on the recovery part. SQLite being ACID compliant, arguably offers better recovery than the WARC format.
- marginalia_nu 3y agoWARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing. A CSV file is arguably harder to recover by comparison.
- kburman 3y ago> WARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing. I just downloaded a sample WARC file to check for any checksums to detect bit rot, but I couldn't find one. Can you share any resources I can explore to understand how WARC is more resistant to hardware errors compared to other file formats?
- marginalia_nu 3y agoIt supports hashes of both the record and record+payload, content length for both, and on top of that if a record is bad, you can easily find the next one from the format itself.