3 ms·

> - Why not use a regular folder and file system with separate metadata files? It seems simpler and more user-friendly. Filesystems don't generally deal well w

> - Why not use a regular folder and file system with separate metadata files? It seems simpler and more user-friendly. Filesystems don't generally deal well with having billions of files. Like you can work around some issues with deep directory structures, but even then it's not very effective and many cases you'll run out of inodes before you even reach a single billion. This is not a use-case filesystems in general tend to optimize for. > - Why not use SQLite? The main reason WARC looks like it does is to be as recoverable as possible if the crawler hard-crashes. There are much fewer things that can go wrong compared other formats. Records are only stored in exactly one location. Data corruption, write faults and other error states are all easy to detect and reason about. You have to work really hard to mess up a WARC file beyond being able to salvage most of it.

kburman 3y ago
> The main reason WARC looks like it does is to be as recoverable as possible if the crawler hard-crashes. I disagree on the recovery part. SQLite being ACID compliant, arguably offers better recovery than the WARC format.
- marginalia_nu 3y ago
  WARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing. A CSV file is arguably harder to recover by comparison.
  - kburman 3y ago
    > WARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing. I just downloaded a sample WARC file to check for any checksums to detect bit rot, but I couldn't find one. Can you share any resources I can explore to understand how WARC is more resistant to hardware errors compared to other file formats?
    - marginalia_nu 3y ago
      It supports hashes of both the record and record+payload, content length for both, and on top of that if a record is bad, you can easily find the next one from the format itself.

kburman 3y ago

> The main reason WARC looks like it does is to be as recoverable as possible if the crawler hard-crashes. I disagree on the recovery part. SQLite being ACID compliant, arguably offers better recovery than the WARC format.

marginalia_nu 3y ago
WARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing. A CSV file is arguably harder to recover by comparison.
- kburman 3y ago
  > WARC is also resistant to hardware errors though, and has this property without requiring constant fsyncing. I just downloaded a sample WARC file to check for any checksums to detect bit rot, but I couldn't find one. Can you share any resources I can explore to understand how WARC is more resistant to hardware errors compared to other file formats?
  - marginalia_nu 3y ago
    It supports hashes of both the record and record+payload, content length for both, and on top of that if a record is bad, you can easily find the next one from the format itself.