It's also sensible to have high quality record markers that make identifying and skipping broken records easy. For example, recordio, a record-based container

It's also sensible to have high quality record markers that make identifying and skipping broken records easy. For example, recordio, a record-based container format used at Google in the early mapreduce days (and probably still used today) was explicitly designed to skip corruption efficiently (you don't wnt a huge index build to fail halfway through because a single document got corrupted). Even so, I avoid parallel appenders to a single file, it can be hard to reason about and debug in ways that having each process appending to its own file doesn't.