4 ms·
'Piles-of-files' is itself a bug in an era where sqlite exists and has bindings for virtually every language. Concurrent writing can almost always be reasonabl
by liability 7y ago
'Piles-of-files' is itself a bug in an era where sqlite exists and has bindings for virtually every language. Concurrent writing can almost always be reasonably avoided by keeping the processing parallel but serializing the writes. In cases where that earnestly isn't sufficient, a 'proper' RDBMS is a good option.
Pile-of-files is 1960's era tech. We've learned much since then and our hardware is much more capable. Unless you're shooting for the retro-UNIX aesthetic for artistic reasons, it should be avoided.
- kardos 7y agoI don't think this is one size fits all advice. Surely once your results are measured in TBs this will fall part.