I'm not saying that one should use a file as long as possible but a real DB from the start for anything the system writes is not always warranted. Yes, with a f

I'm not saying that one should use a file as long as possible but a real DB from the start for anything the system writes is not always warranted. Yes, with a file you'll need to think about either escaping or data validation but there is an option to use json-per-line at a price of higher overhead (compare to TSV) and a JSON library will take care of serialization (including escaping). Locking is not required (on Linux/BSD) if the file is opened in the append mode and each line is written as a single write request (which is atomic). > But now its harder to change than it would have been to do the right thing at the start, and the sunk cost fallacy means that everyone's brain is resistant to throwing away all the work that's been invested With "proper" implementation suck cost fallacy can be even stronger because more effort was spend to make it right from the beginning. IMHO switching from a file to say ClickHouse would be as easy as switching from MySQL to ClickHouse (or my be even easier).