4 ms·
Please don't use csv. At the very least use SQLite. But hosted sqls are probably the smart thing to do.
by dnautics 6y ago
Please don't use csv. At the very least use SQLite. But hosted sqls are probably the smart thing to do.
- ethbr0 6y ago> Please don't use csv Could you elaborate? I'm interested in the specific reasons.
- judge2020 6y agoA quick rundown https://sqlite.org/aff_short.html https://sqlite.org/aff_short.html
- apta 6y agoReliability, atomic updates, roll backs, proper backups, history management, proper schema, etc.
- duskwuff 6y agoHigh-quality software which interacts with real transactional SQL databases is readily available, free, and easy to use. Building an inferior solution yourself, like a CSV-based database, doesn't make your product any better. (If anything, it will probably make it worse.)
- vorticalbox 6y agoIt was never suggested to use a csv database, it was suggested that for small amounts of read only data a csv or other format file is a better option and I agree.
- roenxi 6y agoIf you don't know how to use a database for whatever reason and are doing something as a test-of-concept then CSV is fine. But for anything serious - databases, particularly older featureful ones like postgres, have a lot of efficiency tricks; clever things like indexes and nobody has ever come up with anything that is decisively better organised than the relational model of data. If you use a relational database, the worst-case outcome is you hit tremendous scale and have to do something special later on. The likely case scenario is some wins from the many performance lessons databases have learned. Best case outcome is avoiding a very costly excursion relearning lessons the database community has known about since 1970 (like reinventing transactions). Managing data with a csv (without a great reason) is like programming a complex GUI in assembly without a reason - it isn't going to look like a good decision in hindsight. Most data is not special, and databases are ready for it.
- theamk 6y agoDo use CSV (and other similar formats) for read-only data which fits entirely in the memory. It is great for data safety -- chown/chmod the file, and you can be sure your scripts won't touch this. And if you are accessing live instance, you can be pretty sure that you won't accidentally break it by obtaining a database lock. Now "csv" in particular is kinda bad because it never got standardized, so if you you have complex data (punctuation, newlines, etc..), it you might not be able to get the same data back using a different software. So consider some other storage formats -- there are tons. Like TSV (tab-separated-values) if you want it simple; json if you want great tooling support; jsonlines if you want to use json tools with old-school Unix tools as well; protobufs if you like schemas and speed; numpy's npy if you have millions of fixed-width records; and so on... There is no need to bother with SQL if the app will immediately load every row into memory and work with native objects.
- daniellarusso 6y agoI agree with the JSON suggestion, but what advantage is there to TSV versus CSV? I have experienced pain with both characters (tab and comma), particularly when I am not the one creating the output file.
- junon 6y agoTabs do not appear in common literature. They're easier to justify inputs not have them in order to avoid having a quotation or escaping mess. Commas are _way_ too common. CSV is an awful format anyway.
- theamk 6y agoIf you can make sure the data has no newlines or tabs, the TSV needs no quoting. It is just "split" function, which is present in every language and very easy to use. When I use it, I usually add a check to writer that there is no newlines or tabs in data, and assert if this is not the case. You use this tsv with Unix tools like "cut", "paste", and ad-hoc scripts. There is also "tsv" as defined by excel, which has quoting and stuff. It is basically a dialect of CSV (Python even uses the same module to read it), and has all the disadvantages of CSV. Avoid it.
- hibbelig 6y agoThe way I understand sushsh‘s suggestion is that CSV is an alternative to an API gateway and message queue. I don’t think the suggestion was to replace a database with CSV.