3 ms·
SQLite hides a ton of complexity that lives in the filesystem. It’s incredibly hard to do robust IO correctly with the APIs we have. I almost always choose SQL
by klysm 5y ago
SQLite hides a ton of complexity that lives in the filesystem. It’s incredibly hard to do robust IO correctly with the APIs we have.
I almost always choose SQLite for persisting to disk over JSON files. It essentially removes a large class of bugs and is robust enough that I’m not worried about introducing new problems.
- bob1029 5y agoIn addition to hiding a ton of complexity, SQLite is actually faster than the filesystem in a surprising range of cases.
- anderspitman 5y agoI never questioned whether it was faster or had more features. My whole point is maybe a lot of programs don't need what it offers.
- lokedhs 5y agoMost people don't need everything SQLite offers, but almost all programs needs some of its features. And even if you literally do nothing more than storing rows of text, the API SQLite offers is still more convenient than most filesystem API's once you go beyond the bare minimum. That is to say that there are remarkably few usecases where SQLite isn't better than plain file access.
- anderspitman 5y agoAs I said before, my use cases, while producing useful software, have yet to require more than is offered by a filesystem.
- klysm 5y agoSoftware that isn’t robust, while still useful, can ruin some days.
- newlisp 5y agoSQLite hides a ton of complexity that lives in the filesystem Since they are using Go, couldn't you say the same thing about the Golang std library? As long as they know how to use a local file as database (do the swap, flush, etc...) I don't see the problem.
- gnabgib 5y agoWhat makes you think SQLite uses go? (It's written in C[0]) [0]: https://www.sqlite.org/index.html https://www.sqlite.org/index.html
- newlisp 5y agoWas saying that Go's std lib also "hides a ton of complexity that lives in the filesystem" for you.
- klysm 5y agoNo because it doesn’t give you IO abstractions as robust as SQLite.
- anderspitman 5y agoDo you have an example of a problem I'm likely to encounter at the scale I described in my comment?
- klysm 5y agoIf you are doing IO in a non-atomic manner (seems probable), it’s a matter of time before something happens and a file gets messed up. The frequency of that occurring is certainly a function of scale, but I would argue it’s always a bad idea to deal with your persistent state non-atomically. Additionally, having a schema (even if it’s trivial) protects you from other software problems. It’s much easier to reason about upgrades. And more importantly, if your application complexity suddenly breaches the data complexity threshold where JSON files become a lot more difficult, you already have all the tools you could need at your disposal with SQLite.
- anderspitman 5y ago> If you are doing IO in a non-atomic manner (seems probable) No. I typically use a single mutex for the entire database, and as I said haven't had any performance issues. I'm confident for many classes of software this can scale to at least 100 users (maybe with some slight performance tweaks), which is sufficient for many federated services.
- klysm 5y agoA mutex doesn’t give you atomic or durable IO. I’m referring to proper use of fsync and friends.