8 ms·
I love this article as it shows how fast computers really are. There is one conclusion that I do not agree with. Near the end, the author lists cases where you
by koliber 6mo ago
I love this article as it shows how fast computers really are.
There is one conclusion that I do not agree with. Near the end, the author lists cases where you will outgrow flat files. He then says that "None of these constraints apply to a lot of applications."
One of the constraints is "Multiple processes need to write at the same time." It turns out many early stage products need crons and message queues that execute on a separate worker. These multiple processes often need to write at the same time. You could finagle it so that the main server is the only one writing, but you'd introduce architectural complexity.
So while from the pure scale perspective I agree with the author, if you take a wider perspective, it's best to go with a database. And sqlite is a very sane choice.
If you need scale, cache the most often accessed data in memory and you have the best of both worlds.
My winning combo is sqlite + in-memory cache.
- upmostly 6mo agoSeeing the Rust 1M benches were an amazing reminder as to how fast stuff really is.
- koliber 6mo agoThe reality is that things will be blazing fast in any language if you save things by PK in HashMaps.
- tstenner 6mo agoIn the benchmark Rust is more than 50% faster than the runner up
- koliber 6mo agoCorrect. Order-of-magnitude-wise, it's roughly the same as the alternatives. In the context of writing a new service for a new company, you should not spend one second thinking about whether your technical choices will allow you to serve 100,000 requests per second, or 150,000 requests per second. If you are, you are focusing on the wrong thing. If you get to 1,000 requests per second with a real paying client base you already achieved more than most dream of. On the other hand, if you are optimizing a mature distributed low-latency equity trading system that is consuming ten's of thousands of market data ticks per second, a 50% improvement in performance on a 20 machine cluster might turn into some real $$$ savings. But that's not what this article is about.
- pseudosavant 6mo agoSQLite has become my new go-to when starting any project that needs a DB. The performance is very fast, and if anything is ever successful enough to outgrow SQLite, it wouldn't be that hard to switch it out for Postgres. Not having to maintain/backup/manage a separate database server is cheaper and easier.
- koliber 6mo agoBackups are super-simple as well. I'm also a convert.
- oliver236 6mo agowhy sqlite over postgres?
- Croak 6mo agoYou remove a bit of complexity. Sure Postgres is not hard to set up sn to connect to, but Sqlite is just opening a file. It being a file makes it also very easy to test or debug you application.
- koliber 6mo agopostgres is great and is also a good default choice. It needs a bit more setup than sqlite. Unless I need a capability that postgres provides, I go with sqlite. It just works.
- Hendrikto 6mo agoIt is simpler and removes failure points. You don’t need a separate database server process or network/socket connections. Everything happens in-process.
- RugnirViking 6mo agohave you ever run automated tests on postgres? how long did they take?
- direwolf20 6mo ago
- jerf 6mo agoThe one that gets me a lot, which is similar in practice to your point, is when I need server redundancy, even if one server is otherwise plenty for my task. As soon as I'm not running in one place, you need network data storage, and that kicks pretty hard in the direction of a network-accessible database. S3 works sometimes and the recent work on being able to atomically claim files has helped with some of the worst rough edges but it still doesn't take a lot to disqualify it, at least as the only store.
- nine_k 6mo agoIn short, once you need reliability, your complexity necessarily grows due to the redundancy and failover you need to introduce. If your downtime does not cost much, you can host many things on a single tiny computer.
- ryang2718 6mo agoYeah, files as a database is fun, but I find you ultimately reinvent the wheel when sqlite is pretty battle tested, free and easier to get right or scale up. I can't talk though because I actually find myself doing this a lot
- graemep 6mo agoIf you look at those "When do you actually need a database?" constraints I think its missing consistency which prevents bugs and makes debugging easier. When you combine all those a database is a better alternative for all but the simplest cases.
- econ 6mo agoAt some point you end up writing a lot of code to say something that is a single word on the db query. Perhaps (besides simple things) if you have many millions of users and no money. Or if you need something a DB is truly bad at.
- zaphirplane 6mo agoSQLite is famous for single writer how does that reconcile with you criticism of the article