3 ms·
That depends on what your "full scale loads" are. SQLite can easily scale out to read workloads that can consume an entire (large) machine, especially when usi
by andyjpb 10y ago
That depends on what your "full scale loads" are.
SQLite can easily scale out to read workloads that can consume an entire (large) machine, especially when using its mmap'd IO modes.
However, it doesn't have good concurrency for writers and concurrency for readers also drops when any writer is large enough to spill the cache.
Writer concurrency can be managed down by building a custom write pipeline in front of it but that is not trivial. ...and, when you do that, you then have to manage your own consistency and Read/Modify/Write hazards so you're no better off than any of the first generation NoSQL stores. If you think about your data model up front then that might be OK, otherwise it's a very difficult road.
An important other difference between SQLite and other database engines (RDBMS or otherwise) is that SQLite is in-process rather than client/server.
This has a lot of implications for the latency costs associated with data access. In particular, SQLite allows you to mostly ignore query round-trip times: because there aren't any. If you change your engine to something that's not in-process you suddenly start having to think about batching and prefetching for things that might initially look like a simple point query. This can have a serious performance impact and result in profound refactoring requirements for your app. Things that were previously running at several 10s of thousands of point queries per second and complete in nanoseconds now require 10s of thousands of sockets and take 10s of milliseconds.
For my part, I've built production web apps with lots of users on SQLite and the database engine was the smallest part of our performance work. Reducing garbage generation in the render pipeline and moving query generation out of the request path were where we got most of our performance gains.
- coleifer 10y agoRegarding concurrency, the folks at Oracle welded sqlites front-end onto berkeleydb. Since bdb does page-level locking you can potentially see much greater concurrency with the bdb implementation.