5 ms·
I’m not sure to what degree you want to follow the Redis no concurrency “everything serialized on one thread” model. You can get substantially better performan
by jitl 3y ago
I’m not sure to what degree you want to follow the Redis no concurrency “everything serialized on one thread” model.
You can get substantially better performance out of sqlite by using the lower level https://github.com/crawshaw/sqlite https://github.com/crawshaw/sqlite, turning on WAL etc, using a connection per goroutine for reads, and sending batches of writes over a buffered channel / queue to a dedicated writer thread. That way you can turn off SQLite’s built in per-connection mutex but still be thread safe since each connection is only used on a single thread at a time.
For this use-case you will also probably save a lot of time if you use some large arena-style buffers (probably N per conn?) and copy incoming parameter bytes from the network request/socket to the buffer, or copy straight from sqlite out to the socket, instead of allocating and passing around a bunch of individual strings. Boxing those strings in interface{} (as done by the high level sql stdlib) slows things down even more.
None of this is necessary to get usable perf, even decently good perf, just sharing some tips from my experience trying to get absolutely maximum write throughput from SQLite in Golang.
- nalgeon 3y agoGreat tips, thank you! The thing is, getting maximum throughput is not the goal of the project (at least not at this stage). I'm using reasonable SQLite defaults (including WAL), but that's it for now.
- SJC_Hacker 3y ago> You can get substantially better performance out of sqlite by using the lower level https://github.com/crawshaw/sqlite https://github.com/crawshaw/sqlite, turning on WAL etc, using a connection per goroutine for reads, and sending batches of writes over a buffered channel / queue to a dedicated writer thread. That way you can turn off SQLite’s built in per-connection mutex but still be thread safe since each connection is only used on a single thread at a time. Would this protect against a row update in the middle of a read? e.g. would a row at least be internally consistent at the time it was read?
- lanstin 3y agoYes you would still get SQLite file locking; the thing you aren't using is SQLite thread locking around its own structures (according to GP, I don't know myself). But be sure not to use NFS mount for the files, the locking there is not that great.