10 ms·
> Why is sqlite3 so slow? Well, the benchmark is probably not representative of the typical workload for sqlite (lots of individual insertions and selections).
by Zababa 4y ago
> Why is sqlite3 so slow? Well, the benchmark is probably not representative of the typical workload for sqlite (lots of individual insertions and selections). If we perform the same operations using executemany and one select of all the keys at once, we get:
I think this is because each time you call execute (which is probably sqlite3_exec under the hood), your statement is prepared again, and then deleted, while with executemany it's prepared once and then used with the data. According to the SQLite3 documentation of sqlite3_exec:
> The sqlite3_exec() interface is a convenience wrapper around sqlite3_prepare_v2(), sqlite3_step(), and sqlite3_finalize(), that allows an application to run multiple statements of SQL without having to use a lot of C code.
I knew that when you execute a request a lot, prepared statements are faster, but it seems that it's not exactly the case and that all statements are prepared, the performance improvements come from preparing (and deleting) only once. The documentation page about the Prepared Statement Object has a good explanation of the lifecycle of a prepared statement, which also seems to be the lifecycle of all statements.
[1]: https://www.sqlite.org/c3ref/exec.html https://www.sqlite.org/c3ref/exec.html
[2]: https://www.sqlite.org/c3ref/stmt.html https://www.sqlite.org/c3ref/stmt.html
- quietbritishjim 4y agoThe Python sqlite3 module caches prepared statements, I think it's the last 100 by default but can be overridden. To be effective, this cache requires you to use proper parameters (rather than string interpolation) but the benchmark code for this article does that. So, in effect, it already uses stored procedures. As others have pointed out, it's the lack of a suitable index that's the real problem here.