4 ms·
More than writes, reads is what we struggled with. Within reads the most expensive part was the re-construction of models from the rows returned by SQLite. Wh
by vishnumohandas 3y ago
More than writes, reads is what we struggled with.
Within reads the most expensive part was the re-construction of models from the rows returned by SQLite.
While deserialization is not the responsibility of the database, the overall throughput with SQLite was too low for it to serve our usecase.
- cogman10 3y agohttps://github.com/ente-io/edge-db-benchmarks/blob/43273607d7c55982693d498f10df184a06ed5930/lib/databases/sqlite_db.dart#L84 https://github.com/ente-io/edge-db-benchmarks/blob/43273607d... If this is how you are doing reads, this is your problem. Limit/offset reads are slow. You'll get much better results if you do a proper pagination implementation. This article describes the problem and solution that'd give you much better read results. https://use-the-index-luke.com/sql/partial-results/fetch-next-page https://use-the-index-luke.com/sql/partial-results/fetch-nex... (don't forget page 2 https://use-the-index-luke.com/sql/partial-results/window-functions https://use-the-index-luke.com/sql/partial-results/window-fu... )
- vishnumohandas 3y agoHey yes, thank you for pointing it out! Removing the offsets has unfortunately not sped things up, since it's the step post reading the entries (deserialization) that is the bottleneck[1]. Since SQLite does not support Lists, Protobufs came across as the best way to serialize the data at hand. If there's more native way to solve this problem within SQLite, please let me know! [1]: https://news.ycombinator.com/item?id=39290784 https://news.ycombinator.com/item?id=39290784