3 ms·
Look here: https://github.com/ente-io/edge-db-benchmarks/blob/43273607d7c55982693d498f10df184a06ed5930/lib/databases/sqlite_db.dart#L84 https://github.com/ente-
by SPBS 3y ago
Look here: https://github.com/ente-io/edge-db-benchmarks/blob/43273607d7c55982693d498f10df184a06ed5930/lib/databases/sqlite_db.dart#L84 https://github.com/ente-io/edge-db-benchmarks/blob/43273607d...
SELECT $columnEmbedding FROM $tableName LIMIT 10000 OFFSET $offset
They're not using any indexes, because they're doing offset pagination [1] in a loop. And offset pagination is causing them to scan 550K total rows [2] instead of 100K.
OP, change your code. You're probably wrong about SQLite being slower than Isar, or at least it must be very close.
[1] https://use-the-index-luke.com/no-offset https://use-the-index-luke.com/no-offset
[2] 10K + 20K + 30K + 40K + 50K + 60K + 70K + 80K + 90K + 100K = 550K
- vishnumohandas 3y agoNice catch, thank you! I was playing around with the offsets incorrectly, while attempting to reduce the memory footprint on a low-end Android device. I've updated the code[1] to read all embeddings in one shot, and I'm now measuring the time to read and the time to deserialize separately. On an iPhone simulator running on an Apple M3 Pro, this is what it says: ``` [log] SqliteDB fetch all took: 1572 ms [log] SqliteDB deserialization took: 4268 ms ... [log] Isar: 100000 embeddings retrieved in 616 ms ``` [1]: https://github.com/ente-io/edge-db-benchmarks/commit/e02c3d3811bbfb0296173c960ac219c1f6d90263#diff-f3dd9b936912c331bf9492a92c2f5b6aca5c741b807fab2da43b5f5861fb7077 https://github.com/ente-io/edge-db-benchmarks/commit/e02c3d3...