3 ms·
I second it. I'm regularly using SQLite to look and format data across millions of rows, it's much faster than that, but you have to use transactions and optimi
by charles_f 3y ago
I second it. I'm regularly using SQLite to look and format data across millions of rows, it's much faster than that, but you have to use transactions and optimize queries. It sounds to me like they didn't read the manual and are blaming the tool.
- vishnumohandas 3y agoHello, OP here! The code we used for running the benchmarks is available here[1]. If you have suggestions on how we could speed up the deserialization during reads, please do share. We'd be happy to go back to SQLite. [1]: https://github.com/ente-io/edge-db-benchmarks https://github.com/ente-io/edge-db-benchmarks
- maskros 3y agoThe embeddings are just a flat array of floating point numbers. Try typed arrays/buffers to read and write the typed arrays directly as BLOB data. Protobuf and JSON and other serialization schemes are way overkill for something that should be as simple as a memcpy.
- vishnumohandas 3y agoYes, this[1] did the job, thank you! [1]: https://news.ycombinator.com/item?id=39291411 https://news.ycombinator.com/item?id=39291411