3 ms·
I agree that sqlite has a number of similar benefits - openness, table abstractions and concurrent transactions. It’s also a library so close to how delta, ice
by benjaminwootton 3y ago
I agree that sqlite has a number of similar benefits - openness, table abstractions and concurrent transactions. It’s also a library so close to how delta, iceberg and hudi are implemented.
I’m glad it’s had an uptick in interest recently but I haven’t yet seen it mentioned for analytics yet.
I assume it’s row rather than column oriented?
- speedgoose 3y agoYes, SQLite is row oriented. It's not a very space efficient format because it also doesn't support compression or compact representations of numbers in binary. But it doesn't rely on the JVM and a typical JVM ecosystem. It is a big benefit for some use-cases, like dealing with numerical data on the edge.
- bbuchert 3y agoIf you are searching for the analytics equivalent you should check out duckdb.
- speedgoose 3y agoDuckDB is about querying existing files or am I missing something?
- hobofan 3y agoThe documentation puts a lot of emphasis on loading/querying existing files, as that's the first thing you will want to do for a OLAP use-case, but in general it's the same as sqlite where you maintain a database file.
- speedgoose 3y agoThanks. That's good to know. The file format is not highlighted in the documentation but I found some information about it: https://duckdb.org/internals/storage.html https://duckdb.org/internals/storage.html It seems to be a bit early to rely on it to store data in an object store, but I will do some tests to compare with SQLite: > The DuckDB internal storage format is currently in flux, and is expected to change with each release until we reach v1.0.0.
- dist-epoch 3y agoDuckDB file format is the compressed column oriented equivalent of SQLite files.
- hobofan 3y agoDelta Lake also isn't JVM based. The main implementation is a Rust library with first-party Python bindings.
- gigatexal 3y agoOLAP or column oriented SQLite is DuckDB.