4 ms·
DuckDB's file format is OLAP-based but supports CRUD operations. I've created DBs with it with 10s of millions of records and gotten amazing query speeds. Effi
by marklit 2y ago
DuckDB's file format is OLAP-based but supports CRUD operations. I've created DBs with it with 10s of millions of records and gotten amazing query speeds.
Efficient querying using indices over S3 for their format isn't something I've looked into just yet.
- liquidcarbon 2y agoDuckDB is amazing, I'm using it every day. It would be marginally useful here though. I really just want to get the binary compressed blob. Fun fact: MS SQL Server has "DECOMPRESS" and DuckDB does not.
- datadrivenangel 2y agoWouldn't that blob not be human readable?
- liquidcarbon 2y agono, but it's one step away with gzip.decompress or similar; if we know the exact position, we can retrieve it with range reads, roughly like this: aws s3api get-object --bucket mybucket --key partially_human_friendly_blob --range bytes=start-end | cut or awk | gunzip