3 ms·
My search engine mixes data in MySQL, SQLite-databases and the Parquet-format. The needs and sizes of these data stores are so disparate there's not really one
by marginalia_nu 3y ago
My search engine mixes data in MySQL, SQLite-databases and the Parquet-format. The needs and sizes of these data stores are so disparate there's not really one store that suits them all. I don't need concurrent write safety for for crawl data, for example, and enforcing that would make the crawler slower.
This basically lets me access all this data for ad-hoc data munging. I don't think DuckDB will replace these things, so much as help me integrate them for OLAP-style queries.