3 ms·
I very, very much like the idea of DuckDB, but have had many headaches with what I would consider "low-medium" amounts of log data (400 million rows, 180GB+, af
by mhuffman 6y ago
I very, very much like the idea of DuckDB, but have had many headaches with what I would consider "low-medium" amounts of log data (400 million rows, 180GB+, after stripping out any un-needed text) which works without issue using SQLite, btw, other than the expected slowness of row-oriented storage on the column-aggregate queries that I needed.
I have tried many different ways of importing the data with no luck. Usually it turned into a memory issue. This was irrespective of how I tried to import, whether it was in batches through a script, or using the built-in import functionality, or breaking the data into chunks prior to import, or anything else I tried. I occasionally keep trying to find a way, because I like the idea and features of DuckDB so much.
Of note, MonetDB slurped it right in without issue, but I would much rather use something not so "heavy" like DuckDB.
Also, regarding SQLite, I have DBs with billions of rows that work flawlessly, though not as fast as postgres.
- em500 6y agoYou could try Clickhouse. It's a bit heavier than DuckDB and the default mode is server-client. But you can also use the client (a single binary) without a server to directly query data from csv or parquet files. edit: added a better link, the stand-alone mode is called clickhouse-local https://clickhouse.tech/docs/en/interfaces/cli https://clickhouse.tech/docs/en/interfaces/cli https://altinity.com/blog/2019/6/11/clickhouse-local-the-power-of-clickhouse-sql-in-a-single-command https://altinity.com/blog/2019/6/11/clickhouse-local-the-pow...
- mhuffman 6y agoInteresting! I did not know about the standalone cli feature. Thanks! Will check it out.
- hfmuehleisen 6y agoOne of the DuckDB authors here. Can you open an issue wrt the import problem? Should not happen. Try setting a database file and a memory limit.