4 ms·
Write events in parquet format and use DuckDB for your analytics?
by rodgerd 2y ago
Write events in parquet format and use DuckDB for your analytics?
- notpushkin 2y agoThis. If you’re looking for something portable, DuckDB is hard to beat. It’s pretty much the SQLite for analytics. I’ve had good experience with ClickHouse, too, but it feels a bit more like Postgres rather than SQLite in terms of portability.
- tosh 2y agoTake a look at chDB and clickhouse-local. Clickhouse can do a lot even without running it as a server. https://clickhouse.com/docs/en/chdb https://clickhouse.com/docs/en/chdb https://clickhouse.com/docs/en/operations/utilities/clickhouse-local https://clickhouse.com/docs/en/operations/utilities/clickhou...
- iambateman 2y agoThanks for this...I'd never heard of DuckDB. I'm writing in PHP, so it looks like that's a nonstarter for now, but _very_ interested to see if a PHP extension pops up for this in the future.
- pradeepchhetri 2y agoYou should look at ClickHouse which has good PHP client [0] https://github.com/smi2/phpClickHouse https://github.com/smi2/phpClickHouse
- pradeepchhetri 2y agoThere was a great talk[0] recently from Laravel core team member about ClickHouse, Probably you will enjoy watching it. [0] https://www.youtube.com/watch?v=_jjvaFWWKqg https://www.youtube.com/watch?v=_jjvaFWWKqg
- tosh 2y agoafaiu appending to parquet files is not cheap, you basically have to re-write the file. Do you mean periodically flushing a queue of logs into a new parquet file (e.g. named with a timestamp)?