6 ms·
I wonder what is the best way to process huge amounts of time-series data (several thousand records per second). When I search for "time series database" I get
by renonce 5y ago
I wonder what is the best way to process huge amounts of time-series data (several thousand records per second). When I search for "time series database" I get something like MySQL, InfluxDB, TimescaleDB etc, but all of them are way too powerful and have their own query language and storage engine etc which are hard to learn and manipulate. In case you run out of storage/memory/CPU etc or when you want to query something that the database doesn't support, there is no way out of it without waiting for the database vendor to provide some new feature.
Now I'm just writing all data to plain text files, one JSON object per line, and query and process them with cli tools like jq. Regular compression tools like zstd, pattern matchers like grep works. All the Unix philosophy applies and it's easy to do anything I want without being restricted to the features of a certain database.
- LoriP 5y agoJust wanted to say that TimescaleDB is an extension of PostgreSQL and so it uses SQL. MySQL would use SQL too. InfluxDB does, indeed, have its own query language though.
- Beltalowda 5y agoTimescaleDB "upgrades" fairly gracefully from regular PostgreSQL; there's a bunch of Timescale-specific functions, but it's not that much: mostly it's just regular SQL. I don't know how well it performs with thousand records per second though.
- spaniard89277 5y agoMmm maybe DuckDB?
- ajoseps 5y agoI have a similar issue. I've been using feather files/parquet files for storage, and just using pandas to do analysis. There is an issue where the initial load/convert essentially doubles the memory usage of the file itself as it converts to a pandas dataframe. This can be avoided if you use a feather file and follow its recommendations for a zero-copy conversion (no NaNs/nulls). I think it's a bit more flexible than using cli tools since you can set some sort of time index and query specific timeslices fairly easily