4 ms·

hi there - co-founder of questdb here. The demo on our website hosts a 1.6 billion rows NYC taxi dataset with 10 years of weather data with around 30-minute res

by j1897 6y ago

hi there - co-founder of questdb here. The demo on our website hosts a 1.6 billion rows NYC taxi dataset with 10 years of weather data with around 30-minute resolution and weekly gas prices over the last decade. We've got example of queries in the demo, and you can see the execution times there. We have posted a blog post comparing the ingestion speed of InfluxDB and QuestDB via InfluxDB Line Protocol some time ago: https://questdb.io/blog/2019/12/19/lineprot https://questdb.io/blog/2019/12/19/lineprot

dominotw 6y ago
> We are orders of magnitude faster than TimescaleDB and InfluxDB I think gp might be asking for a source for this claim. I see execution times on the demo but not sure if thats enough to say its faster than timescale.
- sirffuzzylogik 6y ago
  j1897 is referring to https://questdb.io/blog/2020/04/02/using-simd-to-aggregate-billions-of-rows-per-second#perspectives-on-performance https://questdb.io/blog/2020/04/02/using-simd-to-aggregate-b...
  - srini20 6y ago
    (Hard to draw many meaningful conclusions from a single, extremely simple query without much explanation?) Graph shows PostgreSQL as taking a long time, but doesn't say anything about configuration or parallelization. PostgreSQL should be able to parallelize that type of query since 9.6+, but I think they didn't use parallelization in these experiments with PostgreSQL, even though they used a bunch of parallel threads with QuestDB? So would be good to know: - What version of Postgres - How many parallel workers for this query - If employing JIT'ing the query - If pre-warming the cache in PostgreSQL and configuring it to store fully in memory (as benchmarks with QuestDB appeared to do a two-pass to first mmap into memory, and only accounting for the second pass over in-memory data). etc Database benchmarking is pretty complex (and easy to bias), and most queries do not look like this toy one.
    - sirffuzzylogik 6y ago
      I agree that our blog post lacks of details, here are some: - PostgreSQL 12 - 12 - No - We ran the test using the pg_prewarm [0] module, the difference was negligible Regarding the "toy" query, the reason we are showcasing this instead of other more complex queries is because this is a simple, easily reproducible benchmark. It provides a point of reference for performance figures. > Database benchmarking is pretty complex (and easy to bias), and most queries do not look like this toy one. I would say that benchmarking is very hard. We tried not to perform a biased benchmark by running something that is not time-series specific and which does not put us in advantage compared to what Postgres should do. The takeaway from this is that configuration is important and we should expose it. The next benchmark we do will have an associated repository so people can review our config and point non optimal items if any. [0]: https://www.postgresql.org/docs/9.4/pgprewarm.html https://www.postgresql.org/docs/9.4/pgprewarm.html

by j1897 6y ago

dominotw 6y ago

> We are orders of magnitude faster than TimescaleDB and InfluxDB I think gp might be asking for a source for this claim. I see execution times on the demo but not sure if thats enough to say its faster than timescale.

sirffuzzylogik 6y ago
j1897 is referring to https://questdb.io/blog/2020/04/02/using-simd-to-aggregate-billions-of-rows-per-second#perspectives-on-performance https://questdb.io/blog/2020/04/02/using-simd-to-aggregate-b...
- srini20 6y ago
  (Hard to draw many meaningful conclusions from a single, extremely simple query without much explanation?) Graph shows PostgreSQL as taking a long time, but doesn't say anything about configuration or parallelization. PostgreSQL should be able to parallelize that type of query since 9.6+, but I think they didn't use parallelization in these experiments with PostgreSQL, even though they used a bunch of parallel threads with QuestDB? So would be good to know: - What version of Postgres - How many parallel workers for this query - If employing JIT'ing the query - If pre-warming the cache in PostgreSQL and configuring it to store fully in memory (as benchmarks with QuestDB appeared to do a two-pass to first mmap into memory, and only accounting for the second pass over in-memory data). etc Database benchmarking is pretty complex (and easy to bias), and most queries do not look like this toy one.
  - sirffuzzylogik 6y ago
    I agree that our blog post lacks of details, here are some: - PostgreSQL 12 - 12 - No - We ran the test using the pg_prewarm [0] module, the difference was negligible Regarding the "toy" query, the reason we are showcasing this instead of other more complex queries is because this is a simple, easily reproducible benchmark. It provides a point of reference for performance figures. > Database benchmarking is pretty complex (and easy to bias), and most queries do not look like this toy one. I would say that benchmarking is very hard. We tried not to perform a biased benchmark by running something that is not time-series specific and which does not put us in advantage compared to what Postgres should do. The takeaway from this is that configuration is important and we should expose it. The next benchmark we do will have an associated repository so people can review our config and point non optimal items if any. [0]: https://www.postgresql.org/docs/9.4/pgprewarm.html https://www.postgresql.org/docs/9.4/pgprewarm.html