4 ms·

Thanks for the thoughtful reply Paul. I agree that in the end, InfluxDB ended up not being the right tool for the job. We're using it more-so as a backend anal

by LogicX 9y ago

Thanks for the thoughtful reply Paul. I agree that in the end, InfluxDB ended up not being the right tool for the job. We're using it more-so as a backend analytics system for our customer dashboard, making it query-heavy. Sorry, you may have read an earlier version of the post -- I'd since updated it to specify that version 1.3 is the last version we used, a few months ago. We currently do rollups in TimescaleDB via a cronjob that runs a stored procedure. It references a rollup table for the bigid of the last data element in our raw table. It is then rolled up with an upsert. In this way, we don't miss any items while rolling up. They're working to make it more 'native' to the TimescaleDB offering, but it's been working great for us. We use Kafka only as a queue to send data to multiple timescale instances. Kafka has allowed for us to shift to a different timescale instance when making database changes, if an alter table is blocking. Not all table or index changes are blocking though.

pauldix 9y ago
Thanks for the reply. Interesting that you're using crons to query and then upsert. We're going to be doing the same thing in our 2.0 work. That paired with our new query engine should make things much better on our end. So it's one of those things that we know is a weakness and we're working quickly to try to solve and make it a first class citizen in the DB. For many workloads we've had people working successfully, but we keep expanding our focus as we go. If you tested against 1.3 we might have more work on queries and writes at the same time. Although it may be that with the 1.5 and TSI enabled, that problem would largely go away, but not sure. Can you provide more detail about how many QPS you were pushing? For the different Timescale instances, are they share nothing and your write and query pipelines shard based on customer? Didn't think there was a clustering solution available. Thanks again for providing more detail, much appreciated.

by LogicX 9y ago

pauldix 9y ago

Thanks for the reply. Interesting that you're using crons to query and then upsert. We're going to be doing the same thing in our 2.0 work. That paired with our new query engine should make things much better on our end. So it's one of those things that we know is a weakness and we're working quickly to try to solve and make it a first class citizen in the DB. For many workloads we've had people working successfully, but we keep expanding our focus as we go. If you tested against 1.3 we might have more work on queries and writes at the same time. Although it may be that with the 1.5 and TSI enabled, that problem would largely go away, but not sure. Can you provide more detail about how many QPS you were pushing? For the different Timescale instances, are they share nothing and your write and query pipelines shard based on customer? Didn't think there was a clustering solution available. Thanks again for providing more detail, much appreciated.