4 ms·

Fair warning, I had serious scaling issues with Timescale. Solutions like Grafana Mimir, Victoria Metrics, Clickhouse, or yes, the new Influx implementation, a

Fair warning, I had serious scaling issues with Timescale. Solutions like Grafana Mimir, Victoria Metrics, Clickhouse, or yes, the new Influx implementation, are much more scalable and will give you much fewer headaches. ClickhouseDB is realy brilliant, btw, it's a powerhouse. Especially with the fairly recent additions that enable hybrid local + S3 option, pushing older metrics to S3 for cheap long-term storage.

PaulWaldman 3y ago
Can you elaborate on what scaling issues you had with Timescale?
- efxhoy 3y ago
  I ran into issues with TS too. Main issue I recall now was maintaining a grand total count of events that were already rolled up into daily counts was not fast since it always looked back at all the data. There was no way in the ts patterns to efficiently express it without handrolling something. The issue was that a grand total count can’t be expressed in terms of a hypertable because there’s no time column. It’s fantastic for workloads that neatly fit in the hypertable pattern though.
  - lispisok 3y ago
    You tried to use a timeseries database without a time column?
    - efxhoy 3y ago
      > Don't be snarky. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Are you expecting a real answer of how I was hoping timescale's internal watermark system would help me roll up a total count or are you just implying I'm an idiot?
      - mardifoufs 3y ago
        What's the difference between the watermark and a time column? Not the person you were replying to but I'm curious since I also thought that TimeseriesDB had a similar "timestamp" to influx.
        efxhoy 3y ago
        TS requires you to specify a time column in each hypertable or view (continuous aggregate) where you want it to work its magic. It then stores an internal watermark that it compares to the time column in the table to figure out from where to read when refreshing. My issue was that for a grand total I didn’t have a time column, so I couldn’t define my query as a continuous aggregate and the query had to start counting from the start of my underlying series each time.
        pbronez 3y ago
        Perhaps: add a time column with an artificially huge time range? This defines the grand total as an interval sum that happens to include all possible intervals.
mfreed 3y ago
If you are open, would love to hear more about some of the challenges you had with Timescale, esp. with your workload. mike (at) timescale or DM on twitter?
atombender 3y ago
The cloud storage option for CH looks like a game changer for time-based data. Any concerns there about accidentally causing "trashing" when cold data is needed? I believe the MergeTree system works by splitting the table into parts during insertion that then later get merged together, so you have to careful that during merging, only the "hot" data is touched, otherwise you'll start pulling in cloud storage data that's supposed to be effectively read-only.
- the_duke 3y ago
  The merging happens primarily on hot data, I haven't run into any issues there. But there are lots of approaches, depending on your needs. You can (should) define a "cache disk" for S3, which will cache up to X Gb locally to avoid trashing. Another option is is to move data into separate (purely S3 backed) tables after a certain time to avoid accidentally fetching large amounts of data from S3. You can still easily join the data together if needed.
mxstbr 3y ago
Agreed. We initially used Timescale for our GraphQL Metrics product[0] but very quickly ran into scaling & performance issues. We switched to Clickhouse and have scaled 10.000x+ since with almost no issues. [0]: https://stellate.co/graphql-metrics https://stellate.co/graphql-metrics
mfreed 3y ago
Also, Timescale similarly introduced S3 for bottomless data tiering: https://www.timescale.com/blog/expanding-the-boundaries-of-postgresql-announcing-a-bottomless-consumption-based-object-storage-layer-built-on-amazon-s3/ https://www.timescale.com/blog/expanding-the-boundaries-of-p...
- wtfishackernews 3y ago
  Only for their managed solution at the moment.

PaulWaldman 3y ago

Can you elaborate on what scaling issues you had with Timescale?

efxhoy 3y ago
I ran into issues with TS too. Main issue I recall now was maintaining a grand total count of events that were already rolled up into daily counts was not fast since it always looked back at all the data. There was no way in the ts patterns to efficiently express it without handrolling something. The issue was that a grand total count can’t be expressed in terms of a hypertable because there’s no time column. It’s fantastic for workloads that neatly fit in the hypertable pattern though.
- lispisok 3y ago
  You tried to use a timeseries database without a time column?
  - efxhoy 3y ago
    > Don't be snarky. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Are you expecting a real answer of how I was hoping timescale's internal watermark system would help me roll up a total count or are you just implying I'm an idiot?
    - mardifoufs 3y ago
      What's the difference between the watermark and a time column? Not the person you were replying to but I'm curious since I also thought that TimeseriesDB had a similar "timestamp" to influx.
      - efxhoy 3y ago
        TS requires you to specify a time column in each hypertable or view (continuous aggregate) where you want it to work its magic. It then stores an internal watermark that it compares to the time column in the table to figure out from where to read when refreshing. My issue was that for a grand total I didn’t have a time column, so I couldn’t define my query as a continuous aggregate and the query had to start counting from the start of my underlying series each time.
        pbronez 3y ago
        Perhaps: add a time column with an artificially huge time range? This defines the grand total as an interval sum that happens to include all possible intervals.

mfreed 3y ago

If you are open, would love to hear more about some of the challenges you had with Timescale, esp. with your workload. mike (at) timescale or DM on twitter?

atombender 3y ago

The cloud storage option for CH looks like a game changer for time-based data. Any concerns there about accidentally causing "trashing" when cold data is needed? I believe the MergeTree system works by splitting the table into parts during insertion that then later get merged together, so you have to careful that during merging, only the "hot" data is touched, otherwise you'll start pulling in cloud storage data that's supposed to be effectively read-only.

the_duke 3y ago
The merging happens primarily on hot data, I haven't run into any issues there. But there are lots of approaches, depending on your needs. You can (should) define a "cache disk" for S3, which will cache up to X Gb locally to avoid trashing. Another option is is to move data into separate (purely S3 backed) tables after a certain time to avoid accidentally fetching large amounts of data from S3. You can still easily join the data together if needed.

mxstbr 3y ago

Agreed. We initially used Timescale for our GraphQL Metrics product[0] but very quickly ran into scaling & performance issues. We switched to Clickhouse and have scaled 10.000x+ since with almost no issues. [0]: https://stellate.co/graphql-metrics https://stellate.co/graphql-metrics

mfreed 3y ago

Also, Timescale similarly introduced S3 for bottomless data tiering: https://www.timescale.com/blog/expanding-the-boundaries-of-postgresql-announcing-a-bottomless-consumption-based-object-storage-layer-built-on-amazon-s3/ https://www.timescale.com/blog/expanding-the-boundaries-of-p...

wtfishackernews 3y ago
Only for their managed solution at the moment.