14 ms·

TimescaleDB raises $40M

308 points by eloff 6y ago

alexbouchard 6y ago
I'm pretty stoked for this. Timescale ability to use time series on a subset of tables (hypertables) really makes it an interesting choice. I've just dabbled with it but seeing that they'll finally be improving their hosted solution make me want to dive in deeper! Anyone has experience running large DB on timescale, would you recommend it?
- eloff 6y ago
  I haven't tried running the distributed Timescale DB - in general more parts means more things to go wrong, so using their managed cloud service for that is a good idea. But I can attest that the single-server version is rock solid, just like PostgreSQL that it's based on. And it's free and source visible. The pace of innovation has also been really high, it just keeps getting better with every release.
- spmurrayzzz 6y ago
  If you end up transitioning to multinode deployments from a single node deployment, there are a bunch of tradeoffs to consider. A good reference point for this to check out the `multinode` label in their github issues. https://github.com/timescale/timescaledb/labels/multinode https://github.com/timescale/timescaledb/labels/multinode One of the big items that stood out to me is the inability to be able to migrate data from an existing table when creating a distributed hypertable. There were also some significant query performance reports as well. These all may improve with time of course, so watching the dev cycles will give you a good sense of that I think.
AzzieElbab 6y ago
Please free us of kdb
- haolez 6y ago
  Why? I'm genuinely curious, since I've never worked with kdb but I've often heard great things about it (regardless of the weird query language).
  - AzzieElbab 6y ago
    Kdb is like oracle db that somehow won’t die. It could use some competition
    - RyanHamilton 6y ago
      Standard SQL is rubbish for time series queries as it is based on set theory which does not have order. Most SQL datbases exploit that fact to increase performance. Fundamentally kdb is based on ordered lists, which is a much better paradigm for time series data. The contrasting queries can be seen here: http://www.timestored.com/b/kdb-qsql-query-vs-sql/ http://www.timestored.com/b/kdb-qsql-query-vs-sql/ I do agree they could use competition as the overall offering is weak but the core database is very strong.
      - j1897 6y ago
        Have you checked questdb [1]? the data structure is arrays with data that lands in order and SQL queries on top. The fallback was that it was difficult to deal with out of order data, but we have just solved this by re-ordering data on the fly in memory before it hits the disk. Performance wise probably not far from kdb itself (will be sharing some bench results soon vs open source tsdbs) NB: I'm one of the co-founder of questdb [1] https://www.questdb.io https://www.questdb.io
- shin_lao 6y ago
  TimescaleDB isn't an alternative to KDB+. KDB+ is more a programming platform than a database.
  - AzzieElbab 6y ago
KingOfCoders 6y ago
Love timescaledb. Scaled several postgres instances for time data successfully with it.
chirau 6y ago
I love Timescale, and the team as well. Good folks. Congrats!
FirstLvR 6y ago
cheers! i love Timescale
WFHRenaissance 6y ago
Used TimescaleDB in one of my first real DE projects – huge fan. Happy to see this!
somethingAlex 6y ago
Very happy with our choice to use TimescaleDB. The idea to simply make it a Postgres extension was brilliant. The compression release was one of the cooler features I've seen in recent times. Row database for recent transactional data, columnar compressed database for historical OLAP workloads - pretty much automagically.
- halbritt 6y ago
  Curious how large a dataset you're using?
  - somethingAlex 6y ago
    Under 100GB; I'm sure vanilla Postgres would suit our needs too. However, adding TimescaleDB on top was not much of an investment and in exchange we got an interface for operations we do often, effortless continuous aggregation, near-constant time appends, and a native way to leave data mutable for a period of time before marking it immutable and compressing it. The performance is a great feature but its also just an intuitive, familiar (pretty much just SQL) tool that makes life easier.
    - halbritt 6y ago
      Cool. I'm keen to try it. Wondering how well it works with multi-terabyte data sets.
jb1991 6y ago
We switched from Datomic to Timescale and haven’t looked back.
- swamiji 6y ago
  If you could share more details, I'd really appreciate it.
simplify 6y ago
> When we launched TimescaleDB, we met a fair amount of skepticism.... The top voted Hacker News comment at the time called us, “a rather bad idea[0].” Good old HN with its healthy skepticism :) [0] https://news.ycombinator.com/item?id=14036554 https://news.ycombinator.com/item?id=14036554
- eloff 6y ago
  I think it goes to show how impossible it is to judge an idea. YC itself doesn't pretend to do this with the ~15,000 applications they go through each cohort. They try instead to look at the team, look at their progress, imagine what would need to happen for the company to succeed at the level required for them to get the returns they seek. Founders should have a thick skin when it comes to criticism on HN, because we don't know either.
- kirse 6y ago
  Having been on HN long enough, what I look for during any idea/startup launch is polarization and intensity of viewpoints. If people are reacting to the idea (for better or worse), it means its had an impact. Those are often the products that find success. A no-comment launch is far worse than one riddled with criticism. IMO HN's classic "skepticism" is usually just engineering nerd insecurity projected outwards, with enough techno-jargon to maintain plausible deniability. Folks feel threatened by a great idea so it's safer to find some way to tear it down. Not to dismiss all feedback as projected insecurity of course.
  - etaioinshrdlu 6y ago
    I'm still confused why time-series databases are even a thing. It seems to me that time-series just means you have a date/time column plus an index on it. Which is something typical databases already do well, and like the referenced post mentioned, you could use a column store for better performance. But I just don't see anything that makes creating an entire database design for one specific index type worthwhile... I index many tables on my site by num_upvotes so I can find the top ranked items to show. Does this mean that I need an UpvoteDB? I don't think so. A previous time I argued this point, it was mentioned that you rarely need to update or delete old rows. This allows you to tailor the storage solution better. However, this basically means a compressed column store, which again, doesn't really have much to do with time.
neom 6y ago
I was in Grand Central Tech with the Timescale folks, became friends with both Ajay and Mike. Ajay gave me a lot of good thoughts on building my startup(thanks!), and Mike is..well.. just hyper smart. That is to say, I'm not surprised to read this and for what it's worth: they deserve it, really great humans! :)
- mfreed 6y ago
  Thanks, super appreciated! I remember always enjoying our conversations as well!
- akulkarni 6y ago
  Hi John, good to hear from you! (And thank you :-)
ironman1478 6y ago
This is awesome. In certain industries (manufacturing, energy, etc) there are companies (really 1 company) that essentially have monopolies on time series databases / historians. There has been 0 competition in that space and as a result the databases and surrounding client tools are just so awful. It'll be interesting to see if timescaledb can really enter that market and force those companies to adapt.
- ltnublet 6y ago
  What company is that?
  - stevesimmons 6y ago
    KDB in finance?
  - ironman1478 6y ago
    I was employed there so I don't feel entirely comfortable stating their name. Just google around for "operational historian". It was really frustrating because we had these ideas years ago (and many other ideas), but due to reasons we were not allowed to actually make the product genuinely better. Also, the product is in more places than you'd think because they partner with vendors to sell the database as a component in a larger system that the vendor packages up and sells.
  - some0x80070005 6y ago
    I suspect AVEVA PI (formerly OSIsoft) is what they are referring to. PI is excessively popular in Enterprise scenarios either OT or IT.
  - kevinherron 6y ago
    He's probably referring to OSIsoft's PI historian.
    - alexmarcy 6y ago
      Their stuff is a nightmare to use, and is insanely expensive. Always nice when we get to just swap it out for something like Canary or even Ignition although folks are always trying to trash the Ignition Historian when it works well for most use cases people need to solve.
_joel 6y ago
We used TimescaleDB for a flood prediction tool, works really well, kudos.
- msaharia 6y ago
  Hi, I am a hydrologist. Would love to hear about what you have done using TimeScaleDB
  - _joel 6y ago
    We used historical flow meter readings (started out with 15 minute intervals but it worked much better with a higher frequency) and used that to train Recurrent Neural Networks (RNN's) to predict where areas were likely to flood. I was the devops lead on it for the prototype, not the data scientist unfortunately so can't give you the in's and outs but can tell you that we used tensorflow together with pandas/df/timescaledb. We then displayed that using plotly, all this was stuffed inside several containers. It was a great project to work on actually. The whole setup was pretty much a joy to work with.
    - msaharia 6y ago
      That is awesome. I work in flood forecasting and previously worked in NCAR/NASA. If you would be so kind, feel free to share any white papers, links etc to the project you are referring to. msaharia[a]iitd.ac.in
      - _joel 6y ago
        I don't I'm afraid, for starters I contract and no longer work on the project, plus there's IP invovled etc. Also I'm no data scientist, just a hacker really :) The main difficulty was getting access to the data (and ensuring it was valid, as with all ML projects), lucking we managed to get that from several sources (councils, water companies etc). The flow data was in TimescaleDB, pandas dataframes so we could use varying levels of frequency of data and we used HDF5 iirc as well (detail is hazy, it was a few years back now). We did demo it to the Met Office here in the UK too. They were interested but already had their own thing cooking up so the project never really got out of prototype, but it was making accurate predictions. I think there were some other areas that might turn out to be flaky over time using this method (such as rapid changes to catchment areas) but that could maybe be factored in someway with more thought on the model and verification on a larger set/timeframes. Feel free to hit me up if you want any more detail on the tech side, but don't ask me about stats, maths fu I ain't :) joel [at] smashthesystems.com
zomgwat 6y ago
TimescaleDB looks great. I'm interested in using it but I'm concerned about the upgrade path across major PostgreSQL versions. Logical replication is a big help when upgrading PostgreSQL across major versions while minimizing downtime. As far as I understand it, TimescaleDB doesn't support logical replication yet. Major version upgrades with TimescaleDB is obviously a solvable problem but it probably means we'll have a more complicated upgrade path. Upgrading via logical replication is just so nice.
some0x80070005 6y ago
I guess this is an unpopular opinion, but I’ve found InfluxDB to be superb for being trivial to get going in a high performance way. I have never touched InfluxDB Cloud - always just InfluxDB either as an arbitrary process or container. Examples of where I’ve found InfluxDB to be more pleasant: * InfluxDB has way better documentation on functions. For example, look up moving average by time (not points) on TimescaleDB vs InfluxDB. We use these more complex queries and have no problem on Influx. Going further, the number of functions built in is impressive with the same ability to define new functions. * InfluxDB containers are totally self contained which is great for simple architectures. As a process, InfluxDB is a single executable thanks to Go. * This is extremely subjective, but I find Flux easier to comprehend as a separate query vs. the use of SQL to do higher complexity functions; however, I am sure this is due to my lack of experience and know how to write said queries in SQL.
- eloff 6y ago
  I wouldn't choose InfluxDB over TimescaleDB. There's a reasonably balanced comparison here from the Timescale guys: https://blog.timescale.com/blog/timescaledb-vs-influxdb-for-time-series-data-timescale-influx-sql-nosql-36489299877/ https://blog.timescale.com/blog/timescaledb-vs-influxdb-for-... The benchmarks are interesting, showing TimescaleDB to be the clear winner in most scenarios. For me that's nice, but it's a bigger deal to me personally that I already have Postgres and SQL experience that translates directly to TimescaleDB, I don't have to learn a new tool and query language. Development is complex enough and I have to learn too many things as it is. The older I get the less enthusiastic I am about adding something new to the stack.
  - some0x80070005 6y ago
    Fully agreed on having that SQL experience guiding you on a totally reasonable solution. However; our problem space is not high cardinality data; it more closely aligns to the first performance comparison with 10 devices and 10 metrics. The ease of getting high performance with pre implemented functions is great for us. Reliability is obviously a concern, and I can agree that if data is sacred, then choosing something built on Postgres is going to be a better thought. Again, this is just our problem space; small scale deployments on many machines with no preexisting RDMS, low cardinality data, etc. I think it’d be a different story if we were huge, but for us, InfluxDB provides some seriously handy feature and is worth consideration if your problem is similar.
throwaway375 6y ago
I think idea and promise of Timescale is great, but current(well actually I tried it 1 year ago) state of things makes it very hard to choose Timescale over Clickhouse. I've tried to setup simple Twitter parser for trends analysis, so I needed few thousand counters every few seconds. While I did not encounter any perfomance issues, size on disk was a huge deal. I don't remember precise numbers, but Clickhouse used few magnitudes lower disk space. And while Timescale has nice things like materialized views, Clickhouse has them too. And apart from them Clickhouse has excellent data compression algorithms for repeated key value type counters. So it becomes really hard to understand why Timescale. It aims to help you with tables bigger than traditional pg can handle, but at the same time uses same amount of space.
- Croftengea 6y ago
  Did you turn on compression in TSDB?
  - throwaway375 6y ago
    I checked documentation and I don't think I did. Looks like it has same compressing algorithms as Clickhouse, so it should be pretty close in space requirements for old chunks.
  - hagen1778 6y ago
    Is there any reason why it is not turned on by default?
    - Croftengea 6y ago
      You can't delete arbitrary records from compressed tables, only chunks.
- hagen1778 6y ago
  I think ClickHouse is underestimated as a database for time series. Many companies using it for analytics purposes (like Cloudflare [0]), for logs processing (like Uber [1]). I'm just waiting when someone builds something outstanding for monitoring. Articles like [2] shows ClickHouse potential in this area. Btw, ClickHouse is under Apache 2 license, which makes it much easier to use in big companies. [0] https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p... [1] https://eng.uber.com/logging/ https://eng.uber.com/logging/ [2] https://altinity.com/blog/clickhouse-for-time-series https://altinity.com/blog/clickhouse-for-time-series
Croftengea 6y ago
TimescaleDB is a great product, but if you plan to go with them long term, there are few points to consider: * They are still trying to figure out their monetization strategy. Initially, they betted on their on-premise Enterprise version, then abandoned it. Now they are pushing their cloud version. * Even though most of their code licensed under Apache license, some code is under their proprietary license. * I'm sure one can get some ideas about their development directions from their issue tracker and source code, but they don't have any public product roadmap. * Even though the product itself is technically very stable, the version compatibility leaves a lot to be desired. There are removed features and broken APIs from version to version. * Their commercial support terms for on-premise instances don't seem to be well defined, not publicly at least.
- akulkarni 6y ago
  Timescale Co-founder here. Happy to address your concerns! 1. Monetization strategy This funding round is actually a sign that our business model is working really well. To quote Redpoint Ventures, who led this funding round: "The [Timescale] team capitalized on their significant community momentum last year, with their cloud business being one of the fastest-growing database businesses we have seen in the past 20+ years." [0] 2. Licensing Most companies (including open-source companies) actually have both open-source and proprietary software, but the proprietary software is often hidden inside private repos. The difference with Timescale is that we have made the source code for our proprietary software available (on Github), even allowing users to modify it (eg "right to repair), and made all of our software free (ie no paid software features). [1] 3. Public product roadmap We aim to be transparent re: product roadmap via Github, blog posts, etc, but I appreciate the feedback that we could be more transparent. Thanks! 4. Version compatibility / broken APIs Could you say more? AFIAK the only time we "broke" (ie changed) some APIs is with TimescaleDB 2.0, and when we did so we explained why we did that (mostly to improve user experience based on feedback). We take this topic very seriously and even the decision to do so in 2.0 was not something we did lightly (and it was also made after a lot of discussion with users). More about this decision here in our docs: [2] 5. Commercial support for on-premise We offer free support for on-premise instances via Slack (where you can often find our engineers, support team, CTO, and myself). [3] However, if you would like a higher level of support for on premise (e.g., commercial SLAs), please reach out to us directly (e.g., via the form on that same page). [3] Hope this helps! [0] https://medium.com/redpoint-ventures/building-a-next-generation-database-our-investment-in-timescale-59e9a1d1d6f9 https://medium.com/redpoint-ventures/building-a-next-generat... [1] https://blog.timescale.com/blog/building-open-source-business-in-cloud-era-v2/ https://blog.timescale.com/blog/building-open-source-busines... [2] https://docs.timescale.com/timescaledb/latest/overview/release-notes/changes-in-timescaledb-2/ https://docs.timescale.com/timescaledb/latest/overview/relea... [3] https://www.timescale.com/support https://www.timescale.com/support
gsich 6y ago
What I want in TimescaleDB is aggregation of old values in the SAME table. There is no use when I have to do this in a seperate table, Grafana overhead will be insane.
- akulkarni 6y ago
  Have you looked at Real-time aggregates in TimescaleDB? It might help address the problem you are facing: "With real-time aggregation, when you query a continuous aggregate view, rather than just getting the pre-computed aggregate from the materialized table, the query will transparently combine this pre-computed aggregate with raw data from the hypertable that’s yet to be materialized. And, by combining raw and materialized data in this way, you get accurate and up-to-date results, while still enjoying the speedups that come from pre-computing a large portion of the result." https://blog.timescale.com/blog/achieving-the-best-of-both-worlds-ensuring-up-to-date-results-with-real-time-aggregation/ https://blog.timescale.com/blog/achieving-the-best-of-both-w... (You would then just need to query / point to the Continuous Aggregate to get the new and aggregated data in the same query)
  - gsich 5y ago
    Thanks!
sylvain_kerkour 6y ago
Congrats! I really love the approach they took to deliver value: An extension of an existing rock-solid platform (Postgres) instead of building a new server which would require a lot of time to learn and manage. Is TimescaleDB suitable to store logs? If yes, how to architect the tables?
- cevian 6y ago
  (Timescale engineer here). We believe so and we have customers using us for just that. We haven't created our own product for that yet (as we have for metrics -- Promscale) but it is an idea we are playing with. You may want to look at our Promscale design doc[1] for ideas on table layout. [1] https://tsdb.co/prom-design-doc https://tsdb.co/prom-design-doc
  - sylvain_kerkour 6y ago
    Great to hear, thank you!
diveanon 6y ago
I've really enjoyed working with tsdb, we are using it in conjunction with hasura and it has been amazing how productive we feel. Best of luck from a happy customer.
js4ever 6y ago
I'm using TimescaleDB more and more, I just deployed another instance this morning for another customer that need to store timeseries (hundreds of servers metrics and some logs), I do have other instances in production since a year without any issues and the compression and expiration policy are really great for this use case! Thanks to the team.
thunkshift1 6y ago
What are some of the use -cases to consider using any time series db? I cant think of much beyond stock charts and server logs
naltun 6y ago
I remember meeting the TimescaleDB crew back at IoT World 2018. They were a wonderful bunch and took the time to explain to me the technicals behind their fork of PostgreSQL (TimescaleDB). I'm very glad to see that they have carried themselves far, and I am excited for what's next for the company. Congrats!
filmgirlcw 6y ago
Congrats to Timescale! The founders and team are building something really solid and it’s great to see them get the funding to grow even more.
drchaim 6y ago
Congrats to the tsdb team. I've tried the database extension months ago and it works perfect for my use case standalone. I just need to integrate it with Django, which is not easy given the current schema of the database and the way Django creates defaults autoincrements and PK, but I'm sure there will be a workaround for that.
- benn0 6y ago
  I ran into the same problem trying to pair it with Django. From what I remember, some manipulation of primary key constraints in the schema could get them to coexist (although you lose any enforcement of unique PKs, so there's the potential for things to go bad). We ended up moving to FastAPI, so don't have this problem anymore.
prepend 6y ago
> open source (“open core”, to be precise) This is bad. Just be precise and don’t try to conflate open source with proprietary stuff. It’s ok to not be OSS, lots of good software is non-OSS. It’s only bad to say you’re one thing and be another.
kanwisher 6y ago
Awesome we are using TimescaleDB to monitor molting patterns of Crabs on crab farms. https://octo.vision https://octo.vision
- mfreed 6y ago
  I love the use cases of our community!
- akulkarni 6y ago
  Autonomous Crab Farms! Such a cool use case. Thank you for sharing :-)

308 points by eloff 6y ago

I'm pretty stoked for this. Timescale ability to use time series on a subset of tables (hypertables) really makes it an interesting choice. I've just dabbled with it but seeing that they'll finally be improving their hosted solution make me want to dive in deeper! Anyone has experience running large DB on timescale, would you recommend it?

eloff 6y ago
I haven't tried running the distributed Timescale DB - in general more parts means more things to go wrong, so using their managed cloud service for that is a good idea. But I can attest that the single-server version is rock solid, just like PostgreSQL that it's based on. And it's free and source visible. The pace of innovation has also been really high, it just keeps getting better with every release.
spmurrayzzz 6y ago
If you end up transitioning to multinode deployments from a single node deployment, there are a bunch of tradeoffs to consider. A good reference point for this to check out the `multinode` label in their github issues. https://github.com/timescale/timescaledb/labels/multinode https://github.com/timescale/timescaledb/labels/multinode One of the big items that stood out to me is the inability to be able to migrate data from an existing table when creating a distributed hypertable. There were also some significant query performance reports as well. These all may improve with time of course, so watching the dev cycles will give you a good sense of that I think.

AzzieElbab 6y ago

Please free us of kdb

haolez 6y ago
Why? I'm genuinely curious, since I've never worked with kdb but I've often heard great things about it (regardless of the weird query language).
- AzzieElbab 6y ago
  Kdb is like oracle db that somehow won’t die. It could use some competition
  - RyanHamilton 6y ago
    Standard SQL is rubbish for time series queries as it is based on set theory which does not have order. Most SQL datbases exploit that fact to increase performance. Fundamentally kdb is based on ordered lists, which is a much better paradigm for time series data. The contrasting queries can be seen here: http://www.timestored.com/b/kdb-qsql-query-vs-sql/ http://www.timestored.com/b/kdb-qsql-query-vs-sql/ I do agree they could use competition as the overall offering is weak but the core database is very strong.
    - j1897 6y ago
      Have you checked questdb [1]? the data structure is arrays with data that lands in order and SQL queries on top. The fallback was that it was difficult to deal with out of order data, but we have just solved this by re-ordering data on the fly in memory before it hits the disk. Performance wise probably not far from kdb itself (will be sharing some bench results soon vs open source tsdbs) NB: I'm one of the co-founder of questdb [1] https://www.questdb.io https://www.questdb.io
shin_lao 6y ago
TimescaleDB isn't an alternative to KDB+. KDB+ is more a programming platform than a database.
- AzzieElbab 6y ago

KingOfCoders 6y ago

Love timescaledb. Scaled several postgres instances for time data successfully with it.

chirau 6y ago

I love Timescale, and the team as well. Good folks. Congrats!

FirstLvR 6y ago

cheers! i love Timescale

WFHRenaissance 6y ago

Used TimescaleDB in one of my first real DE projects – huge fan. Happy to see this!

somethingAlex 6y ago

Very happy with our choice to use TimescaleDB. The idea to simply make it a Postgres extension was brilliant. The compression release was one of the cooler features I've seen in recent times. Row database for recent transactional data, columnar compressed database for historical OLAP workloads - pretty much automagically.

halbritt 6y ago
Curious how large a dataset you're using?
- somethingAlex 6y ago
  Under 100GB; I'm sure vanilla Postgres would suit our needs too. However, adding TimescaleDB on top was not much of an investment and in exchange we got an interface for operations we do often, effortless continuous aggregation, near-constant time appends, and a native way to leave data mutable for a period of time before marking it immutable and compressing it. The performance is a great feature but its also just an intuitive, familiar (pretty much just SQL) tool that makes life easier.
  - halbritt 6y ago
    Cool. I'm keen to try it. Wondering how well it works with multi-terabyte data sets.

jb1991 6y ago

We switched from Datomic to Timescale and haven’t looked back.

swamiji 6y ago
If you could share more details, I'd really appreciate it.

simplify 6y ago

> When we launched TimescaleDB, we met a fair amount of skepticism.... The top voted Hacker News comment at the time called us, “a rather bad idea[0].” Good old HN with its healthy skepticism :) [0] https://news.ycombinator.com/item?id=14036554 https://news.ycombinator.com/item?id=14036554

eloff 6y ago
I think it goes to show how impossible it is to judge an idea. YC itself doesn't pretend to do this with the ~15,000 applications they go through each cohort. They try instead to look at the team, look at their progress, imagine what would need to happen for the company to succeed at the level required for them to get the returns they seek. Founders should have a thick skin when it comes to criticism on HN, because we don't know either.
kirse 6y ago
Having been on HN long enough, what I look for during any idea/startup launch is polarization and intensity of viewpoints. If people are reacting to the idea (for better or worse), it means its had an impact. Those are often the products that find success. A no-comment launch is far worse than one riddled with criticism. IMO HN's classic "skepticism" is usually just engineering nerd insecurity projected outwards, with enough techno-jargon to maintain plausible deniability. Folks feel threatened by a great idea so it's safer to find some way to tear it down. Not to dismiss all feedback as projected insecurity of course.
- etaioinshrdlu 6y ago
  I'm still confused why time-series databases are even a thing. It seems to me that time-series just means you have a date/time column plus an index on it. Which is something typical databases already do well, and like the referenced post mentioned, you could use a column store for better performance. But I just don't see anything that makes creating an entire database design for one specific index type worthwhile... I index many tables on my site by num_upvotes so I can find the top ranked items to show. Does this mean that I need an UpvoteDB? I don't think so. A previous time I argued this point, it was mentioned that you rarely need to update or delete old rows. This allows you to tailor the storage solution better. However, this basically means a compressed column store, which again, doesn't really have much to do with time.

neom 6y ago

I was in Grand Central Tech with the Timescale folks, became friends with both Ajay and Mike. Ajay gave me a lot of good thoughts on building my startup(thanks!), and Mike is..well.. just hyper smart. That is to say, I'm not surprised to read this and for what it's worth: they deserve it, really great humans! :)

mfreed 6y ago
Thanks, super appreciated! I remember always enjoying our conversations as well!
akulkarni 6y ago
Hi John, good to hear from you! (And thank you :-)

ironman1478 6y ago

This is awesome. In certain industries (manufacturing, energy, etc) there are companies (really 1 company) that essentially have monopolies on time series databases / historians. There has been 0 competition in that space and as a result the databases and surrounding client tools are just so awful. It'll be interesting to see if timescaledb can really enter that market and force those companies to adapt.

ltnublet 6y ago
What company is that?
- stevesimmons 6y ago
  KDB in finance?
- ironman1478 6y ago
  I was employed there so I don't feel entirely comfortable stating their name. Just google around for "operational historian". It was really frustrating because we had these ideas years ago (and many other ideas), but due to reasons we were not allowed to actually make the product genuinely better. Also, the product is in more places than you'd think because they partner with vendors to sell the database as a component in a larger system that the vendor packages up and sells.
- some0x80070005 6y ago
  I suspect AVEVA PI (formerly OSIsoft) is what they are referring to. PI is excessively popular in Enterprise scenarios either OT or IT.
- kevinherron 6y ago
  He's probably referring to OSIsoft's PI historian.
  - alexmarcy 6y ago
    Their stuff is a nightmare to use, and is insanely expensive. Always nice when we get to just swap it out for something like Canary or even Ignition although folks are always trying to trash the Ignition Historian when it works well for most use cases people need to solve.

_joel 6y ago

We used TimescaleDB for a flood prediction tool, works really well, kudos.

msaharia 6y ago
Hi, I am a hydrologist. Would love to hear about what you have done using TimeScaleDB
- _joel 6y ago
  We used historical flow meter readings (started out with 15 minute intervals but it worked much better with a higher frequency) and used that to train Recurrent Neural Networks (RNN's) to predict where areas were likely to flood. I was the devops lead on it for the prototype, not the data scientist unfortunately so can't give you the in's and outs but can tell you that we used tensorflow together with pandas/df/timescaledb. We then displayed that using plotly, all this was stuffed inside several containers. It was a great project to work on actually. The whole setup was pretty much a joy to work with.
  - msaharia 6y ago
    That is awesome. I work in flood forecasting and previously worked in NCAR/NASA. If you would be so kind, feel free to share any white papers, links etc to the project you are referring to. msaharia[a]iitd.ac.in
    - _joel 6y ago
      I don't I'm afraid, for starters I contract and no longer work on the project, plus there's IP invovled etc. Also I'm no data scientist, just a hacker really :) The main difficulty was getting access to the data (and ensuring it was valid, as with all ML projects), lucking we managed to get that from several sources (councils, water companies etc). The flow data was in TimescaleDB, pandas dataframes so we could use varying levels of frequency of data and we used HDF5 iirc as well (detail is hazy, it was a few years back now). We did demo it to the Met Office here in the UK too. They were interested but already had their own thing cooking up so the project never really got out of prototype, but it was making accurate predictions. I think there were some other areas that might turn out to be flaky over time using this method (such as rapid changes to catchment areas) but that could maybe be factored in someway with more thought on the model and verification on a larger set/timeframes. Feel free to hit me up if you want any more detail on the tech side, but don't ask me about stats, maths fu I ain't :) joel [at] smashthesystems.com

zomgwat 6y ago

TimescaleDB looks great. I'm interested in using it but I'm concerned about the upgrade path across major PostgreSQL versions. Logical replication is a big help when upgrading PostgreSQL across major versions while minimizing downtime. As far as I understand it, TimescaleDB doesn't support logical replication yet. Major version upgrades with TimescaleDB is obviously a solvable problem but it probably means we'll have a more complicated upgrade path. Upgrading via logical replication is just so nice.

some0x80070005 6y ago

I guess this is an unpopular opinion, but I’ve found InfluxDB to be superb for being trivial to get going in a high performance way. I have never touched InfluxDB Cloud - always just InfluxDB either as an arbitrary process or container. Examples of where I’ve found InfluxDB to be more pleasant: * InfluxDB has way better documentation on functions. For example, look up moving average by time (not points) on TimescaleDB vs InfluxDB. We use these more complex queries and have no problem on Influx. Going further, the number of functions built in is impressive with the same ability to define new functions. * InfluxDB containers are totally self contained which is great for simple architectures. As a process, InfluxDB is a single executable thanks to Go. * This is extremely subjective, but I find Flux easier to comprehend as a separate query vs. the use of SQL to do higher complexity functions; however, I am sure this is due to my lack of experience and know how to write said queries in SQL.

eloff 6y ago
I wouldn't choose InfluxDB over TimescaleDB. There's a reasonably balanced comparison here from the Timescale guys: https://blog.timescale.com/blog/timescaledb-vs-influxdb-for-time-series-data-timescale-influx-sql-nosql-36489299877/ https://blog.timescale.com/blog/timescaledb-vs-influxdb-for-... The benchmarks are interesting, showing TimescaleDB to be the clear winner in most scenarios. For me that's nice, but it's a bigger deal to me personally that I already have Postgres and SQL experience that translates directly to TimescaleDB, I don't have to learn a new tool and query language. Development is complex enough and I have to learn too many things as it is. The older I get the less enthusiastic I am about adding something new to the stack.
- some0x80070005 6y ago
  Fully agreed on having that SQL experience guiding you on a totally reasonable solution. However; our problem space is not high cardinality data; it more closely aligns to the first performance comparison with 10 devices and 10 metrics. The ease of getting high performance with pre implemented functions is great for us. Reliability is obviously a concern, and I can agree that if data is sacred, then choosing something built on Postgres is going to be a better thought. Again, this is just our problem space; small scale deployments on many machines with no preexisting RDMS, low cardinality data, etc. I think it’d be a different story if we were huge, but for us, InfluxDB provides some seriously handy feature and is worth consideration if your problem is similar.

throwaway375 6y ago

I think idea and promise of Timescale is great, but current(well actually I tried it 1 year ago) state of things makes it very hard to choose Timescale over Clickhouse. I've tried to setup simple Twitter parser for trends analysis, so I needed few thousand counters every few seconds. While I did not encounter any perfomance issues, size on disk was a huge deal. I don't remember precise numbers, but Clickhouse used few magnitudes lower disk space. And while Timescale has nice things like materialized views, Clickhouse has them too. And apart from them Clickhouse has excellent data compression algorithms for repeated key value type counters. So it becomes really hard to understand why Timescale. It aims to help you with tables bigger than traditional pg can handle, but at the same time uses same amount of space.

Croftengea 6y ago
Did you turn on compression in TSDB?
- throwaway375 6y ago
  I checked documentation and I don't think I did. Looks like it has same compressing algorithms as Clickhouse, so it should be pretty close in space requirements for old chunks.
- hagen1778 6y ago
  Is there any reason why it is not turned on by default?
  - Croftengea 6y ago
    You can't delete arbitrary records from compressed tables, only chunks.
hagen1778 6y ago
I think ClickHouse is underestimated as a database for time series. Many companies using it for analytics purposes (like Cloudflare [0]), for logs processing (like Uber [1]). I'm just waiting when someone builds something outstanding for monitoring. Articles like [2] shows ClickHouse potential in this area. Btw, ClickHouse is under Apache 2 license, which makes it much easier to use in big companies. [0] https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p... [1] https://eng.uber.com/logging/ https://eng.uber.com/logging/ [2] https://altinity.com/blog/clickhouse-for-time-series https://altinity.com/blog/clickhouse-for-time-series

Croftengea 6y ago

TimescaleDB is a great product, but if you plan to go with them long term, there are few points to consider: * They are still trying to figure out their monetization strategy. Initially, they betted on their on-premise Enterprise version, then abandoned it. Now they are pushing their cloud version. * Even though most of their code licensed under Apache license, some code is under their proprietary license. * I'm sure one can get some ideas about their development directions from their issue tracker and source code, but they don't have any public product roadmap. * Even though the product itself is technically very stable, the version compatibility leaves a lot to be desired. There are removed features and broken APIs from version to version. * Their commercial support terms for on-premise instances don't seem to be well defined, not publicly at least.

akulkarni 6y ago
Timescale Co-founder here. Happy to address your concerns! 1. Monetization strategy This funding round is actually a sign that our business model is working really well. To quote Redpoint Ventures, who led this funding round: "The [Timescale] team capitalized on their significant community momentum last year, with their cloud business being one of the fastest-growing database businesses we have seen in the past 20+ years." [0] 2. Licensing Most companies (including open-source companies) actually have both open-source and proprietary software, but the proprietary software is often hidden inside private repos. The difference with Timescale is that we have made the source code for our proprietary software available (on Github), even allowing users to modify it (eg "right to repair), and made all of our software free (ie no paid software features). [1] 3. Public product roadmap We aim to be transparent re: product roadmap via Github, blog posts, etc, but I appreciate the feedback that we could be more transparent. Thanks! 4. Version compatibility / broken APIs Could you say more? AFIAK the only time we "broke" (ie changed) some APIs is with TimescaleDB 2.0, and when we did so we explained why we did that (mostly to improve user experience based on feedback). We take this topic very seriously and even the decision to do so in 2.0 was not something we did lightly (and it was also made after a lot of discussion with users). More about this decision here in our docs: [2] 5. Commercial support for on-premise We offer free support for on-premise instances via Slack (where you can often find our engineers, support team, CTO, and myself). [3] However, if you would like a higher level of support for on premise (e.g., commercial SLAs), please reach out to us directly (e.g., via the form on that same page). [3] Hope this helps! [0] https://medium.com/redpoint-ventures/building-a-next-generation-database-our-investment-in-timescale-59e9a1d1d6f9 https://medium.com/redpoint-ventures/building-a-next-generat... [1] https://blog.timescale.com/blog/building-open-source-business-in-cloud-era-v2/ https://blog.timescale.com/blog/building-open-source-busines... [2] https://docs.timescale.com/timescaledb/latest/overview/release-notes/changes-in-timescaledb-2/ https://docs.timescale.com/timescaledb/latest/overview/relea... [3] https://www.timescale.com/support https://www.timescale.com/support

gsich 6y ago

What I want in TimescaleDB is aggregation of old values in the SAME table. There is no use when I have to do this in a seperate table, Grafana overhead will be insane.

akulkarni 6y ago
Have you looked at Real-time aggregates in TimescaleDB? It might help address the problem you are facing: "With real-time aggregation, when you query a continuous aggregate view, rather than just getting the pre-computed aggregate from the materialized table, the query will transparently combine this pre-computed aggregate with raw data from the hypertable that’s yet to be materialized. And, by combining raw and materialized data in this way, you get accurate and up-to-date results, while still enjoying the speedups that come from pre-computing a large portion of the result." https://blog.timescale.com/blog/achieving-the-best-of-both-worlds-ensuring-up-to-date-results-with-real-time-aggregation/ https://blog.timescale.com/blog/achieving-the-best-of-both-w... (You would then just need to query / point to the Continuous Aggregate to get the new and aggregated data in the same query)
- gsich 5y ago
  Thanks!

sylvain_kerkour 6y ago

Congrats! I really love the approach they took to deliver value: An extension of an existing rock-solid platform (Postgres) instead of building a new server which would require a lot of time to learn and manage. Is TimescaleDB suitable to store logs? If yes, how to architect the tables?

cevian 6y ago
(Timescale engineer here). We believe so and we have customers using us for just that. We haven't created our own product for that yet (as we have for metrics -- Promscale) but it is an idea we are playing with. You may want to look at our Promscale design doc[1] for ideas on table layout. [1] https://tsdb.co/prom-design-doc https://tsdb.co/prom-design-doc
- sylvain_kerkour 6y ago
  Great to hear, thank you!

diveanon 6y ago

I've really enjoyed working with tsdb, we are using it in conjunction with hasura and it has been amazing how productive we feel. Best of luck from a happy customer.

js4ever 6y ago

I'm using TimescaleDB more and more, I just deployed another instance this morning for another customer that need to store timeseries (hundreds of servers metrics and some logs), I do have other instances in production since a year without any issues and the compression and expiration policy are really great for this use case! Thanks to the team.

thunkshift1 6y ago

What are some of the use -cases to consider using any time series db? I cant think of much beyond stock charts and server logs

naltun 6y ago

I remember meeting the TimescaleDB crew back at IoT World 2018. They were a wonderful bunch and took the time to explain to me the technicals behind their fork of PostgreSQL (TimescaleDB). I'm very glad to see that they have carried themselves far, and I am excited for what's next for the company. Congrats!

filmgirlcw 6y ago

Congrats to Timescale! The founders and team are building something really solid and it’s great to see them get the funding to grow even more.

drchaim 6y ago

Congrats to the tsdb team. I've tried the database extension months ago and it works perfect for my use case standalone. I just need to integrate it with Django, which is not easy given the current schema of the database and the way Django creates defaults autoincrements and PK, but I'm sure there will be a workaround for that.

benn0 6y ago
I ran into the same problem trying to pair it with Django. From what I remember, some manipulation of primary key constraints in the schema could get them to coexist (although you lose any enforcement of unique PKs, so there's the potential for things to go bad). We ended up moving to FastAPI, so don't have this problem anymore.

prepend 6y ago

> open source (“open core”, to be precise) This is bad. Just be precise and don’t try to conflate open source with proprietary stuff. It’s ok to not be OSS, lots of good software is non-OSS. It’s only bad to say you’re one thing and be another.

kanwisher 6y ago

Awesome we are using TimescaleDB to monitor molting patterns of Crabs on crab farms. https://octo.vision https://octo.vision

mfreed 6y ago
I love the use cases of our community!
akulkarni 6y ago
Autonomous Crab Farms! Such a cool use case. Thank you for sharing :-)