10 ms·
Timescale raises $110M Series C
- matesz 5y agoIs using timeseries database like TimescaleDB a good fit for a chat platform like Slack/Discord/Mattermost? I've heard that Discord is using ScyllaDB, but I have quite a bit of experience in Postgres so it would be nice to use just that. In any case, does anybody know what are the advantages of one over the other? Big thanks!
- ashvardanian 5y agoInsane! $110M towards yet another Postgres extension. Theoretical CS and hardware has advanced so much, but the people are using the same old boring approaches. Truly sad.
- Beltalowda 5y agoWhat do you think better alternatives/approaches are?
- dboreham 5y agoPostgresSQL process-per-connection execution model is limiting.
- ashvardanian 5y agoLiterally anything. There is so much to do better. Faster I/O, kernel bypass and async filesystem, new persistent data-structures, alternative lock-free concurrency resolution schemes… Disclaimer: I am highly biased, as I am funding/researching/developing a DBMS myself. Out of necessity though, as we constantly hit bottlenecks in the persistent I/O layer. We are not selling or offering anything, but will soon share some fresh internal results on aforementioned topics. In the meantime, here is a list of startups re-implementing mostly identical ideas: https://unum.cloud/post/2021-12-31-dbms-startups/ https://unum.cloud/post/2021-12-31-dbms-startups/
- pella 5y ago"boring" == less transactional surprise :-) https://jepsen.io/analyses/postgresql-12.3 https://jepsen.io/analyses/postgresql-12.3
- pella 5y agoproposal for "Solving PostgreSQL wicked problems" https://github.com/orioledb/orioledb/blob/main/orioledb-postgresbuild-2021.pdf https://github.com/orioledb/orioledb/blob/main/orioledb-post...
- ashvardanian 5y agoGreat report, but I am new B-Trees alone will not enough. The simplest common solution is to switch to LSM Trees for higher write throughput. Thats exactly what Yugabute does, by putting Postgres over RocksDB. Same way as Facebook uses MyRocks = MySQL + RocksDB.
- rektide 5y agoNice share. Oriole DB updated the readme 17 days ago to say they'd release in February 2022, so I'll definitely be watching this space. These sounds like some really interesting striking-at-the-heart-of-it changes. I hope we have a long time to see which of these pan out & pay off big. On a side note, nice interesting brief comments & shares elsewhere on this site. :thumbup:
- _joel 5y agoTrue, why use proven technology with decades of production usage with your data when you can use a novel and theoretical CS paper implementations.
- georgewfraser 5y agoI would love to run a time-series benchmark against a good column store like Snowflake to see if purpose-built time series databases are actually faster. I have a sneaking suspicion the time scale databases are just reinventing the column store, and that an appropriate non-sabotaged benchmark would show this.
- ren_engineer 5y agothe main selling point is developer experience I think, rather than building out a bunch of stuff on top of a more general purpose tool you use a specialized DB and save time. They also have some benchmarks against Clickhouse for example
- ashvardanian 5y agoI have just looked up those charts on Timescales website [1] and I am a bit surprised. Never extensively used any of those DBs, but I have seen their sources and must say that expected bigger gaps [2]. Also worth looking: the Taxi Rides Benchmark on Postgres vs Clickhouse [3]. [1]: https://www.timescale.com/blog/what-is-clickhouse-how-does-it-compare-to-postgresql-and-timescaledb-and-how-does-it-perform-for-time-series-data/ https://www.timescale.com/blog/what-is-clickhouse-how-does-i... [2]: https://www.pradeepchhetri.xyz/clickhousevstimescaledb/ https://www.pradeepchhetri.xyz/clickhousevstimescaledb/ [3]: https://tech.marksblogg.com/benchmarks.html https://tech.marksblogg.com/benchmarks.html
- manigandham 5y ago> "timescale ... are just reinventing the column store" Not reinventing but reimplementing it for Postgres, which didn't have serious OLAP capabilities before. Lots of "newsql" systems are combining OLTP and OLAP by starting at one side and adding the other. So far Timescale has column-oriented compressed storage and scale-out partitioning, and they're working on matching the compute part.
- abraxas 5y agoThere is a lot of value in having a columnar storage that is fully ANSI SQL and supports all of the goodies that you get in the Postgres ecosystem. NoSQL databases with their half assed SQL grammar implementations are a real pain to use in real applications where they often have to be handled differently in code vs the RDBMS because either their syntax is slightly different or their connection stack is incompatible.
- CyberDildonics 5y agoI don't understand what is difficult or non trivial about these types of databases and when people try to explain it, it usually just gets more confusing. Filtering values over time is just the same operations that you would find in an audio editor or a one dimensional version of what you find in an image editor (weighted averages of values). I wonder how many customers could just use sqlite but don't know anything about computers and end up buying some sort of subscription to a 'new kind of database'. The web page just drops as many buzzwords as possible - web3, crypto, nfts, monitor soil to fight global warming - it looks like a disaster to anyone who understands the basics of programming.
- cleancoder0 5y agoSQLite does not support column based optimizations. Time series data is insanely compressible.
- deleted 5y ago[deleted]
- jacobr1 5y agoIf your data is small enough, then sure, any number of well tested data platforms will work for you. The problem something like timescale tries to solve is dealing with "high cardinality." When you have many unique values the indexing approaches needed to ensure performance start becoming different. You'll run into write performance issues if need indexes on every single column, and every single combination of columns, and each column has a large number of unique values. While the common factor many of these datasets tend to share is that they are being constantly generated by some kind of sensor/probe/live-system, they tend to have a variety of other dimensions that are also high-cardinality.
- CyberDildonics 5y agoThere are two different things here - the first is people not needing an elaborate solution because computers are fast and the second is that if someone does need a solution with less overhead, why is that difficult? Values over time like audio is a one dimensional signal. Seeking is basic data structure stuff, filtering is basic signal stuff. There aren't going to be other dimensions like time, which makes the other values just other channels. If you need to combine dense values they can be not only filtered, but filtered into individual distributions. People give abstract descriptions like you have here, but I'm just not seeing a difficult problem in all of this.
- aeyes 5y ago> Looking ahead, our goal is to keep innovating on top of PostgreSQL and to continue adding breakthrough capabilities Does Timescale contribute back to PostgreSQL or do they truly only build on top of it? https://www.postgresql.org/community/contributors/ https://www.postgresql.org/community/contributors/ only lists two contributors and they both worked on Postgres before joining Timescale.
- yieldgap 5y agoOn Twitter, they said they're building a team for upstream contributions https://twitter.com/acoustik/status/1496145349735559168?t=HGGOFQ-xUhXAXhLRrvOYBw&s=19 https://twitter.com/acoustik/status/1496145349735559168?t=HG...
- akulkarni 5y ago(Timescale co-founder / CEO) I just want to say that we wouldn't be here without the support, feedback -- and yes, even the honest critiques -- from the HN community. So thank you everyone. As we like to say, we've come a long way in the past 5 years, but we're just getting started :-) And we're hiring globally for our remote-first team! https://www.timescale.com/careers https://www.timescale.com/careers
- sdesol 5y agoCongrats and I was wondering if you can comment on the current team size? I'm looking at the number of contributors that have created pull requests within the last four months and it is shockingly low (in a good way). Based on the following: https://oss.gitsense.com/insights/github?q=pull-age%3A%3C%3D120&t=crc-insights&tb=all&v=timescale%2Ftimescaledb%3A%3Agithub%3Atimescale%2Ftimescaledb%3A%3A%3A%3A https://oss.gitsense.com/insights/github?q=pull-age%3A%3C%3D... It looks like there has only really been 7-10 full time contributors and for you to have raised what you have with such a small team is quite impressive. Is development happening elsewhere or is my hunch correct? Edit: Thanks to feedback from mfreed, below is a more accurate picture of development activity: https://oss.gitsense.com/insights/github?p=authors&q=pull-age%3A%3C%3D120&t=crc-insights&tb=all&v=timescale%2Fcloud-actions%20----%20timescale%2Fdocs%20----%20timescale%2Fdocs-htmltojsx%20----%20timescale%2Fflagsmith-charts%20----%20timescale%2Ffrontend-assignment%20----%20timescale%2Fnft-starter-kit%20----%20timescale%2Fopentelemetry-demo%20----%20tim%3A%3Agithub%3Atimescale%2Fcloud-actions%2Cgithub%3Atimescale%2Fdocs%2Cgithub%3Atimescale%2Fdocs-htmltojsx%2Cgithub%3Atimescale%2Fflagsmith-charts%2Cgithub%3Atimescale%2Ffrontend-assignment%2Cgithub%3Atimescale%2Fnft-starter-kit%2Cgithub%3Atimescale%2Fopentelemetry-demo%2Cgithub%3Atimescale%2Fpg_influx%2Cgithub%3Atimescale%2Fpgx%2Cgithub%3Atimescale%2Fpromscale%2Cgithub%3Atimescale%2Fpromscale_extension%2Cgithub%3Atimescale%2Fstate_of_postgres%2Cgithub%3Atimescale%2Ftimescaledb%2Cgithub%3Atimescale%2Ftimescaledb-backup%2Cgithub%3Atimescale%2Ftimescaledb-docker%2Cgithub%3Atimescale%2Ftimescaledb-docker-ha%2Cgithub%3Atimescale%2Ftimescaledb-kubernetes%2Cgithub%3Atimescale%2Ftimescaledb-toolkit%2Cgithub%3Atimescale%2Ftobs%2Cgithub%3Atimescale%2Ftsbs%3A%3A%3A%3A%3A%3A1645559751079 https://oss.gitsense.com/insights/github?p=authors&q=pull-ag...
- mfreed 5y ago
- mparnisari 5y agoWonder if Timescale would be a good answer to https://stackoverflow.com/questions/70841804/is-aws-timestream-a-good-solution-to-store-an-audit-log https://stackoverflow.com/questions/70841804/is-aws-timestre...
- akulkarni 5y agoIn our benchmarks (which you and others are welcome to replicate), Timescale vastly outperformed AWS Timestream: https://www.timescale.com/blog/timescaledb-vs-amazon-timestream-6000x-higher-inserts-175x-faster-queries-220x-cheaper/ https://www.timescale.com/blog/timescaledb-vs-amazon-timestr...
- mfreed 5y agoTo replicate, please see the Time Series Benchmark Suite, which is open-source and has many vendor-contributed configurations: - https://github.com/timescale/tsbs https://github.com/timescale/tsbs - https://github.com/timescale/tsbs/blob/master/docs/timestream.md https://github.com/timescale/tsbs/blob/master/docs/timestrea...
- mparnisari 5y agoTY!
- jabiko 5y agoWhile I think that TimescaleDB is a great technology, the support experience of their Timescale Cloud offering was quite underwhelming. In one occurrence we wanted to create a VPC peering between the a database hosted on Timescale Cloud and our Kubernetes cluster hosted on Azure. For this you need to put the Azure resource group name in a form on Timescale cloud. Turns out our resource group name contained an uppercase character and the form on Timescale Cloud has a (broken) validation that required the name to be all lowercase. We couldn't easily change that name since that would have required us to re-create our production AKS/K8S cluster. After contacting Timescale support (as a paying customer) the answer was basically: "Well, we require the resource group name to be lowercase, we can't change that, sucks to be you, bye" We can live without that VPC peering, so we didn't push that further, but there are zero technical reasons for that restrictions and I would bet that its just a broken validation regex in their backend that they are unwilling to fix.
- akulkarni 5y agoHi there, sorry you had a negative experience with Timescale Cloud. It's true that on Azure we require both the resource group name and also the Virtual Network name to be in lowercase. But Microsoft names are case-agnostic, so this should be okay. We've had other customers with this issue before and converting the resource group name to be lowercase worked for them. Also, sometimes technical restrictions exist for internal reasons that are not obvious / hard to share externally. :-) That said, I shared your message internally and someone is looking into this. Stay tuned. More soon.
- jabiko 5y agoHi, thanks you. I'm really grateful for your answer. The Azure documentation seems to hint at the resource group name being case insensitive, so I guess that could actually work. To be honest I don't know if we tried just using a lowercase version. Again, I want to emphasize that I find it really great that you took the time to answer here.
- tylerfontaine 5y ago
- riffic 5y agoThis company should have a wikipedia article at this point, right? Where is it? closest I can find are these: https://en.wikipedia.org/wiki/TimescaleDB https://en.wikipedia.org/wiki/TimescaleDB https://en.wikipedia.org/wiki/Draft:TimescaleDB https://en.wikipedia.org/wiki/Draft:TimescaleDB
- LoriP 5y agoTimescale's Community Manager here. Unfortunately that's a longish story of a strongly opinionated Wikipedia editor.
- riffic 5y agoI think I might have bumped into that particular editor recently, lol. Let me know if you want help in putting together a draft, I'm good at finding sources[0] to establish notability. [0] https://www.google.com/search?tbs=bks:1&q="TimescaleDB"+-wikipedia https://www.google.com/search?tbs=bks:1&q="TimescaleDB"+-wik...
- LoriP 5y agoThank you, I might take you up on that...and exchange experiences lol
- riffic 5y agoI just took it upon myself to get the old draft restored and to add sources to assert notability. Moved draft into article namespace. This isn't my first rodeo. We'll see how it goes.
- tkinom 5y agoI have written time series logging db with sqlite believe that approach has following advantages: System performance scales well with latest SSD HW. As compare to cloud base approach that is limited by network/cloud speed. One can store logs per day / week / year in separate db files as needed. Backup of small db files for last few days/weeks are trivial with rsync. Love to hear other pro/con arguments from folks who use Timescale type approach.
- ignoramous 5y agoTry timeseries with duckdb.org or Clickhouse Local? We use the former for analytical queries (queries over columns instead of rows), at which it is better than Sqlite3.
- fabioyy 5y agoi'm using timescale to store sensor/gps log ( 50 inserts/s - 24/7 ). after 2 months still very good )
- hardwaresofton 5y agoHave you written about this anywhere? I'm sure TimescaleDB would love to signal boost that post, and I separately would love to read about how you have it set up and the nitty gritty of the setup. How are you dealing with backups/WAL and general DB administration? Are you using Timescale Cloud?
- fabioyy 5y agoI have done no optimization, no WAL partition, no slave (yet). i'm just using a single t3a.xlarge Aws instance with 1500 IOPS (EBS Volume). we use 1500IOPS just to speed up our read-only jobs and reports. i'm just using AWS EBS snapshot feature ( every hour ) for backup.
- hardwaresofton 5y agoThanks for going into the nitty gritty, super helpful to know what your setup is like. In the I’ve had terrible experiences with on-demand IOPS but now I feel like even 1000 provisioned (the least you can get) is too much for the workload I was running… the app was mostly idle but had bursts that would overwhelm
- fourseventy 5y agoNot OP but I run a Timescale instance with the same order of magnitude inserts/sec and have been running it for about 2 years now. The database is closing in on 1 TB on disk. We don't use Timescale Cloud, we just host it on a VM in Google cloud with 8 CPU's and 32 gigs of ram, which seems adequate for now. We do WAL backup using the WAL-G tool which backs the db up to google storage.
- hardwaresofton 5y ago
- ishikawa 5y agoThis shows that awareness to time series data is huge today, unlike 10 years ago.
- mfringel 5y agoHow does Timescale handle lookup tables? That is, "I have this seldomly-updated list of ~10000 things, and I'm going to need to join it against my time-series data." With other time-series databases I've dealt with, it's an afterthought at best and the answer is typically "Enrich the data via flink/benthos/etc. on import and avoid using any kind of join." Does Timescale's use of PostgreSQL circumvent this issue, both in terms of storage of lookup tables, and performance on join?
- mfreed 5y agoYes, we support the rich set of PostgreSQL's JOIN operations, including against hypertables. It's generally smart enough to only apply these JOINs against the right subset of time-series data if you also have any time predicates (due to the way we perform "constraint exclusion" against our hypertable chunks). There are other common queries related to what you describe, like a "last point query": Tell me the last record for each distinct object. Here, for example, we've built special query optimizations to significantly accelerate such queries: https://www.timescale.com/blog/how-we-made-distinct-queries-up-to-8000x-faster-on-postgresql/ https://www.timescale.com/blog/how-we-made-distinct-queries-...
- manigandham 5y agoThere are plenty of distributed relational columnstores that can do joins. Timescale is bringing that to Postgres but you already have options from Clickhouse to Redshift.
- mfringel 5y agoSure, but the ability to do so is separate from whether it's a good idea or not.
- intellix 5y agoWe enquired about Timescale Cloud, to migrate from Cloud SQL as we could use it to solve a couple of problems in a simpler way and give us a chance to create other solutions when needed. We don't have anything in time series yet but were hoping to have it there to experiment and start dipping our toes but found it a bit expensive. The pricing seemed more of an all or nothing investment and didn't make sense if we weren't ready to start using the extra functionality just yet. Was hoping that it would be similar to Cloud SQL and just cost extra depending on usage.
- akulkarni 5y agoHi there, not sure how you did your price comparison, but typically with native compression, performance improvements, continuous aggregates, etc, you can go much further with the same resources on Timescale Cloud than Cloud SQL (or any other generic PostgreSQL provider). Did your math take those performance improvements into account?