4 ms·

Is this a big deal? What are the noticeable offering here.

scrollaway 9y ago
Aurora is very interesting especially in terms of removing storage management from the equation. I just wish they (and the regular postgres RDS offering) had support for TimescaleDB (http://www.timescale.com http://www.timescale.com).
- samstave 9y ago
  Mind givin some use case scenarios for timescale that you really like? (aside from what one may think obvious)
  - scrollaway 9y ago
    I want to replace InfluxDB with it. Here's what we gain: - Postgres' reliability. Influx is awful and I absolutely do not trust it; I've had far too much data loss and consistency issues with it. - Postgres' internal tooling. Simple things such as "What is my largest measurement on disk" are not possible in Influx - Simplified stack. Timescale makes good on that promise: We no longer would have to ship influxdb libraries for internal apps. Permissions are simplified as well. - Hosted, managed metrics. Our InfluxDB instance is managed by ourselves and is a source of issues. Our RDS instance, comparatively, has always purred. - More advanced operations, including the full power of Postgres user-defined functions. Influx doesn't even support dividing one value by another. - Grafana support! It's finally here!!! :)
    - llsf 9y ago
      I was thinking about the same use-case, but there are features in InfluxDB (downscaling, retention policies) that are cool when dealing with large amount of data. If Timescale bridges the gap, I would be happy to give a try...
      - akulkarni 9y ago
        (CEO TimescaleDB) We already allow for efficient manual deletion of old data[1] (2000x faster than PostgreSQL)[2] and rollups via standard PG aggregates. Automatic retention policies and downsampling are coming soon. But as one of the commentors above points out, we benefit from the reliability (and broad ecosystem) of PostgreSQL, which has allowed various companies to easily deploy us in production with minimal integration efforts. [1] http://docs.timescale.com/latest/api/data-retention http://docs.timescale.com/latest/api/data-retention [2] https://blog.timescale.com/timescaledb-vs-6a696248104e https://blog.timescale.com/timescaledb-vs-6a696248104e
    - linsomniac 9y ago
      Weird, I've had really great experience with InfluxDB. I literally couldn't be happier. The early days were rough, 18 month ago or so. In the last year, it's been very solid. I store metrics (largely from telegraf) on system performance from more than 100 physical/virtual machines into it, and have full resolution data since Feb 2016 stored in around 100GB across 3 Influx instances (dev, stg, prod). Compared to our previous collectd/graphite infrastructure, this is a huge win. Much lower system utilization, easier to add custom metrics. Don't get me wrong, I love Postgres, been using it since '96, but Influx/Telegraf/Grafana has given me the ease of Munin with the high resolution data of collectd while also providing low overhead and low maintenance.
      - scrollaway 9y ago
        It's certainly gotten better but I would not call it a great experience. Right this minute for example, my metrics machine's disk is filling up at higher than usual rate and I have no knowledge of what is causing this to happen, no tools to figure it out. A week ago, I tried to set up a "successful writes per second" metric. I have a "time since start", and "successful writes since start", but no way to operate one with the other. The HTTP input API is dog-slow with HTTPS, and the UDP input API requires a separate socket for every database. It's lots of things like these...
        linsomniac 9y ago
        "successful writes since start" has a time value associated with it, right? I think what you want is not to correlate "time since start" and "successful writes since start", but to just do a derivative of the "successful writes since start" metric. This is just like the networking stats on Linux hosts. I have something like: FROM default net WHERE host =~ /$Host$/ SELECT field(bytes_recv) mean() derivative(1s) alias(In) GROUP BY time($interval) tag(host) tag(interface) fill(null) FORMAT AS Time series ALIAS BY [[col]] [[tag_host]]/[[tag_interface]] That's in my Grafana query. Aside: I sent the TimeseriesDB reference to a friend that just a couple days ago asked about options for storing metrics on storage servers. I had told him about my experiences with Influx, had mentioned Prometheus and also Sentry has some sort of internal TSDB they do on Postgres. In reply to the TimeseriesDB reference, he replied that he was all about InfluxDB now.
- LogicX 9y ago
  Came here to say the same - timescale support would be awesome!
- mrep 9y ago
  Didn't know about timescale DB but it looks awesome and sounds amazing for what my team currently does.
dcosson 9y ago
It's a specially designed networked filesystem as the storage layer of a hosted postgres DB. From what I've read it has some interesting properties like built-in redundancy, expandable storage, you can instantly spin up replicas and copy-on-write clones of the db, plus they claim 2x+ performance compared to vanilla postgres. From what I've gathered it's kind of similar to the Spanner db that google released a few months ago in that there's a pretty tight integration of the software with the hardware & network layer that they control, which is what makes some of these improvements possible. The other thing that I find promising, is RDS has always felt kind of awkward in terms of what's still vanilla postgres that you're on your own for and what AWS does for you. I haven't tried Aurora yet but it seems like it might feel more AWS-native (things like having better performance monitoring tools and using IAM roles to control the db users).

Aurora is very interesting especially in terms of removing storage management from the equation. I just wish they (and the regular postgres RDS offering) had support for TimescaleDB (http://www.timescale.com http://www.timescale.com).

samstave 9y ago
Mind givin some use case scenarios for timescale that you really like? (aside from what one may think obvious)
- scrollaway 9y ago
  I want to replace InfluxDB with it. Here's what we gain: - Postgres' reliability. Influx is awful and I absolutely do not trust it; I've had far too much data loss and consistency issues with it. - Postgres' internal tooling. Simple things such as "What is my largest measurement on disk" are not possible in Influx - Simplified stack. Timescale makes good on that promise: We no longer would have to ship influxdb libraries for internal apps. Permissions are simplified as well. - Hosted, managed metrics. Our InfluxDB instance is managed by ourselves and is a source of issues. Our RDS instance, comparatively, has always purred. - More advanced operations, including the full power of Postgres user-defined functions. Influx doesn't even support dividing one value by another. - Grafana support! It's finally here!!! :)
  - llsf 9y ago
    I was thinking about the same use-case, but there are features in InfluxDB (downscaling, retention policies) that are cool when dealing with large amount of data. If Timescale bridges the gap, I would be happy to give a try...
    - akulkarni 9y ago
      (CEO TimescaleDB) We already allow for efficient manual deletion of old data[1] (2000x faster than PostgreSQL)[2] and rollups via standard PG aggregates. Automatic retention policies and downsampling are coming soon. But as one of the commentors above points out, we benefit from the reliability (and broad ecosystem) of PostgreSQL, which has allowed various companies to easily deploy us in production with minimal integration efforts. [1] http://docs.timescale.com/latest/api/data-retention http://docs.timescale.com/latest/api/data-retention [2] https://blog.timescale.com/timescaledb-vs-6a696248104e https://blog.timescale.com/timescaledb-vs-6a696248104e
  - linsomniac 9y ago
    Weird, I've had really great experience with InfluxDB. I literally couldn't be happier. The early days were rough, 18 month ago or so. In the last year, it's been very solid. I store metrics (largely from telegraf) on system performance from more than 100 physical/virtual machines into it, and have full resolution data since Feb 2016 stored in around 100GB across 3 Influx instances (dev, stg, prod). Compared to our previous collectd/graphite infrastructure, this is a huge win. Much lower system utilization, easier to add custom metrics. Don't get me wrong, I love Postgres, been using it since '96, but Influx/Telegraf/Grafana has given me the ease of Munin with the high resolution data of collectd while also providing low overhead and low maintenance.
    - scrollaway 9y ago
      It's certainly gotten better but I would not call it a great experience. Right this minute for example, my metrics machine's disk is filling up at higher than usual rate and I have no knowledge of what is causing this to happen, no tools to figure it out. A week ago, I tried to set up a "successful writes per second" metric. I have a "time since start", and "successful writes since start", but no way to operate one with the other. The HTTP input API is dog-slow with HTTPS, and the UDP input API requires a separate socket for every database. It's lots of things like these...
      - linsomniac 9y ago
        "successful writes since start" has a time value associated with it, right? I think what you want is not to correlate "time since start" and "successful writes since start", but to just do a derivative of the "successful writes since start" metric. This is just like the networking stats on Linux hosts. I have something like: FROM default net WHERE host =~ /$Host$/ SELECT field(bytes_recv) mean() derivative(1s) alias(In) GROUP BY time($interval) tag(host) tag(interface) fill(null) FORMAT AS Time series ALIAS BY [[col]] [[tag_host]]/[[tag_interface]] That's in my Grafana query. Aside: I sent the TimeseriesDB reference to a friend that just a couple days ago asked about options for storing metrics on storage servers. I had told him about my experiences with Influx, had mentioned Prometheus and also Sentry has some sort of internal TSDB they do on Postgres. In reply to the TimeseriesDB reference, he replied that he was all about InfluxDB now.
LogicX 9y ago
Came here to say the same - timescale support would be awesome!
mrep 9y ago
Didn't know about timescale DB but it looks awesome and sounds amazing for what my team currently does.

dcosson 9y ago

It's a specially designed networked filesystem as the storage layer of a hosted postgres DB. From what I've read it has some interesting properties like built-in redundancy, expandable storage, you can instantly spin up replicas and copy-on-write clones of the db, plus they claim 2x+ performance compared to vanilla postgres. From what I've gathered it's kind of similar to the Spanner db that google released a few months ago in that there's a pretty tight integration of the software with the hardware & network layer that they control, which is what makes some of these improvements possible. The other thing that I find promising, is RDS has always felt kind of awkward in terms of what's still vanilla postgres that you're on your own for and what AWS does for you. I haven't tried Aurora yet but it seems like it might feel more AWS-native (things like having better performance monitoring tools and using IAM roles to control the db users).