8 ms·

Not a surprising result at all. Timestream is another case of AMS trying to make something for the sake of having it (see here Kinesis instead of just doing Kaf

by jpgvm 6y ago

Not a surprising result at all. Timestream is another case of AMS trying to make something for the sake of having it (see here Kinesis instead of just doing Kafka and many other products) instead of just adopting something industry standard. That isn't to say they don't do some good stuff just they also ship a lot of crap, very much 2 tiers of products in the AWS catalog. I would eventually like to see comparison for a more analysis heavy workload vs Druid though. Simple metrics workloads like what Timestream and Influx do aren't really of interest when comparing to Timescale ability to do JOINs and use other SQL and PostgreSQL specific functionality.

that_guy_iain 6y ago
It seems like AWS is just focusing on increasing it's vendor lock in and it makes sense not just from the AWS point of view but if you're a developer at an enterprise company that has authorised AWS the more databases the clone, poorly or badly the more choice you have without having to get authorisation and the cost doesn't matter to the developer since the enterprise company is paying for it. I suspect there are many enterprises that are already so locked into AWS that it would take them a decade to move to a different provider.
- frankietaylr 6y ago
  I don't have much expertise with Timestream. Can't comment on it. But, there is nothing wrong in trying for an alternate implementation. We're using Kinesis and it is a good fit in our stack with lot less overhead. They did bring Kakfa (Amazon MSK) later to the AWS service list. Its upto the consumer to pick Kinesis or MSK. One example which was influential in nature - Dyanmodb. The Dyanmodb paper inspired a number of NoSQL databases, including Apache Cassandra. If they have went with the popular NoSQL database at that time, we might have missed some good contribution.
- Aeolun 6y ago
  But you can just run timescaleDB inside your amazon VPC. No need to use their services at all.
  - mfreed 6y ago
    And we in fact have fully open-source/free k8s helm charts to easily spin up TimescaleDB, including with HA and automated failure detection/failover and streaming replication for incremental backups: https://github.com/timescale/timescaledb-kubernetes https://github.com/timescale/timescaledb-kubernetes Or just use Timescale's managed cloud on AWS: https://www.timescale.com/timescale-signup https://www.timescale.com/timescale-signup
  - that_guy_iain 6y ago
    You can but then you need to spend time setting up and maintaining the service. Which many enterprises aren't willing to do. Also, while you may be authorised to use AWS tech you may not be automatically authorised to use different tech. We literally were told to use MySQL instead of MongoDB soley because RDS was allowed and all services had to be managed by AWS. I am not complaining AWS is doing vendor lock in makes complete sense, I am just commenting that is their main direction for their offering. And their vendor lock is aimed at the enterprises where they can really run wild on it and get lots of money in return. Smaller orginisations will generally never lock themselves in so much because they won't build so many things that are so hard coupled since they just don't have the manpower.
- tracker1 6y ago
  Much like Amazon themselves with breaking away from using Oracle.
superzamp 6y ago
I remember the announcement of Sagemaker at an AWS summit, a moment of profound embarrassment - the whole thing was barely a Jupyter notebook glued to some EC2 stuff with duct tape. You're right, they're definitely doing things for the sole purpose of locking you in.
- whimsicalism 6y ago
  I still do not understand any of the value add of Sagemaker.
- deanCommie 6y ago
  ...have you looked at it recently? It's a MASSIVE suite of services, tools, and capabilities...https://aws.amazon.com/sagemaker/ https://aws.amazon.com/sagemaker/
  - jamesblonde 6y ago
    You still have to compile your own Docker image and then "run it" from a Jupyter notebook if you are not using their Estimators (who uses those?). What is the point of just launching Docker images from Jupyter notebooks and looking a different window to see the log? Serving is ok on Sagemaker, i will admit. The new ones: feature store, pipelines, and data wrangler - we will see, all rushed (some more than others).
atanasovskib 6y ago
In my opinion it would be very hard to justify using Timestream for any analysis heavy workloads for at least three reasons: 1. Queries will need to touch a lot of data - will cost a lot, and there is no ability to optimize the queries in any way (no EXPLAIN, no indexes, no downsampling) 2. No integration with data exploration and visualization tools 3. No ability to have non-timeseries data, or correlate any data in two different tables (no JOINs) (Disclaimer: I work at TimescaleDB)
- jpgvm 6y ago
  I completely agree I think you misunderstood me. I would like to see comparisons between Timescale and Druid. I already know Timestream isn't up to the task lol.
  - valyala 6y ago
    What about comparing TimescaleDB to VictoriaMetrics? There are some old benchmarks [1], but it would be great to see updated benchmarks as well for the latest TimescaleDB and VictoriaMetrics versions. [1] https://valyala.medium.com/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae https://valyala.medium.com/measuring-vertical-scalability-fo...
- siliconmountain 6y ago
  What kind of auth & logging does TimescaleDB offer? Is every request authenticated & authorized?
  - mfreed 6y ago
    TimescaleDB inherits from Postgres a rich role-based access control, so all requests require permissions and roles can be enforced at various levels (database, schema, table, or even rows) and privileges (SELECT, INSERT, DELETE, CREATE, etc). Authentication to the database is typically governed to different security mechanisms, including certificate-based auth, password via SSL, etc.
tester34 6y ago
Working at AWS must be fun You basically reinvent all the stuff - from databases to all avaliable tooling normal companies dont do that, so there' an opportunity to get deep into some specific branch of applied informatics.
- k__ 6y ago
  I think, the difference is that AWS already has a bunch of core services that scale and uses them as a base for new services. (MongoDB->DocumentDB) Also, they hear about the problems some customers have with an existing product and try to (re)invent a better solution. (MySQL->Aurora)
- jrnkntl 6y ago
  They even name their annual event this way[1] [1] https://reinvent.awsevents.com https://reinvent.awsevents.com
- LukeEF 6y ago
  But why do they do it so badly? Seems like they have the engineering muscle to put out a great reinvention of a document db, a time series db, a graph db... but each is relatively poor to the other services on AWS. Is it the combination of MVP-culture and lock-in?
  - nkozyra 6y ago
    Breadth versus depth seems to be the play outside of their absolute core offerings. It allows people to dip their toes into something with minimal commitment and by the time they realize the shortcomings they might conclude that migrating for those features just isn't worth it. So this give them a bit of lock-in. Come for the EC2, stay for the minimally viable queues, machine learning, containers, etc.
    - nautilus12 6y ago
      This exactly, they are just trying to get their foot in the door in literally any idea someone else is already pursuing so that if it starts to make money they can start to put the juice on the engineering for that and take their market share. The business model is solely focused on competing with and taking out other businesses, not actually providing any value. AWS is like a pack of sharks.
shin_lao 6y ago
In a recent webinar for capital markets, AWS didn't even try to advertise Timestream, they talked about the three most popular tsdb in finance and that's it. Source: https://pages.awscloud.com/awsmp-h2-fin-time-series-database.html https://pages.awscloud.com/awsmp-h2-fin-time-series-database...
- pstuart 6y ago
  And those 3 are?
  - a_gopher 6y ago
    OneTick, Kx, and QuasarDB
    - pstuart 6y ago
      Thanks!
troelsSteegin 6y ago
How to tell what service is in what tier, crap or good? What's the metric? I am not a direct user of AWS, so absent an informed technical judgment of "this is a deployment and renaming of that", eg Kinesis for Kafka, how do you know what to avoid? In general, it seems like the value-add for service X is that now that function is integrated to some degree w/ the rest of AWS - so maybe it's based on effective feature set/use cases plus integration points. Alternatively, some function of throughput and unit cost. If I were a SAAS competing with an AWS service, I would want to know how long before a given AWS service might get good enough to be an alternative for my customers. If you run with the idea that AWS offerings get better over time and eventually cross some threshold of "not good" to "good", maybe a crawl of the AWS announcements[0] in combination with a metric could provide a predictor of an AWS service's time to "good"... [0] https://aws.amazon.com/new/ https://aws.amazon.com/new/
zachd1_618 6y ago
I think AWS is early to offer a not-yet-matured product just to get it in front of customers and start gathering real usage feedback. I’d consider dynamo a fantastic tier 1 service, but it wasn’t always that way. This is inline with the Amazon philosophy in general. As for “making something for the sake of having it”, I believe the key reason for building their own applications is that they can build it on the same multi-az incremental ledger/block store that underlies most services (first built for aurora), or a variant thereof. They have some pretty stellar underlying tech built to be reliable at cloud scale, something you won’t get from running a service on your own EC2 fleet. The new stuff built on top of it just isn’t all that mature though. That’s just my take though. If interested, the deep dive talk from reinvent last year on aurora is pretty cool. Disclaimer: I work at Amazon, nothing to do with AWS or timestream. In fact, I’ve been excited for time stream for a while but found its initial performance lackluster for what I needed.
- ramraj07 6y ago
  Which deepdive talk are you recommending?
  - manigandham 6y ago
    AWS re:Invent 2019: [REPEAT 1] Amazon Aurora storage demystified: How it all works (DAT309-R1): https://www.youtube.com/watch?v=DrtwAOND1Pc https://www.youtube.com/watch?v=DrtwAOND1Pc But all of the Deep Dive talks are good for technical details: https://www.youtube.com/results?search_query=aws+reinvent+2019+deep+dive https://www.youtube.com/results?search_query=aws+reinvent+20...
- cheesgrateron 6y ago
  >I think AWS is early to offer a not-yet-matured product just to get it in front of customers and start gathering real usage feedback. It's not just this. In many cases, the services launched are essentially MVPs that were only created to serve a handful of customers' needs. If a huge whale of a customer says they want hosted Jupyter notebooks on AWS (just a theoretical example I picked because I saw someone criticize Sagemaker, but I don't have any specific knowledge of Sagemaker), then AWS will create that for them, but, AWS doesn't actually do "one off" services for a specific customer, so they will also launch those hosted notebooks as a public service. The result is of course that this service was built with one customer's use case in mind, so it might not serve other customers' use cases very well, which leads to some negative perception. But ideally AWS then goes and incorporates feedback into the service, turning it into a "tier 1" service rather than an MVP. It just takes time.
ignoramous 6y ago
> ...see here Kinesis instead of just doing Kafka and many other products Kinesis was a solution to their in-house woes with metering and billing which absolutely drowned in so much data from internal services with myriad metering and billing items and rules that they had to do something about it themselves: https://gigaom.com/2014/03/20/why-amazon-built-its-data-stream-processing-engine-kinesis/ https://gigaom.com/2014/03/20/why-amazon-built-its-data-stre... BTW, Kinesis Data Firehose is a pretty good product.
- jamesblonde 6y ago
  Kinesis Data Firehose is now pretty much Flink. Flink is a pretty good product.

by jpgvm 6y ago

that_guy_iain 6y ago

It seems like AWS is just focusing on increasing it's vendor lock in and it makes sense not just from the AWS point of view but if you're a developer at an enterprise company that has authorised AWS the more databases the clone, poorly or badly the more choice you have without having to get authorisation and the cost doesn't matter to the developer since the enterprise company is paying for it. I suspect there are many enterprises that are already so locked into AWS that it would take them a decade to move to a different provider.

frankietaylr 6y ago
I don't have much expertise with Timestream. Can't comment on it. But, there is nothing wrong in trying for an alternate implementation. We're using Kinesis and it is a good fit in our stack with lot less overhead. They did bring Kakfa (Amazon MSK) later to the AWS service list. Its upto the consumer to pick Kinesis or MSK. One example which was influential in nature - Dyanmodb. The Dyanmodb paper inspired a number of NoSQL databases, including Apache Cassandra. If they have went with the popular NoSQL database at that time, we might have missed some good contribution.
Aeolun 6y ago
But you can just run timescaleDB inside your amazon VPC. No need to use their services at all.
- mfreed 6y ago
  And we in fact have fully open-source/free k8s helm charts to easily spin up TimescaleDB, including with HA and automated failure detection/failover and streaming replication for incremental backups: https://github.com/timescale/timescaledb-kubernetes https://github.com/timescale/timescaledb-kubernetes Or just use Timescale's managed cloud on AWS: https://www.timescale.com/timescale-signup https://www.timescale.com/timescale-signup
- that_guy_iain 6y ago
  You can but then you need to spend time setting up and maintaining the service. Which many enterprises aren't willing to do. Also, while you may be authorised to use AWS tech you may not be automatically authorised to use different tech. We literally were told to use MySQL instead of MongoDB soley because RDS was allowed and all services had to be managed by AWS. I am not complaining AWS is doing vendor lock in makes complete sense, I am just commenting that is their main direction for their offering. And their vendor lock is aimed at the enterprises where they can really run wild on it and get lots of money in return. Smaller orginisations will generally never lock themselves in so much because they won't build so many things that are so hard coupled since they just don't have the manpower.
tracker1 6y ago
Much like Amazon themselves with breaking away from using Oracle.

superzamp 6y ago

I remember the announcement of Sagemaker at an AWS summit, a moment of profound embarrassment - the whole thing was barely a Jupyter notebook glued to some EC2 stuff with duct tape. You're right, they're definitely doing things for the sole purpose of locking you in.

whimsicalism 6y ago
I still do not understand any of the value add of Sagemaker.
deanCommie 6y ago
...have you looked at it recently? It's a MASSIVE suite of services, tools, and capabilities...https://aws.amazon.com/sagemaker/ https://aws.amazon.com/sagemaker/
- jamesblonde 6y ago
  You still have to compile your own Docker image and then "run it" from a Jupyter notebook if you are not using their Estimators (who uses those?). What is the point of just launching Docker images from Jupyter notebooks and looking a different window to see the log? Serving is ok on Sagemaker, i will admit. The new ones: feature store, pipelines, and data wrangler - we will see, all rushed (some more than others).

atanasovskib 6y ago

In my opinion it would be very hard to justify using Timestream for any analysis heavy workloads for at least three reasons: 1. Queries will need to touch a lot of data - will cost a lot, and there is no ability to optimize the queries in any way (no EXPLAIN, no indexes, no downsampling) 2. No integration with data exploration and visualization tools 3. No ability to have non-timeseries data, or correlate any data in two different tables (no JOINs) (Disclaimer: I work at TimescaleDB)

jpgvm 6y ago
I completely agree I think you misunderstood me. I would like to see comparisons between Timescale and Druid. I already know Timestream isn't up to the task lol.
- valyala 6y ago
  What about comparing TimescaleDB to VictoriaMetrics? There are some old benchmarks [1], but it would be great to see updated benchmarks as well for the latest TimescaleDB and VictoriaMetrics versions. [1] https://valyala.medium.com/measuring-vertical-scalability-for-time-series-databases-in-google-cloud-92550d78d8ae https://valyala.medium.com/measuring-vertical-scalability-fo...
siliconmountain 6y ago
What kind of auth & logging does TimescaleDB offer? Is every request authenticated & authorized?
- mfreed 6y ago
  TimescaleDB inherits from Postgres a rich role-based access control, so all requests require permissions and roles can be enforced at various levels (database, schema, table, or even rows) and privileges (SELECT, INSERT, DELETE, CREATE, etc). Authentication to the database is typically governed to different security mechanisms, including certificate-based auth, password via SSL, etc.

tester34 6y ago

Working at AWS must be fun You basically reinvent all the stuff - from databases to all avaliable tooling normal companies dont do that, so there' an opportunity to get deep into some specific branch of applied informatics.

k__ 6y ago
I think, the difference is that AWS already has a bunch of core services that scale and uses them as a base for new services. (MongoDB->DocumentDB) Also, they hear about the problems some customers have with an existing product and try to (re)invent a better solution. (MySQL->Aurora)
jrnkntl 6y ago
They even name their annual event this way[1] [1] https://reinvent.awsevents.com https://reinvent.awsevents.com
LukeEF 6y ago
But why do they do it so badly? Seems like they have the engineering muscle to put out a great reinvention of a document db, a time series db, a graph db... but each is relatively poor to the other services on AWS. Is it the combination of MVP-culture and lock-in?
- nkozyra 6y ago
  Breadth versus depth seems to be the play outside of their absolute core offerings. It allows people to dip their toes into something with minimal commitment and by the time they realize the shortcomings they might conclude that migrating for those features just isn't worth it. So this give them a bit of lock-in. Come for the EC2, stay for the minimally viable queues, machine learning, containers, etc.
  - nautilus12 6y ago
    This exactly, they are just trying to get their foot in the door in literally any idea someone else is already pursuing so that if it starts to make money they can start to put the juice on the engineering for that and take their market share. The business model is solely focused on competing with and taking out other businesses, not actually providing any value. AWS is like a pack of sharks.

shin_lao 6y ago

In a recent webinar for capital markets, AWS didn't even try to advertise Timestream, they talked about the three most popular tsdb in finance and that's it. Source: https://pages.awscloud.com/awsmp-h2-fin-time-series-database.html https://pages.awscloud.com/awsmp-h2-fin-time-series-database...

pstuart 6y ago
And those 3 are?
- a_gopher 6y ago
  OneTick, Kx, and QuasarDB
  - pstuart 6y ago
    Thanks!

troelsSteegin 6y ago

How to tell what service is in what tier, crap or good? What's the metric? I am not a direct user of AWS, so absent an informed technical judgment of "this is a deployment and renaming of that", eg Kinesis for Kafka, how do you know what to avoid? In general, it seems like the value-add for service X is that now that function is integrated to some degree w/ the rest of AWS - so maybe it's based on effective feature set/use cases plus integration points. Alternatively, some function of throughput and unit cost. If I were a SAAS competing with an AWS service, I would want to know how long before a given AWS service might get good enough to be an alternative for my customers. If you run with the idea that AWS offerings get better over time and eventually cross some threshold of "not good" to "good", maybe a crawl of the AWS announcements[0] in combination with a metric could provide a predictor of an AWS service's time to "good"... [0] https://aws.amazon.com/new/ https://aws.amazon.com/new/

zachd1_618 6y ago

I think AWS is early to offer a not-yet-matured product just to get it in front of customers and start gathering real usage feedback. I’d consider dynamo a fantastic tier 1 service, but it wasn’t always that way. This is inline with the Amazon philosophy in general. As for “making something for the sake of having it”, I believe the key reason for building their own applications is that they can build it on the same multi-az incremental ledger/block store that underlies most services (first built for aurora), or a variant thereof. They have some pretty stellar underlying tech built to be reliable at cloud scale, something you won’t get from running a service on your own EC2 fleet. The new stuff built on top of it just isn’t all that mature though. That’s just my take though. If interested, the deep dive talk from reinvent last year on aurora is pretty cool. Disclaimer: I work at Amazon, nothing to do with AWS or timestream. In fact, I’ve been excited for time stream for a while but found its initial performance lackluster for what I needed.

ramraj07 6y ago
Which deepdive talk are you recommending?
- manigandham 6y ago
  AWS re:Invent 2019: [REPEAT 1] Amazon Aurora storage demystified: How it all works (DAT309-R1): https://www.youtube.com/watch?v=DrtwAOND1Pc https://www.youtube.com/watch?v=DrtwAOND1Pc But all of the Deep Dive talks are good for technical details: https://www.youtube.com/results?search_query=aws+reinvent+2019+deep+dive https://www.youtube.com/results?search_query=aws+reinvent+20...
cheesgrateron 6y ago
>I think AWS is early to offer a not-yet-matured product just to get it in front of customers and start gathering real usage feedback. It's not just this. In many cases, the services launched are essentially MVPs that were only created to serve a handful of customers' needs. If a huge whale of a customer says they want hosted Jupyter notebooks on AWS (just a theoretical example I picked because I saw someone criticize Sagemaker, but I don't have any specific knowledge of Sagemaker), then AWS will create that for them, but, AWS doesn't actually do "one off" services for a specific customer, so they will also launch those hosted notebooks as a public service. The result is of course that this service was built with one customer's use case in mind, so it might not serve other customers' use cases very well, which leads to some negative perception. But ideally AWS then goes and incorporates feedback into the service, turning it into a "tier 1" service rather than an MVP. It just takes time.

ignoramous 6y ago

> ...see here Kinesis instead of just doing Kafka and many other products Kinesis was a solution to their in-house woes with metering and billing which absolutely drowned in so much data from internal services with myriad metering and billing items and rules that they had to do something about it themselves: https://gigaom.com/2014/03/20/why-amazon-built-its-data-stream-processing-engine-kinesis/ https://gigaom.com/2014/03/20/why-amazon-built-its-data-stre... BTW, Kinesis Data Firehose is a pretty good product.

jamesblonde 6y ago
Kinesis Data Firehose is now pretty much Flink. Flink is a pretty good product.