12 ms·

AWS Redshift Serverless

64 points by Trisell 5y ago

opjjf 5y ago
So basically BigQuery from AWS. Looks good on first sight, a bit late. Personally worked for a large org which has just moved from BigQuery from Redshift and I have to say that BigQuery is the much better product.
- glogla 5y ago
  Yeah I get that they want to go after Snowflake, Databricks and BigQuery, but AWS it not known for delivering high quality software. They have few things that are very good (EC2, S3, Lambda) and the rest is done by their B-team and barely holding together.
  - fdgsdfogijq 5y ago
    Keep telling yourself that
    - MaxGanzII 5y ago
      I have to say, speaking for Redshift, I significantly agree with the critique. There are too many simple problems which should have been caught in testing, and the problem which have over time been found absolutely imply unprofessional, even amateurish software development standards. For example, recently, the format of the version string was changed. This broke a lot of existing software, which had hard coded parsing - SQL Alchemy stopped working - so did AWS's own JDBC driver. This on the face of it indicates the RS test suite does not include any connections over JDBC. It then turned out the version string had anyway been inaccurate for months, because RS had moved from GCC 3.4.2 (I think it was) to 7.3. But the version string kept reporting the old numbers. I can't even begin to describe how many issues - flat factual errors, and profoundly meaningful gobble-de-gook - in the official docs. The whole thing just feels too much like amateur hour.
- pram 5y ago
  Athena is BigQuery from AWS. I'd have to imagine this is for people already heavily invested in Redshift.
  - tedivm 5y ago
    Athena is not BigQuery. Athena is just a fancy wrapper around Hive (you can see this right from the log output) and just runs map reduce over your S3 data. It's a great tool for what it is. BigQuery is a full database. It is significantly faster than running anything from Athena. The closest comparison on AWS is Redshift.
    - ideonode 5y ago
      Athena is more of a wrapper over Presto / Trino than Hive. It does not run map reduce per se over your data - its a SQL engine.
      - pdelgallego 5y ago
        AWS Glue Data catalog is Hive, and as you mentioned, Athena is a wrapper of Presto
    - mdasen 5y ago
      I think Athena has a lot of value (especially given its pricing), but you're not wrong that it has limitations. Getting data into Athena isn't something that is just done for you. Athena just takes what you've put on S3 and queries over it - and leaves getting it onto S3 (and into an efficient format) as an exercise for the reader. Athena's speed varies a lot depending on what format you put things in. Querying over CSVs will mean that you're slow and reading a lot of data. Querying over ORC (column-store) files is pretty quick. The big thing is Athena's pricing. They price it on how much data you read in reality not how much data would be read if things weren't optimized. BigQuery charges you based on how much data would be read if it weren't optimized. With BigQuery, an integer is always 8 bytes. It doesn't matter if they're able to optimize it down to nothing using RLE (run length encoding). You still pay the full 8 bytes. If your ORC files make that integer column tiny, you get the benefit of that. BigQuery is great, but Athena's pricing is a lot cheaper given that you get to benefit from any storage optimization you do. Out of curiosity, how have you used Athena that you're seeing it be so much slower? In my experience, BigQuery is faster (maybe 2x faster), but I've been using column-oriented data with Athena. If you're using CSVs with Athena, it will be way slower than BigQuery. I'm always a little surprised that AWS doesn't build Athena out more, but I guess if they did they'd want money and margin for the value add. Still, Athena is a pretty decent serverless Presto and Presto can work pretty well over data in column formats.
- tedivm 5y ago
  The only problem with BigQuery is that it's on GCP, so either you have to migrate your whole workload over or you end up eating a lot of data transfer costs. Having used both I do think BigQuery is better in a lot of ways (although it's easier to make it a lot expensive too), but I'm really excited to see Redshift catch up. Adding the serverless options are really great too since my biggest complaint with Redshift was managing the quantity and type of the underlying instances.
  - Scorpiion 5y ago
    I have not used it, but my understanding is that BigQuery Omni is "BigQuery running on AWS/Azure". My understanding is that they are running Anthos on AWS (managed by Google) and they offer BigQuery as a service from that AWS infra managed by Google. See more details here: - https://cloud.google.com/bigquery-omni/docs/introduction https://cloud.google.com/bigquery-omni/docs/introduction - https://cloud.google.com/bigquery-omni/docs/aws https://cloud.google.com/bigquery-omni/docs/aws
- spullara 5y ago
  It isn't. It is more similar to Snowflake's warehouse model.
orf 5y ago
We’ve just moved to Snowflake. I haven’t really been impressed with some of the new features added to Redshift, it seems like too little and too late. The JSON support (SUPER type) is kind of cool, and they are moving towards more “automatic” sorting + partitioning, but it’s just all a bit shit to be honest. We encountered major bugs with data-sharing, our clusters keep insisting that zstd is the best compression format to use for all our data (but then never actually using it), materialised views often fail to update and understanding why is a nightmare, terrible performance if your strings are varchar(max) (guess what Glue sets them to…), Redshift data often just dies (4 hour downtime recently, no status page) and has some really weird semantics around listing queries, before the data API you couldn’t run async queries and it’s eventbridge integration straight up doesn’t work, nightmare bugs in the Java connection library that don’t show up using psql, tiny set of types (no arrays, uuids), unkillable queries, AQUA actually causing everything to slow down hugely, critical release notes posted only in a fucking random forum, etc etc. Snowflake has apparently sorted this, as well as including ingestion tools (snowpipe) that you’d otherwise have to stitch together with AWS Glue or something (a cursed service if ever there was one). That being said, in some cases Redshift absolutely flies. But the real world isn’t filled with ideal schemas and natural sort keys. It’s messy. And Snowflake deals with messy better.
- dominotw 5y ago
  given snowflake also uses s3. Could aws do something with redshift that moves computation closer to s3 and be even faster than snowflake. Basically take advantage of the fact that they own s3.
  - dalyons 5y ago
    they already have https://aws.amazon.com/redshift/features/aqua/ https://aws.amazon.com/redshift/features/aqua/ basically distributing compute down to the actual storage nodes
    - orf 5y ago
      Yeah that’s cool except it made all our queries slower. I’ll add that to my list above, forgot about that.
ggregoire 5y ago
It's not very clear why I should keep using a normal Redshift instead of switching to Serverless? Anyone has more insights?
- bayan1234 5y ago
  If you have a very stable SQL pipeline that runs frequently during the day then it's probably cheaper to have it on a normal Redshift cluster.
- MaxGanzII 5y ago
  The minimum cost for serverless seems quite high - 14.4 USD per hour. The advantage is the flexibility to easily change compute resource. A disadvantage is that your data is now in S3 or something very like it, and this I think alters the characteristics of write performance, for the cluster; I've not yet looked into this, but it's on the list. You absolutely should beware of falling into the trap of imagining that serverless simply gives you flexible compute and that's the only change to behaviour. AWS in their press releases and docs are relentlessly positive - anything which is not a strength is obfuscated - so only actual experimentation and investigation throws light on what you're really getting.
bryan0 5y ago
How is this different than Athena?
- the_af 5y ago
  Athena is not a relational database under the hood. Very different technologies with different implications.
- glogla 5y ago
  Two competing departments inside AWS. Technically, Athena is based on modified Presto while Redshift is (very) heavily modified Postgres.
  - the_af 5y ago
    Competing departments and competing underlying technologies. Postgres (and therefore Redshift) is running a relational database engine under the hood.
    - glogla 5y ago
      Yes, though with ra3 storage being on S3, I think they are getting more and more similar.
- tyingq 5y ago
  Query things other than data in S3 seems to be the big difference.
  - bdcravens 5y ago
    Athena can query non-s3 data sources using federated query.
    - the_af 5y ago
      But that's not to say it's the best idea. Athena is mostly for S3; AWS say so. From their FAQ [1]: Q: What is the difference between Amazon Athena, Amazon EMR, and Amazon Redshift? [...] Amazon Redshift provides the fastest query performance for enterprise reporting and business intelligence workloads, particularly those involving extremely complex SQL with multiple joins and sub-queries. [...] Amazon Athena provides the easiest way to run ad-hoc queries for data in S3 without the need to setup or manage any servers. Elsewhere they also recommend Athena for unstructured data. This and a relational database like Postgres are fundamentally different and you shouldn't use one when your use case is primarily meant for the other.
ndm000 5y ago
I have worked with several companies that have their infrastucture on AWS but consider BigQuery or Snowflake for the serverless model they provide. This brings RedShift much closer to those options. I envision Redshift Serverless becoming the default options for most enterprises, mainly because it stays in the AWS ecosystem and you don't have to work with a different vendor and create different cost governance processes. I beleive the real advantage AWS has here is in cost. Snowflake has positioned itself as price competitive with Redshift but this is primarily due to Snowflake's ability to scale on-demand, whereas prior Redshift versions required you to size for peak usage (RA3 helped with this). In my experience Snowflake is an order of magnitude more expensive if you compare similiar workloads and do not account for idle time. We will need to see the performance of a "Redshift Processing Unit" to be sure of the advantage, but even so AWS will be able provide significant downward cost pressure through this offering.
- ignoramous 5y ago
  > In my experience Snowflake is an order of magnitude more expensive if you compare similiar workloads and do not account for idle time. Cost reasons is why I'm most bullish about DataBricks's FOSS https://delta.io https://delta.io
  - gizmodo59 5y ago
    Delta is not the way. I’d prefer an Apache project like Iceberg https://iceberg.apache.org/ https://iceberg.apache.org/ rather than delta lake.
dreyfan 5y ago
> Just load your data Load my data where? This is "serverless".
- the_af 5y ago
  To "the cloud" :P More seriously, "serverless" usually just means you aren't supposed to worry about server/cluster management, not that there are no servers anywhere. So it really means "load your data to Redshift, wherever that lives".
  - dreyfan 5y ago
    I look forward to the next stage of cloud hype - dataless - so I don't have to worry about the nuances of data management and I can just run my analytics.
    - antod 5y ago
      I've long dreamed of that. Everything becomes stateless and everything to do with GDPR/PCI/Disaster recovery/anonymisation/performance/scaling/etc/etc becomes so much easier.
spullara 5y ago
I wish Amazon would stop naming things "serverless" that clearly have a well defined type and number of servers at any point. That includes Redshift Serverless and Aurora Serverless. If it has a cluster, it isn't serverless it is just autoscaling. Every time they announce a serverless product I'm assuming that it will be like Lambda and am mostly disappointed. For example, a real Aurora Serverless would be more like CockroachDB Cloud or DynamodDB. And a real Redshift Serverless would be more like BigQuery.
- MaxGanzII 5y ago
  AWS hype everything. What they emit is the dance routine of a sugar coated cheerleader squad. I am of the view everything which is not a strength is obfuscated. I have zero faith, confidence and trust is all information AWS emits. I approach press releases and the docs on the basis that they cover up the actual implementation, and so my task is to find out what is actually going on under the hood, so I can actually make sense of what's been provided and operate it correctly (or avoid it completely, as it may be!)
bayan1234 5y ago
I was initially excited about this as I think it might solve our Redshift pain points and potentially avoid us having to deal with a migration to Snowflake but then I remembered when AWS account managers promised Athena and Spectrum would solve these same problems at a previous company I worked for a few years ago. I'm assuming the developer experience will still be terrible with lots of knobs to tune to actually get any decent cost/performance.
- MaxGanzII 5y ago
  I hold a very dim view of all information AWS emits, by whatever channels. My experience with support/account managers is that they always tell you "yes, Redshift can do this", and the and the only way to actually get a "no" out of them is to already know Redshift cannot do something, and to explain to them why. They won't deny reality, but you would never have got that answer from them in any other way. I suspect the problem is the training AWS give its staff. The material they are taught is relentlessly positive and I suspect AWS staff actually have no idea what Redshift is no good for. (Indeed, if you read the official docs for RS, which I strongly advise you never to do, you will come out the other end under the impression there is literally nothing Redshift cannot do; the docs describe everything using positive terms only.)
MaxGanzII 5y ago
Blog post with initial thoughts about internal design of serverless; https://amazonredshiftresearchproject.org/slblog/index.html https://amazonredshiftresearchproject.org/slblog/index.html

64 points by Trisell 5y ago

So basically BigQuery from AWS. Looks good on first sight, a bit late. Personally worked for a large org which has just moved from BigQuery from Redshift and I have to say that BigQuery is the much better product.

glogla 5y ago
Yeah I get that they want to go after Snowflake, Databricks and BigQuery, but AWS it not known for delivering high quality software. They have few things that are very good (EC2, S3, Lambda) and the rest is done by their B-team and barely holding together.
- fdgsdfogijq 5y ago
  Keep telling yourself that
  - MaxGanzII 5y ago
    I have to say, speaking for Redshift, I significantly agree with the critique. There are too many simple problems which should have been caught in testing, and the problem which have over time been found absolutely imply unprofessional, even amateurish software development standards. For example, recently, the format of the version string was changed. This broke a lot of existing software, which had hard coded parsing - SQL Alchemy stopped working - so did AWS's own JDBC driver. This on the face of it indicates the RS test suite does not include any connections over JDBC. It then turned out the version string had anyway been inaccurate for months, because RS had moved from GCC 3.4.2 (I think it was) to 7.3. But the version string kept reporting the old numbers. I can't even begin to describe how many issues - flat factual errors, and profoundly meaningful gobble-de-gook - in the official docs. The whole thing just feels too much like amateur hour.
pram 5y ago
Athena is BigQuery from AWS. I'd have to imagine this is for people already heavily invested in Redshift.
- tedivm 5y ago
  Athena is not BigQuery. Athena is just a fancy wrapper around Hive (you can see this right from the log output) and just runs map reduce over your S3 data. It's a great tool for what it is. BigQuery is a full database. It is significantly faster than running anything from Athena. The closest comparison on AWS is Redshift.
  - ideonode 5y ago
    Athena is more of a wrapper over Presto / Trino than Hive. It does not run map reduce per se over your data - its a SQL engine.
    - pdelgallego 5y ago
      AWS Glue Data catalog is Hive, and as you mentioned, Athena is a wrapper of Presto
  - mdasen 5y ago
    I think Athena has a lot of value (especially given its pricing), but you're not wrong that it has limitations. Getting data into Athena isn't something that is just done for you. Athena just takes what you've put on S3 and queries over it - and leaves getting it onto S3 (and into an efficient format) as an exercise for the reader. Athena's speed varies a lot depending on what format you put things in. Querying over CSVs will mean that you're slow and reading a lot of data. Querying over ORC (column-store) files is pretty quick. The big thing is Athena's pricing. They price it on how much data you read in reality not how much data would be read if things weren't optimized. BigQuery charges you based on how much data would be read if it weren't optimized. With BigQuery, an integer is always 8 bytes. It doesn't matter if they're able to optimize it down to nothing using RLE (run length encoding). You still pay the full 8 bytes. If your ORC files make that integer column tiny, you get the benefit of that. BigQuery is great, but Athena's pricing is a lot cheaper given that you get to benefit from any storage optimization you do. Out of curiosity, how have you used Athena that you're seeing it be so much slower? In my experience, BigQuery is faster (maybe 2x faster), but I've been using column-oriented data with Athena. If you're using CSVs with Athena, it will be way slower than BigQuery. I'm always a little surprised that AWS doesn't build Athena out more, but I guess if they did they'd want money and margin for the value add. Still, Athena is a pretty decent serverless Presto and Presto can work pretty well over data in column formats.
tedivm 5y ago
The only problem with BigQuery is that it's on GCP, so either you have to migrate your whole workload over or you end up eating a lot of data transfer costs. Having used both I do think BigQuery is better in a lot of ways (although it's easier to make it a lot expensive too), but I'm really excited to see Redshift catch up. Adding the serverless options are really great too since my biggest complaint with Redshift was managing the quantity and type of the underlying instances.
- Scorpiion 5y ago
  I have not used it, but my understanding is that BigQuery Omni is "BigQuery running on AWS/Azure". My understanding is that they are running Anthos on AWS (managed by Google) and they offer BigQuery as a service from that AWS infra managed by Google. See more details here: - https://cloud.google.com/bigquery-omni/docs/introduction https://cloud.google.com/bigquery-omni/docs/introduction - https://cloud.google.com/bigquery-omni/docs/aws https://cloud.google.com/bigquery-omni/docs/aws
spullara 5y ago
It isn't. It is more similar to Snowflake's warehouse model.

orf 5y ago

We’ve just moved to Snowflake. I haven’t really been impressed with some of the new features added to Redshift, it seems like too little and too late. The JSON support (SUPER type) is kind of cool, and they are moving towards more “automatic” sorting + partitioning, but it’s just all a bit shit to be honest. We encountered major bugs with data-sharing, our clusters keep insisting that zstd is the best compression format to use for all our data (but then never actually using it), materialised views often fail to update and understanding why is a nightmare, terrible performance if your strings are varchar(max) (guess what Glue sets them to…), Redshift data often just dies (4 hour downtime recently, no status page) and has some really weird semantics around listing queries, before the data API you couldn’t run async queries and it’s eventbridge integration straight up doesn’t work, nightmare bugs in the Java connection library that don’t show up using psql, tiny set of types (no arrays, uuids), unkillable queries, AQUA actually causing everything to slow down hugely, critical release notes posted only in a fucking random forum, etc etc. Snowflake has apparently sorted this, as well as including ingestion tools (snowpipe) that you’d otherwise have to stitch together with AWS Glue or something (a cursed service if ever there was one). That being said, in some cases Redshift absolutely flies. But the real world isn’t filled with ideal schemas and natural sort keys. It’s messy. And Snowflake deals with messy better.

dominotw 5y ago
given snowflake also uses s3. Could aws do something with redshift that moves computation closer to s3 and be even faster than snowflake. Basically take advantage of the fact that they own s3.
- dalyons 5y ago
  they already have https://aws.amazon.com/redshift/features/aqua/ https://aws.amazon.com/redshift/features/aqua/ basically distributing compute down to the actual storage nodes
  - orf 5y ago
    Yeah that’s cool except it made all our queries slower. I’ll add that to my list above, forgot about that.

ggregoire 5y ago

It's not very clear why I should keep using a normal Redshift instead of switching to Serverless? Anyone has more insights?

bayan1234 5y ago
If you have a very stable SQL pipeline that runs frequently during the day then it's probably cheaper to have it on a normal Redshift cluster.
MaxGanzII 5y ago
The minimum cost for serverless seems quite high - 14.4 USD per hour. The advantage is the flexibility to easily change compute resource. A disadvantage is that your data is now in S3 or something very like it, and this I think alters the characteristics of write performance, for the cluster; I've not yet looked into this, but it's on the list. You absolutely should beware of falling into the trap of imagining that serverless simply gives you flexible compute and that's the only change to behaviour. AWS in their press releases and docs are relentlessly positive - anything which is not a strength is obfuscated - so only actual experimentation and investigation throws light on what you're really getting.

bryan0 5y ago

How is this different than Athena?

the_af 5y ago
Athena is not a relational database under the hood. Very different technologies with different implications.
glogla 5y ago
Two competing departments inside AWS. Technically, Athena is based on modified Presto while Redshift is (very) heavily modified Postgres.
- the_af 5y ago
  Competing departments and competing underlying technologies. Postgres (and therefore Redshift) is running a relational database engine under the hood.
  - glogla 5y ago
    Yes, though with ra3 storage being on S3, I think they are getting more and more similar.
tyingq 5y ago
Query things other than data in S3 seems to be the big difference.
- bdcravens 5y ago
  Athena can query non-s3 data sources using federated query.
  - the_af 5y ago
    But that's not to say it's the best idea. Athena is mostly for S3; AWS say so. From their FAQ [1]: Q: What is the difference between Amazon Athena, Amazon EMR, and Amazon Redshift? [...] Amazon Redshift provides the fastest query performance for enterprise reporting and business intelligence workloads, particularly those involving extremely complex SQL with multiple joins and sub-queries. [...] Amazon Athena provides the easiest way to run ad-hoc queries for data in S3 without the need to setup or manage any servers. Elsewhere they also recommend Athena for unstructured data. This and a relational database like Postgres are fundamentally different and you shouldn't use one when your use case is primarily meant for the other.

ndm000 5y ago

I have worked with several companies that have their infrastucture on AWS but consider BigQuery or Snowflake for the serverless model they provide. This brings RedShift much closer to those options. I envision Redshift Serverless becoming the default options for most enterprises, mainly because it stays in the AWS ecosystem and you don't have to work with a different vendor and create different cost governance processes. I beleive the real advantage AWS has here is in cost. Snowflake has positioned itself as price competitive with Redshift but this is primarily due to Snowflake's ability to scale on-demand, whereas prior Redshift versions required you to size for peak usage (RA3 helped with this). In my experience Snowflake is an order of magnitude more expensive if you compare similiar workloads and do not account for idle time. We will need to see the performance of a "Redshift Processing Unit" to be sure of the advantage, but even so AWS will be able provide significant downward cost pressure through this offering.

ignoramous 5y ago
> In my experience Snowflake is an order of magnitude more expensive if you compare similiar workloads and do not account for idle time. Cost reasons is why I'm most bullish about DataBricks's FOSS https://delta.io https://delta.io
- gizmodo59 5y ago
  Delta is not the way. I’d prefer an Apache project like Iceberg https://iceberg.apache.org/ https://iceberg.apache.org/ rather than delta lake.

dreyfan 5y ago

> Just load your data Load my data where? This is "serverless".

the_af 5y ago
To "the cloud" :P More seriously, "serverless" usually just means you aren't supposed to worry about server/cluster management, not that there are no servers anywhere. So it really means "load your data to Redshift, wherever that lives".
- dreyfan 5y ago
  I look forward to the next stage of cloud hype - dataless - so I don't have to worry about the nuances of data management and I can just run my analytics.
  - antod 5y ago
    I've long dreamed of that. Everything becomes stateless and everything to do with GDPR/PCI/Disaster recovery/anonymisation/performance/scaling/etc/etc becomes so much easier.

spullara 5y ago

I wish Amazon would stop naming things "serverless" that clearly have a well defined type and number of servers at any point. That includes Redshift Serverless and Aurora Serverless. If it has a cluster, it isn't serverless it is just autoscaling. Every time they announce a serverless product I'm assuming that it will be like Lambda and am mostly disappointed. For example, a real Aurora Serverless would be more like CockroachDB Cloud or DynamodDB. And a real Redshift Serverless would be more like BigQuery.

MaxGanzII 5y ago
AWS hype everything. What they emit is the dance routine of a sugar coated cheerleader squad. I am of the view everything which is not a strength is obfuscated. I have zero faith, confidence and trust is all information AWS emits. I approach press releases and the docs on the basis that they cover up the actual implementation, and so my task is to find out what is actually going on under the hood, so I can actually make sense of what's been provided and operate it correctly (or avoid it completely, as it may be!)

bayan1234 5y ago

I was initially excited about this as I think it might solve our Redshift pain points and potentially avoid us having to deal with a migration to Snowflake but then I remembered when AWS account managers promised Athena and Spectrum would solve these same problems at a previous company I worked for a few years ago. I'm assuming the developer experience will still be terrible with lots of knobs to tune to actually get any decent cost/performance.

MaxGanzII 5y ago
I hold a very dim view of all information AWS emits, by whatever channels. My experience with support/account managers is that they always tell you "yes, Redshift can do this", and the and the only way to actually get a "no" out of them is to already know Redshift cannot do something, and to explain to them why. They won't deny reality, but you would never have got that answer from them in any other way. I suspect the problem is the training AWS give its staff. The material they are taught is relentlessly positive and I suspect AWS staff actually have no idea what Redshift is no good for. (Indeed, if you read the official docs for RS, which I strongly advise you never to do, you will come out the other end under the impression there is literally nothing Redshift cannot do; the docs describe everything using positive terms only.)

MaxGanzII 5y ago

Blog post with initial thoughts about internal design of serverless; https://amazonredshiftresearchproject.org/slblog/index.html https://amazonredshiftresearchproject.org/slblog/index.html