16 ms·

Amazon Aurora DSQL

187 points by aws_hero 2y ago

redwood 2y ago
In my experience, multi-region databases are used for HA/DR more so than for "writing locally from multiple places". What this means is that the multi-region "consensus cost" is a far easier problem. It looks like DSQL is aiming to change that paradigm: the question is, will this actually mean multiple-region writes become democratized finally? or will this remain more of a database industry marketing talking point that essentially no one uses?
benterix 2y ago
> Amazon Aurora DSQL is a serverless distributed SQL database with virtually unlimited scale, the highest availability, and zero infrastructure management. I will ignore their claimed almost unlimited scalability (which I somewhat believe as it's in their best interest to make it expensive fast), and "highest" (as compared to what?) availability. But my pet peeve is zero infrastructure management. This is blatantly false and it always was - when they first announced their cloud (at that time called "web") services, when they marketed their solutions as "serverless", and now. You need to have well-trained staff to maintain your AWS infrastructure otherwise you will come across many problems.
- gjsman-1000 2y ago
  I think this is way, way too pessimistic and unfair. When they say zero, it's like a food item that says zero calories. It might have up to five calories while still saying zero, but that's very different than three hundred. Managing a SQL Database at scale is more like four thousand calories daily, if I continue stretching this analogy. :)
  - benterix 2y ago
    I don't disagree about the service itself having relatively few calories (I'd argue it's far more then 5, but that's a minor point). The reality is that you need a ton of other things to have it running properly - from a properly created and maintained AWS account, to proper IAM roles with correct permissions, to correct deployment/modification setup and other resources as a database rarely exists in a vacuum. And because Amazon decided not to implement spending cap, if you don't have the right competences, you are at risk losing money - and this is especially true for serverless services. I still remember the folks who set the same bucket for lambda notifications look with panic as their bill increases dramatically as they watch it growing, not knowing how to stop the whole process.[0] [0] They disabled the trap a few weeks ago, finally: https://aws.amazon.com/about-aws/whats-new/2024/10/aws-lambda-detects-stops-recursive-loops-lambda-s3/ https://aws.amazon.com/about-aws/whats-new/2024/10/aws-lambd...
- ndriscoll 2y ago
  > I will ignore their claimed almost unlimited scalability (which I somewhat believe as it's in their best interest to make it expensive fast) I would believe it can scale your cost easily, but I'd be surprised if it can scale well in terms of capability. How many million QPS is "virtually unlimited scale"? The 10k row transaction limit and no FKs (or even views???) also just make this thing sound completely non-serious.
politelemon 2y ago
I am confused, what is the difference between Aurora DSQL and Aurora Serverless? https://aws.amazon.com/rds/aurora/serverless/ https://aws.amazon.com/rds/aurora/serverless/
- nextworddev 2y ago
  Aurora might be tied to region
  - asah 2y ago
    Aurora is much more compatible: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/working-with-postgresql-compatibility-unsupported-features.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
- everfrustrated 2y ago
  Aurora is the postgres binary running with patches to write to their storage backend. This appears wire-compatible with postgres but does things differently like locking. Depends a lot on how you use postgres as to whether its drop-in compatible or not.
octernion 2y ago
love that i can spin this up and spend money before any of the documentation is available. neat-o
- deleted 2y ago
  [deleted]
- aws_hero 2y ago
  it's free during preview
kbumsik 2y ago
So does it scale to zero, and scale from zero with <1s latency like Neon DB? AWS' recent "serverless" products, Aurora Serverless v2 and MSK Serverless for example, had been very misleading with "serverless" title. I hope it is not a lie this time.
- rhodysurf 2y ago
  its supposedly serverless like dyanamo
- everfrustrated 2y ago
  There's nothing in the cluster api to control scale or set instance type so this does look actually serverless for a nice change!
- evanelias 2y ago
  This was announced only a couple weeks ago, but Aurora Serverless v2 can scale to zero now. https://aws.amazon.com/blogs/database/introducing-scaling-to-0-capacity-with-amazon-aurora-serverless-v2/ https://aws.amazon.com/blogs/database/introducing-scaling-to... However, the latency scaling from zero is still painful, "up to 15 seconds".
- vvern 2y ago
  If that’s what you want, try cockroachdb serverless. Scales from zero in well under 1s even for multiregion. Also actually has quite good Postgres support these days.
epaulson 2y ago
It's frustrating that there's no pricing information. The tech looks cool and all, but without knowing how much it's going to cost there's no way to really evaluate it.
- deleted 2y ago
  [deleted]
- gs17 2y ago
  It's really surprising that they don't have a page with information ready to go live when the announcement happens. E.g. the Nova AI models they announced look neat, but the Bedrock page doesn't mention them at all and the page for them has links to non-existent documentation.
estebarb 2y ago
Too bad the details how it works aren't available yet: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/getting-started.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/get... Stay Tuned We're sorry. The documentation you are looking for is not yet available. Please check back soon. -The Amazon Web Services Documentation Team
- lechomp 2y ago
  I think like other products, there will be a tech presentation and there they will talk in details. So hopefully later this week?
- taikahessu 2y ago
  You know true Cutting-edge Technology when you see ... there's no documentation available.
- gs17 2y ago
  Some of the docs are up: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/what-is-aurora-dsql.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wha... EDIT: Your link is up now too.
slackerIII 2y ago
The only announcement I want from AWS about databases is that the price for RDS is going down. Not that they have new chip that is more expensive but offers better price/performance, but that my bill is actually going to drop. I don't trust them enough to use non-portable technology like this until they give me confidence they are committed to lowering prices.
- vosper 2y ago
  > The only announcement I want from AWS about databases is that the price for RDS is going down. If the price hasn't been going up in line with inflation then it's been going down in real terms. Not as nice as the number going down, too, but it's not nothing. Over time AWS prices have significantly declined in real terms simply by not having been raised in nominal value.
  - panarky 2y ago
    The inflation rate for chips, memory, storage and networking is negative.
- steepben 2y ago
  It's Postgres compatible, so not exactly non-portable
  - asah 2y ago
    Cough, not exactly: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/working-with-postgresql-compatibility-unsupported-features.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
    - qvrjuec 2y ago
      Why would this product having a subset of postgresql features make it less portable? If anything, that makes it more portable
taldo 2y ago
Sooo they're finally launching a Spanner contender? I'm itching to read more details into what this actually is under the marketing blab.
- steepben 2y ago
  There's a blog on the technical details here: https://aws.amazon.com/blogs/database/introducing-amazon-aurora-dsql/ https://aws.amazon.com/blogs/database/introducing-amazon-aur...
  - hobs 2y ago
    The most interesting page is always quotas and limits: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/CHAP_quotas.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/CHA... Seeing the No to configurable for some of the settings is the most telling hard limits that we can see up front. Some Very Noteable ones are: Maximum size of all data modified within a write transaction: 10 MiB Maximum transaction time: 5 minutes
    - bobnamob 2y ago
      The 10MiB transaction size limit smells a lot like FoundationDBs transaction size limit
      - bcap 2y ago
        10MiB is an appealing number for humans. Can easily appear across different designs.
- MapleWalnut 2y ago
  Spanner provides serializability. It seems like DSQL only guarantees "repeatable read". https://docs.aws.amazon.com/aurora-dsql/latest/userguide/working-with-postgresql-compatibility-unsupported-features.html#working-with-postgresql-compatibility-unsupported-limitations https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
- 2y ago
chikinpotpi 2y ago
So... its like cockroach db but with the added feature of more vendor lock in?
- Andys 2y ago
  Like cockroachdb but with REPEATABLE READ instead of SERIALIZABLE as the default and only option.
sidcool 2y ago
It doesn't talk too much about data consistency.
- steepben 2y ago
  There's a blog with more technical details: https://aws.amazon.com/blogs/database/introducing-amazon-aurora-dsql/ https://aws.amazon.com/blogs/database/introducing-amazon-aur... It provides strong consistency for cross-region transactions
bradhe 2y ago
DSQL = DHTML for data?
PowerfulWizard 2y ago
Very interesting. My dream is to have something like this, a KV-store, a blob store, and pubsub all behind the same interface.
- mdaniel 2y ago
  "like this" meaning you want wire PG compatibility, so you'd do something like this? UPDATE kv SET value = 'alpha' WHERE key = 'beta.charlie'; UPDATE s3 SET value = $b64$good luck$b64$ WHERE key = '/some/s3/path'; LISTEN whatever;
  - PowerfulWizard 2y ago
    I don't know what I want exactly but I'm thinking along the lines that SQL is already doing a lot so it would make the most sense to start with a database interface and augment from there, to try to build a system to handle all the common forms of durable storage used by applications. The type of situation I'm thinking about is for example storing a blob in S3, storing metadata and a reference to the blob's path in a database row, sending a message into a queue to trigger some async processing, and updating a cache. It would be nice to be able to do this through a single API or service, and it would be really nice to do all this within some type of transaction abstraction that would allow all operations to pass or fail collectively, really really nice if the whole thing could be pay-as-you and scale horizontally-ish on shared infrastructure without managing nodes or slots or whatever. I'm not a Postgres user so I don't know how far you can get currently and I should probably look into it in detail. Coordinating blob/ject storage, database, and pubsub operations is a pain point for me presently. I think that overall system design is going to prevent a database-type system from being a good idea for blob storage but I would still like to see someone try to put three systems in a trenchcoat and try to make it work behind one interface.
    - mdaniel 2y ago
      I was just trying to get a sense for where the line of demarcation was in your mind. PG has "foreign data wrappers"[1] that allow one to treat external ... whatever ... as if it were a table or procedure within PG. Just stupid powerful, IMHO. It is FDW-specific whether "transaction" means anything to the foreign system, so that may break your mental model but could still get you very close (e.g. BEGIN; UPDATE s3 SET ...; ROLLBACK; may not do anything sensible) https://github.com/turbot/steampipe#steampipe-plugins https://github.com/turbot/steampipe#steampipe-plugins and https://steampipe.io/docs/steampipe_postgres/overview https://steampipe.io/docs/steampipe_postgres/overview may be relevant, although watch out for Steampipe's license https://github.com/topics/foreign-data-wrapper https://github.com/topics/foreign-data-wrapper and https://github.com/topics/fdw https://github.com/topics/fdw are some other examples 1: https://www.postgresql.org/docs/17/fdwhandler.html https://www.postgresql.org/docs/17/fdwhandler.html (although strictly speaking that page is for _authoring_ FDW, not a tl;dr of the concept)
e1g 2y ago
Just tried a quick test cluster - Identifies as PG 16.5 No views/triggers/sequences No foreign key constraints No extensions No NOTIFY ("ERROR: Function pg_notify not supported") No nested transactions No json(b) Unsupported PG features are now online https://docs.aws.amazon.com/aurora-dsql/latest/userguide/working-with-postgresql-compatibility-unsupported-features.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
- acedTrex 2y ago
  I feel like no FKs and no jsonb is really burying the lede there lol
- shayonj 2y ago
  oof! that's rough. I was also not particularly impressed with "limitless". Esp. because you can't have partitions on the postgres limitless offering.
- craigkerstiens 2y ago
  It feels very disingenuous to say "Postgres compatible" and have this as a missing feature set. I'm sure they'd quickly argue it's wire compatibility, but even then it's a slippery slope and wire compatible is left open to however the person wants to interpret it. There is no 'standard' or 'spec' for what makes something Postgres wire compatible. This feels like a strong overreach on the marketing front to leverage the love people have for Postgres to help boost what they've built. That is not to say there isn't hard and quality engineering in here, but slapping Postgres compatible on it feels lazy at best.
  - benesch 2y ago
    > I'm sure they'd quickly argue it's wire compatibility, but even then it's a slippery slope and wire compatible is left open to however the person wants to interpret it. I actually think that they'd argue they intend to close the feature gap for full Postgres semantics over time. Indeed their marketing was a bit wishful, but on Bluesky, Marc Brooker (one of the developers on the project) said they reused the parser, planner, and optimizer from Postgres: https://bsky.app/profile/marcbrooker.bsky.social/post/3lcghjbvyx22n https://bsky.app/profile/marcbrooker.bsky.social/post/3lcghj... That means they actually have a very good shot at approaching reasonably full Postgres compatibility (at a SQL semantics level, not just at the wire protocol level) over time.
everfrustrated 2y ago
Postgres compatibility depends a bit on what you rely on. This doesn't appear to be postgres under the hood, more wire-compatible. >Aurora DSQL is PostgreSQL compatible, which means that it provides identical behavior for most supported features, identical query results for all SQL features, and supports many popular PostgreSQL drivers and tools with minor configuration changes. Supported SQL expressions return identical data in query results, including sort order, scale and precision for numeric operations, and equivalence for string operations. With a few documented exceptions, such as synchronous replication, no-lock concurrency control, and asynchronous DDL execution, Aurora DSQL behaves comparably to PostgreSQL. Aurora DSQL supports core relational features like ACID transactions, secondary indexes, joins, insert, and updates. See Supported SQL expressions for an overview of supported SQL features. Aurora DSQL doesn't support all PostgreSQL features. For more information, see Unsupported PostgreSQL features. https://docs.aws.amazon.com/aurora-dsql/latest/userguide/working-with-postgresql-compatibility-unsupported-features.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor... No pricing that I can find
- bragrights 2y ago
  https://aws.amazon.com/rds/aurora/dsql/faqs/ https://aws.amazon.com/rds/aurora/dsql/faqs/ - it is free for preview
Edwinr95 2y ago
I find it to be super limited, and I'm sort of struggling to see the point given all these constraints. No temporary tables, no foreign keys, no views, no more than 10k rows in a transaction. Except for some basic wire compatibility with the postgres protocol, I'd hardly call this a "database", and more a key-value store. https://docs.aws.amazon.com/aurora-dsql/latest/userguide/working-with-postgresql-compatibility.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
- gjsman-1000 2y ago
  > Except for some basic wire compatibility with the postgres protocol, I'd hardly call this a "database", and more a key-value store. Hopefully that keeps the pricing reasonable. :) But seriously, for a smaller CRUD app, this could be sufficient, even "magical," if the price is right. For my part though, the lack of multiple databases per cluster puts multi-tenant systems completely off the table. Now that you mention it, I almost wonder if this is a giant hack on top of Valkey/Redis...
  - everfrustrated 2y ago
    >the lack of multiple databases per cluster I don't think that will matter. This feels much more like DynamoDB where you're charged for GB used/stored and no infra cost, so no reason to nest databases
    - Edwinr95 2y ago
      Isolation can be quite important. I work with a micro services architecture, where we assign a database per service. That makes it very easy to manage access to sensitive data.
      - adobrawy 2y ago
        You do not need logical database isolation if you have even stronger isolation with virtual clusters. The pricing and quota will be important factor for that use case.
Andys 2y ago
The fact that it uses "Firecracker microVMs" and a few other things makes me think this is an S3-backed Lambda-based database that emulates Postgres on the wire. Especially when you consider the recent feature additions to S3.
- deleted 2y ago
  [deleted]
joshstrange 2y ago
I won't even consider this without pricing, I can't imagine how anyone could. Serverless Aurora is stupid-expensive so I can't imagine this is going to be cheap but I keep hunting for that perfect DB that can scale to 0 (or very low) and be priced well. Right now neon.tech is what I'm using and I'm very happy but this looks like it would be interesting IF the pricing is good.
- dbbk 2y ago
  I'm also on Neon. Seems like the primary benefit of this would be multi-region support, which Neon doesn't have. But I do wonder if multi-AZ is actually just enough, has there ever really been a time when all AZs have gone down?
  - dalyons 2y ago
    yes, every couple of years there's a region-wide outage. On us-east anyway.
    - dbbk 2y ago
      Fortunately I'm in Europe ha
  - clarkbw 2y ago
    (neon employee) we have multi-region disaster recovery and replicas coming in 2025. Switch over time will be greater than this active-active system but the overall latency for Neon should be much less on a consistent basis.
    - dbbk 2y ago
      Now we're fucking talking.
bobnamob 2y ago
Oof, press release proof readers must have been hit by layoffs > With its innovative active-active distributed architecture, Aurora DSQL is designed for 99.99% availability in single-Region configuration and 99.999% in multi-Region configuration, with an innovative active-active, distributed architecture, making it ideal for building highly-available applications We get that its architecture is innovative and active-active
- wutwutwat 2y ago
  Reeks of AI generation
zokier 2y ago
Can Amazon please get RDS product management ducks in a row? You got RDS For PostgreSQL, Aurora PostgreSQL (provisioned), Aurora Serverless v1, Aurora Serverless v2, Aurora PostgreSQL Limitless Database, Aurora Global Database, and now Aurora DSQL (with PostgreSQL compatibility). It's very confusing, the docs are confusing, and the even the tools are confused. I think this is pretty big contrast to something like S3 where the product feels far more clearly and coherently managed, and where they have released lots of big improvements/changes over the years without disrupting or muddying the core product in major ways.
- bradhe 2y ago
  RDS generally is a massive, massive business inside AWS. I'd imagine the politics and organization generally is an absolute nightmare.
- korkybuchek 2y ago
  > It's very confusing, the docs are confusing, and the even the tools are confused. Not to worry -- you can use Rufus...uh "Q"....uhh...Nova...to summarize the docs into a decision tree on which confusing choice to use!
  - OJFord 2y ago
    Except you can't, because it will just misundertand you or say something hopelessly generic. Even asking it something on the same page that it's advertised 'now supports asking about' completely failed for me - think it was ENIs or something; just said something completely generic, not at all about the instances I'd asked about which it advertised being able to do (and I happened to have a concurrent need for).
- dakiol 2y ago
  > It's very confusing As a very naive person myself, isn’t “being very confusing” one of the main points of AWS? They know they are big and nobody will get fired for using AWS. So, the more confusing their products are, the more benefits for them (e.g., because you don’t know how much you will pay, because you are misusing their products, because perhaps you don’t actually need their products, etc) So, any manager at AWS probably thinks this is all working very fine.
bobnamob 2y ago
Marc Brooker's blog has a ~bunch~ bit more technical information https://brooker.co.za/blog/2024/12/03/aurora-dsql https://brooker.co.za/blog/2024/12/03/aurora-dsql Specifically, transaction latency is constant relative to transaction statement count, even across region, so I guess that's something
DonnyV 2y ago
[flagged]
- zokier 2y ago
  How many multi-region active-active PostgreSQL clusters have you operated?
  - sgarland 2y ago
    One, and it was a nightmare. Do not recommend.
- dalyons 2y ago
  did you miss the word "distributed"?
qaq 2y ago
Virtually unlimited scale :) Maximum storage GB per cluster: 100GB Maximum size of all data modified within a write transaction 10 MiB Max: 10K rows per transaction
- dbbk 2y ago
  100GB doesn't seem very unlimited
  - bobnamob 2y ago
    While I love to take potshots, the yes(configurable) next to that 100gb does some heavy lifting. I imagine that number is tunable up to multi digit tb if you your monthly bill is worth Jassy's attention https://docs.aws.amazon.com/aurora-dsql/latest/userguide/CHAP_quotas.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/CHA... The more interesting (i.e. constraining) limits are transaction size (both row count and data size) and transaction time, which aren't configurable.
whoevercares 2y ago
Curious why Murat left the project before it’s launched
jwnin 2y ago
Seems like a competitor to Azure's CosmosDB offering , which also now has wire support for postgres.
sorenbs 2y ago
Very interesting announcement! We are also building a hosted Postgres service based on Firecracker and with the ability to create unlimited databases. Will have to dig in and see how we compare to this. https://www.prisma.io/blog/announcing-prisma-postgres-early-access https://www.prisma.io/blog/announcing-prisma-postgres-early-...
sj26 2y ago
If anybody likes to play with Ruby on Rails, I've published a bare bones active record connection adapter: https://github.com/sj26/activerecord-dsql-adapter https://github.com/sj26/activerecord-dsql-adapter
- sj26 2y ago
  Interesting discovery: PG::FeatureNotSupported: ERROR: ddl and dml are not supported in the same transaction
znpy 2y ago
I see many comments about “PostgreSQL compatibility” not being actual compatibility… maybe the PostgreSQL people should compile a list of requirements for a database to call itself “PostgreSQL-compatible”? It feels like the compatibility claim is a bit abused (not only in this case) mostly because there’s no clear and strict reference (maybe with mandatory and optional requirements) for what “compatible” means
rsanheim 2y ago
[flagged]
- deleted 2y ago
  [deleted]
c4pt0r 2y ago
- Yes, the storage is distributed kv. - Yes, it's SQL on distributed transactional kv - SQL layer is stateless, runs in multiple containers - SQL layer would push down compute logic down to the storage(like, filter/predicate) ....Yes, that reminds me of TiDB

187 points by aws_hero 2y ago

In my experience, multi-region databases are used for HA/DR more so than for "writing locally from multiple places". What this means is that the multi-region "consensus cost" is a far easier problem. It looks like DSQL is aiming to change that paradigm: the question is, will this actually mean multiple-region writes become democratized finally? or will this remain more of a database industry marketing talking point that essentially no one uses?

benterix 2y ago

> Amazon Aurora DSQL is a serverless distributed SQL database with virtually unlimited scale, the highest availability, and zero infrastructure management. I will ignore their claimed almost unlimited scalability (which I somewhat believe as it's in their best interest to make it expensive fast), and "highest" (as compared to what?) availability. But my pet peeve is zero infrastructure management. This is blatantly false and it always was - when they first announced their cloud (at that time called "web") services, when they marketed their solutions as "serverless", and now. You need to have well-trained staff to maintain your AWS infrastructure otherwise you will come across many problems.

gjsman-1000 2y ago
I think this is way, way too pessimistic and unfair. When they say zero, it's like a food item that says zero calories. It might have up to five calories while still saying zero, but that's very different than three hundred. Managing a SQL Database at scale is more like four thousand calories daily, if I continue stretching this analogy. :)
- benterix 2y ago
  I don't disagree about the service itself having relatively few calories (I'd argue it's far more then 5, but that's a minor point). The reality is that you need a ton of other things to have it running properly - from a properly created and maintained AWS account, to proper IAM roles with correct permissions, to correct deployment/modification setup and other resources as a database rarely exists in a vacuum. And because Amazon decided not to implement spending cap, if you don't have the right competences, you are at risk losing money - and this is especially true for serverless services. I still remember the folks who set the same bucket for lambda notifications look with panic as their bill increases dramatically as they watch it growing, not knowing how to stop the whole process.[0] [0] They disabled the trap a few weeks ago, finally: https://aws.amazon.com/about-aws/whats-new/2024/10/aws-lambda-detects-stops-recursive-loops-lambda-s3/ https://aws.amazon.com/about-aws/whats-new/2024/10/aws-lambd...
ndriscoll 2y ago
> I will ignore their claimed almost unlimited scalability (which I somewhat believe as it's in their best interest to make it expensive fast) I would believe it can scale your cost easily, but I'd be surprised if it can scale well in terms of capability. How many million QPS is "virtually unlimited scale"? The 10k row transaction limit and no FKs (or even views???) also just make this thing sound completely non-serious.

politelemon 2y ago

I am confused, what is the difference between Aurora DSQL and Aurora Serverless? https://aws.amazon.com/rds/aurora/serverless/ https://aws.amazon.com/rds/aurora/serverless/

nextworddev 2y ago
Aurora might be tied to region
- asah 2y ago
  Aurora is much more compatible: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/working-with-postgresql-compatibility-unsupported-features.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
everfrustrated 2y ago
Aurora is the postgres binary running with patches to write to their storage backend. This appears wire-compatible with postgres but does things differently like locking. Depends a lot on how you use postgres as to whether its drop-in compatible or not.

octernion 2y ago

love that i can spin this up and spend money before any of the documentation is available. neat-o

deleted 2y ago
[deleted]
aws_hero 2y ago
it's free during preview

kbumsik 2y ago

So does it scale to zero, and scale from zero with <1s latency like Neon DB? AWS' recent "serverless" products, Aurora Serverless v2 and MSK Serverless for example, had been very misleading with "serverless" title. I hope it is not a lie this time.

rhodysurf 2y ago
its supposedly serverless like dyanamo
everfrustrated 2y ago
There's nothing in the cluster api to control scale or set instance type so this does look actually serverless for a nice change!
evanelias 2y ago
This was announced only a couple weeks ago, but Aurora Serverless v2 can scale to zero now. https://aws.amazon.com/blogs/database/introducing-scaling-to-0-capacity-with-amazon-aurora-serverless-v2/ https://aws.amazon.com/blogs/database/introducing-scaling-to... However, the latency scaling from zero is still painful, "up to 15 seconds".
vvern 2y ago
If that’s what you want, try cockroachdb serverless. Scales from zero in well under 1s even for multiregion. Also actually has quite good Postgres support these days.

epaulson 2y ago

It's frustrating that there's no pricing information. The tech looks cool and all, but without knowing how much it's going to cost there's no way to really evaluate it.

deleted 2y ago
[deleted]
gs17 2y ago
It's really surprising that they don't have a page with information ready to go live when the announcement happens. E.g. the Nova AI models they announced look neat, but the Bedrock page doesn't mention them at all and the page for them has links to non-existent documentation.

estebarb 2y ago

Too bad the details how it works aren't available yet: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/getting-started.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/get... Stay Tuned We're sorry. The documentation you are looking for is not yet available. Please check back soon. -The Amazon Web Services Documentation Team

lechomp 2y ago
I think like other products, there will be a tech presentation and there they will talk in details. So hopefully later this week?
taikahessu 2y ago
You know true Cutting-edge Technology when you see ... there's no documentation available.
gs17 2y ago
Some of the docs are up: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/what-is-aurora-dsql.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wha... EDIT: Your link is up now too.

slackerIII 2y ago

The only announcement I want from AWS about databases is that the price for RDS is going down. Not that they have new chip that is more expensive but offers better price/performance, but that my bill is actually going to drop. I don't trust them enough to use non-portable technology like this until they give me confidence they are committed to lowering prices.

vosper 2y ago
> The only announcement I want from AWS about databases is that the price for RDS is going down. If the price hasn't been going up in line with inflation then it's been going down in real terms. Not as nice as the number going down, too, but it's not nothing. Over time AWS prices have significantly declined in real terms simply by not having been raised in nominal value.
- panarky 2y ago
  The inflation rate for chips, memory, storage and networking is negative.
steepben 2y ago
It's Postgres compatible, so not exactly non-portable
- asah 2y ago
  Cough, not exactly: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/working-with-postgresql-compatibility-unsupported-features.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
  - qvrjuec 2y ago
    Why would this product having a subset of postgresql features make it less portable? If anything, that makes it more portable

taldo 2y ago

Sooo they're finally launching a Spanner contender? I'm itching to read more details into what this actually is under the marketing blab.

steepben 2y ago
There's a blog on the technical details here: https://aws.amazon.com/blogs/database/introducing-amazon-aurora-dsql/ https://aws.amazon.com/blogs/database/introducing-amazon-aur...
- hobs 2y ago
  The most interesting page is always quotas and limits: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/CHAP_quotas.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/CHA... Seeing the No to configurable for some of the settings is the most telling hard limits that we can see up front. Some Very Noteable ones are: Maximum size of all data modified within a write transaction: 10 MiB Maximum transaction time: 5 minutes
  - bobnamob 2y ago
    The 10MiB transaction size limit smells a lot like FoundationDBs transaction size limit
    - bcap 2y ago
      10MiB is an appealing number for humans. Can easily appear across different designs.
MapleWalnut 2y ago
Spanner provides serializability. It seems like DSQL only guarantees "repeatable read". https://docs.aws.amazon.com/aurora-dsql/latest/userguide/working-with-postgresql-compatibility-unsupported-features.html#working-with-postgresql-compatibility-unsupported-limitations https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...
2y ago

chikinpotpi 2y ago

So... its like cockroach db but with the added feature of more vendor lock in?

Andys 2y ago
Like cockroachdb but with REPEATABLE READ instead of SERIALIZABLE as the default and only option.

sidcool 2y ago

It doesn't talk too much about data consistency.

steepben 2y ago
There's a blog with more technical details: https://aws.amazon.com/blogs/database/introducing-amazon-aurora-dsql/ https://aws.amazon.com/blogs/database/introducing-amazon-aur... It provides strong consistency for cross-region transactions

bradhe 2y ago

DSQL = DHTML for data?

PowerfulWizard 2y ago

Very interesting. My dream is to have something like this, a KV-store, a blob store, and pubsub all behind the same interface.

mdaniel 2y ago
"like this" meaning you want wire PG compatibility, so you'd do something like this? UPDATE kv SET value = 'alpha' WHERE key = 'beta.charlie'; UPDATE s3 SET value = $b64$good luck$b64$ WHERE key = '/some/s3/path'; LISTEN whatever;
- PowerfulWizard 2y ago
  I don't know what I want exactly but I'm thinking along the lines that SQL is already doing a lot so it would make the most sense to start with a database interface and augment from there, to try to build a system to handle all the common forms of durable storage used by applications. The type of situation I'm thinking about is for example storing a blob in S3, storing metadata and a reference to the blob's path in a database row, sending a message into a queue to trigger some async processing, and updating a cache. It would be nice to be able to do this through a single API or service, and it would be really nice to do all this within some type of transaction abstraction that would allow all operations to pass or fail collectively, really really nice if the whole thing could be pay-as-you and scale horizontally-ish on shared infrastructure without managing nodes or slots or whatever. I'm not a Postgres user so I don't know how far you can get currently and I should probably look into it in detail. Coordinating blob/ject storage, database, and pubsub operations is a pain point for me presently. I think that overall system design is going to prevent a database-type system from being a good idea for blob storage but I would still like to see someone try to put three systems in a trenchcoat and try to make it work behind one interface.
  - mdaniel 2y ago
    I was just trying to get a sense for where the line of demarcation was in your mind. PG has "foreign data wrappers"[1] that allow one to treat external ... whatever ... as if it were a table or procedure within PG. Just stupid powerful, IMHO. It is FDW-specific whether "transaction" means anything to the foreign system, so that may break your mental model but could still get you very close (e.g. BEGIN; UPDATE s3 SET ...; ROLLBACK; may not do anything sensible) https://github.com/turbot/steampipe#steampipe-plugins https://github.com/turbot/steampipe#steampipe-plugins and https://steampipe.io/docs/steampipe_postgres/overview https://steampipe.io/docs/steampipe_postgres/overview may be relevant, although watch out for Steampipe's license https://github.com/topics/foreign-data-wrapper https://github.com/topics/foreign-data-wrapper and https://github.com/topics/fdw https://github.com/topics/fdw are some other examples 1: https://www.postgresql.org/docs/17/fdwhandler.html https://www.postgresql.org/docs/17/fdwhandler.html (although strictly speaking that page is for _authoring_ FDW, not a tl;dr of the concept)

e1g 2y ago

Just tried a quick test cluster - Identifies as PG 16.5 No views/triggers/sequences No foreign key constraints No extensions No NOTIFY ("ERROR: Function pg_notify not supported") No nested transactions No json(b) Unsupported PG features are now online https://docs.aws.amazon.com/aurora-dsql/latest/userguide/working-with-postgresql-compatibility-unsupported-features.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...

acedTrex 2y ago
I feel like no FKs and no jsonb is really burying the lede there lol
shayonj 2y ago
oof! that's rough. I was also not particularly impressed with "limitless". Esp. because you can't have partitions on the postgres limitless offering.
craigkerstiens 2y ago
It feels very disingenuous to say "Postgres compatible" and have this as a missing feature set. I'm sure they'd quickly argue it's wire compatibility, but even then it's a slippery slope and wire compatible is left open to however the person wants to interpret it. There is no 'standard' or 'spec' for what makes something Postgres wire compatible. This feels like a strong overreach on the marketing front to leverage the love people have for Postgres to help boost what they've built. That is not to say there isn't hard and quality engineering in here, but slapping Postgres compatible on it feels lazy at best.
- benesch 2y ago
  > I'm sure they'd quickly argue it's wire compatibility, but even then it's a slippery slope and wire compatible is left open to however the person wants to interpret it. I actually think that they'd argue they intend to close the feature gap for full Postgres semantics over time. Indeed their marketing was a bit wishful, but on Bluesky, Marc Brooker (one of the developers on the project) said they reused the parser, planner, and optimizer from Postgres: https://bsky.app/profile/marcbrooker.bsky.social/post/3lcghjbvyx22n https://bsky.app/profile/marcbrooker.bsky.social/post/3lcghj... That means they actually have a very good shot at approaching reasonably full Postgres compatibility (at a SQL semantics level, not just at the wire protocol level) over time.

everfrustrated 2y ago

Postgres compatibility depends a bit on what you rely on. This doesn't appear to be postgres under the hood, more wire-compatible. >Aurora DSQL is PostgreSQL compatible, which means that it provides identical behavior for most supported features, identical query results for all SQL features, and supports many popular PostgreSQL drivers and tools with minor configuration changes. Supported SQL expressions return identical data in query results, including sort order, scale and precision for numeric operations, and equivalence for string operations. With a few documented exceptions, such as synchronous replication, no-lock concurrency control, and asynchronous DDL execution, Aurora DSQL behaves comparably to PostgreSQL. Aurora DSQL supports core relational features like ACID transactions, secondary indexes, joins, insert, and updates. See Supported SQL expressions for an overview of supported SQL features. Aurora DSQL doesn't support all PostgreSQL features. For more information, see Unsupported PostgreSQL features. https://docs.aws.amazon.com/aurora-dsql/latest/userguide/working-with-postgresql-compatibility-unsupported-features.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor... No pricing that I can find

bragrights 2y ago
https://aws.amazon.com/rds/aurora/dsql/faqs/ https://aws.amazon.com/rds/aurora/dsql/faqs/ - it is free for preview

Edwinr95 2y ago

I find it to be super limited, and I'm sort of struggling to see the point given all these constraints. No temporary tables, no foreign keys, no views, no more than 10k rows in a transaction. Except for some basic wire compatibility with the postgres protocol, I'd hardly call this a "database", and more a key-value store. https://docs.aws.amazon.com/aurora-dsql/latest/userguide/working-with-postgresql-compatibility.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...

gjsman-1000 2y ago
> Except for some basic wire compatibility with the postgres protocol, I'd hardly call this a "database", and more a key-value store. Hopefully that keeps the pricing reasonable. :) But seriously, for a smaller CRUD app, this could be sufficient, even "magical," if the price is right. For my part though, the lack of multiple databases per cluster puts multi-tenant systems completely off the table. Now that you mention it, I almost wonder if this is a giant hack on top of Valkey/Redis...
- everfrustrated 2y ago
  >the lack of multiple databases per cluster I don't think that will matter. This feels much more like DynamoDB where you're charged for GB used/stored and no infra cost, so no reason to nest databases
  - Edwinr95 2y ago
    Isolation can be quite important. I work with a micro services architecture, where we assign a database per service. That makes it very easy to manage access to sensitive data.
    - adobrawy 2y ago
      You do not need logical database isolation if you have even stronger isolation with virtual clusters. The pricing and quota will be important factor for that use case.

Andys 2y ago

The fact that it uses "Firecracker microVMs" and a few other things makes me think this is an S3-backed Lambda-based database that emulates Postgres on the wire. Especially when you consider the recent feature additions to S3.

deleted 2y ago
[deleted]

joshstrange 2y ago

I won't even consider this without pricing, I can't imagine how anyone could. Serverless Aurora is stupid-expensive so I can't imagine this is going to be cheap but I keep hunting for that perfect DB that can scale to 0 (or very low) and be priced well. Right now neon.tech is what I'm using and I'm very happy but this looks like it would be interesting IF the pricing is good.

dbbk 2y ago
I'm also on Neon. Seems like the primary benefit of this would be multi-region support, which Neon doesn't have. But I do wonder if multi-AZ is actually just enough, has there ever really been a time when all AZs have gone down?
- dalyons 2y ago
  yes, every couple of years there's a region-wide outage. On us-east anyway.
  - dbbk 2y ago
    Fortunately I'm in Europe ha
- clarkbw 2y ago
  (neon employee) we have multi-region disaster recovery and replicas coming in 2025. Switch over time will be greater than this active-active system but the overall latency for Neon should be much less on a consistent basis.
  - dbbk 2y ago
    Now we're fucking talking.

bobnamob 2y ago

Oof, press release proof readers must have been hit by layoffs > With its innovative active-active distributed architecture, Aurora DSQL is designed for 99.99% availability in single-Region configuration and 99.999% in multi-Region configuration, with an innovative active-active, distributed architecture, making it ideal for building highly-available applications We get that its architecture is innovative and active-active

wutwutwat 2y ago
Reeks of AI generation

zokier 2y ago

Can Amazon please get RDS product management ducks in a row? You got RDS For PostgreSQL, Aurora PostgreSQL (provisioned), Aurora Serverless v1, Aurora Serverless v2, Aurora PostgreSQL Limitless Database, Aurora Global Database, and now Aurora DSQL (with PostgreSQL compatibility). It's very confusing, the docs are confusing, and the even the tools are confused. I think this is pretty big contrast to something like S3 where the product feels far more clearly and coherently managed, and where they have released lots of big improvements/changes over the years without disrupting or muddying the core product in major ways.

bradhe 2y ago
RDS generally is a massive, massive business inside AWS. I'd imagine the politics and organization generally is an absolute nightmare.
korkybuchek 2y ago
> It's very confusing, the docs are confusing, and the even the tools are confused. Not to worry -- you can use Rufus...uh "Q"....uhh...Nova...to summarize the docs into a decision tree on which confusing choice to use!
- OJFord 2y ago
  Except you can't, because it will just misundertand you or say something hopelessly generic. Even asking it something on the same page that it's advertised 'now supports asking about' completely failed for me - think it was ENIs or something; just said something completely generic, not at all about the instances I'd asked about which it advertised being able to do (and I happened to have a concurrent need for).
dakiol 2y ago
> It's very confusing As a very naive person myself, isn’t “being very confusing” one of the main points of AWS? They know they are big and nobody will get fired for using AWS. So, the more confusing their products are, the more benefits for them (e.g., because you don’t know how much you will pay, because you are misusing their products, because perhaps you don’t actually need their products, etc) So, any manager at AWS probably thinks this is all working very fine.

bobnamob 2y ago

Marc Brooker's blog has a ~bunch~ bit more technical information https://brooker.co.za/blog/2024/12/03/aurora-dsql https://brooker.co.za/blog/2024/12/03/aurora-dsql Specifically, transaction latency is constant relative to transaction statement count, even across region, so I guess that's something

DonnyV 2y ago

[flagged]

zokier 2y ago
How many multi-region active-active PostgreSQL clusters have you operated?
- sgarland 2y ago
  One, and it was a nightmare. Do not recommend.
dalyons 2y ago
did you miss the word "distributed"?

qaq 2y ago

Virtually unlimited scale :) Maximum storage GB per cluster: 100GB Maximum size of all data modified within a write transaction 10 MiB Max: 10K rows per transaction

dbbk 2y ago
100GB doesn't seem very unlimited
- bobnamob 2y ago
  While I love to take potshots, the yes(configurable) next to that 100gb does some heavy lifting. I imagine that number is tunable up to multi digit tb if you your monthly bill is worth Jassy's attention https://docs.aws.amazon.com/aurora-dsql/latest/userguide/CHAP_quotas.html https://docs.aws.amazon.com/aurora-dsql/latest/userguide/CHA... The more interesting (i.e. constraining) limits are transaction size (both row count and data size) and transaction time, which aren't configurable.

whoevercares 2y ago

Curious why Murat left the project before it’s launched

jwnin 2y ago

Seems like a competitor to Azure's CosmosDB offering , which also now has wire support for postgres.

sorenbs 2y ago

Very interesting announcement! We are also building a hosted Postgres service based on Firecracker and with the ability to create unlimited databases. Will have to dig in and see how we compare to this. https://www.prisma.io/blog/announcing-prisma-postgres-early-access https://www.prisma.io/blog/announcing-prisma-postgres-early-...

sj26 2y ago

If anybody likes to play with Ruby on Rails, I've published a bare bones active record connection adapter: https://github.com/sj26/activerecord-dsql-adapter https://github.com/sj26/activerecord-dsql-adapter

sj26 2y ago
Interesting discovery: PG::FeatureNotSupported: ERROR: ddl and dml are not supported in the same transaction

znpy 2y ago

I see many comments about “PostgreSQL compatibility” not being actual compatibility… maybe the PostgreSQL people should compile a list of requirements for a database to call itself “PostgreSQL-compatible”? It feels like the compatibility claim is a bit abused (not only in this case) mostly because there’s no clear and strict reference (maybe with mandatory and optional requirements) for what “compatible” means

rsanheim 2y ago

[flagged]

deleted 2y ago
[deleted]

c4pt0r 2y ago

- Yes, the storage is distributed kv. - Yes, it's SQL on distributed transactional kv - SQL layer is stateless, runs in multiple containers - SQL layer would push down compute logic down to the storage(like, filter/predicate) ....Yes, that reminds me of TiDB