3 ms·

Running a Fly app backed by DynamoDB Global Tables is an option. DDB keeps a copy of your data in all the regions you specify, each Fly instance can connect to

Running a Fly app backed by DynamoDB Global Tables is an option. DDB keeps a copy of your data in all the regions you specify, each Fly instance can connect to the nearest region, and writes are propagated with eventual consistency & last write wins. And most Redis commands can be mapped to DDB, I worked on a lib to do that. https://github.com/dbProjectRED/redimo.go https://github.com/dbProjectRED/redimo.go https://github.com/sudhirj/aws-regions.go https://github.com/sudhirj/aws-regions.go https://aws.amazon.com/dynamodb/global-tables/ https://aws.amazon.com/dynamodb/global-tables/

thrwn_frthr_awy 5y ago
Does DynamoDB give transactions, rollbacks, joins?
- sudhirj 5y ago
  Yes, you can apply a set of updates atomically so they all fail or none, with conditions. No save points or nested transactions, though. And the transactions need to be grouped into a single request. Joins, no. Most of the NoSQL systems distribute data by partitioning them on the primary key, and give you scalability by not doing any work across partitions. Which means that traditional joins won’t be supported. You can either join the application, or maintain a secondary schema that has your joined data organised in the access pattern you plan to use. Google DynamoDB Alex DeBrie or Rick Houlihan for a ton of tutorials on how to restructure data for partitioned/distributed NoSQL systems.
  - qaq 5y ago
    and max 100 keys in a transaction or whatever the limit is
mhoad 5y ago
The other option here is using Spanner from Google (https://cloud.google.com/spanner https://cloud.google.com/spanner) which now has full PostgreSQL compatibility. That gets you global scalability with minimal meaningful changes to your application at all. They even made a Rails specific guide for it here https://cloud.google.com/blog/topics/developers-practitioners/scale-your-ruby-applications-active-record-support-cloud-spanner https://cloud.google.com/blog/topics/developers-practitioner... If you combined that with say Cloud Native Buildpacks (buildpacks.io) and Cloud Run you now have basically an infinitely scalable solution that happens to hook into all of Google’s infrastructure at just the right points that requires almost zero work on your behalf. Plus you get to keep the one command to deploy that a lot of the Rails community is used to from the Heroku days. No K8s to configure, no servers to maintain, per second billing on your app server, everything autoscales, very little to learn in the way of new technology. You can complicate things at your own pace as far as the rest of “the cloud” goes but there’s no need to do so unless you want to. It’s a pretty sweet deal in terms of cost to benefit ratios.
- jitl 5y ago
  > full PostgreSQL compatibility By "full", you mean "some"? From [the documentation] on Spanner's PostgreSQL Interface: > This means that applications written against the PostgreSQL interface can be readily ported to another PostgreSQL environment, obviously without the non-functional benefits, like scalability and availability, that are unique to Cloud Spanner. However, 100% compatibility with PostgreSQL is not the goal. > The PostgreSQL interface provides a rich subset of the open-source PostgreSQL SQL dialect, including common query syntax, functions, and operators It seems like existing simple Postgres applications could switch to Spanner, but it does require running a sidecar if your app wants to continue speaking the Postgres protocol on the wire, instead of integrating a Spanner API client. When it comes to [datatypes], Spanner's PostgreSQL interface is very limited. According to the docs, only these types are supported: > bool, bytea (byte array), float8 (64-bit IEEE-754 float), int8 (signed 64-bit int), numeric/decimal, timestampz, text. Notably missing for [my use-case] are UUID, JSON/JSONB, array types like UUID array, and custom enum types. The indexing support is also quite sparse. Full Postgres compatibility, this is not :( [the documentation]: https://cloud.google.com/spanner/docs/postgresql-interface https://cloud.google.com/spanner/docs/postgresql-interface [datatypes]: https://cloud.google.com/spanner/docs/reference/postgresql/data-types https://cloud.google.com/spanner/docs/reference/postgresql/d... [my use-case]: https://www.notion.so https://www.notion.so
  - mhoad 5y ago
    I didn’t know Notion was a Rails product, that’s pretty cool! Just for disclosure, I used to do a bunch of Rails but I haven’t touched it since the start of covid so it’s applicability to some of what I mentioned was not something I’ve actively built and so I haven’t figured out where all the rough parts are yet because it’s just not a use case of mine these days. But that second link I mentioned sounds maybe more promising? I initially skimmed it and thought it was there to handle the PostgreSQL to Spanner integration but looking at it a second time it sounds more like native Spanner integration but keeping the otherwise Rails native ActiveRecord experience. If you’re building against a bunch of PostgreSQL specific functionality then maybe you’re still stuck with the same problem but I don’t think there is a generally speaking more capable publicly available SQL database out there than Spanner than I am aware of is there? But as a side note if you’re Notion in this situation and not a random small team of a few people I don’t imagine you would have much of a problem reaching out to the GCP team to explain your current set up / requirements, they would be more than happy to give you a custom roadmap to migrate to Spanner, they have entire teams dedicated to exactly those kinds of projects.
Scarbutt 5y ago
Yes, lets throw data integrity out the window and make schemas changes painfully tedious and hard.

thrwn_frthr_awy 5y ago

Does DynamoDB give transactions, rollbacks, joins?

sudhirj 5y ago
Yes, you can apply a set of updates atomically so they all fail or none, with conditions. No save points or nested transactions, though. And the transactions need to be grouped into a single request. Joins, no. Most of the NoSQL systems distribute data by partitioning them on the primary key, and give you scalability by not doing any work across partitions. Which means that traditional joins won’t be supported. You can either join the application, or maintain a secondary schema that has your joined data organised in the access pattern you plan to use. Google DynamoDB Alex DeBrie or Rick Houlihan for a ton of tutorials on how to restructure data for partitioned/distributed NoSQL systems.
- qaq 5y ago
  and max 100 keys in a transaction or whatever the limit is

mhoad 5y ago

The other option here is using Spanner from Google (https://cloud.google.com/spanner https://cloud.google.com/spanner) which now has full PostgreSQL compatibility. That gets you global scalability with minimal meaningful changes to your application at all. They even made a Rails specific guide for it here https://cloud.google.com/blog/topics/developers-practitioners/scale-your-ruby-applications-active-record-support-cloud-spanner https://cloud.google.com/blog/topics/developers-practitioner... If you combined that with say Cloud Native Buildpacks (buildpacks.io) and Cloud Run you now have basically an infinitely scalable solution that happens to hook into all of Google’s infrastructure at just the right points that requires almost zero work on your behalf. Plus you get to keep the one command to deploy that a lot of the Rails community is used to from the Heroku days. No K8s to configure, no servers to maintain, per second billing on your app server, everything autoscales, very little to learn in the way of new technology. You can complicate things at your own pace as far as the rest of “the cloud” goes but there’s no need to do so unless you want to. It’s a pretty sweet deal in terms of cost to benefit ratios.

jitl 5y ago
> full PostgreSQL compatibility By "full", you mean "some"? From [the documentation] on Spanner's PostgreSQL Interface: > This means that applications written against the PostgreSQL interface can be readily ported to another PostgreSQL environment, obviously without the non-functional benefits, like scalability and availability, that are unique to Cloud Spanner. However, 100% compatibility with PostgreSQL is not the goal. > The PostgreSQL interface provides a rich subset of the open-source PostgreSQL SQL dialect, including common query syntax, functions, and operators It seems like existing simple Postgres applications could switch to Spanner, but it does require running a sidecar if your app wants to continue speaking the Postgres protocol on the wire, instead of integrating a Spanner API client. When it comes to [datatypes], Spanner's PostgreSQL interface is very limited. According to the docs, only these types are supported: > bool, bytea (byte array), float8 (64-bit IEEE-754 float), int8 (signed 64-bit int), numeric/decimal, timestampz, text. Notably missing for [my use-case] are UUID, JSON/JSONB, array types like UUID array, and custom enum types. The indexing support is also quite sparse. Full Postgres compatibility, this is not :( [the documentation]: https://cloud.google.com/spanner/docs/postgresql-interface https://cloud.google.com/spanner/docs/postgresql-interface [datatypes]: https://cloud.google.com/spanner/docs/reference/postgresql/data-types https://cloud.google.com/spanner/docs/reference/postgresql/d... [my use-case]: https://www.notion.so https://www.notion.so
- mhoad 5y ago
  I didn’t know Notion was a Rails product, that’s pretty cool! Just for disclosure, I used to do a bunch of Rails but I haven’t touched it since the start of covid so it’s applicability to some of what I mentioned was not something I’ve actively built and so I haven’t figured out where all the rough parts are yet because it’s just not a use case of mine these days. But that second link I mentioned sounds maybe more promising? I initially skimmed it and thought it was there to handle the PostgreSQL to Spanner integration but looking at it a second time it sounds more like native Spanner integration but keeping the otherwise Rails native ActiveRecord experience. If you’re building against a bunch of PostgreSQL specific functionality then maybe you’re still stuck with the same problem but I don’t think there is a generally speaking more capable publicly available SQL database out there than Spanner than I am aware of is there? But as a side note if you’re Notion in this situation and not a random small team of a few people I don’t imagine you would have much of a problem reaching out to the GCP team to explain your current set up / requirements, they would be more than happy to give you a custom roadmap to migrate to Spanner, they have entire teams dedicated to exactly those kinds of projects.

Scarbutt 5y ago

Yes, lets throw data integrity out the window and make schemas changes painfully tedious and hard.