12 ms·

Problems with AWS Amplify and its use of DynamoDB

145 points by kevinslin 4y ago

camhart 4y ago
The title of the article isn't a fair representation of what the author actually says. The author dislikes specific technologies within the amplify umbrella, but admits other ones are acceptable. From experience, AWS Amplify auth for interacting with aws cognito is a perfectly acceptable high value (low cost) solution for many use cases. I can't speak to the other technologies.
- MuffinFlavored 4y ago
  > AWS Amplify auth for interacting with aws cognito is a perfectly acceptable high value (low cost) solution for many use cases. https://www.passportjs.org/ https://www.passportjs.org/ is this an alternative or not really because you still have to "build it yourself"?
  - camhart 4y ago
    Passport is nodejs specific it looks like (no personal experience with it). AWS Amplify auth with cognito gives you libs for most major platforms (android, ios, web, etc) in most frameworks (vue, angular, react, native, etc). Plus aws cognito integrates well with AWS.
    - thedougd 4y ago
      I believe Amazon folded the Cognito SDKs into the Amplify SDK. You can use the SDK for Cognito and not use Amplify, the service, at all.
MuffinFlavored 4y ago
> AWS Amplify is a complete solution that lets frontend web and mobile developers easily build, ship, and host full-stack applications on AWS, with the flexibility to leverage the breadth of AWS services as use cases evolve. No cloud expertise needed. How does this compare to say... a $10/mo DigitalOcean Linux VPS that you deploy stuff to with like... Ansible?
- yamtaddle 4y ago
  DO's network performance gets... iffy under serious load. Or sometimes just has consistently-terrible throughput and/or latency to certain addresses, for unclear reasons (I have a suspicion their peering agreements and interconnects are simply awful for a lot of routes—probably as a cost-saving measure). You may not notice any of this on hobbyist or small-time commercial projects, but it's a real problem. So that's one difference. AWS is expensive, and its performance tiers for stuff like networking can be frustratingly opaque, but when you pay enough, you do get more-or-less what you paid for. And fighting DO's networking can get expensive in a hurry, in lost business or technician hours. As for Amplify vs. Ansible, yeah, dunno. [EDIT] Incidentally, this isn't the first time I've seen hosting of this general sort fall apart at the networking level—I ran into very similar problems with RackSpace over a decade ago, there was simply no way to come anywhere near taxing the hardware we were paying for before the pipe got saturated, like, there was an order-of-magnitude mismatch between the two, requests timing out en masse while the hardware ("what a good bargain this price is!") was practically idle. [EDIT EDIT] It's probably no coincidence that it's easy to list and understand hardware specs, but difficult to communicate or evaluate real-world Internet-connected network performance without trying it—so it's probably tempting as hell to scrimp on networking while selling based on bang-for-the-buck in hardware allocation.
  - walls 4y ago
    I've run IRC bots on several VPS services, and connectivity on DO was by far the most unreliable. It's crazy how often they would randomly drop out, even under no load.
    - yamtaddle 4y ago
      Yeah, for some addresses that have no trouble whatsoever with AWS we've seen crazy-high dropped packet rates from DO. Sometimes, but not always, it's the reason for perceived extreme slowness from them, though other times they're just routing through some connection, somewhere along the way, that's about as wide as a straw, for whatever reason. Non-North-American addresses tend to get it the worst, but we've seen it with US and Canadian ones, too. AWS? Smooth sailing for all, from datacenters physically located near the DO ones we were using. With DO we'll have some clients seeing great performance, and others who may as well be on a spotty dial-up connection at the exact same time. Frustrating, and there's not really anything you can do about it except switch to another host. I'd love to know what the story is with 'em.
QuinnyPig 4y ago
It also charges a healthy fee on top of the already high AWS data egress charges...
emptysea 4y ago
Sounds similar to Meteor, works fine for a prototype but collapses under complexity and perf issues once you want more features Mongo has a lot of performance gotchas with things like pipelines / aggregations that don’t scale the same way they do in Postgres & similar Took a while but we eventually migrated to a plain rest api with a react front end. Still used mongo
- indymike 4y ago
  I've used both Meteor and Amplify. Amplify has been fantastic for SPAs that don't need a backend database and require zero SSR. You get nice deployment tooling (especially for jr. devs) and CDN for very little money. When we started on our app, you could call a page generator during build, but doing dynamic SSR was not possible... yet... so we kind of dodged the bullet. Meteor was fantastic for building highly reactive apps with a small team. I think it got a lot of bad run for scaling up because it was dependent (at least when I used it) on tailing MongoDB's op log for reactivity. That lead to having to make very different architecture choices to make it work well, and I seem to recall each server instance getting bogged when they had 70-100 concurrent users , so you ended up using lots of containers for meteor application servers. Not sure where it all ended because I moved on to a different platform three yeas ago.
deleted 4y ago
[deleted]
willio58 4y ago
Speaking of limitations with scan in DynamoDb, I've recently been working with DynamoDB and found that our implementation was solely using scan operations. I wrote a couple lines of code to generate a secondary index, used a query operation instead for our endpoint serving page responses and that's going to save our company ~$15k this year (more as we scale) and decreased our page load times by 10x (3s to .3s) for all of our users. In total that's going to save over a year of page load time for all of our users over the next year. It's just odd to me that I had to make this change in the first place. I wonder how many millions of dollars AWS is raking in just because devs are unaware of things like that. Also makes me wonder how much energy is being wasted on inefficient DB operations..
- camhart 4y ago
  Their documentation is very clear about scans being expensive. Amazon isn't to blame for the poor engineering decision.
  - lofatdairy 4y ago
    That was my initial impression reading the OP as well, if your implementation is slow because you're using scans, well, it's because you're using scans. That said, I honestly don't know enough about the details regarding OP's implementation and why they couldn't refactor to properly Query the PK, but it sounds similar to how you'd authenticate reads on Firebase so maybe it's habits from a different platform kicking in. That said, Amplify allowing users to just get into GraphQL whether or not they actually need it might be a legitimate noob trap.
    - nslog 4y ago
      Eh their documentation is pretty clear about all this stuff and all the new directives released a couple years back (like the PK and index stuff) clearly guide people towards using indexed queries and not scans: https://docs.amplify.aws/cli/graphql/data-modeling/ https://docs.amplify.aws/cli/graphql/data-modeling/ A lot of this is on the author IMO
- 4y ago
othiagocruz 4y ago
I'm curious what was the company doing with Cognito that required anything more than DynamoDB, perhaps storing user profile information or relationships? This is the wrong approach, you use Cognito/Amplify for session management, not data relationship.
oliverrice 4y ago
disclaimer: author of supabase/pg_graphql > What should AWS do -> provide Postgres or similar as a database choice If you're interested a postgres option you could check out Supabase GraphQL[1] which is based on pg_graphql [2], a native PostgreSQL extension. You define your schema in SQL (including any indexes you'd like) and it reflects a full GraphQL API. With that stack, the example you gave about filtering for security would use be solved using a Row Level Security [3] policy where the work all occurs on the DB. While not a part of pg_graphql, using Supabase as a backend also handles Auth [4] & Storage [5] (both OSS) which covers most of the Amplify bases [1] https://supabase.com/docs/guides/api#graphql-api-overview https://supabase.com/docs/guides/api#graphql-api-overview [2] https://github.com/supabase/pg_graphql https://github.com/supabase/pg_graphql [3] https://www.postgresql.org/docs/current/sql-createpolicy.html https://www.postgresql.org/docs/current/sql-createpolicy.htm... [5] https://supabase.com/docs/guides/auth https://supabase.com/docs/guides/auth [5] https://supabase.com/docs/guides/storage https://supabase.com/docs/guides/storage
- smotched 4y ago
  Supabase will never be a realistic option for any company thats heard of mobile. Even if the startup is starting web only, they're kneecapping their future so its not an option Amplify has amazing mobile sdks (esp ios).
  - simlevesque 4y ago
    > Supabase will never be a realistic option What if they make a mobile SDK too ? Seems like your comment is an hyperbole.
    - kiwicopple 4y ago
      (supabase ceo) I'm not sure why GP would think it's never going to be an option. One of the things we're releasing this week is better mobile support. Swift SDK is here: https://supabase.com/docs/reference/swift/introduction https://supabase.com/docs/reference/swift/introduction
trilobyte 4y ago
This is a pretty good write up on the actual implementation problems of the solution vs. just a rant. The company I'm at now built their own custom graph database years ago and is plagued by problems that would be cousins of some of those listed here, so I see the pain the author is dealing with because I'm hearing about similar problems from some team or customer support at least 1x/month.
freedomben 4y ago
We went all in with Dynamo years back for an app because it was exciting. It went fine so we did it again, and again. Over time though (and in some cases not much time) we quickly realized that we were writing a lot of code to do sanity/safety checks, that was basically just re-implementing relational database features. As we'd hit bugs, we'd add more checking to our app, and eventually we just had a ton of relational DB features at the app level. I've just used postgres ever since.
- lightbendover 4y ago
  Dynamo is almost never something you want to use as a "solution" to an open-ended I-don't-quite-know-my-requirements type of problem (which is almost all holistic problems). Sometimes I feel like it's the only tool anyone knows how to reach for, but it comes with dozens of ways to shoot yourself in the foot one way or another if you're trying to use it generally.
- ngc248 4y ago
  DynamoDB is amazing when you know all the access patterns for your data and the access patterns are not liable to change much. The performance it gives in such a case is superlative to just about anything else. But if you have changing access patterns and require flexibility in querying, then DynamoDB is not for you.
nslog 4y ago
Stopped reading when this guy started talking about Single Table DDB being best practice and linking to AWS when he clearly did zero research on AWS recommendations with GraphQL including from the AWS Hero Alex Debrie he mentions: https://www.alexdebrie.com/posts/dynamodb-graphql/ https://www.alexdebrie.com/posts/dynamodb-graphql/ https://aws.amazon.com/blogs/mobile/single-table-vs-multi-table-dynamodb-appsync/ https://aws.amazon.com/blogs/mobile/single-table-vs-multi-ta... https://aws.amazon.com/blogs/mobile/enhancing-dynamodb-single-table-design-with-appsync-access-and-security-features/ https://aws.amazon.com/blogs/mobile/enhancing-dynamodb-singl... Also he mentions the race condition around clients connecting to the backend between queries and mutations, then in the same breath says the DataStore client handles this but also it's mistake is being reliant on Subscriptions? This contradiction doesn't make sense. There are some legit bugs in this article though that I've seen, like that Analytics one. It's really a Pinpoint issue though when you dig under the covers that I've been waiting for them to solve for years.
solatic 4y ago
> provide Postgres or similar as a database choice Amplify's core architectural presumption is to only use serverless options to try to keep costs low. Serverless SQL databases are currently a big hole in AWS's offering. Yes, there's Aurora Serverless, but you need to run it in a VPC, you can't scale to zero (so, for some definitions, not really serverless), v2 threw out their Data API, and v1's Data API had severe scaling issues anyway. v2 is so expensive that you have to very seriously think through the numbers on the small RDS databases and ask what value you're actually going to get out of v2. So what OP proposes can't really be done by the Amplify team. That said, the fact that Planetscale and Neon (which also have branching!) continue without serious competition from AWS is a serious headscratcher.
- nkozyra 4y ago
  That said, nothing prevents you from spinning up services/endpoints that lean on whatever datasource you care about. Assuming it's accessible within the VPC you can shove all your data in there and call it a day. You have another potential problem in your stack, but it can be done.
  - solatic 4y ago
    Of course it can be done, but the minute that you need to spin up a VPC, your infrastructure complexity becomes considerably higher. You need to worry about subnetting, carving up IP address space and IPv4 vs. IPv6, making sure you set up VPC Service Endpoints for various AWS services so that you're not paying for Internet egress despite using AWS services, security groups... none of which relates to business value at all. It's just a hassle that's best avoided unless you really don't have any other choice or the financial math actually works out for doing so.
- Scubabear68 4y ago
  V2 Aurora costs really took me by surprise at my last gig they used it. It seems to be extremely expensive compared to just about anything else.
dmattia 4y ago
I use Amplify in a lot of my personal projects because it's backed by mostly serverless components and I can run my personal websites on it essentially for free. While in my job, I often lean for other infra for the services I manage. So I overall agree with the thesis, but I just couldn't get myself to agree with much of the reasoning in this article :/ Parts I liked: - Callouts of the subscription model are good. I have used Firebase rtdb and firestore quite a bit, and agree it's much easier with those tools to just get an up-to-date view. With amplify, I often avoid subscriptions and just re-call list methods after updates (supported well in Apollo, but still unideal) - The discussions on Amplify/NoSQL/Dynamo being difficult to pivot match my experience, especially when I was less experienced with NoSQL patterns > Amplify's approach in using DynamoDB is literally called out as something you should avoid by AWS Alex DeBrie is a wonderful resource, and I love his books, but I wouldn't go as far as to say that all of AWS calls out using multiple tables. There is lots of active debate on the single-table vs multi-table designs > the fundamental mistake that's made here is that AWS Amplify puts your data into DynamoDB, which is not a general-purpose database Citation needed. It's a fairly standard NoSQL database and can be used generally. > But DynamoDB Scales? I was a bit confused on this article between the frequent mentions of DynamoDB only being worth it at a large scale ("If you're storing that much plain text data, good for you—maybe DynamoDB can help you!", "But you should only use it when a traditional database won't cut it", etc.) while also claiming more broadly that Amplify will not be useful for large apps because it doesn't scale. It seems to contradict itself there? I do think this section brings up a great point on the inflexibility of Amplify and trying to pivot a data model, which can absolutely be tricky! But the core message of this section doesn't seem to support the thesis. > If I was AWS, I would… provide Postgres or similar as a database choice I use RDS with Aurora/postgres with Amplify by using lambda resolvers and it works well. You can't use VTL, but in my mind that's a feature, as VTL is not my favorite for a few reasons like being hard to test. In general, this was one of my larger frustrations with this article: it doesn't mention the flexibility of Amplify. Amplify comes with highly opinionated built-ins, but you don't have to use almost any of it, and it's quite easy to plug-and-play other tools if you want. Lambda resolvers are a good example because they're extremely flexible, you can use many different languages/tools/even docker to run your resolvers. But also, I opt to not use Amplify hosting and it's fine. And you can also use your own auth without Cognito if you'd like (especially if you're already using custom resolvers). It sounds like the author inherited some parts of the codebase in question, so if some of these things were already setup I could see it being time consuming to pivot. > rethink whether GraphQL is actually fit for purpose: It's just not very good Again, while I'm not a huge GraphQL fan, this is very much an opinion and a highly debatable one. For many use cases, GraphQL can work well, and I think from a technical perspective it is at minimum arguable that something like GraphQL makes sense for a tool like Amplify where one of its core use cases is multi-platform usage across web/mobile.
cameroncf 4y ago
The biggest problem with Amplify is not any of it's underlying technologies. These are all great services. DynamoDB is one of the best things in the entire AWS service portfolio. The biggest problem with Amplify is that it hides complexity that the developer really needs to have a firm grasp of when designing their application. You really need to think a lot about your access patterns before using DynamoDB. DynamoDB is OLTP and not OLAP. You aren't going to have a ton of ability to do ad-hoc queries with DynamoDB. that's just not a use case it's well suited for. HOWEVER - AWS Amplify is marketed without that warning - people use it without really understanding that. Then they get stuck when they start trying to treat it like a traditional RDBMS datastore, which it is not. Amplify "dumbs down" something that really should not be dumbed down. That's it's biggest problem and the reason I also stopped using Amplify some time ago.
- nslog 4y ago
  This is spot on. Parts of Amplify are aimed at people just getting started with Cloud and Programming, which is good and bad. You can shoot yourself in the foot if you drive the car too fast. I think a lot of more advanced people use Amplify services along with the Amplify client SDKs and then do the provisioning using CDK rather than the Amplify CLI.
- twawaaay 4y ago
  I would add a bit of my own experience. Products like this are great. But if you already know the technology, know what is happening and you are using it to basically eliminate a bunch of coding or setup tasks that you would have to do. The problem this is only great for a certain type of knowledgeable developer who is also setting up the system. The next person that comes or a junior developer who does not have grasp of the underlying tech will just see a bunch of magic. But he will be required to be productive with it. So if the original devs leave and don't get people with grasp of the underlying tech, the new people will start accumulating technical debt at an ever faster rate without ever being able to pay it back. Because with so little knowledge and expectation of performance, whenever something small happens that requires you to learn stuff you will skip learning and just move stuff around hoping some random change will fix it. So I will amend my original statement: "Products like this are great." But only if you are working on the project alone and don't expect anybody else to join it in the future.
JCharante 4y ago
While Amplify offers a lot, using it for deployments only is really pain-free. If you have a node.js app that you want to deploy with support for feature branches and automatic webhooks listening to your git repo then it's a great serverless deployment choice compared to choosing some serverless frameworks where you still have to deal with CLIs and shell scripts in the repo.
jwsteigerwalt 4y ago
You can pick and choose what you use amplify for. I use it for all the front-end CI/CD tasks and API Gateway/Lambda/etc. however I want for the backend. Use it for what it’s good for without getting locked in.
inssein 4y ago
AWS Amplify is the worst AWS service I have used, by far. The console was constantly broken in chrome, their reverse proxy is slow, the pre-baked auth components with Cognito were pretty useless, and so much more.. It feels like the team is trying to do way too much, all at once, and none of it well.
outworlder 4y ago
> Consider using just a normal database—personally for me that would be Firestore (Google-hosted) or MongoDB (basically self-hotsed), because I strongly prefer scalable NoSQL databases. And then goes on to say: > But for most people, it's probably going to literally just be Postgres. You don't need more than Postgres. Those statements are basically contradictory. Maybe the author only works on applications where NoSQL is a good fit. Otherwise, it's not clear how the author is different from "most people". This idea that SQL databases are not "scalable" has become a meme and is difficult to eradicate. Sure, sharding/replicating Postgres (if you need to) can be more difficult than just increasing a field in some auto-scaling group. But a well-tuned Postgres can service a ridiculous number of requests. If you are doing microservices, then it's even better, since you'll have multiple smaller databases - some of these can even be NoSQL if it fits the application. What 'scalable' NoSQL databases usually means is that you can just throw money at the problem and increase the number of instances, without spending time doing silly things like 'optimizing' queries. Of course, for some applications, NoSQL is a perfect fit. But far too often I see people reaching out to NoSQL based on some future (usually unspecified) "scale" needs. Heck, I've even seen that mindset in internal, corporate network only apps.
ursuscamp 4y ago
One of the things I always thought was funny was that Amazon always tells you to put everything in one DynamoDB table, but the Amplify splits everything across multiple tables.
harrisonjackson 4y ago
I use amplify in lots of projects. It is a nice wrapper for the auth and registration stuff. We also use it for some frontend ci/cd. Nothing else though. Custom authorizer in front of aws lambda with serverless framework works great with amplify’s frontend api client. We also use dynamodb serverless but not with app sync and no graphQL. Would I rather have a good and easy to use serverless Postgres db that played well with aws lambda. Yes. Am I going to switch away because I have to plan my access patterns and indices out carefully… no.
kristianpaul 4y ago
Amplify as other similar software stack choices are very good for a MVP or POC. If your project is thriving start early and migrate your strategy to something that scale better and its easy to change.
ramblinrose 4y ago
I use Amplify for a mobile app that I created and maintain just because I'm not familiar with backend technologies and it seemed like the quickest and easiest option for me, a mobile developer. The app isn't super complex, but I wouldn't describe it as a "toy" application either. I haven't run into any major headaches with DynamoDB or GraphQL limitations... yet. But this article has me thinking about exploring other options. To try to get ahead of the seemingly inevitable wave of tech debt. Does anyone have advice or can point to any resources for embarking on that particular journey? Are there similar technologies that do it better than Amplify? Anything in between Amplify and learning all the disparate AWS tools from scratch?
- spmurrayzzz 4y ago
  > Anything in between Amplify and learning all the disparate AWS tools from scratch? It depends how much of the backend you ultimately would like to avoid (and related to that, how complex your service ecosystem will be for your app). Without knowing much more about your app, I can throw some darts. You could use RDS to get you postgres/maria/etc and then just run an app server on EC2 for your backend REST/SOAP/etc API in front of that. If the latter body of work isn't comfortable for you, there is a swath of low code tooling out there you could use as well to put an API in front of your SQL for you (nocodb comes to mind, or just postgrest if all you need is CRUD ops).

145 points by kevinslin 4y ago

The title of the article isn't a fair representation of what the author actually says. The author dislikes specific technologies within the amplify umbrella, but admits other ones are acceptable. From experience, AWS Amplify auth for interacting with aws cognito is a perfectly acceptable high value (low cost) solution for many use cases. I can't speak to the other technologies.

MuffinFlavored 4y ago
> AWS Amplify auth for interacting with aws cognito is a perfectly acceptable high value (low cost) solution for many use cases. https://www.passportjs.org/ https://www.passportjs.org/ is this an alternative or not really because you still have to "build it yourself"?
- camhart 4y ago
  Passport is nodejs specific it looks like (no personal experience with it). AWS Amplify auth with cognito gives you libs for most major platforms (android, ios, web, etc) in most frameworks (vue, angular, react, native, etc). Plus aws cognito integrates well with AWS.
  - thedougd 4y ago
    I believe Amazon folded the Cognito SDKs into the Amplify SDK. You can use the SDK for Cognito and not use Amplify, the service, at all.

MuffinFlavored 4y ago

> AWS Amplify is a complete solution that lets frontend web and mobile developers easily build, ship, and host full-stack applications on AWS, with the flexibility to leverage the breadth of AWS services as use cases evolve. No cloud expertise needed. How does this compare to say... a $10/mo DigitalOcean Linux VPS that you deploy stuff to with like... Ansible?

yamtaddle 4y ago
DO's network performance gets... iffy under serious load. Or sometimes just has consistently-terrible throughput and/or latency to certain addresses, for unclear reasons (I have a suspicion their peering agreements and interconnects are simply awful for a lot of routes—probably as a cost-saving measure). You may not notice any of this on hobbyist or small-time commercial projects, but it's a real problem. So that's one difference. AWS is expensive, and its performance tiers for stuff like networking can be frustratingly opaque, but when you pay enough, you do get more-or-less what you paid for. And fighting DO's networking can get expensive in a hurry, in lost business or technician hours. As for Amplify vs. Ansible, yeah, dunno. [EDIT] Incidentally, this isn't the first time I've seen hosting of this general sort fall apart at the networking level—I ran into very similar problems with RackSpace over a decade ago, there was simply no way to come anywhere near taxing the hardware we were paying for before the pipe got saturated, like, there was an order-of-magnitude mismatch between the two, requests timing out en masse while the hardware ("what a good bargain this price is!") was practically idle. [EDIT EDIT] It's probably no coincidence that it's easy to list and understand hardware specs, but difficult to communicate or evaluate real-world Internet-connected network performance without trying it—so it's probably tempting as hell to scrimp on networking while selling based on bang-for-the-buck in hardware allocation.
- walls 4y ago
  I've run IRC bots on several VPS services, and connectivity on DO was by far the most unreliable. It's crazy how often they would randomly drop out, even under no load.
  - yamtaddle 4y ago
    Yeah, for some addresses that have no trouble whatsoever with AWS we've seen crazy-high dropped packet rates from DO. Sometimes, but not always, it's the reason for perceived extreme slowness from them, though other times they're just routing through some connection, somewhere along the way, that's about as wide as a straw, for whatever reason. Non-North-American addresses tend to get it the worst, but we've seen it with US and Canadian ones, too. AWS? Smooth sailing for all, from datacenters physically located near the DO ones we were using. With DO we'll have some clients seeing great performance, and others who may as well be on a spotty dial-up connection at the exact same time. Frustrating, and there's not really anything you can do about it except switch to another host. I'd love to know what the story is with 'em.

QuinnyPig 4y ago

It also charges a healthy fee on top of the already high AWS data egress charges...

emptysea 4y ago

Sounds similar to Meteor, works fine for a prototype but collapses under complexity and perf issues once you want more features Mongo has a lot of performance gotchas with things like pipelines / aggregations that don’t scale the same way they do in Postgres & similar Took a while but we eventually migrated to a plain rest api with a react front end. Still used mongo

indymike 4y ago
I've used both Meteor and Amplify. Amplify has been fantastic for SPAs that don't need a backend database and require zero SSR. You get nice deployment tooling (especially for jr. devs) and CDN for very little money. When we started on our app, you could call a page generator during build, but doing dynamic SSR was not possible... yet... so we kind of dodged the bullet. Meteor was fantastic for building highly reactive apps with a small team. I think it got a lot of bad run for scaling up because it was dependent (at least when I used it) on tailing MongoDB's op log for reactivity. That lead to having to make very different architecture choices to make it work well, and I seem to recall each server instance getting bogged when they had 70-100 concurrent users , so you ended up using lots of containers for meteor application servers. Not sure where it all ended because I moved on to a different platform three yeas ago.

deleted 4y ago

[deleted]

willio58 4y ago

Speaking of limitations with scan in DynamoDb, I've recently been working with DynamoDB and found that our implementation was solely using scan operations. I wrote a couple lines of code to generate a secondary index, used a query operation instead for our endpoint serving page responses and that's going to save our company ~$15k this year (more as we scale) and decreased our page load times by 10x (3s to .3s) for all of our users. In total that's going to save over a year of page load time for all of our users over the next year. It's just odd to me that I had to make this change in the first place. I wonder how many millions of dollars AWS is raking in just because devs are unaware of things like that. Also makes me wonder how much energy is being wasted on inefficient DB operations..

camhart 4y ago
Their documentation is very clear about scans being expensive. Amazon isn't to blame for the poor engineering decision.
- lofatdairy 4y ago
  That was my initial impression reading the OP as well, if your implementation is slow because you're using scans, well, it's because you're using scans. That said, I honestly don't know enough about the details regarding OP's implementation and why they couldn't refactor to properly Query the PK, but it sounds similar to how you'd authenticate reads on Firebase so maybe it's habits from a different platform kicking in. That said, Amplify allowing users to just get into GraphQL whether or not they actually need it might be a legitimate noob trap.
  - nslog 4y ago
    Eh their documentation is pretty clear about all this stuff and all the new directives released a couple years back (like the PK and index stuff) clearly guide people towards using indexed queries and not scans: https://docs.amplify.aws/cli/graphql/data-modeling/ https://docs.amplify.aws/cli/graphql/data-modeling/ A lot of this is on the author IMO
4y ago

othiagocruz 4y ago

I'm curious what was the company doing with Cognito that required anything more than DynamoDB, perhaps storing user profile information or relationships? This is the wrong approach, you use Cognito/Amplify for session management, not data relationship.

oliverrice 4y ago

disclaimer: author of supabase/pg_graphql > What should AWS do -> provide Postgres or similar as a database choice If you're interested a postgres option you could check out Supabase GraphQL[1] which is based on pg_graphql [2], a native PostgreSQL extension. You define your schema in SQL (including any indexes you'd like) and it reflects a full GraphQL API. With that stack, the example you gave about filtering for security would use be solved using a Row Level Security [3] policy where the work all occurs on the DB. While not a part of pg_graphql, using Supabase as a backend also handles Auth [4] & Storage [5] (both OSS) which covers most of the Amplify bases [1] https://supabase.com/docs/guides/api#graphql-api-overview https://supabase.com/docs/guides/api#graphql-api-overview [2] https://github.com/supabase/pg_graphql https://github.com/supabase/pg_graphql [3] https://www.postgresql.org/docs/current/sql-createpolicy.html https://www.postgresql.org/docs/current/sql-createpolicy.htm... [5] https://supabase.com/docs/guides/auth https://supabase.com/docs/guides/auth [5] https://supabase.com/docs/guides/storage https://supabase.com/docs/guides/storage

smotched 4y ago
Supabase will never be a realistic option for any company thats heard of mobile. Even if the startup is starting web only, they're kneecapping their future so its not an option Amplify has amazing mobile sdks (esp ios).
- simlevesque 4y ago
  > Supabase will never be a realistic option What if they make a mobile SDK too ? Seems like your comment is an hyperbole.
  - kiwicopple 4y ago
    (supabase ceo) I'm not sure why GP would think it's never going to be an option. One of the things we're releasing this week is better mobile support. Swift SDK is here: https://supabase.com/docs/reference/swift/introduction https://supabase.com/docs/reference/swift/introduction

trilobyte 4y ago

This is a pretty good write up on the actual implementation problems of the solution vs. just a rant. The company I'm at now built their own custom graph database years ago and is plagued by problems that would be cousins of some of those listed here, so I see the pain the author is dealing with because I'm hearing about similar problems from some team or customer support at least 1x/month.

freedomben 4y ago

We went all in with Dynamo years back for an app because it was exciting. It went fine so we did it again, and again. Over time though (and in some cases not much time) we quickly realized that we were writing a lot of code to do sanity/safety checks, that was basically just re-implementing relational database features. As we'd hit bugs, we'd add more checking to our app, and eventually we just had a ton of relational DB features at the app level. I've just used postgres ever since.

lightbendover 4y ago
Dynamo is almost never something you want to use as a "solution" to an open-ended I-don't-quite-know-my-requirements type of problem (which is almost all holistic problems). Sometimes I feel like it's the only tool anyone knows how to reach for, but it comes with dozens of ways to shoot yourself in the foot one way or another if you're trying to use it generally.
ngc248 4y ago
DynamoDB is amazing when you know all the access patterns for your data and the access patterns are not liable to change much. The performance it gives in such a case is superlative to just about anything else. But if you have changing access patterns and require flexibility in querying, then DynamoDB is not for you.

nslog 4y ago

Stopped reading when this guy started talking about Single Table DDB being best practice and linking to AWS when he clearly did zero research on AWS recommendations with GraphQL including from the AWS Hero Alex Debrie he mentions: https://www.alexdebrie.com/posts/dynamodb-graphql/ https://www.alexdebrie.com/posts/dynamodb-graphql/ https://aws.amazon.com/blogs/mobile/single-table-vs-multi-table-dynamodb-appsync/ https://aws.amazon.com/blogs/mobile/single-table-vs-multi-ta... https://aws.amazon.com/blogs/mobile/enhancing-dynamodb-single-table-design-with-appsync-access-and-security-features/ https://aws.amazon.com/blogs/mobile/enhancing-dynamodb-singl... Also he mentions the race condition around clients connecting to the backend between queries and mutations, then in the same breath says the DataStore client handles this but also it's mistake is being reliant on Subscriptions? This contradiction doesn't make sense. There are some legit bugs in this article though that I've seen, like that Analytics one. It's really a Pinpoint issue though when you dig under the covers that I've been waiting for them to solve for years.

solatic 4y ago

> provide Postgres or similar as a database choice Amplify's core architectural presumption is to only use serverless options to try to keep costs low. Serverless SQL databases are currently a big hole in AWS's offering. Yes, there's Aurora Serverless, but you need to run it in a VPC, you can't scale to zero (so, for some definitions, not really serverless), v2 threw out their Data API, and v1's Data API had severe scaling issues anyway. v2 is so expensive that you have to very seriously think through the numbers on the small RDS databases and ask what value you're actually going to get out of v2. So what OP proposes can't really be done by the Amplify team. That said, the fact that Planetscale and Neon (which also have branching!) continue without serious competition from AWS is a serious headscratcher.

nkozyra 4y ago
That said, nothing prevents you from spinning up services/endpoints that lean on whatever datasource you care about. Assuming it's accessible within the VPC you can shove all your data in there and call it a day. You have another potential problem in your stack, but it can be done.
- solatic 4y ago
  Of course it can be done, but the minute that you need to spin up a VPC, your infrastructure complexity becomes considerably higher. You need to worry about subnetting, carving up IP address space and IPv4 vs. IPv6, making sure you set up VPC Service Endpoints for various AWS services so that you're not paying for Internet egress despite using AWS services, security groups... none of which relates to business value at all. It's just a hassle that's best avoided unless you really don't have any other choice or the financial math actually works out for doing so.
Scubabear68 4y ago
V2 Aurora costs really took me by surprise at my last gig they used it. It seems to be extremely expensive compared to just about anything else.

dmattia 4y ago

I use Amplify in a lot of my personal projects because it's backed by mostly serverless components and I can run my personal websites on it essentially for free. While in my job, I often lean for other infra for the services I manage. So I overall agree with the thesis, but I just couldn't get myself to agree with much of the reasoning in this article :/ Parts I liked: - Callouts of the subscription model are good. I have used Firebase rtdb and firestore quite a bit, and agree it's much easier with those tools to just get an up-to-date view. With amplify, I often avoid subscriptions and just re-call list methods after updates (supported well in Apollo, but still unideal) - The discussions on Amplify/NoSQL/Dynamo being difficult to pivot match my experience, especially when I was less experienced with NoSQL patterns > Amplify's approach in using DynamoDB is literally called out as something you should avoid by AWS Alex DeBrie is a wonderful resource, and I love his books, but I wouldn't go as far as to say that all of AWS calls out using multiple tables. There is lots of active debate on the single-table vs multi-table designs > the fundamental mistake that's made here is that AWS Amplify puts your data into DynamoDB, which is not a general-purpose database Citation needed. It's a fairly standard NoSQL database and can be used generally. > But DynamoDB Scales? I was a bit confused on this article between the frequent mentions of DynamoDB only being worth it at a large scale ("If you're storing that much plain text data, good for you—maybe DynamoDB can help you!", "But you should only use it when a traditional database won't cut it", etc.) while also claiming more broadly that Amplify will not be useful for large apps because it doesn't scale. It seems to contradict itself there? I do think this section brings up a great point on the inflexibility of Amplify and trying to pivot a data model, which can absolutely be tricky! But the core message of this section doesn't seem to support the thesis. > If I was AWS, I would… provide Postgres or similar as a database choice I use RDS with Aurora/postgres with Amplify by using lambda resolvers and it works well. You can't use VTL, but in my mind that's a feature, as VTL is not my favorite for a few reasons like being hard to test. In general, this was one of my larger frustrations with this article: it doesn't mention the flexibility of Amplify. Amplify comes with highly opinionated built-ins, but you don't have to use almost any of it, and it's quite easy to plug-and-play other tools if you want. Lambda resolvers are a good example because they're extremely flexible, you can use many different languages/tools/even docker to run your resolvers. But also, I opt to not use Amplify hosting and it's fine. And you can also use your own auth without Cognito if you'd like (especially if you're already using custom resolvers). It sounds like the author inherited some parts of the codebase in question, so if some of these things were already setup I could see it being time consuming to pivot. > rethink whether GraphQL is actually fit for purpose: It's just not very good Again, while I'm not a huge GraphQL fan, this is very much an opinion and a highly debatable one. For many use cases, GraphQL can work well, and I think from a technical perspective it is at minimum arguable that something like GraphQL makes sense for a tool like Amplify where one of its core use cases is multi-platform usage across web/mobile.

cameroncf 4y ago

The biggest problem with Amplify is not any of it's underlying technologies. These are all great services. DynamoDB is one of the best things in the entire AWS service portfolio. The biggest problem with Amplify is that it hides complexity that the developer really needs to have a firm grasp of when designing their application. You really need to think a lot about your access patterns before using DynamoDB. DynamoDB is OLTP and not OLAP. You aren't going to have a ton of ability to do ad-hoc queries with DynamoDB. that's just not a use case it's well suited for. HOWEVER - AWS Amplify is marketed without that warning - people use it without really understanding that. Then they get stuck when they start trying to treat it like a traditional RDBMS datastore, which it is not. Amplify "dumbs down" something that really should not be dumbed down. That's it's biggest problem and the reason I also stopped using Amplify some time ago.

nslog 4y ago
This is spot on. Parts of Amplify are aimed at people just getting started with Cloud and Programming, which is good and bad. You can shoot yourself in the foot if you drive the car too fast. I think a lot of more advanced people use Amplify services along with the Amplify client SDKs and then do the provisioning using CDK rather than the Amplify CLI.
twawaaay 4y ago
I would add a bit of my own experience. Products like this are great. But if you already know the technology, know what is happening and you are using it to basically eliminate a bunch of coding or setup tasks that you would have to do. The problem this is only great for a certain type of knowledgeable developer who is also setting up the system. The next person that comes or a junior developer who does not have grasp of the underlying tech will just see a bunch of magic. But he will be required to be productive with it. So if the original devs leave and don't get people with grasp of the underlying tech, the new people will start accumulating technical debt at an ever faster rate without ever being able to pay it back. Because with so little knowledge and expectation of performance, whenever something small happens that requires you to learn stuff you will skip learning and just move stuff around hoping some random change will fix it. So I will amend my original statement: "Products like this are great." But only if you are working on the project alone and don't expect anybody else to join it in the future.

JCharante 4y ago

While Amplify offers a lot, using it for deployments only is really pain-free. If you have a node.js app that you want to deploy with support for feature branches and automatic webhooks listening to your git repo then it's a great serverless deployment choice compared to choosing some serverless frameworks where you still have to deal with CLIs and shell scripts in the repo.

jwsteigerwalt 4y ago

You can pick and choose what you use amplify for. I use it for all the front-end CI/CD tasks and API Gateway/Lambda/etc. however I want for the backend. Use it for what it’s good for without getting locked in.

inssein 4y ago

AWS Amplify is the worst AWS service I have used, by far. The console was constantly broken in chrome, their reverse proxy is slow, the pre-baked auth components with Cognito were pretty useless, and so much more.. It feels like the team is trying to do way too much, all at once, and none of it well.

outworlder 4y ago

> Consider using just a normal database—personally for me that would be Firestore (Google-hosted) or MongoDB (basically self-hotsed), because I strongly prefer scalable NoSQL databases. And then goes on to say: > But for most people, it's probably going to literally just be Postgres. You don't need more than Postgres. Those statements are basically contradictory. Maybe the author only works on applications where NoSQL is a good fit. Otherwise, it's not clear how the author is different from "most people". This idea that SQL databases are not "scalable" has become a meme and is difficult to eradicate. Sure, sharding/replicating Postgres (if you need to) can be more difficult than just increasing a field in some auto-scaling group. But a well-tuned Postgres can service a ridiculous number of requests. If you are doing microservices, then it's even better, since you'll have multiple smaller databases - some of these can even be NoSQL if it fits the application. What 'scalable' NoSQL databases usually means is that you can just throw money at the problem and increase the number of instances, without spending time doing silly things like 'optimizing' queries. Of course, for some applications, NoSQL is a perfect fit. But far too often I see people reaching out to NoSQL based on some future (usually unspecified) "scale" needs. Heck, I've even seen that mindset in internal, corporate network only apps.

ursuscamp 4y ago

One of the things I always thought was funny was that Amazon always tells you to put everything in one DynamoDB table, but the Amplify splits everything across multiple tables.

harrisonjackson 4y ago

I use amplify in lots of projects. It is a nice wrapper for the auth and registration stuff. We also use it for some frontend ci/cd. Nothing else though. Custom authorizer in front of aws lambda with serverless framework works great with amplify’s frontend api client. We also use dynamodb serverless but not with app sync and no graphQL. Would I rather have a good and easy to use serverless Postgres db that played well with aws lambda. Yes. Am I going to switch away because I have to plan my access patterns and indices out carefully… no.

kristianpaul 4y ago

Amplify as other similar software stack choices are very good for a MVP or POC. If your project is thriving start early and migrate your strategy to something that scale better and its easy to change.

ramblinrose 4y ago

I use Amplify for a mobile app that I created and maintain just because I'm not familiar with backend technologies and it seemed like the quickest and easiest option for me, a mobile developer. The app isn't super complex, but I wouldn't describe it as a "toy" application either. I haven't run into any major headaches with DynamoDB or GraphQL limitations... yet. But this article has me thinking about exploring other options. To try to get ahead of the seemingly inevitable wave of tech debt. Does anyone have advice or can point to any resources for embarking on that particular journey? Are there similar technologies that do it better than Amplify? Anything in between Amplify and learning all the disparate AWS tools from scratch?

spmurrayzzz 4y ago
> Anything in between Amplify and learning all the disparate AWS tools from scratch? It depends how much of the backend you ultimately would like to avoid (and related to that, how complex your service ecosystem will be for your app). Without knowing much more about your app, I can throw some darts. You could use RDS to get you postgres/maria/etc and then just run an app server on EC2 for your backend REST/SOAP/etc API in front of that. If the latter body of work isn't comfortable for you, there is a swath of low code tooling out there you could use as well to put an API in front of your SQL for you (nocodb comes to mind, or just postgrest if all you need is CRUD ops).