5 ms·
In retrospect, going serverless (using Serverless Framework) has been a terrible decision for us. First, exposing an API served by AWS Lambda has the infamous
by thegeomaster 8y ago
In retrospect, going serverless (using Serverless Framework) has been a terrible decision for us.
First, exposing an API served by AWS Lambda has the infamous cold start problem. It's not fun to wait a couple of seconds for a mobile app to respond just because the request hit in an unfortunate time. One solution we found is to use a Serverless Framework plugin to periodically ping lambdas to keep them hot. But each concurrent lambda execution is a separate container, so you have to anticipate a number of concurrent requests you will be receiving at peak, or want to handle without a ~1s latency. Ouch - what happened to effortless scaling? And Amazon API Gateway adds another 100-200ms of latency on top of Lambda.
If you want to use an SQL database, you have to add your lambdas to a VPC, unless you want to expose your database to the Internet. But if you need to access the Internet from your lambdas, then you need a NAT (which you pay for). And you get another couple of seconds to cold starts while AWS attaches a network interface to your lambda. So, if you don't want this, you're stuck with e.g. DynamoDB. DynamoDB is optimized for scaling but it's a poor fit for relational data. Very basic support for indexes, no transactions, and data model migrations are very painful. There is also no spatial data story for DynamoDB, whereas if we had used Postgres, we'd be able to make use of PostGIS which is awesome.
The tooling is terrible. Serverless Framework is very rigid and riddled with bugs - we ended up maintaining our own fork of Serverless alongside forks of 3 plugins and another custom plugin we wrote to support our very simple workflow. There's no faithful offline reproduction of the API Gateway -> Lambda environment. We frequently ran into issues when the code is deployed that wouldn't show up when testing with a "simulation" plugin locally.
There's a lot of other problems we ran into, but these are the biggest ones I could think of. It didn't help that we didn't have much experience with this technology before deciding to serve our API using it (this was probably our biggest mistake). I guess we should have stuck with what we're faimiliar with - normal, "serverful" apps. And with the DyanmoDB provisioned capacity costs, I'm not sure we saved that much in the end.
- tirumaraiselvan 8y agoWhy do the problems that you state with RDS not true for DynamoDB? Does DynamoDB not run in a VPC?
- thegeomaster 8y agoIt's a managed service by Amazon and you access it using the AWS SDK, for which you need an IAM role, so everything is secure. Whereas with RDS, Amazon maintains a database for you, which you just connect to using ordinary TCP. IIRC it's not even possible to run an RDS database exposed to the Internet (well, you can always install Postgres on an EC2 instance, but that's even crazier).
- tirumaraiselvan 8y agoI see, but security aside I am not sure how the latency is impacted by having a serverless on a DynamoDB VPC endpoint. BTW, you can expose RDS out to the internet.
- thegeomaster 8y agoI didn't say it was impacted in that case. Interesting, looks like that might be possible after all, though the docs are a bit unclear how this would work from Lambda specifically, and it looks like it needs a NAT regardless. Guess I misunderstood the docs at the time. (This ties into my comment about inexperience being our largest mistake. :))
- robrtsql 8y agoIf you use DynamoDB, you don't need to run your Lambda in a VPC. With RDS, you do. If you run your Lambda in a VPC, your cold start times get worse, so latency increases.
- tirumaraiselvan 8y agoThis is very good insight and I suppose lot of people would be facing the same issues. I think latency is an issue when you use Lambdas for sync operations. Fundamentally when using serverless, I think your application should become fully async i.e. event-driven. You may look at the 3factor pattern[1] to see if it's any better. Basically, 3factor says that all you should do in your application is simple operations, like CRUD, and emit events which trigger serverless functions (which create more events and so on). You should asynchronously receive updates via something like realtime GraphQL. [1] https://3factor.app https://3factor.app
- matwood 8y agoUnless you are going to run your db on the same server as your server application, are you not going to have a NAT regardless? The Dynamo complaints isn't a Serverless issue, but a NoSQL issue. Hopefully NoSQL first/everywhere is finally dying as people realize all the benefits of a RDBMS (and shortcomings of NoSQL). And, if you knew you were doing something with geo data, picking Dynamo was a tool selection mistake from the get-go.
- thegeomaster 8y agoOur geo data needs still don't require a GIS, but we're slowly getting to the point where they do. At the time, it was unclear if a GIS would ever be needed - the company and product have done a 180° turn, as startups usually do. In hindsight, DynamoDB was a terrible choice, but we didn't have that hindsight back then. Had we stuck to the tools that we knew how to use, we'd have made the right choice, which is what I was trying to say in the parent comment.