5 ms·
I still don’t understand how you do a PostgreSQL insert on a serverless system—please help! Also, this statement is patently silly: > The biggest point to make
by tomcam 8y ago
I still don’t understand how you do a PostgreSQL insert on a serverless system—please help! Also, this statement is patently silly:
> The biggest point to make here is that serverless architecture may well require you to rethink your data layer. That’s not the fault of serverless.
Well, it is the fault of serverless. It’s a shortcoming. Own it. The trade off may we be worthwhile, I just can’t tell yet. I’m trying to wrap my mind around serverless as regards database-backed apps, but obviously it would be preferable to be able to use the RDBMS of my choice.
- matchagaucho 8y agoMoving away from RDBMS just wasn't an option for us. But AWS serverless provided a couple benefits: 1. Aurora Serverless RDBMS (In pilot) 2. Lambda function property caching (for which we cache connections)
- staticassertion 8y agoHow does Aurora Serverless help with the issue of having to reconnect for every lambda? (I am unfamiliar with the Aurora Serverless offering) Also, can you elaborate/ point me to info on 2?
- ersiees 8y agoYou don’t need a reconnect for every invocation actually, only per instance. I know you don’t want to think about instances with lambda, but actually they just run on some VM and you can just open one connection per VM.
- lostcolony 8y agoReally quickly then - Background info: A lambda that spins up stays live for a while, and can continue to handle requests. The function handler is called again, so anything scoped to it, will disappear with the function, but anything scoped outside of the function will persist. Ergo, move your connection creation outside of your handler. Realistically you may need some abstraction to properly close and reopen the connection in the event of failure (so you can't have a 'bad' lambda, where something has happened to the connection and you can't fix it without deploying code again, to trigger a lambda refresh), but for the most part it just works.
- staticassertion 8y agoInteresting. I'd considered this, but was worried about failure cases. Seems tricky.
- philliphaydon 8y agoKeeping connections open on a lambda isn’t a good thing. If you have a lot of Lambdas and a lot of load you can end up hitting max connection limits. Better to open late and close early and take a hit rather than risk taking your db offline.
- lostcolony 8y agoOh, agreed; it doesn't bottleneck on the connection but instead keeps creating them which can take performance down when you spike. Hence why the original article speaks of using something other than a RDBMS, such as Dynamo, since there really is no way of addressing that issue (since you can't set a concurrency limit on one kind of function; else you could have one function that has high concurrency invoke one that is far lower, that exists just to keep connections pooled). If you absolutely must hit an RDBMS, be quick about it to free up the connection, and definitely consider if the latency matters that much; a few dozen ms more to initialize and close the connection is going to be a LOT cleaner. But that said, how badly it penalizes you for scaling connections up depends on the DB.
- tachang 8y agoI agree with this. Having a traditionally relational store can be a huge help: lots of people know it well, the disadvantages are well known, and everything is insanely mature. I dislike how the article tries to say that isn’t serveless’s fault when it is just a trade off.
- actuator 8y agoYeah, RDBMSs do have some big advantages; one of them being the plethora of query patterns possible. If there are other parts of your application which are consuming the same data but in different patterns, it might not be necessary that systems like DynamoDB work in those cases. You might need to make several secondary indexes in those cases and all of them require provisioning of their own. The other alternative being setting up some ETL layer which transforms the data and sends to appropriate sinks, from where they can be consumed; but for small services that adds more moving parts and cost to the system.
- deleted 8y ago[deleted]
- deeg 8y agoAgreed. Perhaps the first thing a "best practices" post should start off with is what kinds of problems serverless is best suited for.
- travbrack 8y agoSo why is rdbms and serverless seemingly mutually exclusive? Are rdbms queries really that slow compared to something like dynamodb?
- actuator 8y agoActually RDBMS queries are quite fast. But those systems weren't designed for use patterns which serverless applications typically have. Like MySQL will limit you on connection count and each new connection puts extra overhead on the server. So if every function negotiates its own connection you might eat away DB resources just on negotiating connections even though the DB queries were fast.
- k__ 8y agoAs far as I know, most RDBMS are connection based, which brings an overhead. When you got your connection they're quite fast.
- whopa 8y agoI wonder why all the predominant RDBMSes are heavyweight connection based, and why this is presumably so hard to change. There's nothing in SQL that requires this to be the case, at least conceptually.
- k__ 8y agoSaves overhead. Crappy application-servers pump queries like nobodies business, so you need to shave every byte of them you can.
- pvg 8y agoPart of it is legacy but also, interactions with an RDBMS are often stateful so it's useful to have a session (in which such things as transactions can live, etc). I'd guess there isn't much reason to change this because there are standard and effective workarounds that just haven't yet made their way to things like lambda.
- k__ 8y ago> It’s a shortcoming. Own it. Well, I don't know much about Oracle and the other players, but AWS at least tries to own it with Aurora Serverless.
- ActVen 8y agoWe use lambda functions(node with pg) to process incoming data. We just connect them to Aurora(Postgres) and make the inserts. We have never had any problem with the process whether we are pulling from a queue or other data source.