5 ms·

I always love a story about a successful strangler pattern migration. I do wonder how they came to Lambda though. I love it for small workloads and highly vari

I always love a story about a successful strangler pattern migration. I do wonder how they came to Lambda though. I love it for small workloads and highly variable demand services, but something like Treezor you'd think has relatively flat and high demand. The cloud cost for Lambda would be much higher than running the equivalent compute, even with something also highly scalable like ECS.

ceejayoz 3y ago
We use Lambda (via Laravel Vapor; https://vapor.laravel.com/ https://vapor.laravel.com/) for a big analytics batch processing job every night; it spawns hundreds of thousands of individual jobs that make various API calls, but only for a few minutes. For the rest of the day it's doing very little. I'd imagine a bank has quite a few of these sorts of bursty bash processing needs; ACHes, end-of-day reconciliation, etc.
- jackconsidine 3y ago
  What's your experience with Vapor? We played around with it and thought about switching from Forge, but it was too complicated for our codebase. I love the idea though. How expensive our your cloud costs? CI/CD pretty good?
  - ceejayoz 3y ago
    Overall quite good. We use their Docker container support (to get ffmpeg, mostly) and it's been pretty smooth running throughout. I'd previously used a lot of Heroku so the habits developed there paid off in our codebase for Vapor; only a few tweaks were really needed. Cost-wise we're pretty bursty, so it's saved us quite a bit. Lambda runtime is money, so a little bit of performance optimization can go a long way. CI we run through Github Actions; an environment secret plus `vapor deploy` and it's on its way up after a successful run. Wasn't a fan of Forge; for that style of things I'd use Ploi nowadays.
- ndriscoll 3y ago
  At my last job, we had an architecture that ran hundreds of thousands of individual jobs with internal API calls for nightly account updates and reporting work, and it was a major battle to finally get it replaced with some simple SQL scripts which ran orders of magnitude faster while being a lot more reliable. As I understand it, banking commonly uses JVM where you can easily do async or multithreaded processing and task scheduling, so lambda doesn't offer much over ECS, and the timeout could be a real killer.
- me_graf 3y ago
  I did some fintech work as a junior / kinda non junior dev. This is correct. I think 90% of the work was taking pictures of checks, and setting up batch jobs. all batch: - money movement (between accounts) - transfers - wires - etc. even the 'non' batch things were done in batch (even if it's do now, and only 1).
- theshrike79 3y ago
  Lambdas are amazing at clearing queues. You can have a single queue gathering workloads and then have Lambda functions grab a bunch of tasks, handle them, grab more etc. until the max runtime is met and they're automatically rebooted. It's also pretty trivial to add more processing capacity when the queue goes over a specified limit just by allowing for more concurrent Lambdas. And when there's nothing to process, you've got one idling Lambda doing nothing and costing nothing.
cogman10 3y ago
This was my thought. Lambda works well for infrequent tasks. But hosting an API? That seems like it could get expensive pretty fast.
- deleugpn 3y ago
  It's cheaper than most options where you need to pay a human to manage something with the same level of scalability and security that AWS provides.
  - ndriscoll 3y ago
    Lambda scalability for an API is awful; it can't run concurrent requests (of course PHP largely can't do this either). An ECS service with pretty much any other language (for async/multithreaded runtimes) will scale far better.
    - cebert 3y ago
      A single lambda can only handle one request at a time, but you can scale up 1,000s of instances if needed.
      - ndriscoll 3y ago
        A single minimal ECS instance can handle thousands of concurrent requests with an async runtime without running into so many cold starts that your p25 time is in the hundreds of ms. 1000 lambdas with 1000 database connections is also going to be a horrible way to do things on the backend. Add in the cost of RDS proxy for lambda and the ECS solution becomes even better.
  - WatchDog 3y ago
    ECS fargate is pretty easy to scale, and much more cost effective. I personally don’t think running on EC2 is particularly difficult to manage, and it’s even cheaper again.
- Sohcahtoa82 3y ago
mnapoli 3y ago
Their clients have very variable traffic patterns because they are in very different industries. They have traffic spikes at lunch, or at the end of every month for example: > infrastructure must be able to scale and be resilient to accommodate various usage patterns. Whether it's a luncheon voucher transaction spike at lunchtime or a monthly batch of transactions by corporate clients
tomrod 3y ago
I'd be more worried about decimal data types being interpreted as floats. That could get gnarly fast on the wrong language runtime, and is partially why Cobol programmers are still in demand in the financial sector.
- chx 3y ago
  the beauty of bref is you work with the same php engine as before. I am a (very) long time PHP developer, my primary work is still Drupal so the ability to make a small serverless app in PHP was very handy for me. My app implements a webhook and I feel Lambda is made for this: the script is small and quite fast, it checks the payload and puts an item into SQS as appropriate. Then another infra can slowly empty the SQS queue. The problem here is the load is insanely spikey -- sometimes tens of thousands of requests in a very short time frame, sometimes nothing. Directly writing this into a transactional database would require very costly infra. The Lambda-SQS model smooths it out. It's fine if some changes take minutes or even hours to be recorded.
Supermancho 3y ago
> I do wonder how they came to Lambda though. Probably some security certification that AWS Lambda comes with. I worked at Cambia Health that used Lambda (Javascript) for dealing with customer mobile data. It was a nightmare solution (cost and complexity) for what should have been a simple fan out queue and process system. They hired me to help fix it. Every other solution was a flat no. AWS Lambda was was HIPPA compliant (for whatever that's worth) and some other security assurances that I don't remember, so I was basically just another developer (with a moron for a manager).
iLoveOncall 3y ago
> I do wonder how they came to Lambda though Probably misplaced hype about being fully serverless. As you said, ECS would be much more suitable while also addressing the issue that they had (spikes in traffic). Lambdas behind API Gateway sounds like a good idea and even looks like one if you don't have experience with ECS and don't do a proper cost comparison analysis. My team uses ECS' autoscaling to scale up to tens (or even hundreds) of thousands of tasks in minutes without issues.

ceejayoz 3y ago

We use Lambda (via Laravel Vapor; https://vapor.laravel.com/ https://vapor.laravel.com/) for a big analytics batch processing job every night; it spawns hundreds of thousands of individual jobs that make various API calls, but only for a few minutes. For the rest of the day it's doing very little. I'd imagine a bank has quite a few of these sorts of bursty bash processing needs; ACHes, end-of-day reconciliation, etc.

jackconsidine 3y ago
What's your experience with Vapor? We played around with it and thought about switching from Forge, but it was too complicated for our codebase. I love the idea though. How expensive our your cloud costs? CI/CD pretty good?
- ceejayoz 3y ago
  Overall quite good. We use their Docker container support (to get ffmpeg, mostly) and it's been pretty smooth running throughout. I'd previously used a lot of Heroku so the habits developed there paid off in our codebase for Vapor; only a few tweaks were really needed. Cost-wise we're pretty bursty, so it's saved us quite a bit. Lambda runtime is money, so a little bit of performance optimization can go a long way. CI we run through Github Actions; an environment secret plus `vapor deploy` and it's on its way up after a successful run. Wasn't a fan of Forge; for that style of things I'd use Ploi nowadays.
ndriscoll 3y ago
At my last job, we had an architecture that ran hundreds of thousands of individual jobs with internal API calls for nightly account updates and reporting work, and it was a major battle to finally get it replaced with some simple SQL scripts which ran orders of magnitude faster while being a lot more reliable. As I understand it, banking commonly uses JVM where you can easily do async or multithreaded processing and task scheduling, so lambda doesn't offer much over ECS, and the timeout could be a real killer.
me_graf 3y ago
I did some fintech work as a junior / kinda non junior dev. This is correct. I think 90% of the work was taking pictures of checks, and setting up batch jobs. all batch: - money movement (between accounts) - transfers - wires - etc. even the 'non' batch things were done in batch (even if it's do now, and only 1).
theshrike79 3y ago
Lambdas are amazing at clearing queues. You can have a single queue gathering workloads and then have Lambda functions grab a bunch of tasks, handle them, grab more etc. until the max runtime is met and they're automatically rebooted. It's also pretty trivial to add more processing capacity when the queue goes over a specified limit just by allowing for more concurrent Lambdas. And when there's nothing to process, you've got one idling Lambda doing nothing and costing nothing.

cogman10 3y ago

This was my thought. Lambda works well for infrequent tasks. But hosting an API? That seems like it could get expensive pretty fast.

deleugpn 3y ago
It's cheaper than most options where you need to pay a human to manage something with the same level of scalability and security that AWS provides.
- ndriscoll 3y ago
  Lambda scalability for an API is awful; it can't run concurrent requests (of course PHP largely can't do this either). An ECS service with pretty much any other language (for async/multithreaded runtimes) will scale far better.
  - cebert 3y ago
    A single lambda can only handle one request at a time, but you can scale up 1,000s of instances if needed.
    - ndriscoll 3y ago
      A single minimal ECS instance can handle thousands of concurrent requests with an async runtime without running into so many cold starts that your p25 time is in the hundreds of ms. 1000 lambdas with 1000 database connections is also going to be a horrible way to do things on the backend. Add in the cost of RDS proxy for lambda and the ECS solution becomes even better.
- WatchDog 3y ago
  ECS fargate is pretty easy to scale, and much more cost effective. I personally don’t think running on EC2 is particularly difficult to manage, and it’s even cheaper again.
Sohcahtoa82 3y ago

mnapoli 3y ago

Their clients have very variable traffic patterns because they are in very different industries. They have traffic spikes at lunch, or at the end of every month for example: > infrastructure must be able to scale and be resilient to accommodate various usage patterns. Whether it's a luncheon voucher transaction spike at lunchtime or a monthly batch of transactions by corporate clients

tomrod 3y ago

I'd be more worried about decimal data types being interpreted as floats. That could get gnarly fast on the wrong language runtime, and is partially why Cobol programmers are still in demand in the financial sector.

chx 3y ago
the beauty of bref is you work with the same php engine as before. I am a (very) long time PHP developer, my primary work is still Drupal so the ability to make a small serverless app in PHP was very handy for me. My app implements a webhook and I feel Lambda is made for this: the script is small and quite fast, it checks the payload and puts an item into SQS as appropriate. Then another infra can slowly empty the SQS queue. The problem here is the load is insanely spikey -- sometimes tens of thousands of requests in a very short time frame, sometimes nothing. Directly writing this into a transactional database would require very costly infra. The Lambda-SQS model smooths it out. It's fine if some changes take minutes or even hours to be recorded.

Supermancho 3y ago

> I do wonder how they came to Lambda though. Probably some security certification that AWS Lambda comes with. I worked at Cambia Health that used Lambda (Javascript) for dealing with customer mobile data. It was a nightmare solution (cost and complexity) for what should have been a simple fan out queue and process system. They hired me to help fix it. Every other solution was a flat no. AWS Lambda was was HIPPA compliant (for whatever that's worth) and some other security assurances that I don't remember, so I was basically just another developer (with a moron for a manager).

iLoveOncall 3y ago

> I do wonder how they came to Lambda though Probably misplaced hype about being fully serverless. As you said, ECS would be much more suitable while also addressing the issue that they had (spikes in traffic). Lambdas behind API Gateway sounds like a good idea and even looks like one if you don't have experience with ECS and don't do a proper cost comparison analysis. My team uses ECS' autoscaling to scale up to tens (or even hundreds) of thousands of tasks in minutes without issues.