28 ms·
A bank runs serverless with PHP and AWS Lambda
- willsmith72 3y agoI always love a story about a successful strangler pattern migration. I do wonder how they came to Lambda though. I love it for small workloads and highly variable demand services, but something like Treezor you'd think has relatively flat and high demand. The cloud cost for Lambda would be much higher than running the equivalent compute, even with something also highly scalable like ECS.
- ceejayoz 3y agoWe use Lambda (via Laravel Vapor; https://vapor.laravel.com/ https://vapor.laravel.com/) for a big analytics batch processing job every night; it spawns hundreds of thousands of individual jobs that make various API calls, but only for a few minutes. For the rest of the day it's doing very little. I'd imagine a bank has quite a few of these sorts of bursty bash processing needs; ACHes, end-of-day reconciliation, etc.
- jackconsidine 3y agoWhat's your experience with Vapor? We played around with it and thought about switching from Forge, but it was too complicated for our codebase. I love the idea though. How expensive our your cloud costs? CI/CD pretty good?
- ceejayoz 3y agoOverall quite good. We use their Docker container support (to get ffmpeg, mostly) and it's been pretty smooth running throughout. I'd previously used a lot of Heroku so the habits developed there paid off in our codebase for Vapor; only a few tweaks were really needed. Cost-wise we're pretty bursty, so it's saved us quite a bit. Lambda runtime is money, so a little bit of performance optimization can go a long way. CI we run through Github Actions; an environment secret plus `vapor deploy` and it's on its way up after a successful run. Wasn't a fan of Forge; for that style of things I'd use Ploi nowadays.
- ndriscoll 3y agoAt my last job, we had an architecture that ran hundreds of thousands of individual jobs with internal API calls for nightly account updates and reporting work, and it was a major battle to finally get it replaced with some simple SQL scripts which ran orders of magnitude faster while being a lot more reliable. As I understand it, banking commonly uses JVM where you can easily do async or multithreaded processing and task scheduling, so lambda doesn't offer much over ECS, and the timeout could be a real killer.
- me_graf 3y agoI did some fintech work as a junior / kinda non junior dev. This is correct. I think 90% of the work was taking pictures of checks, and setting up batch jobs. all batch: - money movement (between accounts) - transfers - wires - etc. even the 'non' batch things were done in batch (even if it's do now, and only 1).
- theshrike79 3y agoLambdas are amazing at clearing queues. You can have a single queue gathering workloads and then have Lambda functions grab a bunch of tasks, handle them, grab more etc. until the max runtime is met and they're automatically rebooted. It's also pretty trivial to add more processing capacity when the queue goes over a specified limit just by allowing for more concurrent Lambdas. And when there's nothing to process, you've got one idling Lambda doing nothing and costing nothing.
- cogman10 3y agoThis was my thought. Lambda works well for infrequent tasks. But hosting an API? That seems like it could get expensive pretty fast.
- deleugpn 3y agoIt's cheaper than most options where you need to pay a human to manage something with the same level of scalability and security that AWS provides.
- ndriscoll 3y agoLambda scalability for an API is awful; it can't run concurrent requests (of course PHP largely can't do this either). An ECS service with pretty much any other language (for async/multithreaded runtimes) will scale far better.
- cebert 3y agoA single lambda can only handle one request at a time, but you can scale up 1,000s of instances if needed.
- ndriscoll 3y agoA single minimal ECS instance can handle thousands of concurrent requests with an async runtime without running into so many cold starts that your p25 time is in the hundreds of ms. 1000 lambdas with 1000 database connections is also going to be a horrible way to do things on the backend. Add in the cost of RDS proxy for lambda and the ECS solution becomes even better.
- WatchDog 3y agoECS fargate is pretty easy to scale, and much more cost effective. I personally don’t think running on EC2 is particularly difficult to manage, and it’s even cheaper again.
- Sohcahtoa82 3y ago
- mnapoli 3y agoTheir clients have very variable traffic patterns because they are in very different industries. They have traffic spikes at lunch, or at the end of every month for example: > infrastructure must be able to scale and be resilient to accommodate various usage patterns. Whether it's a luncheon voucher transaction spike at lunchtime or a monthly batch of transactions by corporate clients
- tomrod 3y agoI'd be more worried about decimal data types being interpreted as floats. That could get gnarly fast on the wrong language runtime, and is partially why Cobol programmers are still in demand in the financial sector.
- chx 3y agothe beauty of bref is you work with the same php engine as before. I am a (very) long time PHP developer, my primary work is still Drupal so the ability to make a small serverless app in PHP was very handy for me. My app implements a webhook and I feel Lambda is made for this: the script is small and quite fast, it checks the payload and puts an item into SQS as appropriate. Then another infra can slowly empty the SQS queue. The problem here is the load is insanely spikey -- sometimes tens of thousands of requests in a very short time frame, sometimes nothing. Directly writing this into a transactional database would require very costly infra. The Lambda-SQS model smooths it out. It's fine if some changes take minutes or even hours to be recorded.
- Supermancho 3y ago> I do wonder how they came to Lambda though. Probably some security certification that AWS Lambda comes with. I worked at Cambia Health that used Lambda (Javascript) for dealing with customer mobile data. It was a nightmare solution (cost and complexity) for what should have been a simple fan out queue and process system. They hired me to help fix it. Every other solution was a flat no. AWS Lambda was was HIPPA compliant (for whatever that's worth) and some other security assurances that I don't remember, so I was basically just another developer (with a moron for a manager).
- iLoveOncall 3y ago> I do wonder how they came to Lambda though Probably misplaced hype about being fully serverless. As you said, ECS would be much more suitable while also addressing the issue that they had (spikes in traffic). Lambdas behind API Gateway sounds like a good idea and even looks like one if you don't have experience with ECS and don't do a proper cost comparison analysis. My team uses ECS' autoscaling to scale up to tens (or even hundreds) of thousands of tasks in minutes without issues.
- mikece 3y agoI don't know which surprises me more: that they continued to use PHP in their move to serverless/microservices or that nobody else on HN has questioned this point yet. I totally get why they would keep PHP: they have the language/platform experience already and AWS supports it (which is really more than enough to silence the "You should have switched to {other_language}!" partisans). Plus, I imaging non-trivial amounts of copy & paste of proven functions was key to moving from monolith to Lambdas... why change the code if it works?
- yaseer 3y agoI'm wondering if PHP's reputation has been rehabilitated somewhat in recent years. At least amongst those who understand the significant changes in modern PHP, and the difference a stable framework like Laravel makes. I've found PHP with Laravel a stable and solid workhorse. Good for getting things done quickly, and then a stable system over time. In contrast, I've gradually become disillusioned with the node.js and JavaScript/TypeScript ecosystem over the years. You can build quickly, but the systems turns into a pain to maintain and update.
- kypro 3y agoIt genuinely amazes me that Node is as popular as it is, especially within large mission critical businesses. If this bank was using TypeScript no one would have questioned it, but you're right, TypeScript is a nightmare to maintain. I probably only use about half of packages I used 5 years ago because either those packages are no longer maintained or the community has switched to some other package that basically does the same thing just in a slightly different way. There's also no real industry standard ways to do anything in Node. You find 10 different Node CRUD projects and they'll all do things completely differently. The PHP ecosystem on the other hand tends to be less fragmented and these days with frameworks like Laravel the code is generally also pretty good. I think it gets a bad reputation because in the 00s there were some truly horrific PHP projects out there and the language itself was quite immature and poorly designed. That's not the case anymore though. These days I struggle to understand the hate it gets. It's certainly no worse than than Node.
- 3y ago
- neatze 3y agoIn second part (Refactoring to serverless microservices) where is database in there ?
- mikece 3y agoI'm going to guess that that database moved to RDS and, once there, was likely refactored as services were decomposed to microservices... but that's not mentioned in the article.
- ThinkBeat 3y ago"" Treezor is an independent provider of outsourcing and white label solutions for electronic payments. "" https://www.crunchbase.com/organization/treezor https://www.crunchbase.com/organization/treezor So, is this a payment processor much like Stripe?
- herpderperator 3y agoI feel like the naming of the bank (Treezor) is slightly unfortunate since it's so close to the "Trezor" bitcoin hardware wallet...
- ganoushoreilly 3y agoI agree, it's what I thought when I read it.
- The_Colonel 3y agoTresor means safe / vault, it's a quite natural name for a bank.
- djbusby 3y agoTresor -> Treasure
- zetalyrae 3y agoAlso a cognate of thesaurus. Like the Trésor de la langue française.
- munk-a 3y agoThe bank isn't named Tresor - it's named Treezor. That's quite a bit closer to Trezor than Tresor. Anyways, if companies are going to use made up words for their names I'm happy to always deny them the benefit of the doubt.
- The_Colonel 3y agoMy point was that both of these companies/products are named after the "Tresor".
- 4ndrewl 3y agoAren't all words made up?
- 3y ago
- joshstrange 3y agoI'd really like to play with Bref but it more or less requires using serverless framework from what I understand as there are no guides on how to get it working with something like SST. I've used serverless framework before and I won't touch it again. It seems like a complete dead end with the team behind it more or less giving up on it to go work on a cloud backend alternative to lambda/etc. Docs were a mess, lot's of half-implemented things, etc. I wouldn't lift-and-shift the PHP app I work on, it requires too many FreeBSD-specific/custom underlying services but I could make use of lambda for certain tasks while leveraging the PHP code we already have. We use SST for for some NodeJS (TypeScript) lambdas that have been very popular but PHP isn't going anywhere in our stack so I'd love to find a way to move the bursty parts of it to lambda.
- mnapoli 3y agoYou can deploy using the CDK for example: https://bref.sh/docs/deploy/aws-cdk https://bref.sh/docs/deploy/aws-cdk
- joshstrange 3y agoOh wow, that’s almost exactly what I need. Thank you! I don’t know if this is newer or if I just totally missed it the last time I looked.
- solardev 3y agoWhat is "SST" in this context? Is it this thing? https://sst.dev/ https://sst.dev/ I'm having trouble understanding where PHP fits into this scenario. If your cloud backend is in PHP, can't you just host that anywhere, separate from your frontends? Where does the serverless come in? And which did you use? (There are so many out there now, from AWS Lambda to Cloudflare to Fastly, etc.) If you're not limited to AWS, Google's Cloud Run lets you containerize a PHP app and auto-scale it up and back down to zero in bursts, for example. It's not really serverless, just an auto-scaling VM that goes up and down as needed. edit: google cloud functions might be even closer: https://cloud.google.com/functions/docs/create-deploy-http-php https://cloud.google.com/functions/docs/create-deploy-http-p...
- agentultra 3y agoCurious if they have spikey latency from cold starts? The authorization calls have a time budget but it's pretty generous so maybe a nothing-burger there? How do they keep authorization context fresh enough for the lambda performing it? Also... what has the bill been like? I've used Lambda for side-car data migration projects but I'd be surprised if it saved money over a beefy co-located or cloud server if your volume is generally constant.
- AndrewDucker 3y agoI'd love to know how much the cost changed.
- padjo 3y ago> The migration to microservices is still an ongoing work. Ain’t that always the way
- Muromec 3y agoOn schedule to be complyted in year, as of three years ago and still 80% in progress
- GaelFG 3y agoI have no clues about what's better but as a french I'm amazed by the idea to delegate banking operations to a cloud provider. Last time I worked in bank IT political requirement and good practices were using their own private physical network infrastructure over all the country and data storage server rooms were literally bunkers with armed security. Is it that common around the world and 'we' just happen to have been overkill on security or do they are not really a true 'bank' and more a payment provider ? that seems such a change from some years ago were cost were totally the last of the issues against security.
- orangepurple 3y agoIt's way worse than you think. A large number of Swiss banks including Swiss banking regulators store all their data on Google Cloud.
- mellowagain 3y agoA lot of swiss government data is also stored in AWS, made a few headlines during covid because we "delegate our data to the americans"
- solardev 3y agoWe have your data, you have our money, sounds fair =)
- orangepurple 3y agoThey don't though. Swiss tax wealth (though its a tiny fraction). IRS makes it a massive PITA to hold substantial financial stakes in foreign companies. See FATCA and GILTI. And many tax deferred foreign retirement funds are not treated as being tax deferred by the IRS. It only gets more complicated from here. It's by far easier to earn and hold cash in America.
- kevinventullo 3y ago
- solardev 3y agoI wonder why they went with AWS Lambda and PHP instead of something serverless-native like Cloudflare Workers? I guess they still have some critical parts of the legacy stack on AWS (DBs especially, but also other AWS services). I wonder if you can build something like this entirely greenfield today, like what you'd use as a permanent data store for all the transactions, without having to manage the scaling of each individual microservice. Would CF Workers + Durable Objects be able to handle something like that?
- solardev 3y agoAlso, I just realized "Bref" is its own project that lets you run PHP on serverless Lambdas... like a Google Cloud Run on Amazon? Is this really such a big niche? It's weird to me to you'd want to use PHP for something like this (and I love PHP, it just doesn't strike me as the best tool for serverless).
- deleugpn 3y agoAs you can see on the website (https://bref.sh/ https://bref.sh/) there's 13 billion monthly AWS Lambda invocations using Bref (PHP).
- solardev 3y agoThat doesn't really say much. Isn't the whole point of serverless to support high-volume, individually-inexpensive, invocations? For context, 13B is like roughly a <strike>$2000</strike> (edit: more like $4000, sorry) spend on Cloudflare Workers, similar cost (harder to calculate but ballpark similar) on Lambda. That could be a single company spending the bulk of that. (Further edit: Lambda pricing is way off too. See reply below.)
- deleugpn 3y agoThat ballpark is not even close to the park. My company spends ~$500/month on AWS Lambda and we're doing about 2M/req/day with Bref. There's still 12B 940M requests to account for
- witnesser2 3y agoThere was once upon a time a discussion about bank/finance software system. You have to keep long history of audit records / logs. It is difficult to peruse such consideration here with the database ballooned to the cloud.
- mschuster91 3y ago> You have to keep long history of audit records / logs. It is difficult to peruse such consideration here with the database ballooned to the cloud. Actually, it's easier. AWS RDS with multi-zone failover and replication + backups with retention schedules, deletion protection and legal holds... easy as a cake, less than 100 lines of Terraform code. And no risk of insider threats deleting, manipulating or ransoming the backups.
- smartusers 3y ago[dead]
- jackconsidine 3y agoLove a good lift-and-shift. I wish bref was this well-supported about 3-5 years ago. At that point I was trying to move some PHP services over to serverless. I think I remember seeing bref but it didn't appear to have the clout then. I even think Laravel Vapor didn't use bref when they were setting up PHP runtime configurations etc
- Implicated 3y agoI've been elbow deep in PHP for well over a decade, and I was genuinely surprised when I first saw this on Twitter. (Insert swaggy p meme here) I have immense respect for the author, but I'm perplexed about the decision to build a bank on this stack. Especially considering that the person who created Laravel Vapor (a serverless Laravel service) has since discussed the advantages of moving a large project from Lambda to EC2 (https://twitter.com/themsaid/status/1716844479817154589 https://twitter.com/themsaid/status/1716844479817154589). I really do wonder what the bullet points of pros/cons in choosing to go serverless for something like this were.
- ceejayoz 3y agoThat's for a large, fairly consistently loaded project. For a smaller site with bursty traffic, Lambda may be very relevant still.
- greatgib 3y agoTreezor is not really directly a bank but more a "bank as a service" company. Lots of companies or "neobank" that want to offer a kind of bank account, for personal or corporate use, but that don't have an official banking agreement, and not really a banking infrastructure, can use them as "white label" solutions. All the bank accounts and banking operations will be done by treezor, but from the user point of view, he will only interact with the company and company frontend, and almost never see that it is Treezor that is operating in the background. That being said, I don't know if their "serverless" move is a good one, because their solution is commonly known to be unreliable, and I think a lot of customers would be happy to go to another provider if it was possible.
- mannyv 3y agoAs a customer of US Bank in the US, I can attest to the fact that they aren’t as reliable as I’d want. Their website and app tend to be “under maintenance” quite often. I think you overestimate the reliability of many bank operations.
- syndicatedjelly 3y agoApp reliability is different from payment service reliability.
- marcus0x62 3y ago> In October 2021, they successfully migrated their first API route to AWS Lambda, the most critical one handling all live credit card transactions. Over the following year, more and more API endpoints were migrated away from the servers to AWS Lambda. Why? Why do people insist on doing this? Move something less important first, prove you can operate in production, then move the crown jewels.
- rewmie 3y agoI'd love to hear about the rationale behind this decision, because operational cost certainly isn't it. It's ludicrous to migrate a high-traffic, high-risk API to a function-as-a-service solution like AWS Lambda. Either they have a very unique requirement or it sounds like this might very well be the poster child of clueless resume-driven development.
- cebert 3y agoWhat specifically makes it ludicrous to you?
- JohnMakin 3y agoIME the rationale behind such decisions is what I call the "sinking ship" phenomenon in infrastructure. Maybe their system was failing in weird ways, hard to maintain, causing all sorts of issues, etc. In that "sinking ship" it's always gonna be "women and children first," or rather, their most critical systems first.
- deleted 3y ago[deleted]
- allan_s 3y agoat Rosaly.com we've been using bref since 2021 and we're extremly happy with it, even if we've stopped to the php-fpm integration. We haven't seen a reason yet to move out of the monolith and bref.sh allow us to have the best of both world (php + lambda ) 1. we're a small team and we want to manage as less infra as possible 2. with their "dev mode" docker images, we can a local version which replicate 99% of the production environment (same runtime, same readonly filesystems etc.) 3. deploying is as simple as doing a zip , uploading to s3, and calling lambda update. 4. we have cronjobs with cloudwatch , Symfony's Command.
- idlewords 3y agoA more accurate title here is "bank moves PHP app to Amazon servers"
- Fischgericht 3y agoWow, that is just amazing. So it means that if I connect to treezor.com, I am not talking to a server, but it's direct communication with god in heaven? And it's not a badly configured server, but the Content "Security" Policy allowing FOURTY different sites, including unsafe inlines (!) is beamed into my brain from a different universe? And the PHP "software" also clearly is running on a server, because PHP isn't a server-side scripting language, it's all happening in your mind! /s Hint: The whole point of a bank is BEING a server, be a central trusted authority accepting and executing transactions from and between clients. A serverless bank is a bank that no longer exists, but got replaced by peer-to-peer transactions.
- andrewxdiamond 3y agoI don’t understand why people nit-pick the wording “serverless” when the benefits of not having to manage servers is very real. Yes, there is a computer. Good job, you pointed it out. No one is paying AWS money because they think amazon is somehow running code without computers being involved. They’re paying to run lambdas to avoid dealing with the computers that run the code. Banks specifically have much higher operational costs due to regulatory requirements. Banks like to focus on their financial strategy and outsource the rest.
- Fischgericht 3y agoOf course us Nerds know that "serverless" is just a BS marketing term. I am nitpicking because there are technically uneducated people (managers) who actually believe in these BS marketing terms. People who already thought that such a thing as a "cloud" exists, resulting in us now having to look at our personal data getting stolen once per week because some manager thought you no longer need competent system administrators, because you are just pushing that personal data "to the cloud", so no risks involved. And in this regard, the term "serverless" is even worse. It's basically the flat earthers of technology. These BS marketing terms like "cloud" and "serverless" are an excuse for managers to act in an irresponsible fashion, thinking competence is no longer needed in-house. But competence IS needed. And that's why every single person of "us" in each and every conversation should point out "you are putting the data of your customers and your businesses existence into the hands of someone you don't know at all" and avoid the BS marketing terms. If we go along with them, we are just hurting us, our employers, customers and/or clients, and humanity. So, in this case the article should be translated into: "Our intern wrote some crappy PHP shit because he is not really a programmer but a baseball player, and we have decided to just upload and expose all our stuff to some random organization at an unknown location under unknown control so we don't have to deal with this stuff anymore once the intern has left." :)
- swingingFlyFish 3y agoWhere's the database staying on? You migrated to AWS RDS or it's elsewhere?
- jheriko 3y agobanks have low requirements with no technical challenges... im honestly amazed how bad almost every bank service is with their tech usage... and this has been true for all of my living memory. i might have to look into these guys... maybe i can have a service thats only millions of times worse than i expected as a child, instead of trillions of times worse