10 ms·

As someone on a two-man-team who runs a lot of little "utility" functions in AWS Lambda with the Serverless Framework[1] to support our DevOps / Build processes

As someone on a two-man-team who runs a lot of little "utility" functions in AWS Lambda with the Serverless Framework[1] to support our DevOps / Build processes, it's been one of the most productive tools in my toolkit (after the initial learning curve, of course). It allows me to stand up a practically maintenance-free endpoint in a matter of hours (usually to glue separate services together): * Want to run a quick ETL process to refresh some tables or pull some data down from a third-party endpoint? * Want to expose a quick usage report for the project manager? * Want to send a slack message when a build finishes from TeamCity so QA can jump on it? * Want to monitor the rate of Crashlytics logs and send a warning to slack if it breeches a threshold? There are tons of "one-off" little processes that now just automatically run and don't require server upkeep. Most of them are functionally independent, so having them as separate functions makes sense (when we do need to coordinate, that's done through Step Functions). Note: I still don't think that they are a great fit for production services (and more fits one-off and cron usage patterns) [1] https://www.serverless.com/ https://www.serverless.com/

Demiurge 6y ago
How is owning an AWS account easier than a cheap dedicates server? You should have a code repo and backups in either case. At least some people mention "but I can scale this lambda x1000" and that's one advantage... But you can do all those tasks on a hetzner server in same amount of type, just python scripts.
- justinram11 6y ago
  Mostly it's about the maintenance costs of owning and managing a server -- on Lambda most of these tools are "set and forget". I don't want to have to manage security updates, disk usage, or any of the other multiple things that come with owning a server. I also could probably debate on the friendly-ness of just dumping all of these one-off services onto a single server (seems like it could get messy pretty quick), but I think you could probably go either way on that. We are also an AWS shop for our "production" stack already, so there is no added overhead for owning an AWS account.
  - chillfox 6y ago
    Servers can also be "set and forget", just enable automatic patching
- mark242 6y ago
  Because that "cheap dedicated server" isn't all that cheap. Just the documentation for backups on your Hetzner server is ridiculous. Why, in the 21st century, should I as an application developer ever have to worry about that? Let's say that Hetzner server goes completely toast at 2am and you get a pingdom message. You log in, realize you have to restore from backup. You fire up a new instance, and go through your restore procedure, and everything works great. You're lucky to be offline for an hour. I cannot even tell you what would happen if an underlying instance running a Lambda goes offline, because it never happens. But let's say that an underlying machine dies in us-east-1. Lambda just starts running more instances of your code on another underlying group of cpus, and you aren't offline.
  - crehn 6y ago
    And I’d argue the mere possibility of the server breaking at any time contributes to non-negligible amounts of sustained psychological stress.
    - mark242 6y ago
      Yes, this exactly. The fact that I can guarantee that my code is running, and have entire massive teams at Amazon whose literal job title is to keep my code running, it is such a freeing feeling. I might be able to delete rows from Dynamo from my code messing up, but Dynamo will never go offline.
      - crehn 6y ago
        The psychological effects of technology seem often overlooked—and they’re hard to quantify. I wish there was some easier way to quantify the “soft” aspects of everything that surrounds us, e.g. the long-term impact of beautiful and usable UI design, the reduction in “ambient” psychological stress, the impacts of chaos/consistency on how we feel, any robust measures of happiness, excitement, relaxation, etc. They seem to be such important attributes, but not as easily measured as a latency. Which in turn makes reaching consensus harder—and probably why design by committee doesn't work?
- konjin 6y ago
  Because you need two servers. One to run the application. One to run the application while the first is being upgraded. Now you need three servers, one to run the application, one as backup and one as load balancer. 99.99% uptime means you can be down no more than an hour a year. Which is really easy to overshoot when you're dicking around with a dist upgrade on three linux boxes. So that's why you end up with a server farm starting from your one small server.
  - akkartik 6y ago
    Notice that grandparent comment explicitly carved out production servers. If you can't tolerate downtime while you're being upgraded, you're production.
  - franciscop 6y ago
    99.99% of the websites out there don't need 99.99% uptime (from a business point of view)
    - klodolph 6y ago
      Once you get to 99.9% uptime you’re still less than 1 hour per month. I used to run apps for companies at work and it was standard operating procedure to have two app servers + load balancer for upgrades. Drain a server, upgrade it, boot it, check that it works, let it accept traffic. Repeat once more. Done. It may be true that most businesses don’t need the uptime, but I’d say it’s usually easier to give them high availability rather than field complaints when some random user tries to look something up during a maintenance window.
      - vanviegen 6y ago
        But where does the data sit? Would you be using some fancy multi-master synchronous replication for that, or is there a database server behind it?
        klodolph 6y ago
- shotgunsparkle 6y ago
  On the flip side, serverless can scale to 0. Makes it very cost effective for one off functions
  - ethanwillis 6y ago
    It doesn't actually scale to 0. The scale to 0 argument is actually just accounting trickery. Servers are still waiting around burning cycles doing nothing waiting to run a serverless function. All that has changed is that Amazon has spread the cost of low utilization across all customers. Which gets baked into your cost as well.
ErikAugust 6y ago
They are good for message queue handlers. But not good for production REST API endpoints in my experience. In the context of AWS Lambda, you are talking a 29 second timeout in API Gateway. API Gateway is fairly expensive. Also, is connection pooling a solved problem? Many times you can do better with an EC2 instance running PM2 and Express, or whatever. Like, what does Serverless Framework do locally anyway? Express? Why not just run Express?
- deleted 6y ago
  [deleted]
- mtnGoat 6y ago
  I have a production REST API endpoints backed by PHP on Lambda that are called 12k/sec on the high end. no problems since ~2017. The best part is, I never have to worry about scaling the underlying resources, if sales brings in large clients. Worst case is I ask AWS for a greater Lambda concurrency limit.
j4ah4n 6y ago
I totally agree with this. My only real complaint is managing these one off items. Team member 1 puts one on a VPC the other doesn’t, another team member ties one to RDS and doesn’t automate auth...etc. It’s great but you have to put management of usage in place first.
notanotherycomb 6y ago
I'm ignorant How does a build trigger the serverless function to notify all of you? I kind of assume serverless means it activates on a timer or when you visit a specific URL. So a build script executes it by visiting a URL when it's finished? And the script executes from any internet connected machine? Why is it better than having a VPS? I currently use a VPS for a few one off scripts. Cron does the timer ones and the URL ones are entries in a nginx config file. Actually that's half true I actually config an app I wrote to do it cause I didn't want to bother finding out how to do shellexecute on nginx
- notsureaboutpg 6y ago
  Serverless functions can be set on a timer exactly like cron jobs. But you can also set them to act on certain triggers, when a message comes into an SQS queue, when a notification goes to an SNS topic, etc. You can do the same with a VPS, but I think VPS cost more because you pay for all the time you're not using them. With serverless functions you only pay for execution time. And with one-off, daily, hourly jobs you end up paying far less, unless your jobs are compute intensive (cron jobs tend to not be...?) I have seen some VPS these days come in at like $2 a month though, so it's worth comparing the cost.
- erikerikson 6y ago
  The serverless framework is an open source tool that you can use to "deploy" "serverless" assets. Basically it covers most of those scripts you usually have to write and gives you a declarative format with some great variable capabilities that help you managed config and declare your components clearly. Running the tool deploys to your cloud of choice and results in functions, APIs, workflows, databases, or whatever you declare now existing and ready for use. That framework is separate from the class of services regarded as serverless. The star that kicked it off was AWS Lambda. Serverless colloquially means you are dealing with an abstract service contract rather than a server (e.g. the oldest is S3). This removes patching and other maintenance that usually does not directly support business value. More formally, serverless includes auto-scaling to zero, paying only for what you use, high availability, and other design patterns most outside the large tech houses cannot use at low to no cost. Your VPS is always on and always paid for. It can crash or get in a bad state. It can go out of date and need patching. It is mutable and more vulnerable. It is limited in resources. Nike reported AWS Lambda scaling in production at 20K RPS/S (0RPS@0s, 20K RPS@1s, 60K@2s, 120K@3s, ...). serverless-artillery, with loosened account "safety" limits can scale from nothing to producing billions of requests per second on target systems. Full disclosure: I contributed to the serverless framework and serverless-artillery. I'm a biased fanboy.
afrojack123 6y ago
A 2 man-team with a project manager. Man1: I'll work on your report. Man2: I'll watch you work on my report.
waheoo 6y ago
That sounds great but if I already have the ability to quickly prototype and stand up one off type services / endpoints in my code base what's the selling point here? To me it's just full of risk and costs where as adding a new function, getting it reviewed and deploying takes all of a few minutes with none of the cost.
x86_64Ubuntu 6y ago
Is anyone else worried about the proliferation of disjoint processes that people can whip up using Serverless? At least with a monolith, you've got a relatively coherent grouped together codebase. But with Serverless, you can have some lambda that is infinitely important, but not documented or well known to anyone else in the org.
- sudhirj 6y ago
  Serverless and microservices aren’t the same thing. You can deploy a monolith to lambda, and make a thousand small services on VMs or orchestration platforms. It’s purely a question of how to organise your code.
  - dtech 6y ago
    I'd say that is misleading at best. Lambdas are inherently unsuited for monoliths due to payload, RAM and execution time limits and cold start times. Lambdas are inherently suited to "real" micro-services, because by nature can only do 1 thing and have strict limits.
    - sudhirj 6y ago
      Depends entirely on workload - there's 3GB RAM available, which is more than what we use for normal monoliths on Heroku. Payload limits are 6MB or something, which isn't hit for a majority of sites on the internet (each request, not total page size). And there's nothing in the nature of Lambdas that force them to do only one thing. HTTP events can be fed through the normal router (lots of adapters for many web frameworks out there). Non-HTTP events can either go through a switch-case-board. There's no inherent "nature" to lambda, it'll execute whatever code you give it. There are limitations, but those are constantly changing.
- marmshallow 6y ago
  I think it all depends on how searchable and generally well-maintained the codebase is. Having some obscure critical function hidden in a monolith is totally possible too.
wokwokwok 6y ago
Sounds like its working for you, which is great, I guess. However in general what you're doing isn't devops, its 'devs creating a big old mess for someone else to cleanup later'. All you're doing is trading long term maintainability and quality for 'getting it done right now'. Sure; it works. I get it. ...but I've also been on the receiving end when that 2-person team has 1 person leave and gets scaled up to a 5-person team, and you have to a) bin everything, and b) write the entire thing from scratch. I personally consider it a very selfish way to build things. Me first. Me now. ...someone else's problem later. There are scalable solutions in this space (eg. pipelines, airflow, azure data factory, glue); doing it using lambdas is... not. ideal. That's not a problem with severless in general; but what you've described isn't any of the things which is good about serverless either: what you've described is the problem with the low barrier to entry with serverless, resulting in production serverless code that skips past 'quality control' to 'let's deploy straight to production!'.
- justinram11 6y ago
  While I think that there is definitely a worry of it turning into a big old mess (especially if they start depending on and talking to each other), I don't really know that I'd consider cron-job style independent one-off tasks any more maintainable in a monolith. I also don't know that I'd consider that specific to serverless (it's easy to create a ball of mud in anything). All of our serverless projects are located in a single repository, and each little service gets its own little google doc that explains what it does, what functions it has, and any resources that it uses. As mentioned below, as soon as an ETL gets multiple steps I agree that pure-lambda is not the right tool for the job (Airflow would be my preference, though we've also successfully used Step Functions)
- nafey 6y ago
  This assumes that survival is guaranteed if you do it the right way. Long term thinking can work in firms that are larger. For smaller firms with limited resources, motivation and technical expertise getting velocity is crucial. This is one of the reasons startups run into issues when they have to scale. I think it has to be a well thought out balance of speed and quality that is required in such environments.
thinkharderdev 6y ago
Right, this is where Lambda really does shine. The problem is when it becomes "everything we build has to be serverless"

Demiurge 6y ago

How is owning an AWS account easier than a cheap dedicates server? You should have a code repo and backups in either case. At least some people mention "but I can scale this lambda x1000" and that's one advantage... But you can do all those tasks on a hetzner server in same amount of type, just python scripts.

justinram11 6y ago
Mostly it's about the maintenance costs of owning and managing a server -- on Lambda most of these tools are "set and forget". I don't want to have to manage security updates, disk usage, or any of the other multiple things that come with owning a server. I also could probably debate on the friendly-ness of just dumping all of these one-off services onto a single server (seems like it could get messy pretty quick), but I think you could probably go either way on that. We are also an AWS shop for our "production" stack already, so there is no added overhead for owning an AWS account.
- chillfox 6y ago
  Servers can also be "set and forget", just enable automatic patching
mark242 6y ago
Because that "cheap dedicated server" isn't all that cheap. Just the documentation for backups on your Hetzner server is ridiculous. Why, in the 21st century, should I as an application developer ever have to worry about that? Let's say that Hetzner server goes completely toast at 2am and you get a pingdom message. You log in, realize you have to restore from backup. You fire up a new instance, and go through your restore procedure, and everything works great. You're lucky to be offline for an hour. I cannot even tell you what would happen if an underlying instance running a Lambda goes offline, because it never happens. But let's say that an underlying machine dies in us-east-1. Lambda just starts running more instances of your code on another underlying group of cpus, and you aren't offline.
- crehn 6y ago
  And I’d argue the mere possibility of the server breaking at any time contributes to non-negligible amounts of sustained psychological stress.
  - mark242 6y ago
    Yes, this exactly. The fact that I can guarantee that my code is running, and have entire massive teams at Amazon whose literal job title is to keep my code running, it is such a freeing feeling. I might be able to delete rows from Dynamo from my code messing up, but Dynamo will never go offline.
    - crehn 6y ago
      The psychological effects of technology seem often overlooked—and they’re hard to quantify. I wish there was some easier way to quantify the “soft” aspects of everything that surrounds us, e.g. the long-term impact of beautiful and usable UI design, the reduction in “ambient” psychological stress, the impacts of chaos/consistency on how we feel, any robust measures of happiness, excitement, relaxation, etc. They seem to be such important attributes, but not as easily measured as a latency. Which in turn makes reaching consensus harder—and probably why design by committee doesn't work?
konjin 6y ago
Because you need two servers. One to run the application. One to run the application while the first is being upgraded. Now you need three servers, one to run the application, one as backup and one as load balancer. 99.99% uptime means you can be down no more than an hour a year. Which is really easy to overshoot when you're dicking around with a dist upgrade on three linux boxes. So that's why you end up with a server farm starting from your one small server.
- akkartik 6y ago
  Notice that grandparent comment explicitly carved out production servers. If you can't tolerate downtime while you're being upgraded, you're production.
- franciscop 6y ago
  99.99% of the websites out there don't need 99.99% uptime (from a business point of view)
  - klodolph 6y ago
    Once you get to 99.9% uptime you’re still less than 1 hour per month. I used to run apps for companies at work and it was standard operating procedure to have two app servers + load balancer for upgrades. Drain a server, upgrade it, boot it, check that it works, let it accept traffic. Repeat once more. Done. It may be true that most businesses don’t need the uptime, but I’d say it’s usually easier to give them high availability rather than field complaints when some random user tries to look something up during a maintenance window.
    - vanviegen 6y ago
      But where does the data sit? Would you be using some fancy multi-master synchronous replication for that, or is there a database server behind it?
      - klodolph 6y ago
shotgunsparkle 6y ago
On the flip side, serverless can scale to 0. Makes it very cost effective for one off functions
- ethanwillis 6y ago
  It doesn't actually scale to 0. The scale to 0 argument is actually just accounting trickery. Servers are still waiting around burning cycles doing nothing waiting to run a serverless function. All that has changed is that Amazon has spread the cost of low utilization across all customers. Which gets baked into your cost as well.

ErikAugust 6y ago

They are good for message queue handlers. But not good for production REST API endpoints in my experience. In the context of AWS Lambda, you are talking a 29 second timeout in API Gateway. API Gateway is fairly expensive. Also, is connection pooling a solved problem? Many times you can do better with an EC2 instance running PM2 and Express, or whatever. Like, what does Serverless Framework do locally anyway? Express? Why not just run Express?

deleted 6y ago
[deleted]
mtnGoat 6y ago
I have a production REST API endpoints backed by PHP on Lambda that are called 12k/sec on the high end. no problems since ~2017. The best part is, I never have to worry about scaling the underlying resources, if sales brings in large clients. Worst case is I ask AWS for a greater Lambda concurrency limit.

j4ah4n 6y ago

I totally agree with this. My only real complaint is managing these one off items. Team member 1 puts one on a VPC the other doesn’t, another team member ties one to RDS and doesn’t automate auth...etc. It’s great but you have to put management of usage in place first.

notanotherycomb 6y ago

I'm ignorant How does a build trigger the serverless function to notify all of you? I kind of assume serverless means it activates on a timer or when you visit a specific URL. So a build script executes it by visiting a URL when it's finished? And the script executes from any internet connected machine? Why is it better than having a VPS? I currently use a VPS for a few one off scripts. Cron does the timer ones and the URL ones are entries in a nginx config file. Actually that's half true I actually config an app I wrote to do it cause I didn't want to bother finding out how to do shellexecute on nginx

notsureaboutpg 6y ago
Serverless functions can be set on a timer exactly like cron jobs. But you can also set them to act on certain triggers, when a message comes into an SQS queue, when a notification goes to an SNS topic, etc. You can do the same with a VPS, but I think VPS cost more because you pay for all the time you're not using them. With serverless functions you only pay for execution time. And with one-off, daily, hourly jobs you end up paying far less, unless your jobs are compute intensive (cron jobs tend to not be...?) I have seen some VPS these days come in at like $2 a month though, so it's worth comparing the cost.
erikerikson 6y ago
The serverless framework is an open source tool that you can use to "deploy" "serverless" assets. Basically it covers most of those scripts you usually have to write and gives you a declarative format with some great variable capabilities that help you managed config and declare your components clearly. Running the tool deploys to your cloud of choice and results in functions, APIs, workflows, databases, or whatever you declare now existing and ready for use. That framework is separate from the class of services regarded as serverless. The star that kicked it off was AWS Lambda. Serverless colloquially means you are dealing with an abstract service contract rather than a server (e.g. the oldest is S3). This removes patching and other maintenance that usually does not directly support business value. More formally, serverless includes auto-scaling to zero, paying only for what you use, high availability, and other design patterns most outside the large tech houses cannot use at low to no cost. Your VPS is always on and always paid for. It can crash or get in a bad state. It can go out of date and need patching. It is mutable and more vulnerable. It is limited in resources. Nike reported AWS Lambda scaling in production at 20K RPS/S (0RPS@0s, 20K RPS@1s, 60K@2s, 120K@3s, ...). serverless-artillery, with loosened account "safety" limits can scale from nothing to producing billions of requests per second on target systems. Full disclosure: I contributed to the serverless framework and serverless-artillery. I'm a biased fanboy.

afrojack123 6y ago

A 2 man-team with a project manager. Man1: I'll work on your report. Man2: I'll watch you work on my report.

waheoo 6y ago

That sounds great but if I already have the ability to quickly prototype and stand up one off type services / endpoints in my code base what's the selling point here? To me it's just full of risk and costs where as adding a new function, getting it reviewed and deploying takes all of a few minutes with none of the cost.

x86_64Ubuntu 6y ago

Is anyone else worried about the proliferation of disjoint processes that people can whip up using Serverless? At least with a monolith, you've got a relatively coherent grouped together codebase. But with Serverless, you can have some lambda that is infinitely important, but not documented or well known to anyone else in the org.

sudhirj 6y ago
Serverless and microservices aren’t the same thing. You can deploy a monolith to lambda, and make a thousand small services on VMs or orchestration platforms. It’s purely a question of how to organise your code.
- dtech 6y ago
  I'd say that is misleading at best. Lambdas are inherently unsuited for monoliths due to payload, RAM and execution time limits and cold start times. Lambdas are inherently suited to "real" micro-services, because by nature can only do 1 thing and have strict limits.
  - sudhirj 6y ago
    Depends entirely on workload - there's 3GB RAM available, which is more than what we use for normal monoliths on Heroku. Payload limits are 6MB or something, which isn't hit for a majority of sites on the internet (each request, not total page size). And there's nothing in the nature of Lambdas that force them to do only one thing. HTTP events can be fed through the normal router (lots of adapters for many web frameworks out there). Non-HTTP events can either go through a switch-case-board. There's no inherent "nature" to lambda, it'll execute whatever code you give it. There are limitations, but those are constantly changing.
marmshallow 6y ago
I think it all depends on how searchable and generally well-maintained the codebase is. Having some obscure critical function hidden in a monolith is totally possible too.

wokwokwok 6y ago

Sounds like its working for you, which is great, I guess. However in general what you're doing isn't devops, its 'devs creating a big old mess for someone else to cleanup later'. All you're doing is trading long term maintainability and quality for 'getting it done right now'. Sure; it works. I get it. ...but I've also been on the receiving end when that 2-person team has 1 person leave and gets scaled up to a 5-person team, and you have to a) bin everything, and b) write the entire thing from scratch. I personally consider it a very selfish way to build things. Me first. Me now. ...someone else's problem later. There are scalable solutions in this space (eg. pipelines, airflow, azure data factory, glue); doing it using lambdas is... not. ideal. That's not a problem with severless in general; but what you've described isn't any of the things which is good about serverless either: what you've described is the problem with the low barrier to entry with serverless, resulting in production serverless code that skips past 'quality control' to 'let's deploy straight to production!'.

justinram11 6y ago
While I think that there is definitely a worry of it turning into a big old mess (especially if they start depending on and talking to each other), I don't really know that I'd consider cron-job style independent one-off tasks any more maintainable in a monolith. I also don't know that I'd consider that specific to serverless (it's easy to create a ball of mud in anything). All of our serverless projects are located in a single repository, and each little service gets its own little google doc that explains what it does, what functions it has, and any resources that it uses. As mentioned below, as soon as an ETL gets multiple steps I agree that pure-lambda is not the right tool for the job (Airflow would be my preference, though we've also successfully used Step Functions)
nafey 6y ago
This assumes that survival is guaranteed if you do it the right way. Long term thinking can work in firms that are larger. For smaller firms with limited resources, motivation and technical expertise getting velocity is crucial. This is one of the reasons startups run into issues when they have to scale. I think it has to be a well thought out balance of speed and quality that is required in such environments.

thinkharderdev 6y ago

Right, this is where Lambda really does shine. The problem is when it becomes "everything we build has to be serverless"