7 ms·

Scaling to 15000 Functions and Beyond

71 points by alexellisuk 3y ago

GiorgioG 3y ago
Or stop chasing the latest fad(s) and build a regular old boring service. The dev experience for building/debugging/troubleshooting distributed functions/lambdas is atrocious. Use boring tech, solve business problems and create value without adding needless complexity because devs want to play with the latest toys.
- SteveNuts 3y ago
  The only time I've ever seen this type of thing put to good use is for gluing things together that don't have a native integration. And even then you better have the configuration defined in code because otherwise you'll spend an entire afternoon reverse engineering how it works.
  - activescott 3y ago
    "you better have the configuration defined in code" - you better do that anyway for anything that is used and will be around for a while :)
    - SteveNuts 3y ago
      100% agree, however the difference between one EC2 instance that was launched manually vs. all of the IAM role mess and configuration needed for a lambda... well let's just say I'd rather have an undocumented EC2 instance.
- omneity 3y ago
  I want a language/runtime that abstracts this away from me. I'll implement and debug functions the regular way, with the runtime in charge of deploying them whichever way fits. Bonus points for deducing infrastructure from statically analyzing code dependency paths. I believe there is very early stage work [0] in this direction, but obviously nothing going mainstream yet. Would love to find more examples in the wild if anyone knows any! [0]: https://egghead.io/blog/how-to-quickly-build-a-backend-with-dark-lang https://egghead.io/blog/how-to-quickly-build-a-backend-with-...
  - jiggawatts 3y ago
    I love how you responded to a post advocating for simpler architectures by linking to an automagic abstraction over microservices, because microservices weren't difficult enough already. The example also mentions Stripe payments, but nowhere does it mention transactions or error recovery because they're stupidly difficult in microservices designs, so we'll just "skip right over that" and go straight to losing money any time there's a hiccup. "Sure it's incomplete and outright dangerous to our bottom line, but I can't wait to tinker with it!" is precisely the problem with the industry.
    - omneity 3y ago
      I mean, I do agree with the parent, insofar as I don't want to deal with the mess of microservices/functions firsthand. Now if a runtime could reap the benefits of this architecture automatically, sort of like a compiler would optimize code for a particular cpu, then that would be a best-of-both-worlds situation.
      - michaelmrose 3y ago
        If you can't run anything remotely like the code you are actually running how will you ever figure out why it isn't doing what you expect?
    - threeseed 3y ago
      The problem with your argument is that microservices and lambdas aren’t new. The industry has had plenty of opportunities to ditch them and go back to monolith services and yet it hasn’t happened. Is it because everyone is wrong except for you ? Or is it because they are better for certain types of use cases.
  - hu3 3y ago
    Perhaps erlang/OTP might tackle some of these challenges of simpler scaling. https://www.erlang.org/doc/design_principles/des_princ https://www.erlang.org/doc/design_principles/des_princ WhatsApp famously scaled a lot with relatively few devs: > WhatsApp scaled to 1B users with only 50 engineers https://news.ycombinator.com/item?id=28985169 https://news.ycombinator.com/item?id=28985169
- deleted 3y ago
  [deleted]
- threeseed 3y ago
  If we are all giving out advice let me join in. It’s not wise to dismiss the technical decisions made by developers as “playing with the latest toys”. Because you are ignorant. You have no understanding of the business requirements, NFRs etc that have led to the decision. So I prefer to give developers the benefit of the doubt. Lambda architectures have trade-offs like every architecture and they are useful in certain situations. Building everything in one service and scaling it when the components have different scaling behaviours usually isn’t ideal.
  - hinkley 3y ago
    > The dev experience for building/debugging/troubleshooting distributed functions/lambdas is atrocious. Address this part, or you’re just deflecting. Comments? Suggestions?
    - threeseed 3y ago
      Lambdas trade developer experience for operational simplicity. And they suit use cases where they are independent from each other and can be tested in isolation eg transcoding. In the early days of AWS Lambda I saw teams trying to apply older techniques to lambdas and get stuck with things like distributed transactions but industry has learnt a lot since then. So as always right tool for the right job.
      - hinkley 3y ago
        If it doesn’t work for the developers, then eventually it won’t work for anybody. Trading developer experience for just about anything does not scale - up or down. It’s a false economy.
        threeseed 3y ago
        And if it can’t scale and run reliably in production then it won’t work at all. Writing some code and throwing it over the fence to some Ops person to manage is not workable long term. It’s why DevOps exist because developers have to take some responsibility for operating their own code. And Lambda architectures help with this.
- jmspring 3y ago
  What do you think are some specific gaps around distributed functions/lambdas?
- jthoaawef 3y ago
  I like FaaS for simple things * 0..N Scaling * Request/Invocation is unit of compute * Deploy anywhere * Don't need to bring a web server * Instant Observability (distributed tracing, metrics, logs, alerts, & dashboards) * Instant Security (AuthN/AuthZ/mTLS) * Instant Cost Monitoring How good "FaaS" is will be highly dependent on how good your ops team is.
- antoniojtorres 3y ago
  These types of grievance comments have really found a permanent home in certain types of HN post. Exhausting.
- latchkey 3y ago
  > The dev experience for building/debugging/troubleshooting distributed functions/lambdas is atrocious. From my own personal experience, Google Cloud Functions, and everything around it, is fantastic. Had 25k "worker" computers hitting GCF 24/7 (~50 rps) reporting their status. New status message went onto Cloud Tasks, which just hits GCF again, and stored into a Cloud SQL Postgres. Cost was about $200/mo. It ran perfectly with zero downtime for over a year. Super easy to develop, test, and deployments were done with github actions. Very little actual code is dependent on anything GC, so vendor lockin was minimal.
activescott 3y ago
Your relentless customer service is commendable. Good work. Serverless works well in many scenarios and being able to run it on any k8s cluster without proprietary platform hooks is a good thing for all of us to have an option for.
latchkey 3y ago
It is hard for me to understand why anyone would build so many individual functions. Just build one function, put a router in it and then call out to as many individual handlers as you want. Then you're just deploying a single function to the PaaS and everything is simple.
- report-to-trees 3y ago
  In that case you are deploying a whole monolithic app to the Faas platform. This needs to be initialized on every cold start which takes longer the larger your code is. The main selling point (that I have experienced) in Faas is that you can have small, individually scalable functions that are quick to initialize and can be torn down after executing. If you have the whole app under one handler you would be much better off with a different architecture where the app is persistent and always running.
  - paulddraper 3y ago
    That seems like a weird reason. You can lazy load your route implementations
    - report-to-trees 3y ago
      If you manage to lazy load your routes it may help with cold starts but I think you effectively lose the benefit of warm handlers since every request could use new code that needs to be loaded fresh. I've never attempted lazy loading inside a Faas platform but I think generally you would be better off creating small deployable chunks and letting the platform handle scaling them up / reusing as necessary. Less fighting against the platform and if you have to maintain strict boundaries to enable lazy loading only the needed code for each route it's a short step to deploying those as their own functions anyway.
      - latchkey 3y ago
        The article is about 15k functions. You know what is a lot worse than lazy loading? Getting 15k functions to deploy accurately. I've seem functions fail to deploy, are you going to fail the whole build if one of the 15k fails? What if functions depend on other functions? How about CI/CD where you're updating multiple times a day? How about when you update a single dependency across those 15k and you need to deploy again? I don't know what PaaS you've used, but at least with Google Cloud Functions, you can do a minimum number of instances. Set it to 1 and you never have cold start issues.
withinboredom 3y ago
Can I just say I was shocked to see a "Pricing" page on something called "openX." Then, on the pricing page, there was no price... so, now I'm just confused. I'm also trying to understand what this 15k functions post is all about, since you'd need an enterprise license to use that many. Whatever that means.
- benjaminwootton 3y ago
  Open core and support packages are very established business models. The OP has also hacked away on OpenFaas in the open for years, giving us a portable alternative to lambda. That seems to have a pretty big value prop considering how lambda ends up being fairly expensive at scale. Though I don’t know their motivation for hiding pricing, I suspect every enterprise level deployment (for instance into an ISP) is pretty bespoke.
  - verdverm 3y ago
    I had a bad experience with the project's authors pricing (not openfaas), lost respect for Alex after that.
    - mardifoufs 3y ago
      What do you mean? What type of issues ?
      - verdverm 3y ago
        scam like, over priced, under delivered, seriously
  - danw1979 3y ago
    > Open core and support packages are very established business models. and “bespoke” pricing for such models is also pretty common. doesn’t mean we have to like it. > That seems to have a pretty big value prop considering how lambda ends up being fairly expensive at scale. Portable is valuable, yes, but at what price ? Does anyone know what the price is ? If it’s customer-specific it’s impossible to say that this proposition is generally competitive with lambda. Open core / commercially support products need to have published pricing for at least a standardised offering IMHO. If I need to get on a call with a salesperson to find that out (so I can do the back-of-the-envelope calculation that says whether or not this thing will cost in) it isn’t happening…
ganzuul 3y ago
With about 20 000 genes to make a human... What are you making that requires 15 000 functions?
- gherkinnn 3y ago
  Horizontally scaleable leftPad. Think of the opportunities. Text can be shaped as en endofunctor, enabling massive parallelisation.

71 points by alexellisuk 3y ago

Or stop chasing the latest fad(s) and build a regular old boring service. The dev experience for building/debugging/troubleshooting distributed functions/lambdas is atrocious. Use boring tech, solve business problems and create value without adding needless complexity because devs want to play with the latest toys.

SteveNuts 3y ago
The only time I've ever seen this type of thing put to good use is for gluing things together that don't have a native integration. And even then you better have the configuration defined in code because otherwise you'll spend an entire afternoon reverse engineering how it works.
- activescott 3y ago
  "you better have the configuration defined in code" - you better do that anyway for anything that is used and will be around for a while :)
  - SteveNuts 3y ago
    100% agree, however the difference between one EC2 instance that was launched manually vs. all of the IAM role mess and configuration needed for a lambda... well let's just say I'd rather have an undocumented EC2 instance.
omneity 3y ago
I want a language/runtime that abstracts this away from me. I'll implement and debug functions the regular way, with the runtime in charge of deploying them whichever way fits. Bonus points for deducing infrastructure from statically analyzing code dependency paths. I believe there is very early stage work [0] in this direction, but obviously nothing going mainstream yet. Would love to find more examples in the wild if anyone knows any! [0]: https://egghead.io/blog/how-to-quickly-build-a-backend-with-dark-lang https://egghead.io/blog/how-to-quickly-build-a-backend-with-...
- jiggawatts 3y ago
  I love how you responded to a post advocating for simpler architectures by linking to an automagic abstraction over microservices, because microservices weren't difficult enough already. The example also mentions Stripe payments, but nowhere does it mention transactions or error recovery because they're stupidly difficult in microservices designs, so we'll just "skip right over that" and go straight to losing money any time there's a hiccup. "Sure it's incomplete and outright dangerous to our bottom line, but I can't wait to tinker with it!" is precisely the problem with the industry.
  - omneity 3y ago
    I mean, I do agree with the parent, insofar as I don't want to deal with the mess of microservices/functions firsthand. Now if a runtime could reap the benefits of this architecture automatically, sort of like a compiler would optimize code for a particular cpu, then that would be a best-of-both-worlds situation.
    - michaelmrose 3y ago
      If you can't run anything remotely like the code you are actually running how will you ever figure out why it isn't doing what you expect?
  - threeseed 3y ago
    The problem with your argument is that microservices and lambdas aren’t new. The industry has had plenty of opportunities to ditch them and go back to monolith services and yet it hasn’t happened. Is it because everyone is wrong except for you ? Or is it because they are better for certain types of use cases.
- hu3 3y ago
  Perhaps erlang/OTP might tackle some of these challenges of simpler scaling. https://www.erlang.org/doc/design_principles/des_princ https://www.erlang.org/doc/design_principles/des_princ WhatsApp famously scaled a lot with relatively few devs: > WhatsApp scaled to 1B users with only 50 engineers https://news.ycombinator.com/item?id=28985169 https://news.ycombinator.com/item?id=28985169
deleted 3y ago
[deleted]
threeseed 3y ago
If we are all giving out advice let me join in. It’s not wise to dismiss the technical decisions made by developers as “playing with the latest toys”. Because you are ignorant. You have no understanding of the business requirements, NFRs etc that have led to the decision. So I prefer to give developers the benefit of the doubt. Lambda architectures have trade-offs like every architecture and they are useful in certain situations. Building everything in one service and scaling it when the components have different scaling behaviours usually isn’t ideal.
- hinkley 3y ago
  > The dev experience for building/debugging/troubleshooting distributed functions/lambdas is atrocious. Address this part, or you’re just deflecting. Comments? Suggestions?
  - threeseed 3y ago
    Lambdas trade developer experience for operational simplicity. And they suit use cases where they are independent from each other and can be tested in isolation eg transcoding. In the early days of AWS Lambda I saw teams trying to apply older techniques to lambdas and get stuck with things like distributed transactions but industry has learnt a lot since then. So as always right tool for the right job.
    - hinkley 3y ago
      If it doesn’t work for the developers, then eventually it won’t work for anybody. Trading developer experience for just about anything does not scale - up or down. It’s a false economy.
      - threeseed 3y ago
        And if it can’t scale and run reliably in production then it won’t work at all. Writing some code and throwing it over the fence to some Ops person to manage is not workable long term. It’s why DevOps exist because developers have to take some responsibility for operating their own code. And Lambda architectures help with this.
jmspring 3y ago
What do you think are some specific gaps around distributed functions/lambdas?
jthoaawef 3y ago
I like FaaS for simple things * 0..N Scaling * Request/Invocation is unit of compute * Deploy anywhere * Don't need to bring a web server * Instant Observability (distributed tracing, metrics, logs, alerts, & dashboards) * Instant Security (AuthN/AuthZ/mTLS) * Instant Cost Monitoring How good "FaaS" is will be highly dependent on how good your ops team is.
antoniojtorres 3y ago
These types of grievance comments have really found a permanent home in certain types of HN post. Exhausting.
latchkey 3y ago
> The dev experience for building/debugging/troubleshooting distributed functions/lambdas is atrocious. From my own personal experience, Google Cloud Functions, and everything around it, is fantastic. Had 25k "worker" computers hitting GCF 24/7 (~50 rps) reporting their status. New status message went onto Cloud Tasks, which just hits GCF again, and stored into a Cloud SQL Postgres. Cost was about $200/mo. It ran perfectly with zero downtime for over a year. Super easy to develop, test, and deployments were done with github actions. Very little actual code is dependent on anything GC, so vendor lockin was minimal.

activescott 3y ago

Your relentless customer service is commendable. Good work. Serverless works well in many scenarios and being able to run it on any k8s cluster without proprietary platform hooks is a good thing for all of us to have an option for.

latchkey 3y ago

It is hard for me to understand why anyone would build so many individual functions. Just build one function, put a router in it and then call out to as many individual handlers as you want. Then you're just deploying a single function to the PaaS and everything is simple.

report-to-trees 3y ago
In that case you are deploying a whole monolithic app to the Faas platform. This needs to be initialized on every cold start which takes longer the larger your code is. The main selling point (that I have experienced) in Faas is that you can have small, individually scalable functions that are quick to initialize and can be torn down after executing. If you have the whole app under one handler you would be much better off with a different architecture where the app is persistent and always running.
- paulddraper 3y ago
  That seems like a weird reason. You can lazy load your route implementations
  - report-to-trees 3y ago
    If you manage to lazy load your routes it may help with cold starts but I think you effectively lose the benefit of warm handlers since every request could use new code that needs to be loaded fresh. I've never attempted lazy loading inside a Faas platform but I think generally you would be better off creating small deployable chunks and letting the platform handle scaling them up / reusing as necessary. Less fighting against the platform and if you have to maintain strict boundaries to enable lazy loading only the needed code for each route it's a short step to deploying those as their own functions anyway.
    - latchkey 3y ago
      The article is about 15k functions. You know what is a lot worse than lazy loading? Getting 15k functions to deploy accurately. I've seem functions fail to deploy, are you going to fail the whole build if one of the 15k fails? What if functions depend on other functions? How about CI/CD where you're updating multiple times a day? How about when you update a single dependency across those 15k and you need to deploy again? I don't know what PaaS you've used, but at least with Google Cloud Functions, you can do a minimum number of instances. Set it to 1 and you never have cold start issues.

withinboredom 3y ago

Can I just say I was shocked to see a "Pricing" page on something called "openX." Then, on the pricing page, there was no price... so, now I'm just confused. I'm also trying to understand what this 15k functions post is all about, since you'd need an enterprise license to use that many. Whatever that means.

benjaminwootton 3y ago
Open core and support packages are very established business models. The OP has also hacked away on OpenFaas in the open for years, giving us a portable alternative to lambda. That seems to have a pretty big value prop considering how lambda ends up being fairly expensive at scale. Though I don’t know their motivation for hiding pricing, I suspect every enterprise level deployment (for instance into an ISP) is pretty bespoke.
- verdverm 3y ago
  I had a bad experience with the project's authors pricing (not openfaas), lost respect for Alex after that.
  - mardifoufs 3y ago
    What do you mean? What type of issues ?
    - verdverm 3y ago
      scam like, over priced, under delivered, seriously
- danw1979 3y ago
  > Open core and support packages are very established business models. and “bespoke” pricing for such models is also pretty common. doesn’t mean we have to like it. > That seems to have a pretty big value prop considering how lambda ends up being fairly expensive at scale. Portable is valuable, yes, but at what price ? Does anyone know what the price is ? If it’s customer-specific it’s impossible to say that this proposition is generally competitive with lambda. Open core / commercially support products need to have published pricing for at least a standardised offering IMHO. If I need to get on a call with a salesperson to find that out (so I can do the back-of-the-envelope calculation that says whether or not this thing will cost in) it isn’t happening…

ganzuul 3y ago

With about 20 000 genes to make a human... What are you making that requires 15 000 functions?

gherkinnn 3y ago
Horizontally scaleable leftPad. Think of the opportunities. Text can be shaped as en endofunctor, enabling massive parallelisation.