9 ms·

For what its worth, ALB is a fantastic product. As a full L7 load balancer it can stand in the place of nginx, provide OIDC auth, replace API gateway especially

For what its worth, ALB is a fantastic product. As a full L7 load balancer it can stand in the place of nginx, provide OIDC auth, replace API gateway especially for high volume Lambdas, and has lots of tunable logic for running diverse auto-scaled workloads. AWS is good about not breaking APIs and contracts, so the "strangler" strategy is really about accommodating existing customers. I'm not going to say Classic Load Balancers are a dog of a product or anything, its just that they pre-date VPC, and the web stack has moved on since then. Having ALB be such a robust, turn-key product does tempt one into giving it a lot of responsibilities which definitely lends itself towards lock-in. I guess its the same with marriage: there are always tradeoffs. I do wish it was better integrated with ECS + AppMesh + API Gateway, so it could be like a managed GetAmbassador.io. Envoy proxies are a hell of a good idea, and I think are going to be something like the React of the backend. While one can dream about a grand unified request router for the cloud, ALB continues to innovate with things like canary deployments and awesome routing rules, all well continuing to 'just work' at web scale. Is there anything else you can throw up to 50K RPS at and not have to think about it that much?

philliphaydon 7y ago
One thing is that ALB to Lambda restrictions suck. Lambdas have a 6mb response size. Api gateway has a 10mb response size. If you go ALB to Lambda. And your response is over 1mb. Your response will fail. Because the ALB team decided to introduce a random 1mb limit, not document it as a limitation, and added it to their trouble shooting notes instead. So if you need to return a small PDF or something from a lambda via ALB. Good luck.
- jcims 7y ago
  Somebody has a war story
  - james_s_tayler 7y ago
    War stories are the best part of HN.
- internalthief 7y ago
  It is documented, at least on this page: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html https://docs.aws.amazon.com/elasticloadbalancing/latest/appl... Right at the top of the page, underneath the header "Limits" > The maximum size of the response JSON that the Lambda function can send is 1 MB.
  - philliphaydon 7y ago
    Ah this must be relatively new. The only documentation I could find or AWS Support Business Support / Lambda team could give me was a link to: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html#target-http-errors https://docs.aws.amazon.com/elasticloadbalancing/latest/appl... Which I complained about because it wasn't mentioned as lambda as a target docs. I guess they amended it.
    - jk563 7y ago
      git blame suggests the change was made 15 months ago, though that doesn't take it to account time to publish. Which I guess just goes to show that it can be tough to find information on the docs, despite them been relatively decent as far as docs go (in my opinion).
- jedberg 7y ago
  FYI for anyone reading, the expected architecture is to return the URL to an S3 object that is your larger than 1mb response. The reason they do this is to keep the alb running efficiently by not having to architect for random large responses.
  - philliphaydon 7y ago
    The problem I have is, is the limit is less than the limit imposed by Lambda itself. Which is less than the limit imposed by API Gateway. API Gateway: 10mb Lambda: 6mb ALB: 1mb Yet if your target is an ec2 instance, there's no limit on the load balancer. So IMO the limit for a lambda target should be the limit imposed by Lambda itself. 6mb.
    - sudhirj 7y ago
      While I'm not disputing the limit numbers or whatever hardship they might cause, it's worth noting that this is a basic limitation of the original Lambda model and maybe FaaS in general. The capability comes from running a giant pseudo-infinite mesh of isolated execution environments that load your code and execute on demand, while having to buffer both the request and response to make sure clients are protected from the details. This buffering means that size of the buffer will always be limited - the team managing might make the buffers bigger based on experience, but it's not a solved problem. ALB to containers or servers is a different beast - here the entire request and response need not be buffered at all (there might still be a very small buffer, mostly negligible), so streaming responses, websockets etc become possible. We use lambda to resize images, so we do push against these limits a bit, but it's a fair tradeoff for the advantages - no worries about CPU throttling from too many requests, no waiting for servers to start for spiky loads etc.
      - inopinatus 7y ago
        Lambda was not designed for request/response. It’s an event driven service. Wrapping API gateway around it is an architectural blunder, and leads to folks like the GP wondering why their use case is a shitty fit.
- fulafel 7y ago
  That's pretty damning, lots of innocent REST messages can be more than 1MB. Can you have larger bodies in chunked encoding?
CoffeeDregs 7y ago
Didn't know that ALBs could front Lambdas... Will be so happy to be able to be rid of APIGW...
- reilly3000 7y ago
  At the volume I was running APIGW I saved 91% in cost moving to ALB.
fosk 7y ago
If you need more capabilities you can also frontend your Lambdas with something like Kong[1]. ALBs and AWS API Gateway can be quite slow and constrained if performance - and a customizable feature set - are important requirements. [1] https://github.com/Kong/kong https://github.com/Kong/kong
sandGorgon 7y ago
can a Envoy be the ingress replacing nginx or traefik ? this is being debated in the k3s distribution - https://github.com/rancher/k3s/issues/817 https://github.com/rancher/k3s/issues/817 will something like https://www.getambassador.io/reference/core/ingress-controller/ https://www.getambassador.io/reference/core/ingress-controll... , support all the usecases that a nginx ingress can ? (old certificates, spdy, etc etc etc)
- reilly3000 7y ago
  https://www.envoyproxy.io/learn/front-proxy https://www.envoyproxy.io/learn/front-proxy outlines the use case for an Envoy proxy being a general purpose ingress, with added routing and observability features. Honestly I can't speak competently to what Nginx and Nginx Plus/Pro are capable of these days, especially in relation to the sidecar proxy paradigm, but I know its much more than I've ever used in practice.
watermelon0 7y ago
It's a great product, but it still imposes limitations on header sizes. It has limitations on the size of the entire header, size of each header line, and size of the request line (i.e. URL length). It's quite unfortunate, since this means that some use cases are limited to the classic ELB. [1] https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html#http-header-limits https://docs.aws.amazon.com/elasticloadbalancing/latest/user...
greyskull 7y ago
Please chime in on https://github.com/aws/aws-app-mesh-roadmap/issues/37 https://github.com/aws/aws-app-mesh-roadmap/issues/37 if you haven't already

philliphaydon 7y ago

One thing is that ALB to Lambda restrictions suck. Lambdas have a 6mb response size. Api gateway has a 10mb response size. If you go ALB to Lambda. And your response is over 1mb. Your response will fail. Because the ALB team decided to introduce a random 1mb limit, not document it as a limitation, and added it to their trouble shooting notes instead. So if you need to return a small PDF or something from a lambda via ALB. Good luck.

jcims 7y ago
Somebody has a war story
- james_s_tayler 7y ago
  War stories are the best part of HN.
internalthief 7y ago
It is documented, at least on this page: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html https://docs.aws.amazon.com/elasticloadbalancing/latest/appl... Right at the top of the page, underneath the header "Limits" > The maximum size of the response JSON that the Lambda function can send is 1 MB.
- philliphaydon 7y ago
  Ah this must be relatively new. The only documentation I could find or AWS Support Business Support / Lambda team could give me was a link to: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html#target-http-errors https://docs.aws.amazon.com/elasticloadbalancing/latest/appl... Which I complained about because it wasn't mentioned as lambda as a target docs. I guess they amended it.
  - jk563 7y ago
    git blame suggests the change was made 15 months ago, though that doesn't take it to account time to publish. Which I guess just goes to show that it can be tough to find information on the docs, despite them been relatively decent as far as docs go (in my opinion).
jedberg 7y ago
FYI for anyone reading, the expected architecture is to return the URL to an S3 object that is your larger than 1mb response. The reason they do this is to keep the alb running efficiently by not having to architect for random large responses.
- philliphaydon 7y ago
  The problem I have is, is the limit is less than the limit imposed by Lambda itself. Which is less than the limit imposed by API Gateway. API Gateway: 10mb Lambda: 6mb ALB: 1mb Yet if your target is an ec2 instance, there's no limit on the load balancer. So IMO the limit for a lambda target should be the limit imposed by Lambda itself. 6mb.
  - sudhirj 7y ago
    While I'm not disputing the limit numbers or whatever hardship they might cause, it's worth noting that this is a basic limitation of the original Lambda model and maybe FaaS in general. The capability comes from running a giant pseudo-infinite mesh of isolated execution environments that load your code and execute on demand, while having to buffer both the request and response to make sure clients are protected from the details. This buffering means that size of the buffer will always be limited - the team managing might make the buffers bigger based on experience, but it's not a solved problem. ALB to containers or servers is a different beast - here the entire request and response need not be buffered at all (there might still be a very small buffer, mostly negligible), so streaming responses, websockets etc become possible. We use lambda to resize images, so we do push against these limits a bit, but it's a fair tradeoff for the advantages - no worries about CPU throttling from too many requests, no waiting for servers to start for spiky loads etc.
    - inopinatus 7y ago
      Lambda was not designed for request/response. It’s an event driven service. Wrapping API gateway around it is an architectural blunder, and leads to folks like the GP wondering why their use case is a shitty fit.
fulafel 7y ago
That's pretty damning, lots of innocent REST messages can be more than 1MB. Can you have larger bodies in chunked encoding?

CoffeeDregs 7y ago

Didn't know that ALBs could front Lambdas... Will be so happy to be able to be rid of APIGW...

reilly3000 7y ago
At the volume I was running APIGW I saved 91% in cost moving to ALB.

fosk 7y ago

If you need more capabilities you can also frontend your Lambdas with something like Kong[1]. ALBs and AWS API Gateway can be quite slow and constrained if performance - and a customizable feature set - are important requirements. [1] https://github.com/Kong/kong https://github.com/Kong/kong

sandGorgon 7y ago

can a Envoy be the ingress replacing nginx or traefik ? this is being debated in the k3s distribution - https://github.com/rancher/k3s/issues/817 https://github.com/rancher/k3s/issues/817 will something like https://www.getambassador.io/reference/core/ingress-controller/ https://www.getambassador.io/reference/core/ingress-controll... , support all the usecases that a nginx ingress can ? (old certificates, spdy, etc etc etc)

reilly3000 7y ago
https://www.envoyproxy.io/learn/front-proxy https://www.envoyproxy.io/learn/front-proxy outlines the use case for an Envoy proxy being a general purpose ingress, with added routing and observability features. Honestly I can't speak competently to what Nginx and Nginx Plus/Pro are capable of these days, especially in relation to the sidecar proxy paradigm, but I know its much more than I've ever used in practice.

watermelon0 7y ago

It's a great product, but it still imposes limitations on header sizes. It has limitations on the size of the entire header, size of each header line, and size of the request line (i.e. URL length). It's quite unfortunate, since this means that some use cases are limited to the classic ELB. [1] https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html#http-header-limits https://docs.aws.amazon.com/elasticloadbalancing/latest/user...

greyskull 7y ago

Please chime in on https://github.com/aws/aws-app-mesh-roadmap/issues/37 https://github.com/aws/aws-app-mesh-roadmap/issues/37 if you haven't already