4 ms·

This is a great article, and sums up my team's frustrations with AWS Lambda and the AWS engineering leadership in general. I worked in the past under a slew of

by JPKab 4y ago

This is a great article, and sums up my team's frustrations with AWS Lambda and the AWS engineering leadership in general. I worked in the past under a slew of former AWS product managers, and I know from them how the AWS roadmap works: 1.) Real, crucial customer feedback is incorporated into roadmap, usually for the core services that are already used extensively like Lambda. 2.) Executives have new, novel, sexy flavor of the month projects like "we need to catch up with that ChatGPT thing my kid was showing the Au Pair the other day!" 3.) Product Management leaders are incentivized to prioritize said new/novel projects, and proceed to do a worker crushing sprint to build what any sane person would call a PoC. 4.) PoC is barely tested and pushed out as a half-baked garbage product. Garbage product languishes. (See AWS Data Science Studio for a great example of giant, useless POS) Meanwhile, developers are siphoned from real projects like, gee, I don't know, getting Python 3.10 out on Lambda? Or how about when you launch a sagemaker notebook instance you still top out at Python 3.8???? And don't get me started on recent versions of CUDA being available on GPU instances. My team is relegated to docker containers deployed to lambda due to the antiquated, ridiculous limits on layer sizes for any of our ML oriented lambdas. I have a buddy who works at AWS and talks about how utterly inefficient they are due to horrific leadership. At this point, you have a choice between a few highly dysfunctional cloud vendors. Google just SUCKS, Azure is a blatant financial rip off, and AWS is always developing new features nobody wants while ignoring the ones we do.

ndneighbor 4y ago
Before my current role, I had to juggle roadmaps and integration partnerships with the Big Clouds. It's insane how much they ship their org chart but also their internal politics? I am convinced that some of the teams there are really trying to angle for releases that ship for PR rather than the feedback. (Although some AWS PMs really are trying despite everything else.)
wallace_g 4y ago
What are your issues with Google cloud?
- JPKab 4y ago
  I had a horrific support experience due to a product (it was a predecessor to Colab that they eventually just cancelled) that was (due to a bug that they eventually fixed) triggering the startup of expensive instances randomly that cost hundreds of dollars a week to run. I couldn't get any help, until I eventually out of desperation found a BigQuery evangelist on a Reddit forum who escalated my issue. Venting rant below: It's blatantly obvious that Google's culture is completely broken, and they are incapable of hiring people who don't think they're too brilliant and amazing to be customer support. They care about their advertising printing press and everything else is secondary. Tensorflow was the first out of the gate, quickly got its ass handed to it by PyTorch, and then they just imitated Pytorch, but in a crappy, crowded API kind of way. Angular is hilarious in a sad, pathetic, "wow that guy still thinks he's doing something useful, doesn't he realize nobody cares" kind of way. Dart..... well, I don't even need to say anything more. I bet there are people on HN who deeply regret investing time in learning it. Colab: "We have paid plans! If you pay, it will be only semi-useless!" (I tried the paid version, utterly useless compared to Paperspace) If anyone from Google is reading this, sorry, but you work at a company that is rotting like early 2000s Microsoft. Even my highly non-technical wife was asking me the other day if I had noticed how shitty the search engine is now. BTW, am I the only one who bought a Google Home speaker and used to listen to Youtube podcasts on it, only to have a software update make it so that now I can't even use Chromecast to send any Youtube video's audio to my speaker? Really awesome Google. First time in my life I've purchased a product and had it literally lose features a year after I bought it. Also, Google Meet sucks, and has become increasingly bug ridden over the years.
  - jfim 4y ago
    > BTW, am I the only one who bought a Google Home speaker [...] and had it literally lose features a year after I bought it There are tons of other features that are rotting on Google home/assistant. It used to be that one could set a location based reminder on the assistant ("hey Google remind me when I get home to do the laundry") but they removed that, for example.
- mdaniel 4y ago
  Whooowee, pull up a chair and grab a drink Their IAM is a raging tire fire. Disturbingly stupid for such an advanced organization. I'm on my phone so the rant will have to be left at that for now, but people can bitch about AWS IAM all they like, at least it was designed and thought out versus the GCP "works on my machine, ohh I got promoted now it's someone else's problem". Words fail me how angry GCP IAM makes me I know this isn't a "cool kid" thing to say but the lack of CloudFormation makes deploying into other people's GCP more painful than necessary, which has an intersection with my IAM comment because Role bindings are authoritative (in both gcloud and terraform) so oh, you want to add a member to `role/editor` but leave the rest of the members alone? Tough! I had high hopes for Deployment Manager but it doesn't support hardly any of the resources and there's no one click URL to roll out the yaml anyway And congratulations, you just created your first Project in GCP: now you want to run Cloud Functions? Well prepare to spend the next hours of your life turning on the necessary APIs for that project because unlike AWS where everything works to your credit limit, GCP is that blank room in the Matrix. For ever project, every time. It's like GCP has amnesia. What APIs are necessary, you may ask? Who knows, just try something, read the 403 it gives you back, rinse and repeat
Twirrim 4y ago
From my experiences working there, AWS is a feature mill. They've built their entire sense of self around shipping features. Stuff for Jeff Barr to blog about (that's no diss on Jeff, I think he does an awesome job). They can't get off that horse they've decided to ride because they're concerned if they do they'll be seen as having stopped innovating, but they're also not willing to provide headcount to meet both innovation goals and properly look after what they've already got.
- metadat 4y ago
  > that's no diss on Jeff, I think he does an awesome job Honest question: How is Lambda runtime version rot acceptable to an "awesome" top exec like Jeff Barr? Is someone else demanding he siphon resources away from maintaining key services? What data-driven metric says maintaining service quality shouldn't be continually prioritized? I'm open to any explanation, but the logical conclusion I see if the lack of prioritization only really makes sense if internally, Lambda is considered a gimmicky toy. Where is the "Customer Obsession" mantra here? A large portion of the most valuable work in software is of the boring, unsexy variety needed to keep key pillars working smoothly.
  - BFLpL0QNek 4y ago
    They’ve provided custom runtimes which removes the need completely for managed runtimes giving you whatever version you like. They also did it in a way with Docker that’s simple and powerful. If I was an AWS exec I’d say managed runtimes are legacy, with the introduction of containers move to containers for new deploys. I’ve been critical of Lambda in the past and critical when people use Kubernetes/Docker because it’s Kubernetes/Docker. The container support in Lambda has been a game changer, it’s one of them now they’ve done it you think why didn’t they think of that earlier, it’s a great fit.
    - Dunedan 4y ago
      > If I was an AWS exec I’d say managed runtimes are legacy, with the introduction of containers move to containers for new deploys. If they do that they remove one of the key benefits of AWS Lambda and that's that customers don't have to care about the underlying operating system and just have to care about their function code. And I'm sure if they'd would look at the numbers of how many AWS Lambda Functions use zip packages vs. Docker images they'd see that something like >90% use zip packages. Based on that they'd be insane to deprecate the use of zip packages.
KRAKRISMOTT 4y ago
I mean has AWS ever been easy to use outside of S3? If they were simple, companies like Vercel and Heroku would not even exist (both runs on AWS infrastructure).
- sokoloff 4y ago
  I feel like SQS, API Gateway, DynamoDB, (largely hidden) SimpleDB, and vanilla Lambda and EC2 are pretty damn easy. (IAM is admittedly semi-baroque layer holding them all together. I struggle to call that easy.)
- matus_congrady 4y ago
  The complexity and amount of AWS-specific knowledge required to efficiently use it is pretty much absurd. Even though AWS offers multiple different solutions (AWS SAM, Elastic Beanstalk, AWS CDK, AWS App Runner) that in theory should make consuming AWS services easier, these options still assume you understand and know how to use AWS. In my previous job, I've been tasked to create an abstraction that will make using these AWS services easier and more developer-friendly. I failed. Yet I realized how hard it is, and how many people/companies are struggling with it. That's why I've founded https://stacktape.com/ https://stacktape.com/. It makes using AWS simple to a point where any developer can efficiently use it. Let me know what you think.
h0l0cube 4y ago
> Meanwhile, developers are siphoned from real projects like, gee, I don't know, getting Python 3.10 out on Lambda? Why isn't competition putting pressure on AWS? A cursory googling around seems to show that Google Cloud Functions is at Python 3.11 and Azure is at 3.8. Which then makes me wonder why people don't use Google Cloud more? And why would anyone use Azure at all? [0] https://cloud.google.com/functions/docs/create-deploy-gcloud-1st-gen#deploy_a_function https://cloud.google.com/functions/docs/create-deploy-gcloud... [1] https://learn.microsoft.com/en-us/azure/azure-functions/create-first-function-cli-python?tabs=azure-cli%2Cbash&pivots=python-mode-decorators#create-venv https://learn.microsoft.com/en-us/azure/azure-functions/crea... > Google just SUCKS, Azure is a blatant financial rip off If AWS is the least worst option, and also the most popular, what motivation do they have to improve?
- wussboy 4y ago
  I’m a very happy GCP user. Cloud Functions are ass but Cloud Run is amazing, and as far as Python versions go, if you can make a Docker image of it you can use Cloud Run. Scale to zero. Automatic load balancing. Any Docker image. Yes please.

by JPKab 4y ago

ndneighbor 4y ago

Before my current role, I had to juggle roadmaps and integration partnerships with the Big Clouds. It's insane how much they ship their org chart but also their internal politics? I am convinced that some of the teams there are really trying to angle for releases that ship for PR rather than the feedback. (Although some AWS PMs really are trying despite everything else.)

wallace_g 4y ago

What are your issues with Google cloud?

JPKab 4y ago
I had a horrific support experience due to a product (it was a predecessor to Colab that they eventually just cancelled) that was (due to a bug that they eventually fixed) triggering the startup of expensive instances randomly that cost hundreds of dollars a week to run. I couldn't get any help, until I eventually out of desperation found a BigQuery evangelist on a Reddit forum who escalated my issue. Venting rant below: It's blatantly obvious that Google's culture is completely broken, and they are incapable of hiring people who don't think they're too brilliant and amazing to be customer support. They care about their advertising printing press and everything else is secondary. Tensorflow was the first out of the gate, quickly got its ass handed to it by PyTorch, and then they just imitated Pytorch, but in a crappy, crowded API kind of way. Angular is hilarious in a sad, pathetic, "wow that guy still thinks he's doing something useful, doesn't he realize nobody cares" kind of way. Dart..... well, I don't even need to say anything more. I bet there are people on HN who deeply regret investing time in learning it. Colab: "We have paid plans! If you pay, it will be only semi-useless!" (I tried the paid version, utterly useless compared to Paperspace) If anyone from Google is reading this, sorry, but you work at a company that is rotting like early 2000s Microsoft. Even my highly non-technical wife was asking me the other day if I had noticed how shitty the search engine is now. BTW, am I the only one who bought a Google Home speaker and used to listen to Youtube podcasts on it, only to have a software update make it so that now I can't even use Chromecast to send any Youtube video's audio to my speaker? Really awesome Google. First time in my life I've purchased a product and had it literally lose features a year after I bought it. Also, Google Meet sucks, and has become increasingly bug ridden over the years.
- jfim 4y ago
  > BTW, am I the only one who bought a Google Home speaker [...] and had it literally lose features a year after I bought it There are tons of other features that are rotting on Google home/assistant. It used to be that one could set a location based reminder on the assistant ("hey Google remind me when I get home to do the laundry") but they removed that, for example.
mdaniel 4y ago
Whooowee, pull up a chair and grab a drink Their IAM is a raging tire fire. Disturbingly stupid for such an advanced organization. I'm on my phone so the rant will have to be left at that for now, but people can bitch about AWS IAM all they like, at least it was designed and thought out versus the GCP "works on my machine, ohh I got promoted now it's someone else's problem". Words fail me how angry GCP IAM makes me I know this isn't a "cool kid" thing to say but the lack of CloudFormation makes deploying into other people's GCP more painful than necessary, which has an intersection with my IAM comment because Role bindings are authoritative (in both gcloud and terraform) so oh, you want to add a member to `role/editor` but leave the rest of the members alone? Tough! I had high hopes for Deployment Manager but it doesn't support hardly any of the resources and there's no one click URL to roll out the yaml anyway And congratulations, you just created your first Project in GCP: now you want to run Cloud Functions? Well prepare to spend the next hours of your life turning on the necessary APIs for that project because unlike AWS where everything works to your credit limit, GCP is that blank room in the Matrix. For ever project, every time. It's like GCP has amnesia. What APIs are necessary, you may ask? Who knows, just try something, read the 403 it gives you back, rinse and repeat

Twirrim 4y ago

From my experiences working there, AWS is a feature mill. They've built their entire sense of self around shipping features. Stuff for Jeff Barr to blog about (that's no diss on Jeff, I think he does an awesome job). They can't get off that horse they've decided to ride because they're concerned if they do they'll be seen as having stopped innovating, but they're also not willing to provide headcount to meet both innovation goals and properly look after what they've already got.

metadat 4y ago
> that's no diss on Jeff, I think he does an awesome job Honest question: How is Lambda runtime version rot acceptable to an "awesome" top exec like Jeff Barr? Is someone else demanding he siphon resources away from maintaining key services? What data-driven metric says maintaining service quality shouldn't be continually prioritized? I'm open to any explanation, but the logical conclusion I see if the lack of prioritization only really makes sense if internally, Lambda is considered a gimmicky toy. Where is the "Customer Obsession" mantra here? A large portion of the most valuable work in software is of the boring, unsexy variety needed to keep key pillars working smoothly.
- BFLpL0QNek 4y ago
  They’ve provided custom runtimes which removes the need completely for managed runtimes giving you whatever version you like. They also did it in a way with Docker that’s simple and powerful. If I was an AWS exec I’d say managed runtimes are legacy, with the introduction of containers move to containers for new deploys. I’ve been critical of Lambda in the past and critical when people use Kubernetes/Docker because it’s Kubernetes/Docker. The container support in Lambda has been a game changer, it’s one of them now they’ve done it you think why didn’t they think of that earlier, it’s a great fit.
  - Dunedan 4y ago
    > If I was an AWS exec I’d say managed runtimes are legacy, with the introduction of containers move to containers for new deploys. If they do that they remove one of the key benefits of AWS Lambda and that's that customers don't have to care about the underlying operating system and just have to care about their function code. And I'm sure if they'd would look at the numbers of how many AWS Lambda Functions use zip packages vs. Docker images they'd see that something like >90% use zip packages. Based on that they'd be insane to deprecate the use of zip packages.

KRAKRISMOTT 4y ago

I mean has AWS ever been easy to use outside of S3? If they were simple, companies like Vercel and Heroku would not even exist (both runs on AWS infrastructure).

sokoloff 4y ago
I feel like SQS, API Gateway, DynamoDB, (largely hidden) SimpleDB, and vanilla Lambda and EC2 are pretty damn easy. (IAM is admittedly semi-baroque layer holding them all together. I struggle to call that easy.)
matus_congrady 4y ago
The complexity and amount of AWS-specific knowledge required to efficiently use it is pretty much absurd. Even though AWS offers multiple different solutions (AWS SAM, Elastic Beanstalk, AWS CDK, AWS App Runner) that in theory should make consuming AWS services easier, these options still assume you understand and know how to use AWS. In my previous job, I've been tasked to create an abstraction that will make using these AWS services easier and more developer-friendly. I failed. Yet I realized how hard it is, and how many people/companies are struggling with it. That's why I've founded https://stacktape.com/ https://stacktape.com/. It makes using AWS simple to a point where any developer can efficiently use it. Let me know what you think.

h0l0cube 4y ago

> Meanwhile, developers are siphoned from real projects like, gee, I don't know, getting Python 3.10 out on Lambda? Why isn't competition putting pressure on AWS? A cursory googling around seems to show that Google Cloud Functions is at Python 3.11 and Azure is at 3.8. Which then makes me wonder why people don't use Google Cloud more? And why would anyone use Azure at all? [0] https://cloud.google.com/functions/docs/create-deploy-gcloud-1st-gen#deploy_a_function https://cloud.google.com/functions/docs/create-deploy-gcloud... [1] https://learn.microsoft.com/en-us/azure/azure-functions/create-first-function-cli-python?tabs=azure-cli%2Cbash&pivots=python-mode-decorators#create-venv https://learn.microsoft.com/en-us/azure/azure-functions/crea... > Google just SUCKS, Azure is a blatant financial rip off If AWS is the least worst option, and also the most popular, what motivation do they have to improve?

wussboy 4y ago
I’m a very happy GCP user. Cloud Functions are ass but Cloud Run is amazing, and as far as Python versions go, if you can make a Docker image of it you can use Cloud Run. Scale to zero. Automatic load balancing. Any Docker image. Yes please.