15 ms·

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics

My word. I'm sort of gob smacked this article exists. I know there are nuances in the article, but my first impression was it's saying "we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better". Which is good lesson, and a good story, but there's a kind of irony it's come from an internal Amazon team. As another poster commented, I wouldn't be surprised if it's taken down at some point.

abrookewood 3y ago
Yep, expect the Lambda team to raise hell.
- oblio 3y ago
  Especially now that it's on the frontpage of HN :-)))
- djtango 3y ago
  Probably an unpopular take and my experience is almost 10 years old, but I would be surprised to see the Amazon I worked at try to bury something like this. If the product isn't what the customer wants, it isn't what the customer wants - move on and build something the customer wants. Yes agreed there were some funny business like not selling Chromecast, but the guiding principle was generally to make things customers want...
- onion2k 3y ago
  Do you think the Lambda team want people to use as many of their services as possible even when it's not actually appropriate and there are better architectures and approaches available? I doubt that. They probably understand that Lambda is a good service for some things and not for others, and using it as a part of deploying things to AWS is a great idea but using it where it doesn't fit makes all of AWS look bad (in particular, hard to use and expensive.)
  - adql 3y ago
    > Do you think the Lambda team want people to use as many of their services as possible even when it's not actually appropriate and there are better architectures and approaches available? They most definitely want to as that would most likely mean more money (and promotions) is flowing there.
- datadeft 3y ago
  Why would that be? Somebody figured out that AWS Lambda is not the answer to every single question?
donalhunt 3y ago
Archived: https://archive.is/LFtNg https://archive.is/LFtNg http://web.archive.org/web/20230504060528/https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90 http://web.archive.org/web/20230504060528/https://www.primev...
- deleted 3y ago
  [deleted]
benjaminwootton 3y ago
That was my reaction too. I know Microservices doesn’t equal cloud, but putting a big monolith on a big server is tangential to AWS interests to say the least!
Aeolun 3y ago
I feel like it’s an object lesson in using the right solution for a problem. Step functions do not appear to me to be something that you’d use for things that need to be executed multiple times per second.
- Hamuko 3y ago
  Yeah, this is my takeaway too. I'm pretty happy with the monolith that we run at our business and this seems to validate our decision to stick to that monolith, but I'm also pretty confident that where we use AWS Lambda, serverless is absolutely the right way to go. For example, I've written a Lambda application to reply to webhook calls and send API calls whenever those come in. It costs maybe $2 per month to run in compute and requests. Would that make more sense to rewrite as a monolith and run on EC2? I really doubt it.
  - raxxorraxor 3y ago
    For some Alexa integrations it is neat and convenient, but I went back to hosting such small interfaces as a service on another server. Not an EC2 instance, just another hosted unmanaged server. They are as cheap as they can get right now.
    - Hamuko 3y ago
      I'm not really interested in the operational overhead that brings for these small services. Cost-wise they might be just about neck-and-neck, but at least I don't need to worry about the server going down, or having outdated software. Lambda gives me scaling, load balancing and redundancy for that $2.
      - 411111111111111 3y ago
        > these small services I think you might be unintentionally arguing with a strawman, as everyone else here is talking about using monoliths instead of that. Few people want to administer a bunch of micro services themselves, but running a single service on a box is pretty low effort, even if you duplicate it for fail over/redundancy
seydor 3y ago
Maybe they'll publish the opposite results in 6 months
- goodpoint 3y ago
  And someone else will get promoted.
dragonwriter 3y ago
> Which is good lesson, and a good story, but there’s a kind of irony it’s come from an internal Amazon team. As another poster commented, I wouldn’t be surprised if it’s taken down at some point. Why? Using the model they switched to (which uses a different set of AWS services) instead of the model they switched from is a recommendation that the AWS tech advisers that are made available to enterprise customers will make for certain workloads. Now when they do that, they can also point to this article as additional backing.
- lucideer 3y ago
  Have you had AWS tech advisers advise teams in your company to go with this stack? Because I haven't. AWS doesn't have an equally distributed interest in selling all of its products. Some AWS products exist because customers need/demand them and others exist because they provide higher margins and tighter lock-in to Amazon: the first type of products are great for customer acquisition, the role of their sales folk is to then convince people using the former to migrate to the latter.
  - Paul-Craft 3y ago
    > Some AWS products exist because customers need/demand them and others exist because they provide higher margins and tighter lock-in to Amazon I'm still trying to figure out which one Aurora and Cognito fall under.
    - m_mueller 3y ago
      Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution.
      - Paul-Craft 3y ago
        Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employer was looking at moving from RDS to Aurora.
motbus3 3y ago
I think it is fine. There are scenarios were you need distributed and there are scenarios that you don't. IMO, distributed software is more practical for working development than for technical reasons. We all know from basic stuff that performing software comes from single structures that does not require packing and unpacking data But scaling large applications is hard, and it was much more expensive back then. Now that we overreacted to microservices we will overreact to monoliths again. And we will bounce many more times until AI take our jobs and do the loop itself
- goodpoint 3y ago
  [flagged]
  - motbus3 3y ago
    I do not care about the downvoting it is unfortunate that there is no comment back saying why the person disagrees
    - goodpoint 3y ago
      That's my point. Silent downvotes.
joelhaasnoot 3y ago
Half of the AWS certifications isn't about what's what but what to use when and using it for the right use case.
- datadeft 3y ago
  Exactly right. Most cloud victims are people who have faith instead of cost calculations. DHH & co. are the prime example. It seems even Amazon has such people. I guess hiring is much harder nowadays.
BbzzbB 3y ago
There was an article not long ago from AWS saying they'll be focussing on cutting cost for customers. Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.
- bootsmann 3y ago
  Still waiting for python 3.11 on lambdas so must not be that big of a focus. (They finally delivered 3.10 last month at least)
  - IshKebab 3y ago
    If you're sensitive to a 25% performance improvement, why not switch languages and get a 1000% performance improvement?
    - techdragon 3y ago
      An easily tested compatible upgrade that gets a free performance boost… vs lots of engineering effort to rewrite… yeah that’s just not going to fly with management. Who are probably looking at the 25% performance boost as a 20% cost reduction not a 20% speed increase
  - datadeft 3y ago
    It is possible[1]. You are better off this way than the AWS "native" way. 1. https://dev.l1x.be/posts/2023/02/28/using-python-3.11-with-aws-lambda/ https://dev.l1x.be/posts/2023/02/28/using-python-3.11-with-a...
    - easton 3y ago
      The only problem with that is that Docker lambdas boot slower than lambdas with the built in runtime (not ridiculously slow, but could be 2x or something). God help you anyway if you’re trying to do something latency sensitive on Lambda, but if you are then you probably don’t want to add more time for a docker pull.
fbn79 3y ago
But they migrated to AWS ECS that still is an expensive serverless AWS stuff, just fully managed by Amazon.
- nevon 3y ago
  This is simply incorrect. ECS doesn't cost anything other than what you're paying for the EC2 instances that you place your tasks on. Fargate does, but that's not what they're using.
j45 3y ago
Around 2008 the idea of microseconds were looked down on, until they weren’t. The key is to look down on nothing, become competent with multiple architects and know which ones not to implement in a use case if the one to use isn’t clear right away
djtango 3y ago
>Microservices and serverless components are tools that do work at high scale, but whether to use them over monolith has to be made on a case-by-case basis. Tldr build the right thing. >"AWS sales and support teams continue to spend much of their time helping customers optimize AWS spend so they can weather this uncertain economy," Brian Olsavsky, Amazon's finance chief, said on a conference call with analysts.[0] Amazon isn't afraid of this trend, they're embracing it. Better to cannibalise yourself than be disrupted by someone else https://twitter.com/DanRose999/status/1287944667414196225?s=20 https://twitter.com/DanRose999/status/1287944667414196225?s=... [0] https://www.cnbc.com/2023/04/27/aws-q1-earnings-report-2023.html https://www.cnbc.com/2023/04/27/aws-q1-earnings-report-2023....
seanhandley 3y ago
It's been online for 2 weeks already.
Jenk 3y ago
The cynic in me (so like 93% of me) reads this as a "Instead of abandoning AWS altogether, we changed how we use AWS, but most importantly we're still on AWS"
helsinkiandrew 3y ago
> I wouldn't be surprised if it's taken down at some point ... Why? they're still using "AWS stuff" - EC2 and ECS etc. Serverless is a fraction of the services AWS offers. AWS actively promote ways of reducing customers bills. This article could be considered a puff piece for the AWS Compute Savings Plan: https://aws.amazon.com/savingsplans/compute-pricing/ https://aws.amazon.com/savingsplans/compute-pricing/
- blowski 3y ago
  Exactly. You could easily frame it as "if AWS seems expensive, you're using it wrong". That an internal team could get it so wrong is testament to how difficult it is to get right, but of course, there's a consultant for helping with that.
dmw_ng 3y ago
The smoking gun is probably the box that was previously labelled "Media Conversion Service" (Elemental MediaConvert - easily 5-6 figures/mo. for a small amount of snappy on-demand capacity, or crippled slow-as-molasses reserved queues) now labelled "Media Converter" running on ECS. For example, vt1 instances are <$200/mo. spot and each instance packs enough transcode to power a small galaxy, for fine-grained tuning an equivalent CPU-only transcode solution isn't that much more expensive either. At some point the industry will wake up to the fact the AWS pricing pages are the real API docs, meanwhile dumb shit like this will keep happening over and over again, and AWS absolutely are not to blame for it, any more than e.g. a vendor of cabling is guilty of burning down the house of someone who plugged 10 electric heaters into a chain of double-gang power extension cords
pauby 3y ago
It's already on the Wayback Machine https://web.archive.org/web/20230504060528/https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90 https://web.archive.org/web/20230504060528/https://www.prime...
steveBK123 3y ago
Yeah this article seems like heresy for someone at Amazon to have written about AWS, no way it lives long.
credit_guy 3y ago
I don't read it like that at all. Both solutions use the Amazon cloud. Only in one solution you distribute a lot of processes, just because it's possible, and easy to code. When they figured out that rampant distribution was costly, they put more thinking in keeping a lot of computation in the same place (so, "monolith", but still in the cloud). No surprise, they found great savings. If they hadn't, they wold not have written about it. But they had to put some (most likely major) effort into redesigning the application.
jasonlotito 3y ago
> but there's a kind of irony it's come from an internal Amazon team Not at all. My time working with AWS reps, they never pushed a particular way of doing things. Rather, they tried to make what we wanted to do easier. And the caveat was always to test and make decisions on what was important to us. This isn't an anti-AWS article. Rather, it's exactly the type of thing I'd expect from them. Use the right tool for the right job.
fnordpiglet 3y ago
As an exaws senior dude we never looked at our service stack as a sell at any cost, but as a continuum of service offerings that could be assembled to be more cost optimal at higher operational burden to (mostly) ops free at a higher premium. The goal was to provide a lego kit of power tools and disappear from view tools. At least in my org we never tried to upsell or convince customers of architectures that accreted revenues at their expense, we tried to honestly assess their sophistication and desire for ops burden and complexity vs cost savings by building it themselves with the lower level kit. By our measure using aws brought us business, and we were generally more motivated by customer obsession over soaking them. I know Andy definitely had that view and drilled it into our collective heads. In many ways as an engineering minded person I appreciated the sentiment as I enjoy solving problems more than screwing people out of their money for sport.
emodendroket 3y ago
I don’t really agree that this somehow exposes those tools as bad. It more shows that they weren’t that well suited for this particular use case.
deleted 3y ago
[deleted]

abrookewood 3y ago

Yep, expect the Lambda team to raise hell.

oblio 3y ago
Especially now that it's on the frontpage of HN :-)))
djtango 3y ago
Probably an unpopular take and my experience is almost 10 years old, but I would be surprised to see the Amazon I worked at try to bury something like this. If the product isn't what the customer wants, it isn't what the customer wants - move on and build something the customer wants. Yes agreed there were some funny business like not selling Chromecast, but the guiding principle was generally to make things customers want...
onion2k 3y ago
Do you think the Lambda team want people to use as many of their services as possible even when it's not actually appropriate and there are better architectures and approaches available? I doubt that. They probably understand that Lambda is a good service for some things and not for others, and using it as a part of deploying things to AWS is a great idea but using it where it doesn't fit makes all of AWS look bad (in particular, hard to use and expensive.)
- adql 3y ago
  > Do you think the Lambda team want people to use as many of their services as possible even when it's not actually appropriate and there are better architectures and approaches available? They most definitely want to as that would most likely mean more money (and promotions) is flowing there.
datadeft 3y ago
Why would that be? Somebody figured out that AWS Lambda is not the answer to every single question?

donalhunt 3y ago

Archived: https://archive.is/LFtNg https://archive.is/LFtNg http://web.archive.org/web/20230504060528/https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90 http://web.archive.org/web/20230504060528/https://www.primev...

deleted 3y ago
[deleted]

benjaminwootton 3y ago

That was my reaction too. I know Microservices doesn’t equal cloud, but putting a big monolith on a big server is tangential to AWS interests to say the least!

Aeolun 3y ago

I feel like it’s an object lesson in using the right solution for a problem. Step functions do not appear to me to be something that you’d use for things that need to be executed multiple times per second.

Hamuko 3y ago
Yeah, this is my takeaway too. I'm pretty happy with the monolith that we run at our business and this seems to validate our decision to stick to that monolith, but I'm also pretty confident that where we use AWS Lambda, serverless is absolutely the right way to go. For example, I've written a Lambda application to reply to webhook calls and send API calls whenever those come in. It costs maybe $2 per month to run in compute and requests. Would that make more sense to rewrite as a monolith and run on EC2? I really doubt it.
- raxxorraxor 3y ago
  For some Alexa integrations it is neat and convenient, but I went back to hosting such small interfaces as a service on another server. Not an EC2 instance, just another hosted unmanaged server. They are as cheap as they can get right now.
  - Hamuko 3y ago
    I'm not really interested in the operational overhead that brings for these small services. Cost-wise they might be just about neck-and-neck, but at least I don't need to worry about the server going down, or having outdated software. Lambda gives me scaling, load balancing and redundancy for that $2.
    - 411111111111111 3y ago
      > these small services I think you might be unintentionally arguing with a strawman, as everyone else here is talking about using monoliths instead of that. Few people want to administer a bunch of micro services themselves, but running a single service on a box is pretty low effort, even if you duplicate it for fail over/redundancy

seydor 3y ago

Maybe they'll publish the opposite results in 6 months

goodpoint 3y ago
And someone else will get promoted.

dragonwriter 3y ago

> Which is good lesson, and a good story, but there’s a kind of irony it’s come from an internal Amazon team. As another poster commented, I wouldn’t be surprised if it’s taken down at some point. Why? Using the model they switched to (which uses a different set of AWS services) instead of the model they switched from is a recommendation that the AWS tech advisers that are made available to enterprise customers will make for certain workloads. Now when they do that, they can also point to this article as additional backing.

lucideer 3y ago
Have you had AWS tech advisers advise teams in your company to go with this stack? Because I haven't. AWS doesn't have an equally distributed interest in selling all of its products. Some AWS products exist because customers need/demand them and others exist because they provide higher margins and tighter lock-in to Amazon: the first type of products are great for customer acquisition, the role of their sales folk is to then convince people using the former to migrate to the latter.
- Paul-Craft 3y ago
  > Some AWS products exist because customers need/demand them and others exist because they provide higher margins and tighter lock-in to Amazon I'm still trying to figure out which one Aurora and Cognito fall under.
  - m_mueller 3y ago
    Aurora is I think pretty simple to move away from, since it's just fully compatible Postgres or Mysql. We even use a local postgres for development purposes against an Aurora solution.
    - Paul-Craft 3y ago
      Nope. AWS makes it dead simple to move from RDS to Aurora by clicking a button. There's no way to move data from Aurora to RDS short of doing a SQL dump and reloading everything that way. I found this out when my previous employer was looking at moving from RDS to Aurora.

motbus3 3y ago

I think it is fine. There are scenarios were you need distributed and there are scenarios that you don't. IMO, distributed software is more practical for working development than for technical reasons. We all know from basic stuff that performing software comes from single structures that does not require packing and unpacking data But scaling large applications is hard, and it was much more expensive back then. Now that we overreacted to microservices we will overreact to monoliths again. And we will bounce many more times until AI take our jobs and do the loop itself

goodpoint 3y ago
[flagged]
- motbus3 3y ago
  I do not care about the downvoting it is unfortunate that there is no comment back saying why the person disagrees
  - goodpoint 3y ago
    That's my point. Silent downvotes.

joelhaasnoot 3y ago

Half of the AWS certifications isn't about what's what but what to use when and using it for the right use case.

datadeft 3y ago
Exactly right. Most cloud victims are people who have faith instead of cost calculations. DHH & co. are the prime example. It seems even Amazon has such people. I guess hiring is much harder nowadays.

BbzzbB 3y ago

There was an article not long ago from AWS saying they'll be focussing on cutting cost for customers. Maybe the next step of that process will be pushing their clients off of AWS and telling them to just host on prem.

bootsmann 3y ago
Still waiting for python 3.11 on lambdas so must not be that big of a focus. (They finally delivered 3.10 last month at least)
- IshKebab 3y ago
  If you're sensitive to a 25% performance improvement, why not switch languages and get a 1000% performance improvement?
  - techdragon 3y ago
    An easily tested compatible upgrade that gets a free performance boost… vs lots of engineering effort to rewrite… yeah that’s just not going to fly with management. Who are probably looking at the 25% performance boost as a 20% cost reduction not a 20% speed increase
- datadeft 3y ago
  It is possible[1]. You are better off this way than the AWS "native" way. 1. https://dev.l1x.be/posts/2023/02/28/using-python-3.11-with-aws-lambda/ https://dev.l1x.be/posts/2023/02/28/using-python-3.11-with-a...
  - easton 3y ago
    The only problem with that is that Docker lambdas boot slower than lambdas with the built in runtime (not ridiculously slow, but could be 2x or something). God help you anyway if you’re trying to do something latency sensitive on Lambda, but if you are then you probably don’t want to add more time for a docker pull.

fbn79 3y ago

But they migrated to AWS ECS that still is an expensive serverless AWS stuff, just fully managed by Amazon.

nevon 3y ago
This is simply incorrect. ECS doesn't cost anything other than what you're paying for the EC2 instances that you place your tasks on. Fargate does, but that's not what they're using.

j45 3y ago

Around 2008 the idea of microseconds were looked down on, until they weren’t. The key is to look down on nothing, become competent with multiple architects and know which ones not to implement in a use case if the one to use isn’t clear right away

djtango 3y ago

>Microservices and serverless components are tools that do work at high scale, but whether to use them over monolith has to be made on a case-by-case basis. Tldr build the right thing. >"AWS sales and support teams continue to spend much of their time helping customers optimize AWS spend so they can weather this uncertain economy," Brian Olsavsky, Amazon's finance chief, said on a conference call with analysts.[0] Amazon isn't afraid of this trend, they're embracing it. Better to cannibalise yourself than be disrupted by someone else https://twitter.com/DanRose999/status/1287944667414196225?s=20 https://twitter.com/DanRose999/status/1287944667414196225?s=... [0] https://www.cnbc.com/2023/04/27/aws-q1-earnings-report-2023.html https://www.cnbc.com/2023/04/27/aws-q1-earnings-report-2023....

seanhandley 3y ago

It's been online for 2 weeks already.

Jenk 3y ago

The cynic in me (so like 93% of me) reads this as a "Instead of abandoning AWS altogether, we changed how we use AWS, but most importantly we're still on AWS"

helsinkiandrew 3y ago

> I wouldn't be surprised if it's taken down at some point ... Why? they're still using "AWS stuff" - EC2 and ECS etc. Serverless is a fraction of the services AWS offers. AWS actively promote ways of reducing customers bills. This article could be considered a puff piece for the AWS Compute Savings Plan: https://aws.amazon.com/savingsplans/compute-pricing/ https://aws.amazon.com/savingsplans/compute-pricing/

blowski 3y ago
Exactly. You could easily frame it as "if AWS seems expensive, you're using it wrong". That an internal team could get it so wrong is testament to how difficult it is to get right, but of course, there's a consultant for helping with that.

dmw_ng 3y ago

The smoking gun is probably the box that was previously labelled "Media Conversion Service" (Elemental MediaConvert - easily 5-6 figures/mo. for a small amount of snappy on-demand capacity, or crippled slow-as-molasses reserved queues) now labelled "Media Converter" running on ECS. For example, vt1 instances are <$200/mo. spot and each instance packs enough transcode to power a small galaxy, for fine-grained tuning an equivalent CPU-only transcode solution isn't that much more expensive either. At some point the industry will wake up to the fact the AWS pricing pages are the real API docs, meanwhile dumb shit like this will keep happening over and over again, and AWS absolutely are not to blame for it, any more than e.g. a vendor of cabling is guilty of burning down the house of someone who plugged 10 electric heaters into a chain of double-gang power extension cords

pauby 3y ago

It's already on the Wayback Machine https://web.archive.org/web/20230504060528/https://www.primevideotech.com/video-streaming/scaling-up-the-prime-video-audio-video-monitoring-service-and-reducing-costs-by-90 https://web.archive.org/web/20230504060528/https://www.prime...

steveBK123 3y ago

Yeah this article seems like heresy for someone at Amazon to have written about AWS, no way it lives long.

credit_guy 3y ago

I don't read it like that at all. Both solutions use the Amazon cloud. Only in one solution you distribute a lot of processes, just because it's possible, and easy to code. When they figured out that rampant distribution was costly, they put more thinking in keeping a lot of computation in the same place (so, "monolith", but still in the cloud). No surprise, they found great savings. If they hadn't, they wold not have written about it. But they had to put some (most likely major) effort into redesigning the application.

jasonlotito 3y ago

> but there's a kind of irony it's come from an internal Amazon team Not at all. My time working with AWS reps, they never pushed a particular way of doing things. Rather, they tried to make what we wanted to do easier. And the caveat was always to test and make decisions on what was important to us. This isn't an anti-AWS article. Rather, it's exactly the type of thing I'd expect from them. Use the right tool for the right job.

fnordpiglet 3y ago

As an exaws senior dude we never looked at our service stack as a sell at any cost, but as a continuum of service offerings that could be assembled to be more cost optimal at higher operational burden to (mostly) ops free at a higher premium. The goal was to provide a lego kit of power tools and disappear from view tools. At least in my org we never tried to upsell or convince customers of architectures that accreted revenues at their expense, we tried to honestly assess their sophistication and desire for ops burden and complexity vs cost savings by building it themselves with the lower level kit. By our measure using aws brought us business, and we were generally more motivated by customer obsession over soaking them. I know Andy definitely had that view and drilled it into our collective heads. In many ways as an engineering minded person I appreciated the sentiment as I enjoy solving problems more than screwing people out of their money for sport.

emodendroket 3y ago

I don’t really agree that this somehow exposes those tools as bad. It more shows that they weren’t that well suited for this particular use case.

deleted 3y ago

[deleted]