4 ms·
Sounds about right, this is really a hidden 'extra cost' with AWS. Once you get to a certain size, you really need to spend dedicated engineering resources to
by siliconc0w 9y ago
Sounds about right, this is really a hidden 'extra cost' with AWS. Once you get to a certain size, you really need to spend dedicated engineering resources to build extra tooling like this or do other 'cost engineering' like RI capacity planning - which you totally thought you didn't need to do anymore right?
You can offload some of that to paid services like cloud-health but it still takes engineering resources to manage the costs you find out about. I.e You may want a 'fully immutable' data pipeline (i.e DPL) - with each job running on a fresh instance each time but that usually means getting hit with a full billing hour even if the job takes <1 minute. So you have to use schedulers/containers but then you're working outside the 'instance' paradigm and get hit with the problems segment talks about.
To control costs and provide good UX for end users we eventually started regressing to mainframe-style computing with the ridiculous X1 RIs which was an interesting experiment. These take like a house down-payment but on the plus side users are really pleased when they see 128 cpus in htop.
- cakeface 9y agoI'm curious what you mean by fully immutable data pipeline or DPL. Is this a data processing methodology or an actual technology?
- siliconc0w 9y agoAh probably sound have said 'idempotent'. Often, you want a fresh and clean resource to pop into existence, process something, persist the result someplace, and then go away. To align with the rest of AWS in terms of billing, security/IAM, networking, etc you kinda need to use the 'ec2 instance' as this resource but this causes problems so usually the customer has to create and manage another level of abstraction (like EMR, kube, mesos, ECS, what have you). It'd be real nice if the customer didn't have to do this but AWS would rather spend their engineering resources to compete with the G at beagle identification APIs or try to vertically integrate with poor clones of github. (I realize this is slightly unfair ) I know, I know - you're saying 'lambda' is the answer - won't get into that here.
- RhodesianHunter 9y agoI don't think anyone would say Lambda is the answer for anything but very quick jobs, not until they pump the available run time up to an hour.
- zitterbewegung 9y agoCan you go into detail of what you mean by "mainframe style computing "?
- will_hughes 9y agoAt a guess: Get the biggest box you can, run a ton of things on it (eg containers, or something like that). This is opposed to the 'cloud' thing of get an instance just big enough to run your job, run lots of them if you need to scale.