5 ms·

Ah so its not only me that uses AWS primitives for hackily implementing all sorts of synchronization primitives. My other favorite pattern is implementing a po

Ah so its not only me that uses AWS primitives for hackily implementing all sorts of synchronization primitives. My other favorite pattern is implementing a pool of workers by quering ec2 instances with a certain tag in a stopped state and starting them. Starting the instance can succeed only once - that means I managed to snatch the machine. If it fails, I try again, grabbing another one. This is one of those things that I never advertised out of professional shame, but it works, its bulletproof and dead simple and does not require additional infra to work.

_zoltan_ 2y ago
this actually sounds interesting. do you precreate the workers beforehand and then just keep them in a stopped state?
- torginus 2y ago
  yeah. one of the goals was startup time, so It made sense to precreate them. In practice we never ran out of free machines (and if we did, I have a cdk script to make more), and inifnite scaling is a pain in the butt anyways due to having to manage subnets etc. Cost-wise we're only paying for the EBS volumes for the stopped instances which are like 4GB each, so they cost practically nothing, we spend less than a dollar per month for the whole bunch.
  - zild3d 2y ago
    Warm pools are a supported feature in AWS on auto scaling groups. Works as you're describing (have a pool of instances in stopped state ready to use, only pay for EBS volume if relevant) https://aws.amazon.com/blogs/compute/scaling-your-applications-faster-with-ec2-auto-scaling-warm-pools/ https://aws.amazon.com/blogs/compute/scaling-your-applicatio...
    - zerd 2y ago
      If you want even fast startups restarting stopped instances is apparently faster https://depot.dev/blog/faster-ec2-boot-time https://depot.dev/blog/faster-ec2-boot-time
  - merb 2y ago
    I always thought that stopped instances will cost money as well?!
    - torginus 2y ago
      You're only paying for the hard drive (and the VPC stuff, if you want to be pedantic). The downside is that if you try to start your instance, they might not start if AWS doesn't have the capacity (rare but have seen it happen, particularly with larger, more exotic instances.)
  - rfoo 2y ago
    > we spend less than a dollar per month for the whole bunch This does not change the point, I'm just being pedantic, but: 4GB of gp3 EBS takes $0.32 per month, assuming a 50% discount (not unusual), less than a dollar gives only... 6 instances.
belter 2y ago
If you use hourly billed machines...Sounds like the world most expensive semaphore :-)
- torginus 2y ago
  except we are actually using them :)
  - belter 2y ago
    Just don't call them before the hour and start a different one again.Because otherwise within the hour, you will be billed for hundreds of hours...If they are of the type billed by the hour....
- messe 2y ago
  EC2 bills by the second.
  - belter 2y ago
    Some... "Your Amazon EC2 usage is calculated by either the hour or the second based on the size of the instance, operating system, and the AWS Region where the instances are launched" - https://repost.aws/knowledge-center/ec2-instance-hour-billing https://repost.aws/knowledge-center/ec2-instance-hour-billin... https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/pricing/on-demand/
    - QuinnyPig 2y ago
      MacOS instances appear to be the sole remaining exception since RHEL got on board.
      - redeux 2y ago
        Thanks Corey. Always nice to get the TL;DR from an authority on the subject.
  - 2y ago
williamdclt 2y ago
What would you say would be the "clean" way to implement a pool of workers (using EC2 instances too)?
- ndjdjddjsjj 2y ago
  etcd?
- torginus 2y ago
  not sure, probably either an eks cluster with a job scheduler pod that creates jobs via the batch api. The scheduler pod might be replaced by a lambda. Another possibility is something cooked up with a lambda creating ec2 instances via cdk and the whole thing is kept track by a dynamodb table. the first one is probably cleaner (though I don't like it, it means that I need the instance to be a kubernetes node, and that comes with a bunch of baggage).
- Cthulhu_ 2y ago
  Autoscaling and task queue based workloads, if my cloud theory is still relevant.
  - twodave 2y ago
    Agreed. Scaling based on the length of the queue, up to some maximum.
    - giovannibonetti 2y ago
      Even better, based on queue latency instead of length
      - jcrites 2y ago
        The single best metric I've found for scaling things like this is the percent of concurrent capacity that's in use. I wrote about this in a previous HN comment: https://news.ycombinator.com/item?id=41277046 https://news.ycombinator.com/item?id=41277046 Scaling on things like the length of the queue doesn't work very well at all in practice. A queue length of 100 might be horribly long in some workloads and insignificant in others, so scaling on queue length requires a lot of tuning that must be adjusted over time as the workload changes. Scaling based on percent of concurrent capacity can work for most workloads, and tends to remain stable over time even as workloads change.

_zoltan_ 2y ago

this actually sounds interesting. do you precreate the workers beforehand and then just keep them in a stopped state?

torginus 2y ago
yeah. one of the goals was startup time, so It made sense to precreate them. In practice we never ran out of free machines (and if we did, I have a cdk script to make more), and inifnite scaling is a pain in the butt anyways due to having to manage subnets etc. Cost-wise we're only paying for the EBS volumes for the stopped instances which are like 4GB each, so they cost practically nothing, we spend less than a dollar per month for the whole bunch.
- zild3d 2y ago
  Warm pools are a supported feature in AWS on auto scaling groups. Works as you're describing (have a pool of instances in stopped state ready to use, only pay for EBS volume if relevant) https://aws.amazon.com/blogs/compute/scaling-your-applications-faster-with-ec2-auto-scaling-warm-pools/ https://aws.amazon.com/blogs/compute/scaling-your-applicatio...
  - zerd 2y ago
    If you want even fast startups restarting stopped instances is apparently faster https://depot.dev/blog/faster-ec2-boot-time https://depot.dev/blog/faster-ec2-boot-time
- merb 2y ago
  I always thought that stopped instances will cost money as well?!
  - torginus 2y ago
    You're only paying for the hard drive (and the VPC stuff, if you want to be pedantic). The downside is that if you try to start your instance, they might not start if AWS doesn't have the capacity (rare but have seen it happen, particularly with larger, more exotic instances.)
- rfoo 2y ago
  > we spend less than a dollar per month for the whole bunch This does not change the point, I'm just being pedantic, but: 4GB of gp3 EBS takes $0.32 per month, assuming a 50% discount (not unusual), less than a dollar gives only... 6 instances.

belter 2y ago

If you use hourly billed machines...Sounds like the world most expensive semaphore :-)

torginus 2y ago
except we are actually using them :)
- belter 2y ago
  Just don't call them before the hour and start a different one again.Because otherwise within the hour, you will be billed for hundreds of hours...If they are of the type billed by the hour....
messe 2y ago
EC2 bills by the second.
- belter 2y ago
  Some... "Your Amazon EC2 usage is calculated by either the hour or the second based on the size of the instance, operating system, and the AWS Region where the instances are launched" - https://repost.aws/knowledge-center/ec2-instance-hour-billing https://repost.aws/knowledge-center/ec2-instance-hour-billin... https://aws.amazon.com/ec2/pricing/on-demand/ https://aws.amazon.com/ec2/pricing/on-demand/
  - QuinnyPig 2y ago
    MacOS instances appear to be the sole remaining exception since RHEL got on board.
    - redeux 2y ago
      Thanks Corey. Always nice to get the TL;DR from an authority on the subject.
- 2y ago

williamdclt 2y ago

What would you say would be the "clean" way to implement a pool of workers (using EC2 instances too)?

ndjdjddjsjj 2y ago
etcd?
torginus 2y ago
not sure, probably either an eks cluster with a job scheduler pod that creates jobs via the batch api. The scheduler pod might be replaced by a lambda. Another possibility is something cooked up with a lambda creating ec2 instances via cdk and the whole thing is kept track by a dynamodb table. the first one is probably cleaner (though I don't like it, it means that I need the instance to be a kubernetes node, and that comes with a bunch of baggage).
Cthulhu_ 2y ago
Autoscaling and task queue based workloads, if my cloud theory is still relevant.
- twodave 2y ago
  Agreed. Scaling based on the length of the queue, up to some maximum.
  - giovannibonetti 2y ago
    Even better, based on queue latency instead of length
    - jcrites 2y ago
      The single best metric I've found for scaling things like this is the percent of concurrent capacity that's in use. I wrote about this in a previous HN comment: https://news.ycombinator.com/item?id=41277046 https://news.ycombinator.com/item?id=41277046 Scaling on things like the length of the queue doesn't work very well at all in practice. A queue length of 100 might be horribly long in some workloads and insignificant in others, so scaling on queue length requires a lot of tuning that must be adjusted over time as the workload changes. Scaling based on percent of concurrent capacity can work for most workloads, and tends to remain stable over time even as workloads change.