2 ms·
Why not set up a render farm that's always running and then send it job after job to keep the pipeline full? You said that making it take an hour was ok, so I
by jedberg 11y ago
Why not set up a render farm that's always running and then send it job after job to keep the pipeline full?
You said that making it take an hour was ok, so I assume the processing time is somewhat flexible. If you don't have enough workload to always keep one machine busy, you could always wait until you did have enough workload and then spin up a one machine render farm, have it work for an hour and then shut it back down until you had enough work.
It's true that per minute billing would save you money, but assuming what you're doing is meant to scale up, once you get past enough work to keep a single machine busy, you've gained nothing.
However if you put in the work now to separate the jobs from the machines, you're on the path to much better future scalability, because you can size the render farm to the work instead of having the farm and the number of jobs directly linked.
- fpgaminer 11y ago> You said that making it take an hour was ok, so I assume the processing time is somewhat flexible. My comment was unclear. It's a balancing act between running time of a job, time before a job starts, and cost. 1 and 2 are both important and appreciated parameters, but only if 3 allows. > It's true that per minute billing would save you money, but assuming what you're doing is meant to scale up, once you get past enough work to keep a single machine busy, you've gained nothing. Completely true, but at that point we're running a cloud within a cloud ... which seems silly. Per-minute billing would get us what we need, without the complexity of having to engineer a cloud within a cloud. Plus, it would allow scaling horizontally, which improves parameter #2 above, without impacting #3. In other words, a farm of huge machines will need a queue to keep them busy. Having a queue means there's some delay before a job starts. Whereas if we just spin up a per-minute billed machine immediately for each job, there is no delay. That requires less engineering, and handles any workload whether it be heavy or sporadic.
- jedberg 11y ago> Completely true, but at that point we're running a cloud within a cloud You're doing that anyway, even if you run one job per machine, you just loose granularity (one of the big advantages of any "cloud"). > In other words, a farm of huge machines will need a queue to keep them busy. Having a queue means there's some delay before a job starts But that's only true if you can't fill one machine. Once you're filling one machine, you're queue processing time will be the same. Presumably you still have a queue, it's just that the queue drives the start of machines instead of processes on the machine. > That requires less engineering Right, like I said, you can do less engineering now but pay more later as your workloads scale up in both money and maintenance costs.
- Retric 11y ago1.1 machines worth of processing means some things wait 1hour, or your wasting a lot of time on the second machine. So, shorter increments still save you as the workload grows, just less as the system keeps growing. (More so if demand is random.)