3 ms·
That is probably the primary use case for spot instances and certainly how I tend to used them. I have a few thousand files that need processing and each files
by dagw 5y ago
That is probably the primary use case for spot instances and certainly how I tend to used them. I have a few thousand files that need processing and each files takes 30 minutes. So I spin up 100 spot instances and use a small script to copy the file from a control machine to the next available remote machine, start the processing script, and then copy the result back when done.
I've also got another script that keeps track of how many spot instances are currently running and spins up some new ones (possibly in a different region) if they fall below a certain level.