4 ms·

Why would DO be upset about spinning up 10 VMs then spinning them down again? Isn't this exactly the point of cloud providers? This is what they bill me for, ri

by aftbit 7y ago

Why would DO be upset about spinning up 10 VMs then spinning them down again? Isn't this exactly the point of cloud providers? This is what they bill me for, right?

cptskippy 7y ago
I guess it depends on what each of those VMs is doing. I don't trust the Dev's explanation tbh. > to make it faster we execute it in parallel on multiple droplets ~10 that we set up only for this pipeline and shut down once it’s done. Tildes preceding numbers means "approximately". Why doesn't he know exactly how many VMs he spun up? Was he actually spinning up 1 VM per record and only allowing 10 VMs to be running concurrently? I'm not a Pythong dev but why can't you execute 10 instances of Python on a single VM? If you need to dedicate an entire VM to processing 1 row, what the hell is it doing?
- developer2 7y ago
  My gut feeling agrees with "I don't trust the Dev's explanation tbh." Companies and developers often try to get away with doing Bad Things, and cry wolf publicly without offering up specifics about what they were trying to get away with. "Spin up 10 VMs for 500k rows of data" offers no explanation to just what those 10 VMs are doing. There is a big difference between "using memory and cpu" and "saturating the network in abusive ways". Random speculation of one possibility: each of those 10 instances were suddenly doing something unexpected and spammy with the network. Maybe sending 500k+ emails (one per row of data claimed by the developer) over SMTP in a very short period, or jumping to massive spikes in torrent traffic, or crawling sites to scrape data (maybe each row of 500k is just a top-level domain name, and they crawl every URL on those domains, possibly turning 500k rows into hundreds of million of http requests). The postmortem will be interesting. If DO is truly at fault here, that email after the second lockout saying the account is locked after review, no further details required... bad.
SteveNuts 7y ago
Smaller VPS providers like Linode or DO oversubscribe like crazy. Last time I used Linode, they would email us telling us we're using too much CPU or memory, and we'd need to move to a larger tier VM.
- Aaronn 7y ago
  I think you misunderstood those emails. They are just there to help you if you didn't realize some process was stuck or something, they specifically say "This is not meant as a warning or a representation that you are misusing your resources." and you can also change the value that triggers those emails or disable them completely.
  - SteveNuts 7y ago
    No it was definitely a ticket from their support, telling us we were noisy neighbors. They told us we needed to increase the size of our VMs.

by aftbit 7y ago

Why would DO be upset about spinning up 10 VMs then spinning them down again? Isn't this exactly the point of cloud providers? This is what they bill me for, right?

cptskippy 7y ago

I guess it depends on what each of those VMs is doing. I don't trust the Dev's explanation tbh. > to make it faster we execute it in parallel on multiple droplets ~10 that we set up only for this pipeline and shut down once it’s done. Tildes preceding numbers means "approximately". Why doesn't he know exactly how many VMs he spun up? Was he actually spinning up 1 VM per record and only allowing 10 VMs to be running concurrently? I'm not a Pythong dev but why can't you execute 10 instances of Python on a single VM? If you need to dedicate an entire VM to processing 1 row, what the hell is it doing?

developer2 7y ago
My gut feeling agrees with "I don't trust the Dev's explanation tbh." Companies and developers often try to get away with doing Bad Things, and cry wolf publicly without offering up specifics about what they were trying to get away with. "Spin up 10 VMs for 500k rows of data" offers no explanation to just what those 10 VMs are doing. There is a big difference between "using memory and cpu" and "saturating the network in abusive ways". Random speculation of one possibility: each of those 10 instances were suddenly doing something unexpected and spammy with the network. Maybe sending 500k+ emails (one per row of data claimed by the developer) over SMTP in a very short period, or jumping to massive spikes in torrent traffic, or crawling sites to scrape data (maybe each row of 500k is just a top-level domain name, and they crawl every URL on those domains, possibly turning 500k rows into hundreds of million of http requests). The postmortem will be interesting. If DO is truly at fault here, that email after the second lockout saying the account is locked after review, no further details required... bad.

SteveNuts 7y ago

Smaller VPS providers like Linode or DO oversubscribe like crazy. Last time I used Linode, they would email us telling us we're using too much CPU or memory, and we'd need to move to a larger tier VM.

Aaronn 7y ago
I think you misunderstood those emails. They are just there to help you if you didn't realize some process was stuck or something, they specifically say "This is not meant as a warning or a representation that you are misusing your resources." and you can also change the value that triggers those emails or disable them completely.
- SteveNuts 7y ago
  No it was definitely a ticket from their support, telling us we were noisy neighbors. They told us we needed to increase the size of our VMs.