3 ms·
You had a mandate for 5-7 nines uptime and you couldn't tolerate a problem on a single node?
by closeparen 5y ago
You had a mandate for 5-7 nines uptime and you couldn't tolerate a problem on a single node?
- rytis 5y agoyup, that sounds strange. if the environment is that advanced, then there shouldn't be a problem of taking the node in question out of the service pool and let the devs loose on it (assuming there's no info that they aren't supposed to see). when they're done, just recycle it.
- theptip 5y agoWhere did they say it was a single node issue?
- tuananh 5y agoi think they mean, in case he mess up a single node, it should still be fine
- theptip 5y agoI see, that makes more sense, thanks for the clarification. I’d still rather not have to debug arbitrary mutations to the env or file system in a production container though. It seems to me that shelling into a prod container/VM is discouraged not because you might cause it to fail, but because you might produce undefined behavior while claiming it is healthy (more like Byzantine faults). For example if you unset a single env var by mistake, then 1/N of your requests will potentially fail. Debugging this kind of issue is a nightmare. Not to mention that developers can often run arbitrary SQL from a prod shell when the app is backed by a DB.
- yjftsjthsd-h 5y ago> For example if you unset a single env var by mistake, then 1/N of your requests will potentially fail. Or far worse, not fail.
- deleted 5y ago[deleted]