3 ms·
> 4. an code-focused ops team (as opposed to a mouse-driven ops team) with extremely strong diagnostic skills and the bandwidth to babysit a service with a pote
by tetha 8y ago
> 4. an code-focused ops team (as opposed to a mouse-driven ops team) with extremely strong diagnostic skills and the bandwidth to babysit a service with a potential pain-in-the-ass ceiling around that of a Cassandra cluster
Here it is running fine... running fine.. running fine... aaaaand there's a compaction-and-gc cycle of death and fire and lost data and tears. Thank you for this terrible memory.
- eropple 8y agoI was going to say "we've all been there," but we haven't, and that's the deceptive thing about the five-minute-demo culture that a lot of "devops" has gotten into. Everything is easy when it has nothing riding on it. When it isn't is where the value of a tool comes into focus.
- tetha 8y agoYeah I'm dealing with crap out of that area atm. I've been recently asked why I'm extremely restrictive and careful with our primary production cluster. Well, we got 20k+ full time employees of our customers depending on this system for their everyday work. An hour of downtime of this thing will cost our end customers 20k man-hours of work done in a worse way. We're not touching the tooling this system sits on without good reason and a lot of testing. And even then I'll be bloody scared. Sorry modern world, but in this case, I'll be wearing my hard ops hat.