3 ms·
Not necessarily. I've been at some places where you just "do it live." We had dozens of worker nodes. An engineer would "reserve" one for debugging purposes, ed
by icedchai 5y ago
Not necessarily. I've been at some places where you just "do it live." We had dozens of worker nodes. An engineer would "reserve" one for debugging purposes, editing python code live, adding logs or prints as needed and restarting services. Worst case a small percentage of requests was lost, though generally they would be retried except in the most extreme cases. This risk was considered acceptable.
Obviously, depending on the business and maturity level of the company, this could be a big no-no. In general, I find those "wild west" environments more fun. Waiting for an hour long build, deploys, and "approvals" to add a couple of prints is a big drag.
- invalidname 5y agoFor that you have developer observability which doesn't carry the HUGE gaping security risks or performance problems. But still lets you debug the cluster at scale.
- icedchai 5y agoI’ve worked in that space. Observability tools may not give you all you need to debug certain classes of problems. I saw your other comments about Lightrun and will check it out though!