4 ms·

I was at a place where we had "worker" machines that would handle incoming data with fluctuating volume. If the queues got too long we would automatically spin

I was at a place where we had "worker" machines that would handle incoming data with fluctuating volume. If the queues got too long we would automatically spin up new worker instances and when it came time to spin down we would kill the older ones first. You can probably see where this is going. The workers had some problem where they would bog down if left running too long. Causing the queues to back up and indirectly causing themselves to eventually be culled. Never did figure out why they would bog down. We just ran herky jerky like this for a few years till I left. Might still be doing it for all I know.

aequitas 3y ago
> The workers had some problem where they would bog down if left running too long. So you just automatically replace the instances after a certain amount of runtime and your problem is gone.
- lucianbr 3y ago
  Yeah, fixing a problem without understanding it has some disadvantages. It works sometimes, but the "with understanding" strategy works much more often. Is this really a prevailing attitude now? Who cares what happened, as long as we can paper over it with some other maneuver/resources? For me it's both intellectually rewarding and skill-building to figure out what caused the problem in the first place. I mean, I hear plenty of managers with this attitude. But I really expect better on a forum called hacker news.
  - FridgeSeal 3y ago
    For me, there’s a certain threshold involved. If it happens extremely rarely (like, once every 6 months) or it’s super transient and low impact, we kick it and move on. If it starts happening a 3rd or 4th time, or the severity increases we start to dig in and actually fix it. So we’re not giving up, and losing all diagnosis/bugfixing ability, just setting a threshold. There’ll always be issues, some of them will always be mystery issues, so you can’t solve everything, so you’ve got to triage appropriately.
  - aequitas 3y ago
    I said nothing about not understanding the issue. Even with understanding just “turning it on and off again” might be the better solution at the moment. Because going for the “real” solution means making a trade-off somewhere else.
  - account42 3y ago
    Plus just because the only visible symptom of the bug is a perfomance issue right now doesn't mean that there won't also be other consequences. If something is behaving contrary to expectations you should always figure out why.

aequitas 3y ago

> The workers had some problem where they would bog down if left running too long. So you just automatically replace the instances after a certain amount of runtime and your problem is gone.

lucianbr 3y ago
Yeah, fixing a problem without understanding it has some disadvantages. It works sometimes, but the "with understanding" strategy works much more often. Is this really a prevailing attitude now? Who cares what happened, as long as we can paper over it with some other maneuver/resources? For me it's both intellectually rewarding and skill-building to figure out what caused the problem in the first place. I mean, I hear plenty of managers with this attitude. But I really expect better on a forum called hacker news.
- FridgeSeal 3y ago
  For me, there’s a certain threshold involved. If it happens extremely rarely (like, once every 6 months) or it’s super transient and low impact, we kick it and move on. If it starts happening a 3rd or 4th time, or the severity increases we start to dig in and actually fix it. So we’re not giving up, and losing all diagnosis/bugfixing ability, just setting a threshold. There’ll always be issues, some of them will always be mystery issues, so you can’t solve everything, so you’ve got to triage appropriately.
- aequitas 3y ago
  I said nothing about not understanding the issue. Even with understanding just “turning it on and off again” might be the better solution at the moment. Because going for the “real” solution means making a trade-off somewhere else.
- account42 3y ago
  Plus just because the only visible symptom of the bug is a perfomance issue right now doesn't mean that there won't also be other consequences. If something is behaving contrary to expectations you should always figure out why.