3 ms·

I have a saying that isn't perfectly true but often will apply to "fixes" like this - If you don't know why the fix worked, you may not have actually fixed it.

bicijay 3y ago
But you may have
- lbhdc 3y ago
  Ahhh, Schrodinger's patch.
  - HPsquared 3y ago
    Schrödinger's bug?
gwbas1c 3y ago
But they couldn't fix the bug: The bug was in another product that they couldn't access source code or submit patches. The best they could do was work around it. Sometimes workarounds are the best you can do until your vendor provides a real fix.
- JohnMakin 3y ago
  I didn't mean literally fix the underlying bug. They also don't really know why their workaround worked, which means it could not really be fixed at all.
tetha 3y ago
Someone once said, there is accidental function, and deliberate function. If your system doesn't work, and you just plonk around at values, until, very surprisingly, the system starts behaving well and you the call it working... well it might be working now. But it's just accidental correctness. As soon as something causes the system to bank left, something's gonna break and no one knows how to fix it - and you're back to square one. On the other hand, as hard as it is, if you can clearly tell why your fix will restore function to the system without even applying it, you have deliberate correctness and function. If done right, it is very boring, because exactly and only the expected thing will happen. You should know about the unknowns and plan around those as well, so even if an unknown bites you, it's a known and handled unknown. This can be exhausting to make happen, because it is much harder, but those systems will just work. But this is a fight I have with some development teams probably forever. "But we poked at the values, and that stopped the flames. It is fixed!" "but why?" "Dunny. But no fire anymore. All good." And then 2 weeks pass, and there is more fire and everyone is like "Oh but why would this happen? How should we have known for this to happen again"
- saulpw 3y ago
  On the other hand, I've spent weeks with a team looking for a bug, and by the time we found something that appeared to fix it, we were way behind on everything else that really needed to get done. How long would it take to find the root cause? We tried. It wasn't worth weeks or months of effort, to anyone. This isn't JPL and human lives weren't on the line. We just needed it not to crash so we could all get on with the "real" task of shipping useful and profitable software.
  - tetha 3y ago
    Yeah, that is why software engineering and system operations is hard. For example, the article doesn't get to a root cause in an absolute way. There is no absolute SEGFAULT of the OS causing the misbehavior. However, they nail down the crash to a gif, and if the gif is in, it crashes, and if the gif is out it doesn't. If the gif is loaded otherwise it crashes, too. At that level, to me, that would be enough, because we're users of the browser's rendering there. Finding a solid cause that can demonstrate and reproduce a problem, and basing a workaround around that at a boundary you're unwilling to cross can be fine. If it's within the company, it absolutely is fine as long as you escalate beyond that boundary. However, I have enough teams who are like "Oh, we set all values to 25 one by one and when we arrived at flum-value at 25 it stopped crashing. Fixed." Why 25? Who knows. Why flum? Who knows. Maybe the other value changed at the same time fixed it? Who knows. Do we use 26 once it starts crashing again? Fuck knows. Maybe 24 is better? We have no explanation for 25, so why would 25 be a good fix?
derefr 3y ago
And in fact, I don't think they have fixed it. I've seen "Error 5" plenty of times in Chrome. It seemingly occurs whenever I have a lot (100+) of tabs open for any site where each page allocates at least one accelerated drawing canvas (a literal <canvas>, or a <video>, or a .gif <img>.) I've seen it happen on Reddit (but only new reddit, not old reddit!) and on a number of other sites. I hypothesize that Chrome simply has a global (i.e. cross-tab) per-toplevel-origin limit to the number of allocated accelerated drawing canvases it's willing to allow; and that when you go over it, Chrome forcibly de-allocates all the existing drawing canvases used by other tabs that have that toplevel origin loaded, thereby causing them to crash. It's probably a measure designed to prevent a site from from DoSing your computer by just allocating an infinite number of canvases.
- nonethewiser 3y ago
  They said it also crashes when they just load the gif. It seems hard to believe they always had tons of tabs open for these tests.
  - derefr 3y ago
    It doesn't have to be tons of open tabs; it can just as well be one tab that makes tons of accel draw contexts all on its own — maybe temporary hidden <canvas> draw contexts that it never cleans up. But a dev would usually end up hitting that kind of problem during development; whereas, if the problem never arises through "normal use" but instead requires having 100+ open tabs on the site to trigger, that bug might go unnoticed all throughout the QA process. I can also extend my speculation about all the times I've personally seen "Error 5", although this is probably dipping into superstition territory: I think the per-toplevel-origin draw-context limit isn't static, but is influenced by the amount of VRAM available on the computer as a whole. So on an extremely VRAM-constrained system like a Raspberry Pi (or on a system that's running Cyberpunk 2077 on one display and Chrome on another, sharing a GPU) the per-toplevel-origin limit on accel draw contexts might be dropped as low as, say, 5: an amount low enough that just having one extra .gif on the single tab of the site you have open might be enough to make the tab fall over.
nonethewiser 3y ago
I agree 100% and the observe this all the time with things that “magically fix themselves.” But Im not sure it applies here since they seemed to have actually quarantined it. I suppose since they dont know the root cause it’s impossible to say. But I think the saying would fit better if they kept the gif but made some change that seemed to fix it without knowing why.
- JohnMakin 3y ago
  Maybe, but they don’t know another GIF couldn’t eventually cause the same issue again.

bicijay 3y ago

But you may have

lbhdc 3y ago
Ahhh, Schrodinger's patch.
- HPsquared 3y ago
  Schrödinger's bug?

gwbas1c 3y ago

But they couldn't fix the bug: The bug was in another product that they couldn't access source code or submit patches. The best they could do was work around it. Sometimes workarounds are the best you can do until your vendor provides a real fix.

JohnMakin 3y ago
I didn't mean literally fix the underlying bug. They also don't really know why their workaround worked, which means it could not really be fixed at all.

tetha 3y ago

Someone once said, there is accidental function, and deliberate function. If your system doesn't work, and you just plonk around at values, until, very surprisingly, the system starts behaving well and you the call it working... well it might be working now. But it's just accidental correctness. As soon as something causes the system to bank left, something's gonna break and no one knows how to fix it - and you're back to square one. On the other hand, as hard as it is, if you can clearly tell why your fix will restore function to the system without even applying it, you have deliberate correctness and function. If done right, it is very boring, because exactly and only the expected thing will happen. You should know about the unknowns and plan around those as well, so even if an unknown bites you, it's a known and handled unknown. This can be exhausting to make happen, because it is much harder, but those systems will just work. But this is a fight I have with some development teams probably forever. "But we poked at the values, and that stopped the flames. It is fixed!" "but why?" "Dunny. But no fire anymore. All good." And then 2 weeks pass, and there is more fire and everyone is like "Oh but why would this happen? How should we have known for this to happen again"

saulpw 3y ago
On the other hand, I've spent weeks with a team looking for a bug, and by the time we found something that appeared to fix it, we were way behind on everything else that really needed to get done. How long would it take to find the root cause? We tried. It wasn't worth weeks or months of effort, to anyone. This isn't JPL and human lives weren't on the line. We just needed it not to crash so we could all get on with the "real" task of shipping useful and profitable software.
- tetha 3y ago
  Yeah, that is why software engineering and system operations is hard. For example, the article doesn't get to a root cause in an absolute way. There is no absolute SEGFAULT of the OS causing the misbehavior. However, they nail down the crash to a gif, and if the gif is in, it crashes, and if the gif is out it doesn't. If the gif is loaded otherwise it crashes, too. At that level, to me, that would be enough, because we're users of the browser's rendering there. Finding a solid cause that can demonstrate and reproduce a problem, and basing a workaround around that at a boundary you're unwilling to cross can be fine. If it's within the company, it absolutely is fine as long as you escalate beyond that boundary. However, I have enough teams who are like "Oh, we set all values to 25 one by one and when we arrived at flum-value at 25 it stopped crashing. Fixed." Why 25? Who knows. Why flum? Who knows. Maybe the other value changed at the same time fixed it? Who knows. Do we use 26 once it starts crashing again? Fuck knows. Maybe 24 is better? We have no explanation for 25, so why would 25 be a good fix?

derefr 3y ago

And in fact, I don't think they have fixed it. I've seen "Error 5" plenty of times in Chrome. It seemingly occurs whenever I have a lot (100+) of tabs open for any site where each page allocates at least one accelerated drawing canvas (a literal <canvas>, or a <video>, or a .gif <img>.) I've seen it happen on Reddit (but only new reddit, not old reddit!) and on a number of other sites. I hypothesize that Chrome simply has a global (i.e. cross-tab) per-toplevel-origin limit to the number of allocated accelerated drawing canvases it's willing to allow; and that when you go over it, Chrome forcibly de-allocates all the existing drawing canvases used by other tabs that have that toplevel origin loaded, thereby causing them to crash. It's probably a measure designed to prevent a site from from DoSing your computer by just allocating an infinite number of canvases.

nonethewiser 3y ago
They said it also crashes when they just load the gif. It seems hard to believe they always had tons of tabs open for these tests.
- derefr 3y ago
  It doesn't have to be tons of open tabs; it can just as well be one tab that makes tons of accel draw contexts all on its own — maybe temporary hidden <canvas> draw contexts that it never cleans up. But a dev would usually end up hitting that kind of problem during development; whereas, if the problem never arises through "normal use" but instead requires having 100+ open tabs on the site to trigger, that bug might go unnoticed all throughout the QA process. I can also extend my speculation about all the times I've personally seen "Error 5", although this is probably dipping into superstition territory: I think the per-toplevel-origin draw-context limit isn't static, but is influenced by the amount of VRAM available on the computer as a whole. So on an extremely VRAM-constrained system like a Raspberry Pi (or on a system that's running Cyberpunk 2077 on one display and Chrome on another, sharing a GPU) the per-toplevel-origin limit on accel draw contexts might be dropped as low as, say, 5: an amount low enough that just having one extra .gif on the single tab of the site you have open might be enough to make the tab fall over.

nonethewiser 3y ago

I agree 100% and the observe this all the time with things that “magically fix themselves.” But Im not sure it applies here since they seemed to have actually quarantined it. I suppose since they dont know the root cause it’s impossible to say. But I think the saying would fit better if they kept the gif but made some change that seemed to fix it without knowing why.

JohnMakin 3y ago
Maybe, but they don’t know another GIF couldn’t eventually cause the same issue again.