15 ms·

Help the Graphics team track down an interesting WebRender bug

427 points by Santosh83 7y ago

miohtama 7y ago
But hunting is one of the easy to apply kinds of crowd wisdom. Let's hope Mozilla gets enough data to nail down the root cause.
egfx 7y ago
Looks a lot like older graphics cards over heating. I worked in graphics QA.
- trynumber9 7y ago
  I get this frequently. Usually when an update fires and I scroll. Have not yet been able to reproduce reliably. I really doubt it's overheating, since it takes a fraction of a second to happen and it couldn't heat up that quickly.
  - inetknght 7y ago
    > it couldn't heat up that quickly It could, actually, especially if it's not well maintained.
    - trynumber9 7y ago
      It's a $1200 GPU, I do make time to clean. I'm not saying it's Firefox specifically, perhaps a strange interaction between Firefox and the latest Nvidia drivers. But you're welcome to come check my machine.
- Lammy 7y ago
  I experience this bug and my GPU is definitely not overheating.
- chrisan 7y ago
  I will get the black box bug in outlook, temps are fine. I've never seen this in Firefox but makes me wonder if its windows related instead of FF/Outlook specific
- rplnt 7y ago
  My Windows 7 + nvidia does these black boxes. The card also can't handle 2 monitors anymore (crashes the system from time to time).
Rapzid 7y ago
I had issues like this in May of last year in Chrome on Windows 10. Oddly it happened most frequently on TravisCI's website, and I thought at first that it was a TravisCI issue! It turned out a Windows updated over night and replaced my AMD driver with some old compatibility driver. After updating back to the most recent driver the issue went away.
lucideer 7y ago
Given this is the mozillagfx blog, one would hope the "gfx" displayed on the blog were a little better: I can't make out the graphical glitches described in those tiny low-resolution screenshots.
- VWWHFSfQ 7y ago
  you dont see the glitches in this image https://mozillagfx.files.wordpress.com/2020/02/screen-shot-glitch.jpg https://mozillagfx.files.wordpress.com/2020/02/screen-shot-g...
  - ilikehurdles 7y ago
    Is that not how reddit looks?
    - speedgoose 7y ago
      There is old.reddit.com for the boomers.
    - k1t 7y ago
      Interesting to note that even the address bar appears to be affected. Several characters are missing from the URL in that screenshot.
      - capableweb 7y ago
        Yeah, when I had it happen to me, it happens to the entire Firefox instance, settings page, url bar, tab names, context menu (right click menu) and everything in between.
        cpeterso 7y ago
        Much of Firefox's UI is HTML and can be affected by bugs like this. But it is very interesting to see that bug appears to affect both the app UI and web page at the same time.
j0e1 7y ago
I have seen this in the past (I use FF-nightly on MacOS) but subsequent updates seemed to have fixed it. Although maybe some other factor affecting the frequency. Will keep an eye out.
modeless 7y ago
The state of debugging for graphics code is deplorable. When you run code on a GPU you almost always have a gigantic closed source binary blob underneath you, the graphics driver. Graphics drivers contain complete compilers that munge your code with a focus on speed over correctness. The graphics driver is often years out of date, and you can't even run it yourself to reproduce issues unless you go out and buy the exact hardware that each user has. The state of the art in debugging GPU issues is to have a closet full of random hardware purchased used on eBay that hopefully contains something close enough to what you need to reproduce the issue you're working on. Most fancy debugging tools pretend that GPUs don't exist and GPU-focused debuggers are incredibly buggy and feature-poor, not to mention hardware specific in many cases.
- _bxg1 7y ago
  I've never understood the logic behind closed-source drivers. Nobody pays for a driver, people pay for hardware. If you want your hardware to get better support, and therefore become more popular, aren't you incentivized to open-source the drivers?
  - ensiferum 7y ago
    It'd expose trade secrets and give away the details basically giving away any competitive edge.
  - mschuster91 7y ago
    There is a disincentive: patents/IP as one thing (competitors could deduce how your stuff works from oss drivers), and the other thing is exclusivity with games: especially nvidia has deals with game developers to tweak their games in drivers (!) - things like hand optimized shaders or other rendering settings, that provide superior quality for the gamer, so that gamers prefer to buy nvidia cards over AMD. With open source drivers, AMD or Intel could just walk in and steal whatever optimizations nvidia has made for a game.
    - JohnFen 7y ago
      Everything that is patented is already fully disclosed to the public. It's a requirement of getting a patent. Keeping drivers closed-source is more likely to be due to the presence of trade secrets or that the code includes licensed bits that the manufacturer doesn't have the right to disclose the source code to.
dmead 7y ago
is this a real request for help, or a marketing-look-at-our-blog thing?
- Lammy 7y ago
  If you can suggest a better way to get word about a tricky bug out to people who don't usually look at Mozilla's bug tracker I'm sure they'd be happy to hear it :)
- jbonisteel 7y ago
  This is a real request for help!
- rafaelvasco 7y ago
  When in doubt assume the best of people, not the worst by default.
tvelichkov 7y ago
This very much sound like a similar issue Linux users are facing from more than a year and it's still not resolved? https://askubuntu.com/questions/1088324/firefox-becoming-unresponsive-after-suspending-on-kubuntu-18-10 https://askubuntu.com/questions/1088324/firefox-becoming-unr...
- twic 7y ago
  I don't think so - in the Linux bug, Firefox becomes unresponsive, whereas in the bug in the article, Firefox runs as normal, and the blanks can be cleared by scrolling or mousing over. I see a somewhat similar bug on my Linux box, where windows flicker between black and normal, and the flickering stops if i focus on the window. I don't think it's specific to Firefox, though - i believe i've seen all sorts of apps do it. I do have an nvidia graphics card, FWIW.
  - tvelichkov 7y ago
    As someone who experience this very often I can't say that Firefox is becoming unresponsive, you can still scroll up/down, change tabs, etc.. but the all pages are glitched similar (even worse) to the screenshots.
    - jbonisteel 7y ago
      Can you file a bug? https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Graphics%3A+WebRender https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&comp...
petargyurov 7y ago
This bug happens to me somewhat often. In fact it happened to me yesterday. It fixes itself after a couple of seconds. No idea how it happens - I haven't noticed a pattern. I have a GTX 1070.
- godelski 7y ago
  This is interesting. I have a 1060Q and used to get the text tearing all the time. In the last (2 or 3?) FF updates I have yet to see it happen. (Currently v73.0)
- jbonisteel 7y ago
  Could you file a bug for us? https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Graphics%3A+WebRender https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&comp...
  - capableweb 7y ago
    I'm in the same boat with Windows 10 + nvidia 2080ti. Not sure why you want a bug report filled when neither of us have any way of reproducing the issue nor more information about where/when it happens.
    - jdashg 7y ago
      Because that's what bug reports are helpful for: organizing and collecting data, which helps with the whole "finding a repro case" and eventually fixing this issue. Did you think bugs got fixed by mentioning "me too" in a HN comment thread? :)
    - floatingatoll 7y ago
      Bugs are useful because it’s not possible to attach the about:support data requested to an HN comment, and because in general bug trackers have the opportunity to offer enhanced integration of support attachments to saved searches and so forth.
    - ric2b 7y ago
godelski 7y ago
I used to have this problem __ALL__ the time. So I can give some specs to help anyone, because another comment suggests old hardware. I __only__ experienced the text tearing. It would basically be a slash through the screen, usually two or three where text would just be displaced. A small scroll would fix it. This has happened to me in both Manjaro and Ubuntu and on two different computers. The Manjaro is a Inspiron with a 1060Qmax. The Ubuntu has a GTK Titan Black (so old). I have not experienced these glitches in fairly recent memory. So probably last two versions of FF. These computers are my daily drivers and I keep them fairly up to date. Usually where these glitches would happen is in long form text articles. I don't recall seeing them on Reddit, but more like NYT. Hope this helps.
- rahuldottech 7y ago
  I've been experiencing this too! On only one of my machines. I did a complete Windows reinstall but the problem persisted. It doesn't happen too often, but iirc I saw a text glitch sometime in the past few days. Moving my mouse over the glitched area or a small scroll solves it for me too.
  - godelski 7y ago
    Yeah I could get rid of it by scrolling or highlighting the text. I remember looking it up and only find like one small bug report a year ago. But I don't see it anymore. Are you on the newest FF? I don't use nightly and I haven't seen this problem in awhile.
    - rahuldottech 7y ago
      Just checked, I'm on latest regular FF, non-nightly. I don't recall seeing it in the past couple days so maaaaybe it's resolved now? I'll keep an eye out for it in case I see it again.
rvz 7y ago
This reminds me of the comments section or sometimes a mailing list for a particular bug in a driver or a low-level component and most of all the replies were from 'users' who weren't helping to solving the actual problem in the discussion. Then one day an unknown contributor not only was able to reproduce the bug but also sent an actual fix and ended up gaining commit access to the project after staying there for a while. Whenever I hear the 'mythical legend of the 10X rockstar developer', I think of them. Mozilla needs more of those developers to reproduce AND send patches to interesting bugs like this. Not the whining copypasta forum-posting 'users' calling themselves 'developers' who take many open-source projects like this for granted and don't appreciate the time spent by the maintainers of the project. Nowadays they're usually doing this on a developer social network through every maintainers nightmare 'The Issues Tab'.
- paxys 7y ago
  Hah! As an open source maintainer for a mildly popular repo a big chunk of my time is spent weeding out people filing issues and opening PRs for trivial changes (like adding a line to the README file) just so they can claim to be a "contributor". Meanwhile there are a ton of issues that are ready to code with a full explanation and are even marked as such ("good first issue" or whatever else) that stay untouched. Regardless of community engagement, perceived interest, Github stars or whatever else — finding a developer willing to spend actual time and effort to make a project better is very, very rare.
  - jpxw 7y ago
    Which repo, out of interest?
  - sarah180 7y ago
    Yeah, people who drive-by PRs to unnecessarily fix a minuscule style or dependency-version issue is one of my pet peeves. I don't spend a lot of time on it, but this kind of cynical game playing sucks. These kinds of contributions produce a net loss for OSS, not a net win.
    - 7y ago
Const-me 7y ago
I’m almost sure it’s incorrect usage of D3D. I’ve encountered and fixed similar bugs in my 3D graphics code. Based on the symptoms and difficulties to reproduce, I think what you see on the screenshots are incomplete renders. The FF code submitted some draw calls to GPU, and copied the render target texture without waiting for draw calls to complete. A good way to fix is use a D3D11_QUERY_EVENT query to wait for completion of rendering. If all GPU access happens through D3D11 this shouldn’t happen i.e. the API guarantees to wait for the completion of previously submitted draw calls. It may happen in practice when mixing multiple GPU APIs, e.g. when using DXGI surface sharing to pass textures between D3D11 and DX9. Also, it may happen when using D3D from multiple threads. Easy way to detect the latter, set D3D11_CREATE_DEVICE_DEBUG flag when creating the device, and read warnings in debug output of the process. Another possibility is just bugs in rendering code. To troubleshoot them, https://renderdoc.org/ https://renderdoc.org/ is awesome. Unfortunately, if the FF doesn’t use D3D to present, only rendering to textures, some changes to FF’s source code is required to capture frames with RD, see this page for details: https://renderdoc.org/docs/in_application_api.html https://renderdoc.org/docs/in_application_api.html
- deleted 7y ago
  [deleted]
- ehsankia 7y ago
  Makes sense that a delay sensitive bug is hard to repro too. I have definitely seen similar artifacts throughout the years in Chrome too come and go, seems like it's a common problem in browser D3D implementations.
  - Const-me 7y ago
    A few years ago a customer reported a bug in my software which only happened with a specific 3D scene, on specific GPUs (my desktop rendered that scene fine), and was only happening rarely. That’s how I have learned about that GPU synchronization issue across graphics APIs which may cause incomplete renders.
- core-questions 7y ago
electrotype 7y ago
Why the Nightly version only, I'm curious?
- sleavey 7y ago
  The Firefox developers only patch security vulnerabilities in standard Firefox releases. All minor bug fixes and new features go into Nightly and only after a few months into stock Firefox.
- cpeterso 7y ago
  Presumably there are other graphics bugs that have been fixed in Firefox Nightly but still exist in the Beta or Stable Release channels. Best to debug and reproduce the bug in the code closest to version the Mozilla engineers are using.
dx87 7y ago
I was having a weird WebRender issue last year, except it was causing sudden reboots until the computer stayed powered off long enough for the GPU memory to be cleared. I had enabled WebRender in Linux (it's experimental, I know), and the computer would run fine for weeks at a time, then reboot out of nowhere. It would continue to reboot any time I opened firefox, sometimes a few minutes after starting firefox. The reboots would persist even when I would boot into Windows instead of Linux after one of the forced reboots. Once I narrowed it down to firefox causing the reboots, I started it in safe mode and disabled WebRender, and I haven't had any issues since. This was with a 1050TI graphics card.
- deleted 7y ago
  [deleted]
- jamienicol 7y ago
  That sounds more like a Linux or hardware issue than a webrender one! (Likewise if a website causes Firefox to crash, that's a Firefox issue rather than the website's)
tomlong 7y ago
Heh, i was talking about this today. I do three days a month for a client where I use their computers and it's the only time I ever use Windows 10 and the only time I see this. I was asking around about it but no one else was using Firefox. Next time I'm in I'll pay more attention.
- jbonisteel 7y ago
  If you can grab the info in about:support for that machine and file a bug: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Graphics%3A+WebRender https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&comp... that would be helpful!
rmccue 7y ago
I am fairly certain I’ve seen this bug, but on macOS using a MBP’s integrated graphics. I opted into WebRender a while ago via about:config, so I had figured it was just a random beta issue. Will have to see if I can capture it next time it happens!
- skept 7y ago
  Me too. I've seen it happen on more than a couple of occasions while visiting Trello.com and YouTube.com. Firefox isn't my primary browser, partly for this reason. MBP 2013 with integrated graphics, running Mojave and the latest stable public release of Firefox. I haven't messed with about:config.
- dawnerd 7y ago
  Same here as well, but in fairness I've also seen it happen in Chrome too on the same sites.
  - nicoburns 7y ago
    Came here to say this. I've seen this on my (2015 13 inch MBP), in both Chrome and Firefox.
thefurman 7y ago
You should ask the people having the issue to switch graphics cards, PSU and RAM since this >90% smells like a memory/power issue. I think you've just gotten unlucky and collected a few similar reports of people having memory corruptions. I am guessing you cache renderings and that is why the issue sticks around until you re-render. If you want to reproduce it yourselves then perhaps try pointing a hairdryer from a distance at the various components until they start to create trouble, or alternatively just overclock them towards the breaking points.
- thefurman 7y ago
  Why the downvotes? Anything contradicting this which you might want to share in text instead?
pg_is_a_butt 7y ago
I thought I was the only one seeing this... they claim it's only windows 10, but I'm also seeing it on a 2012 macbook pro i5 intel graphics 3000. Not very often, but enough to be very annoyed and concerned by it. My fear is some other program is gaining access to portions of the screen or video memory to scrape for banking information using ocr or something like that. Sometimes the glitches will be filled with upside down and stretched/keystone shifted versions of an area of a rendered page from a different tab. The keystone shift triggers my off-by-1 memory allocation bug sense... But it might be some other driver issue or exploit causing that on purpose somehow (like rowhammer). I fear this is going to take years to track down.
reuben_scratton 7y ago
I haven't seen this particular issue but I've seen very similar bugs in my own work where GPU and CPU are not properly sync'd when using glyph or small image cache textures. E.g. gpu has a frame pending, CPU updates glyph cache, GPU renders pending frame using incorrect cache data.
j1elo 7y ago
I'm curious, why does a team working for Mozilla need to host their blog as a free account on wordpress.com? They really don't have a unified blog system to provide a more official feeling to all their different departments? Or is this just an unofficial blog from people that is unrelated to Mozilla?
- staktrace 7y ago
  In this respect (and probably others) Mozilla is more like a ragtag army of rebels than a uniform-wearing platoon of soldiers. The blog is hosted on WordPress because the person who started it found it most convenient, and it never got migrated anywhere else. We also have hundreds of different github organizations and repos (despite also hosting our own "official" mercurial server), use lots of different communication tools, spread documentation over multiple wikis and bug trackers, etc. But yes, this is an "official" blog from Mozilla employees on the graphics team, of which I too am a member.
  - j1elo 7y ago
    That's both strange, very confusing, and amazing at the same time :-) It surprised me that Mozilla is able to simultaneously have things like the MDN, basically the best resource for HTML and Web APIs, but then other references or publications scattered around everywhere.
MayeulC 7y ago
> Update: we have created the channel #gfx-wr-glitch on Matrix Funnily, this alias is incomplete and therefore not enough to join. A room ID might have been enough, but just an alias without a server part is worthless, a bit like someone left an e-mail adress as gfx-wr-glitch@, without specifying the rest. I've successfully joined trough #gfx-wr-glitch:mozilla.org, (roomid is !qaAtjMWizEgMcVSbNH:mozilla.org, no alias on matrix.org) and told them so. Edit: they fixed it. I wonder how often that happens with those new to Matrix?
- Dylan16807 7y ago
  I don't think that's a "new to Matrix" thing. They run their own server, just like they ran their own IRC server before, and at first didn't really consider the need to make that explicit. I doubt there was any misunderstanding of the tech, or that the channel names are per-server.
AlexMax 7y ago
Son of a gun, I have been running into this issue sporadically ever since I built my new PC with a Radeon 5700 XT, and I thought it might have been a bug in the drivers or a bad card, but turns out it's this instead. For what it's worth, I downgraded my drivers to the last set of drivers from 2019 (19.9.2) and the problem seems to have gone away, but I'm having other issues instead (for instance, half the screen flashes dark when Night Light turns on). Guess it's time to upgrade and file some bugs.
shitloadofbooks 7y ago
I had a similar issue in Chrome and a regular crash-to-desktop in the main game I was playing at the time (Divinity 2) on Windows 10 with a GTX 970. I also had major problems with YouTube. My fix was to turn off all the Dynamic Clock and GPU Boost stuff in the NVIDIA control panel.
deleted 7y ago
[deleted]
irrational 7y ago
> Without having a way to reliably reproduce this bug, we are at a loss on how to solve it. I wish non-technical people understood this.
zamalek 7y ago
Black rectangle rings true here. I get it pretty reliably on my phone (preview nightly, Android). If I click a GitHub link and FastHub automatically opens (any GitHub link), when I go back on the phone I see a black rectangle for the whole page. Pixel 3. Hopefully it's a common codepath.
- jamienicol 7y ago
  That sounds like a separate issue. Similar to one I had before on android, but fixed. If you go to settings, about Firefox nightly, what version does it say? Does the black go away by itself after a couple seconds, or when you scroll, or does it stay forever?
nonbirithm 7y ago
I wonder if Mozilla has enough resources to test on Windows machines, or if it's just bad luck with hardware configuration, or something else entirely. I reported an unrelated bug in Nightly where I couldn't use smartcard authentication and since it only reproed on Windows we had a back and forth exchanging screenshots and logs until the root cause was narrowed down.
nootropicat 7y ago
I get it all the time but I have gfx.webrender.all set to false. gfx.webrender.blob-images, webrender.dcomp-win.enabled, force-angle, picture-caching, triple-buffering.enabled are set to true However gfx.webrender.enabled is set to false so are they even active? FF 72.0.2
- jamienicol 7y ago
  If you got to about:support and check the value of "compositing", you will see if webrender is enabled.
Izkata 7y ago
Hmm... This looks identical to something that occurred on Ubuntu 14.04 when I came out of suspend, except it happened to all gtk apps. Switching to another virtual terminal then back to 7 seemed to reinitialize something and fix it. Even nowadays, on rare occasion on Ubuntu 18.04, something like this occurs during startup of Android Studio. I don't recall seeing it on Firefox since at least 16.04 though.
tomaszs 7y ago
A classical race condiction double bug. What happens here is there is a a special case of the data. So other routine of processing the data is used. It causes a race condition that results in proving incomplete data to the next step of the process. There is surely logic to prevent it from happening. But the data initially used the bad routine so no one expects the race condition to occur. And at the end we have a bad routine that processes data badly because data are bad and incomplete. This routine does not expect how it receives the data and in what time. In a result rendering breaks, but for the logic of the code it seems fine. So the error block is displayed. Gor GPU rendering it is a black box. What is interesting if you did deep enought to find all these 3 issues that cause it each of them separately is fine. It is what they do together causes the issue. Very nasty thing to track. And also fix. Because on a high level there is little to about it. And a workaround is mostly to detect the situation and process data slightly different way to avoid the issue, or to just do processing two times in a rather hacky way.
bfdm 7y ago
This is so bizarre for me because it's like a flashback. I was seeing errors exactly like this in Firefox years ago, around 4-5 years or more I'd say. This was on windows 7 64bit with a probably outdated install because my favorite extension for vertical tab tree was always behind on version support. This error would manifest when I had many tabs open for a long time. I'd often keep sessions of hundreds of tabs open for weeks. The errors would start to appear and gradually get worse until I had to force crash ffx and restart (and recover my tabs!) Not really actionable or reoro, but anecdotally maybe it helps to hear this isn't strictly new.
looperhacks 7y ago
OT: This is the first time firefox has warned me about blocking social media trackers. Funny that it happened on one of mozillas own websites.
- nical 7y ago
  Yeah that's a wordpress blog quickly put together outside of mozilla's official blogging infrastructure. There was no really good reason to do that, it was just laziness on our end.
pja 7y ago
I have seen very similar bugs (the same text glitches for sure) under Linux (Radeon RX580, open source drivers) in the past. I had put it down to open source drive stack bugs, but maybe it was actually a Firefox bug? I’ll try and take a screenshot next time it happens & report it.

427 points by Santosh83 7y ago

But hunting is one of the easy to apply kinds of crowd wisdom. Let's hope Mozilla gets enough data to nail down the root cause.

egfx 7y ago

Looks a lot like older graphics cards over heating. I worked in graphics QA.

trynumber9 7y ago
I get this frequently. Usually when an update fires and I scroll. Have not yet been able to reproduce reliably. I really doubt it's overheating, since it takes a fraction of a second to happen and it couldn't heat up that quickly.
- inetknght 7y ago
  > it couldn't heat up that quickly It could, actually, especially if it's not well maintained.
  - trynumber9 7y ago
    It's a $1200 GPU, I do make time to clean. I'm not saying it's Firefox specifically, perhaps a strange interaction between Firefox and the latest Nvidia drivers. But you're welcome to come check my machine.
Lammy 7y ago
I experience this bug and my GPU is definitely not overheating.
chrisan 7y ago
I will get the black box bug in outlook, temps are fine. I've never seen this in Firefox but makes me wonder if its windows related instead of FF/Outlook specific
rplnt 7y ago
My Windows 7 + nvidia does these black boxes. The card also can't handle 2 monitors anymore (crashes the system from time to time).

Rapzid 7y ago

I had issues like this in May of last year in Chrome on Windows 10. Oddly it happened most frequently on TravisCI's website, and I thought at first that it was a TravisCI issue! It turned out a Windows updated over night and replaced my AMD driver with some old compatibility driver. After updating back to the most recent driver the issue went away.

lucideer 7y ago

Given this is the mozillagfx blog, one would hope the "gfx" displayed on the blog were a little better: I can't make out the graphical glitches described in those tiny low-resolution screenshots.

VWWHFSfQ 7y ago
you dont see the glitches in this image https://mozillagfx.files.wordpress.com/2020/02/screen-shot-glitch.jpg https://mozillagfx.files.wordpress.com/2020/02/screen-shot-g...
- ilikehurdles 7y ago
  Is that not how reddit looks?
  - speedgoose 7y ago
    There is old.reddit.com for the boomers.
  - k1t 7y ago
    Interesting to note that even the address bar appears to be affected. Several characters are missing from the URL in that screenshot.
    - capableweb 7y ago
      Yeah, when I had it happen to me, it happens to the entire Firefox instance, settings page, url bar, tab names, context menu (right click menu) and everything in between.
      - cpeterso 7y ago
        Much of Firefox's UI is HTML and can be affected by bugs like this. But it is very interesting to see that bug appears to affect both the app UI and web page at the same time.

j0e1 7y ago

I have seen this in the past (I use FF-nightly on MacOS) but subsequent updates seemed to have fixed it. Although maybe some other factor affecting the frequency. Will keep an eye out.

modeless 7y ago

The state of debugging for graphics code is deplorable. When you run code on a GPU you almost always have a gigantic closed source binary blob underneath you, the graphics driver. Graphics drivers contain complete compilers that munge your code with a focus on speed over correctness. The graphics driver is often years out of date, and you can't even run it yourself to reproduce issues unless you go out and buy the exact hardware that each user has. The state of the art in debugging GPU issues is to have a closet full of random hardware purchased used on eBay that hopefully contains something close enough to what you need to reproduce the issue you're working on. Most fancy debugging tools pretend that GPUs don't exist and GPU-focused debuggers are incredibly buggy and feature-poor, not to mention hardware specific in many cases.

_bxg1 7y ago
I've never understood the logic behind closed-source drivers. Nobody pays for a driver, people pay for hardware. If you want your hardware to get better support, and therefore become more popular, aren't you incentivized to open-source the drivers?
- ensiferum 7y ago
  It'd expose trade secrets and give away the details basically giving away any competitive edge.
- mschuster91 7y ago
  There is a disincentive: patents/IP as one thing (competitors could deduce how your stuff works from oss drivers), and the other thing is exclusivity with games: especially nvidia has deals with game developers to tweak their games in drivers (!) - things like hand optimized shaders or other rendering settings, that provide superior quality for the gamer, so that gamers prefer to buy nvidia cards over AMD. With open source drivers, AMD or Intel could just walk in and steal whatever optimizations nvidia has made for a game.
  - JohnFen 7y ago
    Everything that is patented is already fully disclosed to the public. It's a requirement of getting a patent. Keeping drivers closed-source is more likely to be due to the presence of trade secrets or that the code includes licensed bits that the manufacturer doesn't have the right to disclose the source code to.

dmead 7y ago

is this a real request for help, or a marketing-look-at-our-blog thing?

Lammy 7y ago
If you can suggest a better way to get word about a tricky bug out to people who don't usually look at Mozilla's bug tracker I'm sure they'd be happy to hear it :)
jbonisteel 7y ago
This is a real request for help!
rafaelvasco 7y ago
When in doubt assume the best of people, not the worst by default.

tvelichkov 7y ago

This very much sound like a similar issue Linux users are facing from more than a year and it's still not resolved? https://askubuntu.com/questions/1088324/firefox-becoming-unresponsive-after-suspending-on-kubuntu-18-10 https://askubuntu.com/questions/1088324/firefox-becoming-unr...

twic 7y ago
I don't think so - in the Linux bug, Firefox becomes unresponsive, whereas in the bug in the article, Firefox runs as normal, and the blanks can be cleared by scrolling or mousing over. I see a somewhat similar bug on my Linux box, where windows flicker between black and normal, and the flickering stops if i focus on the window. I don't think it's specific to Firefox, though - i believe i've seen all sorts of apps do it. I do have an nvidia graphics card, FWIW.
- tvelichkov 7y ago
  As someone who experience this very often I can't say that Firefox is becoming unresponsive, you can still scroll up/down, change tabs, etc.. but the all pages are glitched similar (even worse) to the screenshots.
  - jbonisteel 7y ago
    Can you file a bug? https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Graphics%3A+WebRender https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&comp...

petargyurov 7y ago

This bug happens to me somewhat often. In fact it happened to me yesterday. It fixes itself after a couple of seconds. No idea how it happens - I haven't noticed a pattern. I have a GTX 1070.

godelski 7y ago
This is interesting. I have a 1060Q and used to get the text tearing all the time. In the last (2 or 3?) FF updates I have yet to see it happen. (Currently v73.0)
jbonisteel 7y ago
Could you file a bug for us? https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Graphics%3A+WebRender https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&comp...
- capableweb 7y ago
  I'm in the same boat with Windows 10 + nvidia 2080ti. Not sure why you want a bug report filled when neither of us have any way of reproducing the issue nor more information about where/when it happens.
  - jdashg 7y ago
    Because that's what bug reports are helpful for: organizing and collecting data, which helps with the whole "finding a repro case" and eventually fixing this issue. Did you think bugs got fixed by mentioning "me too" in a HN comment thread? :)
  - floatingatoll 7y ago
    Bugs are useful because it’s not possible to attach the about:support data requested to an HN comment, and because in general bug trackers have the opportunity to offer enhanced integration of support attachments to saved searches and so forth.
  - ric2b 7y ago

godelski 7y ago

I used to have this problem __ALL__ the time. So I can give some specs to help anyone, because another comment suggests old hardware. I __only__ experienced the text tearing. It would basically be a slash through the screen, usually two or three where text would just be displaced. A small scroll would fix it. This has happened to me in both Manjaro and Ubuntu and on two different computers. The Manjaro is a Inspiron with a 1060Qmax. The Ubuntu has a GTK Titan Black (so old). I have not experienced these glitches in fairly recent memory. So probably last two versions of FF. These computers are my daily drivers and I keep them fairly up to date. Usually where these glitches would happen is in long form text articles. I don't recall seeing them on Reddit, but more like NYT. Hope this helps.

rahuldottech 7y ago
I've been experiencing this too! On only one of my machines. I did a complete Windows reinstall but the problem persisted. It doesn't happen too often, but iirc I saw a text glitch sometime in the past few days. Moving my mouse over the glitched area or a small scroll solves it for me too.
- godelski 7y ago
  Yeah I could get rid of it by scrolling or highlighting the text. I remember looking it up and only find like one small bug report a year ago. But I don't see it anymore. Are you on the newest FF? I don't use nightly and I haven't seen this problem in awhile.
  - rahuldottech 7y ago
    Just checked, I'm on latest regular FF, non-nightly. I don't recall seeing it in the past couple days so maaaaybe it's resolved now? I'll keep an eye out for it in case I see it again.

rvz 7y ago

This reminds me of the comments section or sometimes a mailing list for a particular bug in a driver or a low-level component and most of all the replies were from 'users' who weren't helping to solving the actual problem in the discussion. Then one day an unknown contributor not only was able to reproduce the bug but also sent an actual fix and ended up gaining commit access to the project after staying there for a while. Whenever I hear the 'mythical legend of the 10X rockstar developer', I think of them. Mozilla needs more of those developers to reproduce AND send patches to interesting bugs like this. Not the whining copypasta forum-posting 'users' calling themselves 'developers' who take many open-source projects like this for granted and don't appreciate the time spent by the maintainers of the project. Nowadays they're usually doing this on a developer social network through every maintainers nightmare 'The Issues Tab'.

paxys 7y ago
Hah! As an open source maintainer for a mildly popular repo a big chunk of my time is spent weeding out people filing issues and opening PRs for trivial changes (like adding a line to the README file) just so they can claim to be a "contributor". Meanwhile there are a ton of issues that are ready to code with a full explanation and are even marked as such ("good first issue" or whatever else) that stay untouched. Regardless of community engagement, perceived interest, Github stars or whatever else — finding a developer willing to spend actual time and effort to make a project better is very, very rare.
- jpxw 7y ago
  Which repo, out of interest?
- sarah180 7y ago
  Yeah, people who drive-by PRs to unnecessarily fix a minuscule style or dependency-version issue is one of my pet peeves. I don't spend a lot of time on it, but this kind of cynical game playing sucks. These kinds of contributions produce a net loss for OSS, not a net win.
  - 7y ago

Const-me 7y ago

I’m almost sure it’s incorrect usage of D3D. I’ve encountered and fixed similar bugs in my 3D graphics code. Based on the symptoms and difficulties to reproduce, I think what you see on the screenshots are incomplete renders. The FF code submitted some draw calls to GPU, and copied the render target texture without waiting for draw calls to complete. A good way to fix is use a D3D11_QUERY_EVENT query to wait for completion of rendering. If all GPU access happens through D3D11 this shouldn’t happen i.e. the API guarantees to wait for the completion of previously submitted draw calls. It may happen in practice when mixing multiple GPU APIs, e.g. when using DXGI surface sharing to pass textures between D3D11 and DX9. Also, it may happen when using D3D from multiple threads. Easy way to detect the latter, set D3D11_CREATE_DEVICE_DEBUG flag when creating the device, and read warnings in debug output of the process. Another possibility is just bugs in rendering code. To troubleshoot them, https://renderdoc.org/ https://renderdoc.org/ is awesome. Unfortunately, if the FF doesn’t use D3D to present, only rendering to textures, some changes to FF’s source code is required to capture frames with RD, see this page for details: https://renderdoc.org/docs/in_application_api.html https://renderdoc.org/docs/in_application_api.html

deleted 7y ago
[deleted]
ehsankia 7y ago
Makes sense that a delay sensitive bug is hard to repro too. I have definitely seen similar artifacts throughout the years in Chrome too come and go, seems like it's a common problem in browser D3D implementations.
- Const-me 7y ago
  A few years ago a customer reported a bug in my software which only happened with a specific 3D scene, on specific GPUs (my desktop rendered that scene fine), and was only happening rarely. That’s how I have learned about that GPU synchronization issue across graphics APIs which may cause incomplete renders.
core-questions 7y ago

electrotype 7y ago

Why the Nightly version only, I'm curious?

sleavey 7y ago
The Firefox developers only patch security vulnerabilities in standard Firefox releases. All minor bug fixes and new features go into Nightly and only after a few months into stock Firefox.
cpeterso 7y ago
Presumably there are other graphics bugs that have been fixed in Firefox Nightly but still exist in the Beta or Stable Release channels. Best to debug and reproduce the bug in the code closest to version the Mozilla engineers are using.

dx87 7y ago

I was having a weird WebRender issue last year, except it was causing sudden reboots until the computer stayed powered off long enough for the GPU memory to be cleared. I had enabled WebRender in Linux (it's experimental, I know), and the computer would run fine for weeks at a time, then reboot out of nowhere. It would continue to reboot any time I opened firefox, sometimes a few minutes after starting firefox. The reboots would persist even when I would boot into Windows instead of Linux after one of the forced reboots. Once I narrowed it down to firefox causing the reboots, I started it in safe mode and disabled WebRender, and I haven't had any issues since. This was with a 1050TI graphics card.

deleted 7y ago
[deleted]
jamienicol 7y ago
That sounds more like a Linux or hardware issue than a webrender one! (Likewise if a website causes Firefox to crash, that's a Firefox issue rather than the website's)

tomlong 7y ago

Heh, i was talking about this today. I do three days a month for a client where I use their computers and it's the only time I ever use Windows 10 and the only time I see this. I was asking around about it but no one else was using Firefox. Next time I'm in I'll pay more attention.

jbonisteel 7y ago
If you can grab the info in about:support for that machine and file a bug: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Graphics%3A+WebRender https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&comp... that would be helpful!

rmccue 7y ago

I am fairly certain I’ve seen this bug, but on macOS using a MBP’s integrated graphics. I opted into WebRender a while ago via about:config, so I had figured it was just a random beta issue. Will have to see if I can capture it next time it happens!

skept 7y ago
Me too. I've seen it happen on more than a couple of occasions while visiting Trello.com and YouTube.com. Firefox isn't my primary browser, partly for this reason. MBP 2013 with integrated graphics, running Mojave and the latest stable public release of Firefox. I haven't messed with about:config.
dawnerd 7y ago
Same here as well, but in fairness I've also seen it happen in Chrome too on the same sites.
- nicoburns 7y ago
  Came here to say this. I've seen this on my (2015 13 inch MBP), in both Chrome and Firefox.

thefurman 7y ago

You should ask the people having the issue to switch graphics cards, PSU and RAM since this >90% smells like a memory/power issue. I think you've just gotten unlucky and collected a few similar reports of people having memory corruptions. I am guessing you cache renderings and that is why the issue sticks around until you re-render. If you want to reproduce it yourselves then perhaps try pointing a hairdryer from a distance at the various components until they start to create trouble, or alternatively just overclock them towards the breaking points.

thefurman 7y ago
Why the downvotes? Anything contradicting this which you might want to share in text instead?

pg_is_a_butt 7y ago

I thought I was the only one seeing this... they claim it's only windows 10, but I'm also seeing it on a 2012 macbook pro i5 intel graphics 3000. Not very often, but enough to be very annoyed and concerned by it. My fear is some other program is gaining access to portions of the screen or video memory to scrape for banking information using ocr or something like that. Sometimes the glitches will be filled with upside down and stretched/keystone shifted versions of an area of a rendered page from a different tab. The keystone shift triggers my off-by-1 memory allocation bug sense... But it might be some other driver issue or exploit causing that on purpose somehow (like rowhammer). I fear this is going to take years to track down.

reuben_scratton 7y ago

I haven't seen this particular issue but I've seen very similar bugs in my own work where GPU and CPU are not properly sync'd when using glyph or small image cache textures. E.g. gpu has a frame pending, CPU updates glyph cache, GPU renders pending frame using incorrect cache data.

j1elo 7y ago

I'm curious, why does a team working for Mozilla need to host their blog as a free account on wordpress.com? They really don't have a unified blog system to provide a more official feeling to all their different departments? Or is this just an unofficial blog from people that is unrelated to Mozilla?

staktrace 7y ago
In this respect (and probably others) Mozilla is more like a ragtag army of rebels than a uniform-wearing platoon of soldiers. The blog is hosted on WordPress because the person who started it found it most convenient, and it never got migrated anywhere else. We also have hundreds of different github organizations and repos (despite also hosting our own "official" mercurial server), use lots of different communication tools, spread documentation over multiple wikis and bug trackers, etc. But yes, this is an "official" blog from Mozilla employees on the graphics team, of which I too am a member.
- j1elo 7y ago
  That's both strange, very confusing, and amazing at the same time :-) It surprised me that Mozilla is able to simultaneously have things like the MDN, basically the best resource for HTML and Web APIs, but then other references or publications scattered around everywhere.

MayeulC 7y ago

> Update: we have created the channel #gfx-wr-glitch on Matrix Funnily, this alias is incomplete and therefore not enough to join. A room ID might have been enough, but just an alias without a server part is worthless, a bit like someone left an e-mail adress as gfx-wr-glitch@, without specifying the rest. I've successfully joined trough #gfx-wr-glitch:mozilla.org, (roomid is !qaAtjMWizEgMcVSbNH:mozilla.org, no alias on matrix.org) and told them so. Edit: they fixed it. I wonder how often that happens with those new to Matrix?

Dylan16807 7y ago
I don't think that's a "new to Matrix" thing. They run their own server, just like they ran their own IRC server before, and at first didn't really consider the need to make that explicit. I doubt there was any misunderstanding of the tech, or that the channel names are per-server.

AlexMax 7y ago

Son of a gun, I have been running into this issue sporadically ever since I built my new PC with a Radeon 5700 XT, and I thought it might have been a bug in the drivers or a bad card, but turns out it's this instead. For what it's worth, I downgraded my drivers to the last set of drivers from 2019 (19.9.2) and the problem seems to have gone away, but I'm having other issues instead (for instance, half the screen flashes dark when Night Light turns on). Guess it's time to upgrade and file some bugs.

shitloadofbooks 7y ago

I had a similar issue in Chrome and a regular crash-to-desktop in the main game I was playing at the time (Divinity 2) on Windows 10 with a GTX 970. I also had major problems with YouTube. My fix was to turn off all the Dynamic Clock and GPU Boost stuff in the NVIDIA control panel.

deleted 7y ago

[deleted]

irrational 7y ago

> Without having a way to reliably reproduce this bug, we are at a loss on how to solve it. I wish non-technical people understood this.

zamalek 7y ago

Black rectangle rings true here. I get it pretty reliably on my phone (preview nightly, Android). If I click a GitHub link and FastHub automatically opens (any GitHub link), when I go back on the phone I see a black rectangle for the whole page. Pixel 3. Hopefully it's a common codepath.

jamienicol 7y ago
That sounds like a separate issue. Similar to one I had before on android, but fixed. If you go to settings, about Firefox nightly, what version does it say? Does the black go away by itself after a couple seconds, or when you scroll, or does it stay forever?

nonbirithm 7y ago

I wonder if Mozilla has enough resources to test on Windows machines, or if it's just bad luck with hardware configuration, or something else entirely. I reported an unrelated bug in Nightly where I couldn't use smartcard authentication and since it only reproed on Windows we had a back and forth exchanging screenshots and logs until the root cause was narrowed down.

nootropicat 7y ago

I get it all the time but I have gfx.webrender.all set to false. gfx.webrender.blob-images, webrender.dcomp-win.enabled, force-angle, picture-caching, triple-buffering.enabled are set to true However gfx.webrender.enabled is set to false so are they even active? FF 72.0.2

jamienicol 7y ago
If you got to about:support and check the value of "compositing", you will see if webrender is enabled.

Izkata 7y ago

Hmm... This looks identical to something that occurred on Ubuntu 14.04 when I came out of suspend, except it happened to all gtk apps. Switching to another virtual terminal then back to 7 seemed to reinitialize something and fix it. Even nowadays, on rare occasion on Ubuntu 18.04, something like this occurs during startup of Android Studio. I don't recall seeing it on Firefox since at least 16.04 though.

tomaszs 7y ago

A classical race condiction double bug. What happens here is there is a a special case of the data. So other routine of processing the data is used. It causes a race condition that results in proving incomplete data to the next step of the process. There is surely logic to prevent it from happening. But the data initially used the bad routine so no one expects the race condition to occur. And at the end we have a bad routine that processes data badly because data are bad and incomplete. This routine does not expect how it receives the data and in what time. In a result rendering breaks, but for the logic of the code it seems fine. So the error block is displayed. Gor GPU rendering it is a black box. What is interesting if you did deep enought to find all these 3 issues that cause it each of them separately is fine. It is what they do together causes the issue. Very nasty thing to track. And also fix. Because on a high level there is little to about it. And a workaround is mostly to detect the situation and process data slightly different way to avoid the issue, or to just do processing two times in a rather hacky way.

bfdm 7y ago

This is so bizarre for me because it's like a flashback. I was seeing errors exactly like this in Firefox years ago, around 4-5 years or more I'd say. This was on windows 7 64bit with a probably outdated install because my favorite extension for vertical tab tree was always behind on version support. This error would manifest when I had many tabs open for a long time. I'd often keep sessions of hundreds of tabs open for weeks. The errors would start to appear and gradually get worse until I had to force crash ffx and restart (and recover my tabs!) Not really actionable or reoro, but anecdotally maybe it helps to hear this isn't strictly new.

looperhacks 7y ago

OT: This is the first time firefox has warned me about blocking social media trackers. Funny that it happened on one of mozillas own websites.

nical 7y ago
Yeah that's a wordpress blog quickly put together outside of mozilla's official blogging infrastructure. There was no really good reason to do that, it was just laziness on our end.

pja 7y ago

I have seen very similar bugs (the same text glitches for sure) under Linux (Radeon RX580, open source drivers) in the past. I had put it down to open source drive stack bugs, but maybe it was actually a Firefox bug? I’ll try and take a screenshot next time it happens & report it.