7 ms·
A large crash spike affecting Firefox users on Linux
- tgv 3y ago20000 variables in a function? Even for machine generated code that sounds like an exaggeration. Anybody here who knows the reason?
- idle_zealot 3y agoI don't know for sure, but it sounds like the result of a JS obfuscater. I know Google Docs ships obfuscated code so I wouldn't be surprised if Google Image Search does too.
- viraptor 3y agoIt's not exactly variables. The Bugzilla entry is more specific: "... entails copying all the values that are currently on the interpreter's stack (arguments, local variables, intermediate results) from the heap onto the native stack." https://bugzilla.mozilla.org/show_bug.cgi?id=1839139#c8 https://bugzilla.mozilla.org/show_bug.cgi?id=1839139#c8 So it may be a large function with lots of temporary values visible in scope. If I understand correctly, local lambdas would also count and they have their own captured context. I'm sure it's possible to find a pathological-but-not-unreasonable way to reproduce it.
- IainIreland 3y agoI wrote that comment. One other possibility: if you use spread syntax to call a function with a large array (eg `foo(...Array(20000))`), then all those arguments will be pushed on the stack. (I didn't dig into the specifics of the Google code because, as weird as it is to have 20000 stack values, we really should be able to handle it. This was, at the end of the day, a bug in our stack probing code.)
- hulitu 3y agoWhen you have 16GB of RAM, why bother ? /s
- im3w1l 3y agoThis sounds like blameshifting to me. Javascript is untrusted code, so it's on Firefox to make sure to handle any craziness gracefully. Stack probing is kind of a weird thing though. I'm kind of surprised C++ compiler isn't doing that properly. Are they using inline assembly for speed?
- fathyb 3y agoAgreed that the problem here is Firefox, untrusted JS code should not cause a crash. For the probing, according to the code: // Can't push large frames blindly on windows, so we must touch frame memory // incrementally, with no more than 4096 - 1 bytes between touches. // // This is used across all platforms for simplicity. https://searchfox.org/mozilla-central/rev/c936f47f3a629ae49a4d528d3366bf29f2d4e4a7/js/src/jit/MacroAssembler.cpp#6585-6612 https://searchfox.org/mozilla-central/rev/c936f47f3a629ae49a...
- im3w1l 3y agoYeah I got that far but then I wasn't sure what exactly that was doing. Normal C++ doesn't directly move stack pointers around and do manual probing, so I was idly wondering whether those calls hid some inline assembly or how they worked. Edit: Or perhaps that is not the stack of the C++ program but rather than stack of the Javascript program.
- fathyb 3y agoIt does hide some assembly. Those calls are calling an assembler to generate native code at runtime for JIT compilation. The C++ compiler compiles an assembler, but this assembler runs at runtime. `MacroAssembler` itself is architecture independent, and calls into functions implemented in back-ends such as `MacroAssemblerARM` and `MacroAssemblerX64`. So the code in this function is not performing the stack-probing, it generates code to perform it instead.
- IainIreland 3y agoIn general, there's no difference between the stack of the C++ program and the stack of the JS program. When SpiderMonkey just-in-time compiles a JS function, the result is native code that creates a stack frame on the same stack as the C++ code that implements SpiderMonkey. JS code and C++ code can be semi-arbitrarily interleaved on the stack: JS calls C++ calls JS calls C++... The one exception about sharing the same stack is that the first few iterations of a function run in an interpreter implemented in C++, and that interpreter has its own stack that we heap-allocate. This particular bug occurred during the transition from the C++ interpreter to JIT code.
- semiquaver 3y ago~64 KiB of stack ought to be enough for anybody. https://github.com/torvalds/linux/blob/84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d/arch/x86/mm/fault.c#L1359 https://github.com/torvalds/linux/blob/84df9525b0c27f3ebc2eb...
- bibanez 3y agoThe article mentions this was fixed in Kernel 4.20
- mike_hock 3y agoYes, it ought to be. I can hardly see that as a "bug" in Linux. Clearly, allocating 20k stack variables can't be right and the bug is on Google's side. The stack isn't for bulk storage, I thought that was common knowledge.
- veave 3y agoIf a javascript program crashes the interpreter that is a bug in the interpreter. I don't understand how this is even a discussion. In the era of IE6 there were lots of one-liners that could crash the browser and no one would blame whoever wrote them.
- mike_hock 3y agoIn the interpreter, yes, not the kernel that enforces a reasonable stack limit.
- realitythreek 3y agoIt had already been fixed in the kernel. Linus tends to agree with it being a bug.
- AshamedCaptain 3y agoIts even worse, since they are not increasing the stack pointer and thus firefox is violating some redzone abi somewhere.
- kramerger 3y agoKind of off topic: How can we get Mastodon links like this to open in the Mastodon app? On android this is decided by the url (apps can request certain urls to be forwarded to them) but since there are many different servers in fediverse that becomes impractical
- 2Gkashmiri 3y agohttps://ibb.co/v4pw6rw https://ibb.co/v4pw6rw This URL actually resolves in my android fedilab app so don't know if this has hardcodrd instances or uses regex. It shouldn't be difficult to build a regrx, I know of an app that did for peertube
- shrx 3y agoI'm using Link Eye [1] which lets you open any url in the application of your choice. It's fantastic, also enabling you to open youtube links in NewPipe. [1] https://f-droid.org/packages/kuesji.link_eye.fdroid/ https://f-droid.org/packages/kuesji.link_eye.fdroid/
- Roark66 3y agoCome on now! "a peculiar interpreted code" crashes your interpreter and you blame those that (allegedly) auto generated it? I realise there is a bit of tongue in cheek in there and Firefox is an amazing product, but although it's certainly not normal for a function to declare 20k variables it is not outside the realm of the language. Furthermore, 20k is not that much if you take the amounts of RAM current devices have. Don't get me wrong. I don't think assigning blame is the most important thing to do when troubleshooting. I'd rather not, but when that process starts it should be factual. So, sorry, its not Google's fault... Then we also throw Linux under the bus. "it's not our code, it's Linus" here is the code. But, that Linux kernel code that kills a process if it accesses too far from it's stack pointer has the following comment: "Accessing the stack below %sp is always a bug (...)" I haven't got time to look at the history how it was changed and why in the Linux kernel and it became "not a bug". If someone knows more, please do explain. So is it a "bug in Firefox" Or "bug in old Linux"? I can't say with absolute certainty without researching how exactly the stack allocation in old Linux kernel works, how is it documented etc. So if anything I'd thank Google for exposing the bug ;-) On a side note, I've recently experienced a similar JS/firefox/web site bug. There is this open-source ecommerce software called shopware. They use symfony (yes, PHP i know...) and the most recent major version simply freezes Firefox when one goes to the admin interface and looses connectivity. Not just freezing one tab, no, freezes entire Firefox, multiple open windows. This is on up to date Arch with a new Linux kernel so it's definitely not this issue,but it does happen in Firefox and not in Chrome. JavaScript bugs like this are hard to find. I think AI may be one tool that will help us find them faster (intentionally or not).
- outwit 3y agoIt does seem to me that Google is intentionally refusing to test their websites in anything other than Chrome. Even without AI... It can afford to pay a handful of people to test in Firefox and Safari. But will it?
- db48x 3y agoTo be fair this crash is only happening with older kernels. I wouldn’t expect them to test their website in every version of Firefox running on every version of the Linux kernel. Not even Mozilla does that.
- Tade0 3y ago> It is interesting though that we find ourselves working around a bug we did not introduce triggered by code we do not control. I used to be part of a team developing a popular browser WYSIWYG editor. Every release of any of the supported browsers was a coin toss regarding introducing new bugs. From this perspective developing for the still supported back then IE8 was easier, because there was no chance for it to ever change.
- hulitu 3y agoYes. But. We. Need. New. Features. Every. F...ing. Week.
- wkat4242 3y agoWell... I was damn happy with 114.0 when it FINALLY introduced Webauthn support on Linux and Mac. That was a really long wait. Some features really are worth it. The problem is more that they prioritise the glossy fluff ones. Like time-limited "inspirational" colour schemes.
- webstrand 3y agoI'm confused. I have been using webauthn with a USB authenticator for at least a year now with Firefox. How is this a new feature?
- remram 3y agoIt was when it came out, which is the event GP recalls.
- derkades 3y agoFirefox 114 was just released
- anonymouskimmer 3y agoWhat operating system? The 114.0 (June 6th 2023) release notes say: https://www.mozilla.org/en-US/firefox/114.0/releasenotes/ https://www.mozilla.org/en-US/firefox/114.0/releasenotes/ > Users on macOS, Linux, and Windows 7 can now use FIDO2 / WebAuthn authenticators over USB. Some advanced features, such as fully passwordless logins, require a PIN to be set on the authenticator.
- barrkel 3y agoCompilers generating stack probes for large stack allocations has been a thing for decades. It was required in Windows 95 for 32-bit applications, and you'd do that on a page (4k) granularity. I am still in the dark as to what the bug was here. Did Firefox stop doing probes for JIT code? Not do them at all, because most JS stack frames are small?
- db48x 3y agoFirefox was probing ahead in increments of 2048 bytes in order to ensure that the stack was allocated, but it left the stack pointer at the end of the stack for that whole time. This usually worked ok, but certain versions of Linux will bail if the stack probe is greater than 64kb+256b away from the stack pointer. The new code moves the stack pointer incrementally so that the probe is never more than a single page from the stack pointer. https://phabricator.services.mozilla.com/rMOZILLACENTRAL304d01f5488b384023a1c647e8c5c2f5cee32a5b https://phabricator.services.mozilla.com/rMOZILLACENTRAL304d...
- kingofkyiv 3y ago[dead]
- arun-mani-j 3y agoI use Firefox downloaded from the official website as well as the Flatpak version on my Debian laptop. It crashes frequently, as in every 2 hours or so. I don't do anything heavy on the browser except using it to read documentation. This has been an issue in the last two months and I don't know what's wrong... Also, does anybody known why Firefox still depends on the deprecated libdbus-glib-1-2 [1] in Debian and based distros? For example, try to uninstall the package. Then download the latest Firefox from their website [2]. Extract the archive, launch the executable inside it from a terminal. You will see an error message that it is unable to load the DBus library. 1 - https://packages.debian.org/bookworm/libdbus-glib-1-2 https://packages.debian.org/bookworm/libdbus-glib-1-2 2 - https://www.mozilla.org/en-US/firefox/new/ https://www.mozilla.org/en-US/firefox/new/
- autoexec 3y agoI've got Firefox on a widows 10 box for work and ever since whichever ESR update made it just difficult enough to tell which tab is active to be annoying, Firefox crashes for me all the time and it never just takes out a single tab either. I really blame myself though. It only happens after I've had the browser open for weeks, with at least 3 windows open at once, and literally hundreds of open tabs with hundreds more come and gone since Firefox was restarted. I have about:memory bookmarked, and hitting everything in the Free Memory box seems to delay the inevitable a bit while allowing Javascript seems to make it worse. I doubt many others have the problem and I'm impressed Firefox holds up as well as it does!
- deleted 3y ago[deleted]
- dTP90pN 3y agoHave you looked at the crash reports via `about:crashes`? This should show you if there's any open bug report associated with the crash(es) you encounter. Regarding libdbus-glib-1-2, you may want to open a bug. It looks like [1] it's mainly used by the ~12 year old UPowerClient, and more recently, for wake/sleep and timezone change notifications (nsAppShell). [1] https://searchfox.org/mozilla-central/search?q=DBusG.*&path=&case=false®exp=true https://searchfox.org/mozilla-central/search?q=DBusG.*&path=...
- msla 3y agoI use the Firefox from the Ubuntu Mozilla Team's PPA and I haven't seen any crashes at all. https://launchpad.net/~mozillateam/+archive/ubuntu/ppa https://launchpad.net/~mozillateam/+archive/ubuntu/ppa
- atoav 3y agoI think I had one crash with Firefox on Linux during the past 6 years and that was a weird edge case where I tried to do something during an update process.
- abwizz 3y agosame here, rock solid. not a beta tester thou
- bmicraft 3y agoThe last two times Firefox crashed on me, both of the computers had bad memory
- 0134340 3y agoSame here. Firefox has always been really stable for me but after moving to Linux I had some crashing which coincided with a new bios update. Turns out my memory timings were a bit too high, it couldn't handle the XMP profile with the new bios.
- edflsafoiewq 3y agoSame for me until 114, which has been crashing constantly. I can't open deepl.com for example. 114.0.2 fixes several crashes but not mine.
- pb82 3y agoUsing Google Maps causes Firefox (114.0.2) and the whole OS to completely freeze within minutes on my Laptop. This started happening after I upgraded to Fedora 38. Not sure if this is caused by the same bug. Could be Gnome, Wayland, who knows. Has anyone had a similar experience?
- db48x 3y agoThe bug reported here is a crash, not a freeze, so it’s not the same bug.
- superkuh 3y agoProbably a different issue. Similarly I haven't been able to use Firefox since version 97 without it freezing when using complex applications like Google Maps. Older versions still work but all new (post 97) versions freeze up at the drop of a hat.
- classified 3y agoThey're probably rolling out a new ad engine. Or a bitcoin miner.
- gsatic 3y agoSwitched to Brave on Linux and Android. It's not bad and even find myself using their Search more than Google quite often.
- slig 3y agoBrave Search feels like the good, old Google search from mid-00s.
- em-bee 3y agoand it even works from any browser. as a staunch firefox user i switched to brave search but not brave. very happy with the results. i only wish they would allow linking image and video search to duckduckgo and other search engines, or at least proxy the search instead of linking directly to google and bing
- isaacremuant 3y agoSame. It's increasingly my default browser in all my devices. I went from Yahoo to Google and really valued their product but they became so poor overtime. So focused on spiking their results with ads and then, much worse to me, hiding news and other content based on US centric biased narratives that apparently we all need to follow. It was very apparent and annoying.
- rvz 3y agoGreat choice. Brave seems much more better and does more than what Firefox can do.
- thyrsus 3y agoThe NYTimes animation of the Canadian fires air quality effect froze Firefox, mouse, and keyboard on CentOS 7 for me. I could ssh in from another system, kill -9 Firefox and it recovered.
- XorNot 3y agoThat's on X windows honestly. It shouldn't be possible for a badly behaved program to bring down the window manager and/or display server.
- wkat4242 3y agoIf you can do that you should also be able to just open up a virtual terminal with Control-Alt-F1, just saying :)
- thyrsus 3y agoNope: I tried that. No response to any common keyboard combo. As suggested in a sibling comment, X11 shares the blame.
- deleted 3y ago[deleted]
- BenjiWiebe 3y agoOddly enough, I've had a lot of issues with Firefox on Windows crashing with OOM issues in spite of having GBs of free memory. Multiple different computers, multiple different versions of Windows.
- LordShredda 3y agoIf I make a function with 200 000 stack values and compile it with gcc, people call me an idiot. But when google does it it's fine?
- deleted 3y ago[deleted]
- gigel82 3y agoInterestingly, Mozilla openly talks about their telemetry telling them not only about the spike in crashes but apparently also about the specific website and activity users are doing at the time of the crash (Google Image Search). That's pretty wild, I need to look deeper into how to disable telemetry reporting in Mozilla. I'm pretty sure even Microsoft is sanitizing their crash reports to exclude as much information as possible that could identify the user.