5 ms·
Mozilla has been fighting the MSVC linker's 3GB virtual address space limit for years. Mozilla has asked Microsoft for a 64-bit linker that can produce 32-bit c
by cpeterso 13y ago
Mozilla has been fighting the MSVC linker's 3GB virtual address space limit for years. Mozilla has asked Microsoft for a 64-bit linker that can produce 32-bit code, but Microsoft apparently has no plans for such a configuration.
https://groups.google.com/forum/?fromgroups#!topic/mozilla.dev.platform/wLHTg_moymM https://groups.google.com/forum/?fromgroups#!topic/mozilla.d...
- dthunt 13y agoWhat I want to know is how someone can see that their software project has become unlinkable on a 32-bit OS and somehow think that this is purely a compiler problem.
- jlgreco 13y agoKeep in mind that this is the group that thinks de-prioritizing 64-bit windows builds is a-okay while at the same time they need 64-bit windows machines running 32-bit linkers to build their 32-bit windows binaries. ...phew, what a mouthfull. Mozilla has never struck me as the most... forward looking.
- yuhong 13y agoFor one thing, I don't think their JIT produces code that properly follows the Windows x64 ABI yet. It really should be disabled in 64-bit builds until they do.
- jlgreco 13y agoTheir stated reason for de-prioritizing 64-bit builds wasn't technical; rather they say they don't have the manpower to do both. Perhaps the manpower issue is that they don't have enough people to work on the JIT issues on x64 and do everything else, but I do not believe they specified.
- terrymah 13y agoFollowing the ABI is only an issue at module boundaries. If you control every callsite of a function, you can invent whatever calling convention you want.
- yuhong 13y agoBut you still need the function tables and follow prolog/epilog conventions with x64 for SEH to work.
- cpeterso 13y agobtw, Google Chrome doesn't have a 64-bit version* for Windows. The Chrome team also disabled PGO on Windows because of the same MSVC linker limitations that Mozilla hit. * Edit: oops, I corrected a typo where I wrote "32-bit build".
- yuhong 13y agoYou mean a 32-bit build system?
- cpeterso 13y agoSorry, I had a typo in my post. I mean Chrome for Windows is a 32-bit application. There is no 64-bit Chrome for Windows.
- jlgreco 13y agoYes, though that is far less a pain in the ass since they use multiple processes. I can't count the number of times I have run out of memory with firefox on windows despite having far more than firefox was using.
- terrymah 13y agoI'm actually about to sit down and write a blog post about VS compiler memory issues, and I had a long talk with the Firefox guys a few weeks ago about this issue. It's not all their fault.
- mbrubeck 13y agoIt's not just Firefox; other large projects like Chromium also exceed this limit when using MSVC's profile-guided optimization (PGO) on 32-bit Windows. (Chrome's solution is to build without PGO; Firefox's is to limit PGO to only a portion of source directories.)
- ryanmolden 13y agoI am a bit confused. That thread seems to claim that Microsoft isn't producing a 64 bit linker, my install seems to disagree with them (vc\bin\amd64). Then you claim that the linker won't produce 32 bit code, but /machine:x86 seems to disagree with that. I tested this by building as 32 bit, then just repeating the link using the 64 bit linker and /machine:x86, worked fine. So what are their problems exactly? That they are building with VS 2005 (from what I saw in that thread) and MS isn't backporting changes to the 2005 toolset?
- yuhong 13y agoThe problem is cl.exe for x86 is only 32-bit, and the design of most build systems don't make it easy, especially as Mozilla was using PGO.
- ryanmolden 13y agoYou mean it only produces 32 bit code? This disagrees (http://msdn.microsoft.com/en-US/library/x4d2c09s(v=vs.80).aspx http://msdn.microsoft.com/en-US/library/x4d2c09s(v=vs.80).as...) going back to at least 2005. The following list describes the various versions of cl.exe (the Visual C++ compiler): ... x64 on x86 (x64 cross-compiler) Allows you to create output files for x64. This version of cl.exe runs as a 32-bit process, native on an x86 machine and under WOW64 on a 64-bit Widows operating system.
- jlgreco 13y agoWhat they are currently doing is running a 32-bit linker to produce 32-bit output on 64-bit windows. This 32-bit linker is apparently LARGE_ADDRESS_AWARE, so running it on 64-bit windows gets them 4GB to play with (which they are apparently rapidly burning through.) What they need is a 64-bit linker to run on 64-bit windows that can produce 32-bit output. OS Linker Output Mem 1 32 32 32 3GB 2 32 32 64 3GB 3 64 32 32 4GB 4 64 64 32 8TB They use to be doing 1, now they are doing 3. They need to do 4, but apparently cannot. My understanding of this may be wrong, or they may be wrong, I don't know.
- marshray 13y agoDisclosure: I'm a new guy at Microsoft, not in the VS team, just speaking from prior experience as a Win32 developer. There are still internal limits in the PE32 format which mean parts of the output as a whole will be limited to 2 GiB or so, regardless of the address space available to the linker. I too have hit internal linker limits from time-to-time, but the span of computing history in which it makes sense to have a 64-bit linker building a load module output with 32-bit internal limits to run on 64-bit processors doesn't seem like the place to be in the great scheme of things. Just my personal observation.
- bzbarsky 13y agoIn Mozilla's case the _output_ is nowhere close to 2GiB. What hits the 32-bit limit is the link-time code generation, which has to have the entire program's AST, plus all the profiling information, plus whatever other data structures it's using in memory all at once.
- malkia 13y agoSame with Qt5 and /LTCG - visual studio can't compile (I've tried out the 64-bit compiler targeting 64-bit since there is a 32-bit compiler targeting 64-bit too).
- malkia 13y agoAfter checking again with my logs, it was failing while building QtWebkit (ahem - other posts explain it too). For my custom Qt5 build, I've disabled /LTCG only for QtWebkit (32/64bit debug/release) and was able to deal with this limitation (at some other cost I guess)
- mitchi 13y agoIf you compile with /LARGEADDRESSAWARE and run the executable on win7 64 bits, your process has a maximum of 4 gigabytes of ram. I know, I tested it up to 3.6 g. Not bad at all!