12 ms·

Do Not Use Go for 32bit Development

117 points by abtinf 15y ago

nikcub 15y ago
> On 32bit Windows machines, processes only have access to 2GB user space memory. You can change this in boot options to 3 or 4 (optimized) or enable PAE. see: http://msdn.microsoft.com/en-us/library/aa366778.aspx http://msdn.microsoft.com/en-us/library/aa366778.aspx i'd try that before rewriting your code (also kill services you don't need, which is usually a lot of them. run -> msconfig). Edit: you also don't mention probably the most important piece of info: which version of windows you are running. Edit 2: make sure go is being treated as a background process Edit 3: it seems a lot of these tips should be in the Go docs for win32 - i'll add it when I have a moment
- trimbo 15y ago
  Importantly, you will need to recompile with /LARGEADDRESSAWARE.
  - 4ad 15y ago
    Yeah, that kernel flag is useless without stamping the binaries, and the Go linker doesn't know about this.
    - barrkel 15y ago
      It's just a flag in the PE headers. It can be modified after the fact with a tool; the Go linker doesn't need to know.
      - doug363 15y ago
        But the reason why there is an explicit flag in the first place is that there's a lot of code out there that doesn't work with pointers >= 0x80000000, or when pointers could differ by more than that. Things like pointer "tagging", and signed integer overflow bugs, are examples of the sort of code that can be affected. Without knowing details about the Go code generation and runtime/garbage collection, it would be a bit risky to just set the flag and hope for the best.
        barrkel 15y ago
        Oh, I know; I've written bounds-checking logic for RTLs that use casts to integers and tests for negativity to check for overrun, implicitly assuming 31-bit address space. My only point is that a limitation of the Go linker does not preclude setting the flag.
        4ad 15y ago
        No such problems in Go. I just checked and the flag is set by default in all Windows binaries.
    - 4ad 15y ago
      > the Go linker doesn't know about this. I was wrong, it does, it's on by default :).
- yuhong 15y ago
  It has nothing to do with PAE.
  - Bootvis 15y ago
    It seems to me it does: With PAE Windows can use more memory. This extra memory can be used to allocate the 512mb chunck. Can you clarify?
    - deleted 15y ago
      [deleted]
    - tankenmate 15y ago
      It's a VM (mapping) issue not a RAM (free memory) issue.
      - 4ad 15y ago
        Also, with PAE, each process gets the same 32 address space as before.
    - deleted 15y ago
      [deleted]
- malkia 15y ago
  Enabling /3GB and LARGEADDRESSAWARE might work, but I have found that because of certain video drivers (XP) would not work correctly with D3D or OpenGL apps (looks like some assumptions were made there that the space would be below 2GB). For one we could not get correctly our game and some of our tools to work on XP with /3GB (that was few years ago) Nowadays since we moved to Windows Vista/7 64-bit, we no longer have the problems (LARGEADDRESSAWARE) and our 32-bits sometimes go up to 3.5gb of usage.
- xpaulbettsx 15y ago
  One thing to note, is that his problem is lack of contiguous VA space, not physical frames (i.e. "memory"). Killing services will do nada to help that, the only thing that will help is mapping less DLLs.
  - 4ad 15y ago
    DLLs are loaded in a process address space only because they are referenced in the PE header or the process calls LoadLibrary(), I'm afraid killing processes won't cause mapping of less DLLs either.
- georgemcbay 15y ago
  "Just set these boot options" is fine if you're just deploying this code to a server you control. I get the sense from various aspects of this guy's story that he's shipping executables to end users, which makes this "solution" a non-starter. You can't expect the user to go in and mess with these sorts of options just to run your program, and I wouldn't feel comfortable changing something like this automatically during install since it is such a fundamental OS setting (not to mention testing the installer against all versions of Windows would then become a nightmare you'd have to redo every release).
xal 15y ago
Just compile with the gcc go compiler and you are set. Running 32 bit on servers is just like running IE6 on desktops. It's time to upgrade. Also every language that uses a conservative GC behaves this way, it's not just a GO issue.
- mikeash 15y ago
  What other popular languages use a conservative GC, though?
  - sausagefeet 15y ago
    I believe Mono uses Boehm GC, which is conservative: http://www.mono-project.com/Mono:Runtime#Mono.27s_use_of_Boehm_GC http://www.mono-project.com/Mono:Runtime#Mono.27s_use_of_Boe...
    - kingkilr 15y ago
      They've introduced S-Gen which is a good, precise, generational collector. I don't know if it's the default, but it is considered production ready IIRC.
      - jstedfast 15y ago
        It'll be the default in Mono 3.0 (or 2.12, whatever we decide to call it). Currently, in Mono 2.10, Boehm GC is the default, although we've been using SGen for our commercial products (which is why we feel that SGen is now ready to be the default).
    - jstedfast 15y ago
      And yet Mono works fine in environments with < 512MB of RAM, even when using Boehm GC :-)
      - chc 15y ago
        That's because the conservative GC issue is separate from the low-RAM issue.
algolicious 15y ago
At runtime init, Go programs attempt to reserve 512mb of RAM. If they can’t do this, they crash. Go needs the space because it is garbage collected - presumably, all GC languages, or any program that takes a similar approach to memory management, will be susceptible to the same problem. While it is true that most garbage collected environments will reserve some memory at startup, a hard minimum of 512 MB is not the norm. Do note that embedded Java has run on cell phones with minimal amounts of RAM for many years.
- _delirium 15y ago
  Fwiw, Go doesn't need a hard minimum of 512 MB physical memory, but it does seem to assume a contiguous 512 MB of virtual address space will be available. Since every process has 2 GB of virtual address space and grabbing the 512-MB chunk is the first thing Go does, this should almost always be safe, but on Windows it appears that can fail to be the case because in some configurations, DLLs are loaded during initialization and fragment the virtual address space. I can run Go fine on a 32-bit (Linux) VPS with 256 MB of RAM, though.
  - abtinf 15y ago
    You are correct - it was an error in my post. Go wants 512mb of virtual space.
  - ismarc 15y ago
    They only get scattered throughout when using cgo with the DLLs loaded by the Windows system by the application loader rather than by the go runtime. It appears to be very specific to how Windows loads DLLs at runtime.
    - 4ad 15y ago
      Yes, though in this particular case the author told me cgo wasn't involved. This makes it very very peculiar. Usually when I see system DLLs (not 3rd party) rebasing it's either mallware in other processes that lazily loaded ntdll.dll and that had to be rebased or some legit program or mallware that does user mode hooking by injecting DLLs. It is NOT a Go problem, but Go could implement a workaround making the address space reservation in teh PE header.
dhconnelly 15y ago
I'm a big fan of Go, but it would be nice to see a caveat about 32 bit systems on the Go website or blog. Or at least some official warning or notice that isn't just a mailing list discussion.
ismarc 15y ago
Looking through the linked bug and the testing they did, the issue doesn't appear to be using Go for 32 bit systems. It's using 32 bit compiled Go programs that use cgo on 64 bit Windows systems. Understanding the why of the issue is extremely important in this case. Based on the discussion in the bug report, I would expect that using the 32 bit compiler for 32 bit architectures and 64 bit for 64 bit architectures would not result in the issue occurring. Before abandoning Go, I would very much identify that this is indeed the case. With Go's young age, I would very much use the architecture specific compilers for the specific targets which is different than normal expectations, but not at all unreasonable.
- abtinf 15y ago
  Unfortunately, this is not the case. 32bit compiled runs great on 64bit machines, where it gets the full 4gb of space and memory fragmentation at init is very unlikely to be a problem. Edit: similarly, 64bit also is immune because of how the main platforms allocate virtual address space to clients.
  - ismarc 15y ago
    Hrm, that actually makes the issue tricky to solve. I don't have any Windows machines to work with, so I can't test, and I'm not very familiar with using Go on Windows in general, but is it possible to specify delayed load of the DLLs? That should allow the main Go system to get fully initialized (and allocate the chunk of memory it needs) before loading the DLLs needed for the cgo portions.
    - 4ad 15y ago
      Nobody is using cgo, it's the system DLLs which get loaded in the middle of the address space. This is very very unusual, Windows tries hard to avoid such things. It could be solved by making the initial virtual address space reservation in the PE header, not requesting it in the initialization path, that would always work, even with cgo.
- sausagefeet 15y ago
Dylan16807 15y ago
How many dlls are loaded into the process? Have you looked at all the loaded dlls at the moment allocation fails to look at the potential for remapping? Edit: Okay I see a memory map on the bug report. So why not poke at a copy of kernelbase.dll?
malkia 15y ago
If 512mb are needed as contiguous space, then one hacky solution is do define a BSS section of 512mb, or in "C" terms - global array - char mem[512 * 1024 * 1024]; the executable loader would "allocate" this memory before loading any dlls, and later dlls that were supposed to be in that memory would be rebased Oh, and somehow Go would need to use this memory rather than allocate it.
- moldbug 15y ago
  I wouldn't call this "hacky" - it's exactly the right way to do it. For one thing, it makes the address of your arena constant. Unfortunately my experience is that a lot of C environments handle this very poorly.
  - malkia 15y ago
    I called this hacky, as it would only work for executables, if this was in a dll/dylib/so shared library, then it might even make the things worse, as now even more stricter contiguous space would be neeeded. For example, if the language runtime was loaded as plugin or something. So one has to be aware that is not always the good solution for all purposes, hence hacky :)
- barrkel 15y ago
  This would, however, allocate memory rather than just address space. (Yes, it may be lazily committed, but it's still not as cheap as address space.)
  - malkia 15y ago
    Once this is done, you can VirtualFree it (starting/ending from page boundaries), and let it back to the OS. Yes for some short amount of time, while it loads and goes to your code the memory is reserved.
someCTO 15y ago
It seems like the actual problem is you chose to use windows...
- eridius 15y ago
  It does seem rather surprising to me that Windows is so broken that a program can fail to reserve 512MB of virtual address space as the very first thing it does at launch. Of course, the 2GB limit on 32bit Windows programs also seems terribly broken to me.
  - Mavrik 15y ago
    You mean the identical limit that also 32-bit Linux applications on x86 have without reconfiguring kernel for different split? It seems Windows bashing without any background knowledge is becoming popular again -_-
    - someCTO 15y ago
      Running a PAE kernel on 32 bit is pretty standard at this point, I believe. Please correct me if I missed something.
      - yuhong 15y ago
        It has nothing to do with PAE, but 3G/1G split I think is default on Linux.
        someCTO 15y ago
        That does not address the 512MB of virtual address problem, which is not an issue with PAE.
        4ad 15y ago
        Yes it is because PAE doesn't change the address space of processes. PAE let's you use more physical memory, the address space of each process is still the same, because 2^32 is still 2^32 with PAE.
ExpiredLink 15y ago
> I’m sitting here rewriting a ton of Go code in C. I don't believe a single word. He pretends to have written "a ton of Go code" without discovering the "real show stopper"? I'm not a Go fan (quite the contrary, Go isn't the step forward from C++ and Java I expected from Google). But this Anti-Go campaign few days after 1.0 smells.
- abtinf 15y ago
  Believe what you want. The problem is intermittent and didn't appear on my test machines/VMs.
  - trimbo 15y ago
    What's "a ton"?
- eternalban 15y ago
  > Go isn't the step forward from C++ and Java I expected from Google Go is a lovely language, imo. Quirky in parts, but definitely a charmer, very fun to code in, and once you get the Go zen, simple and elegant code effortlessly follows. I haven't had this much fun since Java came out in mid 90s. Of course, I do agree with the other comments that would like to see Golang.org be more clear as to the platform coverage and the current state of Scheduler and MM -- but they are certainly upfront in the source tree (c.f. proc.c header comment), forums, and the issue tracker. So hacker fine print is there and available, and frankly (IMHO) that approach to disclosure may turn out to be a good filter for the growth of Go community. The issue at hand is nothing that time and resources will not solve -- they just ran out of time (c.f. RSC's comment before Abtin's in the issue.)
  - pjmlp 15y ago
    >> Go isn't the step forward from C++ and Java I expected from Google > >Go is a lovely language, imo. Quirky in parts, but definitely a charmer, very fun to code in, and once you get the Go zen, simple and elegant code effortlessly follows. I haven't had this much fun since Java came out in mid 90s. Go is a step backwards in language abstractions, throwing away many of the abstractions that have become mainstream in the last decades.
pjmlp 15y ago
This only goes to show that Go is still not production ready.
- jemeshsu 15y ago
  YouTube is using Golang http://code.google.com/p/vitess/ http://code.google.com/p/vitess/. So 10% of Internet traffic now depends on it. I wouldn't call it not production ready. From the project goal page: "Go is miles ahead of C++ and Java in terms of expressibility and close in terms of performance. It is also relatively simple and has a straightforward interaction with Linux system calls. The main drawback is also its strength - the garbage collector. vtocc has made spot optimizations to minimize most of the adverse effects of Go’s stop-the-world gc. At this point, we are trading some amount of performance for greater creativity and efficiency at lower layers. unless you’re trying to max out on qps for your servers, you should see acceptable performance from vtocc. Also, go’s garbage collector is being improved. So, this should only get better over time. Go’s existing mark-and-sweep garbage collector is sub-optimal for systems that use large amounts of static memory (like caches). In the case of vtocc, this would be the row cache. To alleviate this, we intend to use memcache for the time being. If the gc ends up addressing this, it should be fairly trivial to switch to an in-memory row cache. Note that the row cache functionality is not fully ready yet."
  - pjmlp 15y ago
    "Go is miles ahead of C++ and Java in terms of expressibility and close in terms of performance. It is also relatively simple and has a straightforward interaction with Linux system calls." How can this be when Go lacks: - enumerations - exceptions - generics - dynamic loading The short compilation times are possible in any language with modules. Channels are available as part of concurrency libraries in Java, .NET, C++ and Erlang. Goroutines are also possible in other languages, in form of continuations or task pools.
    - agentS 15y ago
      - It has a more general analogue to enumerations, one that does not add a lot of verbiage to the language spec. (http://play.golang.org/p/HSh4Ke3pCJ http://play.golang.org/p/HSh4Ke3pCJ) - It has an "exception" mechanism that is rarely used, in favour of error-valued returns. (http://blog.golang.org/2010/08/defer-panic-and-recover.html http://blog.golang.org/2010/08/defer-panic-and-recover.html) - Apparently, vitess didn't need generics. You'll notice a bit of casting here in their implementation of an LRU cache, but writing containers isn't exactly the main purpose of the library. I do admit I want generics, for the sole purpose of stopping people parroting that criticism without actually using the language. (http://code.google.com/p/vitess/source/browse/go/cache/lru_cache.go http://code.google.com/p/vitess/source/browse/go/cache/lru_c...) - Not sure what you mean here. If you mean a Go library is being loaded, then there's an issue for that, but one unlikely to be fixed in the short-term because of Go's unusual calling convention. If you mean Go loading a library at runtime, I don't know why you'd want this. Either way, static linking seems cleaner and more self contained to me. "Short compilation times" being available in any language is total bull. Try building chromium or firefox in a reasonable amount of time, without a build cluster. Don't you think if there was a way to speed that up, the developers would make that priority 1? Channel/Goroutines are of course available in all languages, Turing-completeness implies that they must. However, in practice, does all code in those languages make use of the same concurrency primitives? Do they read as nicely as Go does?
jesstaa 15y ago
The strategy the allocator uses is to reserve a large contiguous block of address space(not memory) on start up and then allocate actual memory from that block. On 32bit this is around 512MB, on 64bit it's something like 16TB. This makes allocating and garbage collection far simpler and it really shouldn't be a big ask for an operating system to give a process a contiguous block of address space and not interfere with it. But just like the pinning issue the smaller address space on 32bit means that things that expect to be at a certain address have a higher likelihood of getting in the way of the contiguous block. The fix for both this issue and the pinning issue is big changes to the garbage collector. Since the language and standard library have stabilised with Go1, changes to the runtime are going to be the focus of development so this issue will probably get a fix.
- xpaulbettsx 15y ago
  This is highly dependent on the video card and other PCI devices you are using, as they will often reserve huge chunks of VA space to map to device memory (and furthermore, require it to be in low addresses i.e. user space), making a contiguous reservation difficult. If you want to see exactly what's happening, SysInternals VMMap (http://technet.microsoft.com/en-us/sysinternals/dd535533?ppud=4 http://technet.microsoft.com/en-us/sysinternals/dd535533?ppu...) will tell you exactly the VA space layout. Note that Linux has exactly the same problems with VA space, except they default to 3/1, instead of NT's default (but configurable) of 2/2. It seems that a quick fix to this issue would be to lower that number from 512MB to something like 128MB, though I know nothing about Go's internals.
  - JoshTriplett 15y ago
    Getting 512MB of contiguous virtual address space does not require 512MB of contiguous free physical memory; contiguous virtual addresses can map to discontiguous physical memory.
    - astrodust 15y ago
      That's what I thought. The address your user-space program uses and the actual address in memory are two different things, aren't they? Isn't this what enables the operating system to scramble the allocations it gives you to make it harder to implement a buffer-overflow attack?
cjensen 15y ago
In my opinion, the problem is that 32-bit Windows is intentionally broken by Microsoft. Unlike other 32-bit x86 operating systems, the consumer version of Windows refuses to addresses more than the first 4GB of physical memory, and on most machines a quarter to half of that space is consumed by PCI address space. Yes there are hacks around this. But the right way to deal with this is to just fold and accept what Microsoft is imposing: use a 64-bit version of Windows whenever possible and enjoy your memory. Once on a 64-bit OS, you can either write a 64-bit program, or you can write a 32-bit program and use a compiler option to get the full 32-bit address space. (Microsoft's compilers assume by default that you are an idiot who uses signed values for addresses and your program will break if they give you more than 2GB of virtual address space) The other lesson is don't use experimental languages you are uncertain of for actual product.
- pjmlp 15y ago
  The same also applies to other 32 bit OS, this is not Windows specific.
  - leon_ 15y ago
    Yes, Amiga OS.
  - tankenmate 15y ago
    Umm no, other OSes have different defaults for mapping and have different heuristics for how and where they map devices and libraries. Go on 32bit Linux rarely has the problems that the Windows versions do.
    - 4ad 15y ago
      The PCI mappings are the same, either in Linux or Windows. I do agree that this particular report is a Windows problem, 512MB contiguous address space should be available at program initialization in every case. I do agree that Go on 32 bit Linux rarely has any problems, but this is true for Windows as well. What happened in the last few days is blown out of proportion.
fleitz 15y ago
Somewhere in the source of Go will be the number 536870912, change this number to 134217728. Problem should largely be solved. Note: I'm not a Go dev, or have even programmed in it, but logically it should be a quick fix that you can try to save you rewriting your codebase. Given the importance of the number it's probably a #define or constant so should be pretty easy to find in one of the header files.
- 4ad 15y ago
  It's 512<<20, and a proper way to fix it, on Windows, at least, would be to reserve the address space in the PE header, not ask the operating system in the process initialization path.
JoeAltmaier 15y ago
x86 32-bit is actually capable of addressing any amount of memory, using the right compilation model. Flat 32-bit has a 4GB limit sure. But segemented, it can access any amount of virtual memory, with the caveat of only 4GB per allocation. Probably no OS supports this any more. But the instructions set and memory management units support it, or used to in early Pentium days.
- ghost91 15y ago
  Take a look at PAE nearly every modern kernel supports it.
  - JoeAltmaier 15y ago
    But do any languages? When was the last time you compiled to anything but flat?
- caf 15y ago
  Segment:Offset addresses are converted to 32 bit linear addresses; the segment base address in the segment descriptor is a 32 bit value.
  - zvrba 15y ago
    > Segment:Offset addresses are converted to 32 bit linear addresses That's true, but segment descriptors have a not-present bit, which allows you to implement segment-level swapping.
  - JoeAltmaier 15y ago
    On a 64-bit architecture its 64-bit. 32-bit programs run there. Also, as noted elsewhere, on 32-bit architectures there's segment swapping too. In fact I wrote an OS that segment swapped before the 386 came out, when the 286 was king. Probably the only one out there; a pretty crazy notion and the 386 came out a year later with paging.
    - caf 15y ago
caf 15y ago
This same Go implementation design earlier ran into some issues on Linux, too - in this case, with distributions setting conservative RLIMIT_AS limits. The technical details are interesting (https://groups.google.com/forum/#!msg/golang-dev/EpUlHQXWykg/LN2o9fV6R3wJ https://groups.google.com/forum/#!msg/golang-dev/EpUlHQXWykg...) and Linus' response (http://lkml.indiana.edu/hypermail/linux/kernel/1102.1/00233.html http://lkml.indiana.edu/hypermail/linux/kernel/1102.1/00233....) is also instructive.
- aidenn0 15y ago
  I had a similar issue with the default configuration of SBCL on a 64-bit vhost; the 64-bit version asks for 8gb of VAS and the vhost had a fairly small limit on VSIZE. Fortunately the heap allocation is configurable in SBCL; perhaps doing something like that on go would be the right thing?
tashmahalic 15y ago
This is my understanding of the problem: when a Go program is initialized on 32-bit windows or linux systems, it tries to allocate 512MB of contiguous virtual ram. When the virtual address space is too fragmented, the initialization will fail and the Go program won't start. Is this correct? We're considering using Go for an end-user desktop application for Mac, Windows, and Linux. Of course, we don't have control over whether our users have a 32-bit or 64-bit system. And, of course, our app has to work 100% of the time. What can we do to make Go work 100% of the time on these 32-bit systems?
- JoeAltmaier 15y ago
  Don't know what it means for the virtual address space to be fragmented. Each process begins with a pristine virtual address space.
  - 4ad 15y ago
    Mostly, but not really, apart from the binary image itself, its dependent shared libraries are also loaded, and the various images loaded might have virtual address space reservations marked in the header. In normal scenarios there should be enough contiguous address space, Go programs only link to ntdll.dll and winmm.dll. While it's true that on recent Windows versions these two shared objects pull around 30 other shared objects, all the DLLs shipped with Windows have a base address chosen so that they only take a very small chunk of address space and most address space is contiguous. 512MB of contiguous reservation shouldn't be a problem to get. The fact that this is not happening means that 3rd party DLLs are loaded, this happens if you use a program, knowingly or not, that install user mode hooks.
kokoloko 15y ago
I think Go, Mono, Ruby, Python et al should team up to create a reusable garbage collector library that can compete with the OpenJDK one. For me, it's the single biggest piece falling behind on non JVM languages. The performance and "tunnability" are unmatched right now. Bonus points would be for a real time garbage collector, a la IBM Metronome.
tashmahalic 15y ago
We need to decide if Go is a suitable language to build our app with. Our app will be deployed on end-user 32-bit systems, and needs to work 100% of the time. Is there someone here who can say either: (1) Go will fail sometimes on 32-bit systems, don't use it until this problem has been fixed, or (2) there are things you can do to always avoid the 32-bit problems. ?
- tashmahalic 15y ago
  (and if there's no one here with an informed answer, where should i ask this question? Thanks.)

117 points by abtinf 15y ago

> On 32bit Windows machines, processes only have access to 2GB user space memory. You can change this in boot options to 3 or 4 (optimized) or enable PAE. see: http://msdn.microsoft.com/en-us/library/aa366778.aspx http://msdn.microsoft.com/en-us/library/aa366778.aspx i'd try that before rewriting your code (also kill services you don't need, which is usually a lot of them. run -> msconfig). Edit: you also don't mention probably the most important piece of info: which version of windows you are running. Edit 2: make sure go is being treated as a background process Edit 3: it seems a lot of these tips should be in the Go docs for win32 - i'll add it when I have a moment

trimbo 15y ago
Importantly, you will need to recompile with /LARGEADDRESSAWARE.
- 4ad 15y ago
  Yeah, that kernel flag is useless without stamping the binaries, and the Go linker doesn't know about this.
  - barrkel 15y ago
    It's just a flag in the PE headers. It can be modified after the fact with a tool; the Go linker doesn't need to know.
    - doug363 15y ago
      But the reason why there is an explicit flag in the first place is that there's a lot of code out there that doesn't work with pointers >= 0x80000000, or when pointers could differ by more than that. Things like pointer "tagging", and signed integer overflow bugs, are examples of the sort of code that can be affected. Without knowing details about the Go code generation and runtime/garbage collection, it would be a bit risky to just set the flag and hope for the best.
      - barrkel 15y ago
        Oh, I know; I've written bounds-checking logic for RTLs that use casts to integers and tests for negativity to check for overrun, implicitly assuming 31-bit address space. My only point is that a limitation of the Go linker does not preclude setting the flag.
      - 4ad 15y ago
        No such problems in Go. I just checked and the flag is set by default in all Windows binaries.
  - 4ad 15y ago
    > the Go linker doesn't know about this. I was wrong, it does, it's on by default :).
yuhong 15y ago
It has nothing to do with PAE.
- Bootvis 15y ago
  It seems to me it does: With PAE Windows can use more memory. This extra memory can be used to allocate the 512mb chunck. Can you clarify?
  - deleted 15y ago
    [deleted]
  - tankenmate 15y ago
    It's a VM (mapping) issue not a RAM (free memory) issue.
    - 4ad 15y ago
      Also, with PAE, each process gets the same 32 address space as before.
  - deleted 15y ago
    [deleted]
malkia 15y ago
Enabling /3GB and LARGEADDRESSAWARE might work, but I have found that because of certain video drivers (XP) would not work correctly with D3D or OpenGL apps (looks like some assumptions were made there that the space would be below 2GB). For one we could not get correctly our game and some of our tools to work on XP with /3GB (that was few years ago) Nowadays since we moved to Windows Vista/7 64-bit, we no longer have the problems (LARGEADDRESSAWARE) and our 32-bits sometimes go up to 3.5gb of usage.
xpaulbettsx 15y ago
One thing to note, is that his problem is lack of contiguous VA space, not physical frames (i.e. "memory"). Killing services will do nada to help that, the only thing that will help is mapping less DLLs.
- 4ad 15y ago
  DLLs are loaded in a process address space only because they are referenced in the PE header or the process calls LoadLibrary(), I'm afraid killing processes won't cause mapping of less DLLs either.
georgemcbay 15y ago
"Just set these boot options" is fine if you're just deploying this code to a server you control. I get the sense from various aspects of this guy's story that he's shipping executables to end users, which makes this "solution" a non-starter. You can't expect the user to go in and mess with these sorts of options just to run your program, and I wouldn't feel comfortable changing something like this automatically during install since it is such a fundamental OS setting (not to mention testing the installer against all versions of Windows would then become a nightmare you'd have to redo every release).

xal 15y ago

Just compile with the gcc go compiler and you are set. Running 32 bit on servers is just like running IE6 on desktops. It's time to upgrade. Also every language that uses a conservative GC behaves this way, it's not just a GO issue.

mikeash 15y ago
What other popular languages use a conservative GC, though?
- sausagefeet 15y ago
  I believe Mono uses Boehm GC, which is conservative: http://www.mono-project.com/Mono:Runtime#Mono.27s_use_of_Boehm_GC http://www.mono-project.com/Mono:Runtime#Mono.27s_use_of_Boe...
  - kingkilr 15y ago
    They've introduced S-Gen which is a good, precise, generational collector. I don't know if it's the default, but it is considered production ready IIRC.
    - jstedfast 15y ago
      It'll be the default in Mono 3.0 (or 2.12, whatever we decide to call it). Currently, in Mono 2.10, Boehm GC is the default, although we've been using SGen for our commercial products (which is why we feel that SGen is now ready to be the default).
  - jstedfast 15y ago
    And yet Mono works fine in environments with < 512MB of RAM, even when using Boehm GC :-)
    - chc 15y ago
      That's because the conservative GC issue is separate from the low-RAM issue.

algolicious 15y ago

At runtime init, Go programs attempt to reserve 512mb of RAM. If they can’t do this, they crash. Go needs the space because it is garbage collected - presumably, all GC languages, or any program that takes a similar approach to memory management, will be susceptible to the same problem. While it is true that most garbage collected environments will reserve some memory at startup, a hard minimum of 512 MB is not the norm. Do note that embedded Java has run on cell phones with minimal amounts of RAM for many years.

_delirium 15y ago
Fwiw, Go doesn't need a hard minimum of 512 MB physical memory, but it does seem to assume a contiguous 512 MB of virtual address space will be available. Since every process has 2 GB of virtual address space and grabbing the 512-MB chunk is the first thing Go does, this should almost always be safe, but on Windows it appears that can fail to be the case because in some configurations, DLLs are loaded during initialization and fragment the virtual address space. I can run Go fine on a 32-bit (Linux) VPS with 256 MB of RAM, though.
- abtinf 15y ago
  You are correct - it was an error in my post. Go wants 512mb of virtual space.
- ismarc 15y ago
  They only get scattered throughout when using cgo with the DLLs loaded by the Windows system by the application loader rather than by the go runtime. It appears to be very specific to how Windows loads DLLs at runtime.
  - 4ad 15y ago
    Yes, though in this particular case the author told me cgo wasn't involved. This makes it very very peculiar. Usually when I see system DLLs (not 3rd party) rebasing it's either mallware in other processes that lazily loaded ntdll.dll and that had to be rebased or some legit program or mallware that does user mode hooking by injecting DLLs. It is NOT a Go problem, but Go could implement a workaround making the address space reservation in teh PE header.

dhconnelly 15y ago

I'm a big fan of Go, but it would be nice to see a caveat about 32 bit systems on the Go website or blog. Or at least some official warning or notice that isn't just a mailing list discussion.

ismarc 15y ago

Looking through the linked bug and the testing they did, the issue doesn't appear to be using Go for 32 bit systems. It's using 32 bit compiled Go programs that use cgo on 64 bit Windows systems. Understanding the why of the issue is extremely important in this case. Based on the discussion in the bug report, I would expect that using the 32 bit compiler for 32 bit architectures and 64 bit for 64 bit architectures would not result in the issue occurring. Before abandoning Go, I would very much identify that this is indeed the case. With Go's young age, I would very much use the architecture specific compilers for the specific targets which is different than normal expectations, but not at all unreasonable.

abtinf 15y ago
Unfortunately, this is not the case. 32bit compiled runs great on 64bit machines, where it gets the full 4gb of space and memory fragmentation at init is very unlikely to be a problem. Edit: similarly, 64bit also is immune because of how the main platforms allocate virtual address space to clients.
- ismarc 15y ago
  Hrm, that actually makes the issue tricky to solve. I don't have any Windows machines to work with, so I can't test, and I'm not very familiar with using Go on Windows in general, but is it possible to specify delayed load of the DLLs? That should allow the main Go system to get fully initialized (and allocate the chunk of memory it needs) before loading the DLLs needed for the cgo portions.
  - 4ad 15y ago
    Nobody is using cgo, it's the system DLLs which get loaded in the middle of the address space. This is very very unusual, Windows tries hard to avoid such things. It could be solved by making the initial virtual address space reservation in the PE header, not requesting it in the initialization path, that would always work, even with cgo.
sausagefeet 15y ago

Dylan16807 15y ago

How many dlls are loaded into the process? Have you looked at all the loaded dlls at the moment allocation fails to look at the potential for remapping? Edit: Okay I see a memory map on the bug report. So why not poke at a copy of kernelbase.dll?

malkia 15y ago

If 512mb are needed as contiguous space, then one hacky solution is do define a BSS section of 512mb, or in "C" terms - global array - char mem[512 * 1024 * 1024]; the executable loader would "allocate" this memory before loading any dlls, and later dlls that were supposed to be in that memory would be rebased Oh, and somehow Go would need to use this memory rather than allocate it.

moldbug 15y ago
I wouldn't call this "hacky" - it's exactly the right way to do it. For one thing, it makes the address of your arena constant. Unfortunately my experience is that a lot of C environments handle this very poorly.
- malkia 15y ago
  I called this hacky, as it would only work for executables, if this was in a dll/dylib/so shared library, then it might even make the things worse, as now even more stricter contiguous space would be neeeded. For example, if the language runtime was loaded as plugin or something. So one has to be aware that is not always the good solution for all purposes, hence hacky :)
barrkel 15y ago
This would, however, allocate memory rather than just address space. (Yes, it may be lazily committed, but it's still not as cheap as address space.)
- malkia 15y ago
  Once this is done, you can VirtualFree it (starting/ending from page boundaries), and let it back to the OS. Yes for some short amount of time, while it loads and goes to your code the memory is reserved.

someCTO 15y ago

It seems like the actual problem is you chose to use windows...

eridius 15y ago
It does seem rather surprising to me that Windows is so broken that a program can fail to reserve 512MB of virtual address space as the very first thing it does at launch. Of course, the 2GB limit on 32bit Windows programs also seems terribly broken to me.
- Mavrik 15y ago
  You mean the identical limit that also 32-bit Linux applications on x86 have without reconfiguring kernel for different split? It seems Windows bashing without any background knowledge is becoming popular again -_-
  - someCTO 15y ago
    Running a PAE kernel on 32 bit is pretty standard at this point, I believe. Please correct me if I missed something.
    - yuhong 15y ago
      It has nothing to do with PAE, but 3G/1G split I think is default on Linux.
      - someCTO 15y ago
        That does not address the 512MB of virtual address problem, which is not an issue with PAE.
        4ad 15y ago
        Yes it is because PAE doesn't change the address space of processes. PAE let's you use more physical memory, the address space of each process is still the same, because 2^32 is still 2^32 with PAE.

ExpiredLink 15y ago

> I’m sitting here rewriting a ton of Go code in C. I don't believe a single word. He pretends to have written "a ton of Go code" without discovering the "real show stopper"? I'm not a Go fan (quite the contrary, Go isn't the step forward from C++ and Java I expected from Google). But this Anti-Go campaign few days after 1.0 smells.

abtinf 15y ago
Believe what you want. The problem is intermittent and didn't appear on my test machines/VMs.
- trimbo 15y ago
  What's "a ton"?
eternalban 15y ago
> Go isn't the step forward from C++ and Java I expected from Google Go is a lovely language, imo. Quirky in parts, but definitely a charmer, very fun to code in, and once you get the Go zen, simple and elegant code effortlessly follows. I haven't had this much fun since Java came out in mid 90s. Of course, I do agree with the other comments that would like to see Golang.org be more clear as to the platform coverage and the current state of Scheduler and MM -- but they are certainly upfront in the source tree (c.f. proc.c header comment), forums, and the issue tracker. So hacker fine print is there and available, and frankly (IMHO) that approach to disclosure may turn out to be a good filter for the growth of Go community. The issue at hand is nothing that time and resources will not solve -- they just ran out of time (c.f. RSC's comment before Abtin's in the issue.)
- pjmlp 15y ago
  >> Go isn't the step forward from C++ and Java I expected from Google > >Go is a lovely language, imo. Quirky in parts, but definitely a charmer, very fun to code in, and once you get the Go zen, simple and elegant code effortlessly follows. I haven't had this much fun since Java came out in mid 90s. Go is a step backwards in language abstractions, throwing away many of the abstractions that have become mainstream in the last decades.

pjmlp 15y ago

This only goes to show that Go is still not production ready.

jemeshsu 15y ago
YouTube is using Golang http://code.google.com/p/vitess/ http://code.google.com/p/vitess/. So 10% of Internet traffic now depends on it. I wouldn't call it not production ready. From the project goal page: "Go is miles ahead of C++ and Java in terms of expressibility and close in terms of performance. It is also relatively simple and has a straightforward interaction with Linux system calls. The main drawback is also its strength - the garbage collector. vtocc has made spot optimizations to minimize most of the adverse effects of Go’s stop-the-world gc. At this point, we are trading some amount of performance for greater creativity and efficiency at lower layers. unless you’re trying to max out on qps for your servers, you should see acceptable performance from vtocc. Also, go’s garbage collector is being improved. So, this should only get better over time. Go’s existing mark-and-sweep garbage collector is sub-optimal for systems that use large amounts of static memory (like caches). In the case of vtocc, this would be the row cache. To alleviate this, we intend to use memcache for the time being. If the gc ends up addressing this, it should be fairly trivial to switch to an in-memory row cache. Note that the row cache functionality is not fully ready yet."
- pjmlp 15y ago
  "Go is miles ahead of C++ and Java in terms of expressibility and close in terms of performance. It is also relatively simple and has a straightforward interaction with Linux system calls." How can this be when Go lacks: - enumerations - exceptions - generics - dynamic loading The short compilation times are possible in any language with modules. Channels are available as part of concurrency libraries in Java, .NET, C++ and Erlang. Goroutines are also possible in other languages, in form of continuations or task pools.
  - agentS 15y ago
    - It has a more general analogue to enumerations, one that does not add a lot of verbiage to the language spec. (http://play.golang.org/p/HSh4Ke3pCJ http://play.golang.org/p/HSh4Ke3pCJ) - It has an "exception" mechanism that is rarely used, in favour of error-valued returns. (http://blog.golang.org/2010/08/defer-panic-and-recover.html http://blog.golang.org/2010/08/defer-panic-and-recover.html) - Apparently, vitess didn't need generics. You'll notice a bit of casting here in their implementation of an LRU cache, but writing containers isn't exactly the main purpose of the library. I do admit I want generics, for the sole purpose of stopping people parroting that criticism without actually using the language. (http://code.google.com/p/vitess/source/browse/go/cache/lru_cache.go http://code.google.com/p/vitess/source/browse/go/cache/lru_c...) - Not sure what you mean here. If you mean a Go library is being loaded, then there's an issue for that, but one unlikely to be fixed in the short-term because of Go's unusual calling convention. If you mean Go loading a library at runtime, I don't know why you'd want this. Either way, static linking seems cleaner and more self contained to me. "Short compilation times" being available in any language is total bull. Try building chromium or firefox in a reasonable amount of time, without a build cluster. Don't you think if there was a way to speed that up, the developers would make that priority 1? Channel/Goroutines are of course available in all languages, Turing-completeness implies that they must. However, in practice, does all code in those languages make use of the same concurrency primitives? Do they read as nicely as Go does?

jesstaa 15y ago

The strategy the allocator uses is to reserve a large contiguous block of address space(not memory) on start up and then allocate actual memory from that block. On 32bit this is around 512MB, on 64bit it's something like 16TB. This makes allocating and garbage collection far simpler and it really shouldn't be a big ask for an operating system to give a process a contiguous block of address space and not interfere with it. But just like the pinning issue the smaller address space on 32bit means that things that expect to be at a certain address have a higher likelihood of getting in the way of the contiguous block. The fix for both this issue and the pinning issue is big changes to the garbage collector. Since the language and standard library have stabilised with Go1, changes to the runtime are going to be the focus of development so this issue will probably get a fix.

xpaulbettsx 15y ago
This is highly dependent on the video card and other PCI devices you are using, as they will often reserve huge chunks of VA space to map to device memory (and furthermore, require it to be in low addresses i.e. user space), making a contiguous reservation difficult. If you want to see exactly what's happening, SysInternals VMMap (http://technet.microsoft.com/en-us/sysinternals/dd535533?ppud=4 http://technet.microsoft.com/en-us/sysinternals/dd535533?ppu...) will tell you exactly the VA space layout. Note that Linux has exactly the same problems with VA space, except they default to 3/1, instead of NT's default (but configurable) of 2/2. It seems that a quick fix to this issue would be to lower that number from 512MB to something like 128MB, though I know nothing about Go's internals.
- JoshTriplett 15y ago
  Getting 512MB of contiguous virtual address space does not require 512MB of contiguous free physical memory; contiguous virtual addresses can map to discontiguous physical memory.
  - astrodust 15y ago
    That's what I thought. The address your user-space program uses and the actual address in memory are two different things, aren't they? Isn't this what enables the operating system to scramble the allocations it gives you to make it harder to implement a buffer-overflow attack?

cjensen 15y ago

In my opinion, the problem is that 32-bit Windows is intentionally broken by Microsoft. Unlike other 32-bit x86 operating systems, the consumer version of Windows refuses to addresses more than the first 4GB of physical memory, and on most machines a quarter to half of that space is consumed by PCI address space. Yes there are hacks around this. But the right way to deal with this is to just fold and accept what Microsoft is imposing: use a 64-bit version of Windows whenever possible and enjoy your memory. Once on a 64-bit OS, you can either write a 64-bit program, or you can write a 32-bit program and use a compiler option to get the full 32-bit address space. (Microsoft's compilers assume by default that you are an idiot who uses signed values for addresses and your program will break if they give you more than 2GB of virtual address space) The other lesson is don't use experimental languages you are uncertain of for actual product.

pjmlp 15y ago
The same also applies to other 32 bit OS, this is not Windows specific.
- leon_ 15y ago
  Yes, Amiga OS.
- tankenmate 15y ago
  Umm no, other OSes have different defaults for mapping and have different heuristics for how and where they map devices and libraries. Go on 32bit Linux rarely has the problems that the Windows versions do.
  - 4ad 15y ago
    The PCI mappings are the same, either in Linux or Windows. I do agree that this particular report is a Windows problem, 512MB contiguous address space should be available at program initialization in every case. I do agree that Go on 32 bit Linux rarely has any problems, but this is true for Windows as well. What happened in the last few days is blown out of proportion.

fleitz 15y ago

Somewhere in the source of Go will be the number 536870912, change this number to 134217728. Problem should largely be solved. Note: I'm not a Go dev, or have even programmed in it, but logically it should be a quick fix that you can try to save you rewriting your codebase. Given the importance of the number it's probably a #define or constant so should be pretty easy to find in one of the header files.

4ad 15y ago
It's 512<<20, and a proper way to fix it, on Windows, at least, would be to reserve the address space in the PE header, not ask the operating system in the process initialization path.

JoeAltmaier 15y ago

x86 32-bit is actually capable of addressing any amount of memory, using the right compilation model. Flat 32-bit has a 4GB limit sure. But segemented, it can access any amount of virtual memory, with the caveat of only 4GB per allocation. Probably no OS supports this any more. But the instructions set and memory management units support it, or used to in early Pentium days.

ghost91 15y ago
Take a look at PAE nearly every modern kernel supports it.
- JoeAltmaier 15y ago
  But do any languages? When was the last time you compiled to anything but flat?
caf 15y ago
Segment:Offset addresses are converted to 32 bit linear addresses; the segment base address in the segment descriptor is a 32 bit value.
- zvrba 15y ago
  > Segment:Offset addresses are converted to 32 bit linear addresses That's true, but segment descriptors have a not-present bit, which allows you to implement segment-level swapping.
- JoeAltmaier 15y ago
  On a 64-bit architecture its 64-bit. 32-bit programs run there. Also, as noted elsewhere, on 32-bit architectures there's segment swapping too. In fact I wrote an OS that segment swapped before the 386 came out, when the 286 was king. Probably the only one out there; a pretty crazy notion and the 386 came out a year later with paging.
  - caf 15y ago

caf 15y ago

This same Go implementation design earlier ran into some issues on Linux, too - in this case, with distributions setting conservative RLIMIT_AS limits. The technical details are interesting (https://groups.google.com/forum/#!msg/golang-dev/EpUlHQXWykg/LN2o9fV6R3wJ https://groups.google.com/forum/#!msg/golang-dev/EpUlHQXWykg...) and Linus' response (http://lkml.indiana.edu/hypermail/linux/kernel/1102.1/00233.html http://lkml.indiana.edu/hypermail/linux/kernel/1102.1/00233....) is also instructive.

aidenn0 15y ago
I had a similar issue with the default configuration of SBCL on a 64-bit vhost; the 64-bit version asks for 8gb of VAS and the vhost had a fairly small limit on VSIZE. Fortunately the heap allocation is configurable in SBCL; perhaps doing something like that on go would be the right thing?

tashmahalic 15y ago

This is my understanding of the problem: when a Go program is initialized on 32-bit windows or linux systems, it tries to allocate 512MB of contiguous virtual ram. When the virtual address space is too fragmented, the initialization will fail and the Go program won't start. Is this correct? We're considering using Go for an end-user desktop application for Mac, Windows, and Linux. Of course, we don't have control over whether our users have a 32-bit or 64-bit system. And, of course, our app has to work 100% of the time. What can we do to make Go work 100% of the time on these 32-bit systems?

JoeAltmaier 15y ago
Don't know what it means for the virtual address space to be fragmented. Each process begins with a pristine virtual address space.
- 4ad 15y ago
  Mostly, but not really, apart from the binary image itself, its dependent shared libraries are also loaded, and the various images loaded might have virtual address space reservations marked in the header. In normal scenarios there should be enough contiguous address space, Go programs only link to ntdll.dll and winmm.dll. While it's true that on recent Windows versions these two shared objects pull around 30 other shared objects, all the DLLs shipped with Windows have a base address chosen so that they only take a very small chunk of address space and most address space is contiguous. 512MB of contiguous reservation shouldn't be a problem to get. The fact that this is not happening means that 3rd party DLLs are loaded, this happens if you use a program, knowingly or not, that install user mode hooks.

kokoloko 15y ago

I think Go, Mono, Ruby, Python et al should team up to create a reusable garbage collector library that can compete with the OpenJDK one. For me, it's the single biggest piece falling behind on non JVM languages. The performance and "tunnability" are unmatched right now. Bonus points would be for a real time garbage collector, a la IBM Metronome.

tashmahalic 15y ago

We need to decide if Go is a suitable language to build our app with. Our app will be deployed on end-user 32-bit systems, and needs to work 100% of the time. Is there someone here who can say either: (1) Go will fail sometimes on 32-bit systems, don't use it until this problem has been fixed, or (2) there are things you can do to always avoid the 32-bit problems. ?

tashmahalic 15y ago
(and if there's no one here with an informed answer, where should i ask this question? Thanks.)