12 ms·
Go: Severe memory problems on 32bit Linux
- wtallis 15y agoWow. That third message, with suggestions for avoiding the bug, reads like a twisted joke. Highlights: "avoid struct types which contain both integer and pointer fields" "avoid data structures which form densely interconnected graphs at run-time" "avoid integer values which may alias at run-time to an address; make sure most integer values are fairly low (such as: below 10000)" I understand that this isn't a completely brain-dead garbage collector, but warnings like that really scream "I'm just a toy language". It doesn't seem wise to call such a fragile programming tool production-ready or 1.0; the 32-bit implementation should be tagged as experimental, if only to lessen the damage to Go's reputation.
- moilolita 15y agoUnfortunately you are right. For me this is a very unpleasant surprise after I put aside D and rewrote my little framework in Go. Everybody said that the gc is not final, that there are some performance issues and they are working on it but I never imagined that such catastrophic bugs are not solved by now.
- eternalban 15y agoI also looked at D/Go -- I predict the reign of JVM is over (quote me). I wouldn't bother with the 32 bit hiccups. Go hits the sweet spot pretty well. You have chosen well. Hang in there.
- moilolita 15y agoD has a nice allocation management(i loved the manual+gc approach) and in my benchmarks, optimized D was slightly faster than Go at almost anything and consumed up to 70% less memory (I assumed that was because the Go gc kicked in later and was a bit lazy) but D was weak at threading/synchronization and the documentation of the standard lib was quite messy and lacking. So I decided that in the long run Go will be better (cheap goroutines, channels, good stdlib /documentation + support from google and the prospect of a better GC, all indicated a clear winner).I really hope they fix this cause I can't throw away my atom box and ARM is becoming more and more important.
- nakkiel 15y agoI wrote a couple of D programs long ago and I still have a taste of an unfinished language. If I recall correctly, arrays manipulation is pretty weird and the language feels a bit the same PHP feels: a bunch of different things pieced together with no coherence. Compared to Python or Go, it's a whole different world.
- moilolita 15y agoThe Phobos standard library is quite unfinished indeed and poorly documented in some areas but the D2 language is rather complete. It has sh*tloads of features but this also makes it a bit harder to master. Go is lighter (very easy to take on) and has an impressive library. Long story short: with D2 I needed ~1 month to get a good grasp of the language and the std library (the library was the hardest part) while with Go I needed around a week.
- el_muchacho 15y agoAgreed, D is a much bigger and complex language than Go. Comparing the standard libraries, Phobos doesn't look far behind Go in scope. There are big holes though, like crypto, which is entirely missing, and a complete SQL driver (was in development, but we haven't heard from it for a while now), although there is a binding for SQLite3 and several drivers for major RDBMSs (not in the standard lib, tough). Logging will be included soon. Most of the rest is included (networking uses libCurl), and Phobos quality is continuously improving, some parts of it being excellent both in terms of functionality and performance, like the new regex library. On some other parts, like containers, Phobos seems much more advanced than Go. OTOH, there seems to be more 3-party libraries for Go than for D, but we can't comment on their quality. And of course, both languages allow to bind C libraries.
- rubashov 15y ago> The Phobos standard library is quite unfinished indeed How so? I think it's got pretty much what belongs in the standard library.
- drey08 15y ago
- silon3 15y agoI strongly prefer ceylon/rust... But they're not ready yet.
- moilolita 15y agoIf Ceylon/Rust are not ready today it means they will be immature for at least 2 years from their release date. Without a large community behind them, they will slowly fade away like D did.In order to get community, they have to have a good/clean/pragmatic design and a library as comprehensive as possible. If they don't have a good library they better have excellent interop with other platforms or they will be doomed. IMO Go is a solid step in the right direction and if the authors will resist the temptation to complicate it with (too many) new features¶digms I think it will fare well.
- he_the_great 15y ago> Without a large community behind them, they will slowly fade away like D did. This is a rather strange assessment. The history of "successful" languages has been a mixture of "cool jump onboard" and "who can stay alive the longest to get a community." D is in the latter camp. It seems to me, unreasonable to expect a language to be coming out the gate with guns blazing. People expect Nukes now!
- moilolita 15y agoThe market will decide. As someone said on HN: the best language in the world would be a mix of Go and D and would be called GoD. When there were few options, yes, it was enough to survive long enough to get a community. But when you have plenty of similar options, some of them backed by some major actors (at least in the early stages), i think no one will sacrifice his productivity (other than for hobby projects) for the sake of one language. I really hope D will develop into something, but looking to various sources (abandoned projects for D1 on dsource.org) on the net and google results, it seemed to me that D peaked somewhere around 2007-2008 with a slight revival in 2010 when D2 was released.
- el_muchacho 15y agoThe drop after 2007/2008 probably corresponds to the Phobos/Tango debacle, and the fact that Walter decided to fork with D2. One of the main problems was, he was almost the sole compiler developer, and could hardly keep up with the tasks of maintaining 2 parallel branches and developing new ideas at the same time. People complained that they couldn't get involved as much as they wanted. It's understandable that many people thought that D didn't have a solid future with such uncertainties. Nowadays, these problems are mostly overcome with a much better organization: there are several committers for the compiler, and several committers for the standard library. Phobos is the standard library, it's maturing, D2 has shown its strengths over D1 and the community is united again, because not only it is deeply involved with the design of the language and standard lib(through the m-l), it is also involved with the implementation of essential parts of it. 2011 has been a very good year for D, and I think that more than ever, the whole project feels like it's going in the right direction. edit: I guess another reason D isn't gaining as much traction as it could is, it has been removed from the Alioth computer language shootout. For a language which is aimed at raw speed (and was brilliant at that when it was still on the shootout), it's a severe blow.
- BarkMore 15y agoGo 1 defines and the backwards-compatibility guarantees one can expect as the language matures. It's more a statement about the language specification than it is about the implementations. I know from experience that the gc 64-bit version is production ready. It's unfortunate that the limitations of the 32-bit version are not called out clearly on the website.
- nakkiel 15y agoClearly, there was a need for "Hey, things won't change as much from now on". Now, I'm still afraid little work will appear on the GC, which is IMO Go's biggest weakness to date.
- dchest 15y agoNow, I'm still afraid little work will appear on the GC No need to guess, look at golang-dev https://groups.google.com/forum/?fromgroups#!forum/golang-dev https://groups.google.com/forum/?fromgroups#!forum/golang-de...
- nakkiel 15y agoOh yeah thanks. I guess I'm doing it wrong :) Glad to see some work on that front.
- batista 15y ago>I know from experience that the gc 64-bit version is production ready. Is it really production ready though, or is it the same half-arsed implementation as the 32-bit one, just taking advantage of the larger availability of virtual memory on a 64 bit system?
- oskarth 15y agoIf Google is, and has been for quite some time, using it for large-scale systems in production, I would call it production ready.
- singular 15y agoKeep in mind that the comments on that thread are open to all people and thus that doesn't entail an 'official' response to the problem, though it does unfortunately seem to entail sound (but obviously crazy given the restrictions it puts you under) advice for avoiding the issue.
- 0xe2-0x9a-0x9b 15y agoI am sorry if the post puts Go in a bad light. That wasn't my intention. You shouldn't read the post as "32-bit Go is unusable" - read it as "If you encounter a garbage collection issue in your 32-bit program then you can use the rules mentioned in the post to fix the issue".
- sp332 15y agoWe know. But that does put Go in a bad light. If your garbage collector falls down if you use "big numbers", it looks bad.
- _delirium 15y agoThis is the relevant bug: http://code.google.com/p/go/issues/detail?id=909 http://code.google.com/p/go/issues/detail?id=909 As the discussion there and in the thread says, the root problem is that Go uses a conservative garbage collector, and on 32-bit a lot more values look like pointers than on 64-bit, so many more things don't get freed in long-running processes. Seems not to be easy to fix.
- sounds 15y agoPublicizing the bug may speed the fix along. Here's the real test: is hitting the front page of HN going to make a difference?
- _delirium 15y agoI don't think this one is lacking interest, but an obvious fix. It looks like Russ Cox did commit a fix for some portion of the cases. But a better fix would either need to significantly modify the way things are laid out to reduce the odds of false positives, or else move to a precise GC. (Precise GCs are possible in C-like languages, but trickier to implement. Here's a recent paper on one: http://www.cs.utah.edu/~regehr/papers/ismm15-rafkind.pdf http://www.cs.utah.edu/~regehr/papers/ismm15-rafkind.pdf)
- eternalban 15y agoVon Neumann architecture falsely lead us to the path with the singular notion of "memory". We need to distinguish at the hardware architecture level between "working memory" and "persistent memory". I doodle all the time when working. Pristine set of diagrams emerges from the chaos. I wish my computer would play along with this regime ..
- eternalban 15y agoI haven't mentioned downvotes to date, but the down votes on this is puzzling. Would the down voters care to comment as to why the comment is deemed down vote worthy? (I would think those who work/grok on memory managers do get the point.)
- sunyc 15y agoto go back to 32 bit hardware, you will be running very old machines that you can buy off ebay for <200$ and the power cost offset the new hardware cost in several months.
- wtallis 15y agoRunning a 32-bit operating system on 64-bit hardware is still pretty common. And this problem afflicts ARM as well.
- jergason 15y agoThis is really disappointing. I was hoping to do some stuff with Go on ARM, but it looks like they are not too concerned about this bug.
- luser001 15y agoI would run a 32-bit OS (even if the hardware supported 64-bit) if I wanted to squeeze every last bit of usable memory out of a VPS.
- derleth 15y agoThis sounds like precisely the use-case for x32: Running userland programs that only use 32-bit pointers on x86-64 bit hardware in 64-bit mode; you only get to address 4GB of RAM, but your pointers fit two to a register and you have all the x86-64 registers, opcodes, and special hardware. Sadly, it isn't here yet. https://sites.google.com/site/x32abi/ https://sites.google.com/site/x32abi/ http://en.wikipedia.org/wiki/X32_ABI http://en.wikipedia.org/wiki/X32_ABI (To forestall the obvious objection: No, there's no way Torvalds will allow this to re-introduce the 2038 Problem into a new ABI. None.)
- Aissen 15y agox32 has been accepted(and merged) for Linux 3.4. Kernel support is here, now it's userspace's turn.
- 15y ago
- deleted 15y ago[deleted]
- jedbrown 15y agoHowever, it does not address the base issue, which is that Go uses a conservative garbage collector, and more values look like pointers in a 32-bit world. The only real fix would be to improve the garbage collector's understanding of which values are pointers and which are something else (e.g., floating point numbers that happen to look like pointers). And that is not an easy fix. How does this GC work? Is it literally just marching through the heap looking for pointer-sized values in the range that has been mapped to the process?
- krakensden 15y agoThe heap AND the data section. Although apparently, gcc-go doesn't do that for the data section.
- jedbrown 15y agoThat would seem to also present a DOS vector (even on 64 bit) if a user can get the program to store data (of any type, e.g. char or floating point) that happen to be binary-equivalent to pointers to large allocations.
- agentS 15y agoThis is realistically not an issue. First, on a 64 bit machine, the range of actually mapped addresses is small relative to all the possible values that can fit into 64 bits. Second, from an attacker's perspective, the values corresponding to mapped memory are extremely difficult to predict, and the values binary-equivalent to large allocations are impossible to predict, even with access to the source. If you think you can still do it, all of *.golang.org and golang.org are running Go on Appengine, with the source code being freely available. This is your opportunity to get a back door into Google's servers.
- jedbrown 15y agoIf you make a huge allocation (many pages), isn't the Go runtime very likely to call malloc()? For large allocations, malloc() is going to get you a bunch of fresh pages and you will generally get the address of the start of a page. The offset of the pointer within the page is then likely to be deterministic, so you probably only need one unit of pointer-equivalent data per page. If you have enabled huge pages (e.g. 2MB, not uncommon), then you have already soaked up 21 bits of the 48 bits of address space that are actually used by x86-64 implementations, leaving only 27 bits for a collision. The stack grows down from 2^46 and typical heap values on x86-64 are still well within 32 bits. Finally, a collision need not be frequent to be a serious DOS concern.
- 0xABADC0DA 15y agoIt's amazing how nonchalant they are about this 'oh sure we'll fix it in a year or so' when the garbage collector is basically all there needs to be in a Google Go runtime Some comments were asking why early Java's GC wasn't this bad even though it was conservative. The reason is that Java can't take references to fields of an object, so the data mistaken as a pointer has to actually point to an object header. In Google Go you can take a reference to a field, locking the whole object, so the faux pointer can point to any field as well (or in this probably any location in the object). Not exactly the wisest choice in semantics, as they are seeing now that it complicates the GC.
- 4ad 15y agoIt's not Google Go, it's simply Go, there's not a single Google reference on the web page and that's intentional. Looking at your previous posts here and on Reddit I see you are a notorious anti-Google troll that adds "Google" to "Go" so that your negative posts will be associated with both Google and Go. It's absurd to call it Google Go when there's a first class GNU implementation that has been in development from the beginning, there's at least a closed source implementation and another distinct BSD license implementation in its infancy. It makes even less sense than saying Apple LLVM, Juniper FreeBSD, or AT&T C.
- deleted 15y ago[deleted]
- burgerbrain 15y agoThe number of times people have needed to tell this guy this is staggering.
- rand_r 15y agoAny recommendation on how we should google for Go related pages?
- 4ad 15y agoGolang or Go programming works fine. If searching for Google Go produces meaningful results, by all means, use it. I was merely complaining about referring to the project as such. There's also this: http://go-lang.cat-v.org/go-search http://go-lang.cat-v.org/go-search
- luriel 15y agogccgo is better on 32bit systems: https://groups.google.com/d/msg/golang-nuts/qxlxu5RZAl0/NS718AK297cJ https://groups.google.com/d/msg/golang-nuts/qxlxu5RZAl0/NS71... Note that work on the garbage collector is ongoing post-Go1, a faster parallel GC is in process being merged: http://codereview.appspot.com/5279048/ http://codereview.appspot.com/5279048/ But ultimately 32bit systems don't seem to be a big issue for Google or anyone else using Go in production (and there are quite a few big organizations using it: http://go-lang.cat-v.org/organizations-using-go http://go-lang.cat-v.org/organizations-using-go ). Most people moved to 64bits a while ago so the amount of attention the 32bit port gets will never be the same.
- leon_ 15y agoyeah, if there wasn't ARM ...
- bad_user 15y agoYou know, that kind of sucks ... there are many machines around that are still on 32 bits. I keep my own Ubuntu laptop (dual-boots to Windows) on 32-bit builds (both Linux and Windows), simply because I have less problems that way (mostly with hardware drivers, but also with software). I keep the Amazon EC2 instances I maintain on 32-bit images, simply because they are cheaper. My Android phone is also 32-bit and will be so for a long time. My other phone, an older iPhone 3GS, is also 32-bit. My servers, prior to Amazon EC2, built with ARM processors, were also 32-bit. And when I was playing with MongoDB, do you know what I did when I discovered that the 32-bit build was basically unusable? I ditched it and never looked back.
- 4ad 15y agoI've been exclusively using 64 bit computers and operating systems, both Windows and Linux, for about 7 years now, never had the reported driver problems, well, never had any issue, really. For me at least, 32 bit is only important for ARM. On the other hand the issue is very much blown out of proportion, most people haven't seen it, even if they run 32 bit servers. Most usual servers written in Go, like web servers, use very little memory. I process 4k requests per second using 7MB of resident memory. There are many memory intensive applications, but you usually don't run those on 32 bit.
- stiff 15y agoThe problem itself seems quite serious and I would seriously reconsider using Go if I was interested in it in the first place, but after seeing the way this is treated by the Go "community", I am pretty sure I will never ever even think about using Go for anything.
- dchest 15y agothe way this is treated by the Go "community" What do you mean?
- stiff 15y agoWell, since the people behind Go did not up front say anything about the language/compiler being specifically targeted at 64-bit platforms, I would expect someone more mature from the Go team to step up and say something like "We are sorry, we didn't foresee the consequences of some design decisions and hence screwed it up." and either "We will fix it ASAP" or "We cannot fix it because XYZ". They might not take money for their work directly from the users, but there is still some moral obligation if you create something, release it to the world, praise its virtues and persuade people to use it. As it can be seen in the thread, lots of people already invested lots of time into building things with Go and now they're in serious trouble. Instead, many people in this thread try to somehow downplay the problem, advocate changing hardware (that's something quite new in the programming language world) or following some pretty absurd guidelines. This might or might not be representative of the whole community around Go, but it surely leaves a bad taste, hence the slight irony.
- dchest 15y agoThere is an "official" response: the post references a bug # where Russ Cox said "the rest of the issue will have to wait until after Go 1". Note that Go 1 is a "language freeze", not the implementation freeze. Now that it's known there's a bug that will be worked on later, people proposed possible workarounds, the easiest of which is to switch to 64-bit platform. I agree that downplaying the issue is wrong, but only one person did that. There is occasional rudeness, mostly caused by strong opinions, but overall I think Go community is pretty good.
- ww520 15y agoAre most Androids 32-bit? Doesn't this issue impact the GO Android SDK adoption?
- dchest 15y ago...if there was the Go Android SDK.
- masklinn 15y ago> Are most Androids 32-bit? By virtue of there being no AArch64 implementation announced so far, all androids are 32-bit because there is no 64b ARM core. > Doesn't this issue impact the GO Android SDK adoption? Any Go application should be able to run into this issue if it has a similar usage pattern.
- tbsdy 15y agoIs there a reproducible test case? I can well believe that there might be issues, the post isn't really actionable without some sort of specific code to go along with it :(
- zvrba 15y agoInteresting piece of information below in the thread: in reply to "Go being advertised as a systems PL", David Symonds replies with: "It's not. It used to be, but it's not any more."
- el_muchacho 15y agoWhat is the definition of a system PL ?
- talentdeficit 15y agodeterministic memory usage is a good start
- zvrba 15y agoThe ability to access the raw memory underlying any object.
- binarycrusader 15y agoExcept, it still is advertised as one (or at least implied to be): "Go is a general-purpose language designed with systems programming in mind." http://golang.org/ref/spec#Introduction You could nit-pick and say that it says "in mind", but the point still stands that they're trying to imply that it's suitable for systems programming. It is not; certainly not at the level of an operating system kernel, etc.
- blinkingled 15y agoI suspect the issue exists on 64-bit platforms as well, it's just that it doesn't impact as easily. In theory it is possible/common to have 64-bit machine with less than a ton of physical memory (VPS) and running 64-bit Go program which triggers this bug would result in similar impact as the 32-bit version. Why yes - Issue #909 essentially confirms this - running 64-bit doesn't fundamentally change anything - it just buys more time for impact because of larger address space and hope of more physical memory. Which is sad on many levels - just mind blowing that the language designers did not think of this upfront! (Oh and Go's built in packages trigger this problem too - per #909 commenting Unicode package makes the program run!)
- 4ad 15y agoPhysical memory does not have anything to do with it, the virtual address space is all that matters. It might be useful to consider that 2^64 is 2^32 * 2^32. This means the problem becomes important on 64 bit only if an application will use 10 orders of magnitude more memory. By considering the historic growth in memory capacity and usage, this will only happen around 2060.
- blinkingled 15y agoCheck out http://code.google.com/p/go/issues/detail?id=909#c32 http://code.google.com/p/go/issues/detail?id=909#c32 and the following comment that agrees with it. This isn't just a address space leak - it is a real memory leak. On 64-bit the GC may not be so easily fooled as on 32-bit but it can still be fooled and that is a fundamental problem that will result in memory leaks - if I have 2GB RAM VPS - it doesn't help to have 2^64 bits of address space (actually it is more like 2^48 (http://en.wikipedia.org/wiki/X86-64#Virtual_address_space_details http://en.wikipedia.org/wiki/X86-64#Virtual_address_space_de... ) - if the GC leaks memory sooner or later my process will be killed by the OS.
- 4ad 15y agoAgain, this has nothing to do with physical memory. It only has to do with the virtual address space. Of course if you bump into this, it's a real leak, who said otherwise? And yes, it's possible to artificially generate the collision on 64 bit as it's the same mechanism as with 32 bit. It's about whether it happens frequent enough under normal usage patterns to be a concern. Youtube, and everybody who tried Go in production say it isn't, and that's because of reasons outlined in my first reply to you.
- ezyang 15y agoFunnily enough, I understood why they went the conservative GC route. It has to do with the overall Go philosophy, which is that they really do not want features to affect data representation. This has meant no boxing (and no easy polymorphism), and a decision like that has logical consequences for GC too. Here's to hoping they find a cool solution! It's been a problem for GC's since forever, and if they find a general way of handling the problem I'm sure it will be picked up by many other runtimes.
- pcwalton 15y agoIn Rust we're working on a solution for this problem. Essentially, the plan is to have RTTI on garbage-collected data (this is already completed and is used in the cycle collector) and precise stack information for every root on the stack. The latter is in a fork of LLVM: https://github.com/pcwalton/llvm https://github.com/pcwalton/llvm But aside from the shameless plug, C# and D (I believe), have had precise garbage collectors for quite a while now, and they have similar memory management to Go. It's well-known how to implement it (but that doesn't make it any less hard — I can totally understand why Google opted for conservative GC in the first version).
- ezyang 15y agoI know how to do precise GC if you allow me to add a (pointer-size) header to all data living in the heap; i.e. to maintain the RTTI. I don't know how you do that if you're not allowed a header. Do C# and D have headers?
- pcwalton 15y agoI'm sure they do. There are a few things to note here: (1) In order for malloc to work, you need a header anyway (at least, unless your allocation fits in one of the fixed-size bins). (2) You can get around the header to some extent by sorting the fields of your objects so that pointers come first, and then all you need to do is to store the number of pointers (or a sentinel value). This is what Haskell does. Of course, this prevents low-level control over data representation. (3) You can tag (or NaN box) all your values. This is what most MLs do, as well as JS, many Lisps, etc. (4) You can use a map on the side from pointer to type info to avoid a header. This is what Rust in its early days did. It's worse than a header for memory consumption though, so it doesn't really buy much.
- ungerik 15y agoAfter switching to 64 bit, we haven't had a single crash. So I would describe the 64 bit version as production ready. But the problems on 32 bit systems should have been documented in the release notes. Beside this hiccup (and a week without sleep with the website on life support), Go has been a real joy to work with.