6 ms·

However, it does not address the base issue, which is that Go uses a conservative garbage collector, and more values look like pointers in a 32-bit world. The

However, it does not address the base issue, which is that Go uses a conservative garbage collector, and more values look like pointers in a 32-bit world. The only real fix would be to improve the garbage collector's understanding of which values are pointers and which are something else (e.g., floating point numbers that happen to look like pointers). And that is not an easy fix. How does this GC work? Is it literally just marching through the heap looking for pointer-sized values in the range that has been mapped to the process?

krakensden 15y ago
The heap AND the data section. Although apparently, gcc-go doesn't do that for the data section.
- jedbrown 15y ago
  That would seem to also present a DOS vector (even on 64 bit) if a user can get the program to store data (of any type, e.g. char or floating point) that happen to be binary-equivalent to pointers to large allocations.
  - agentS 15y ago
    This is realistically not an issue. First, on a 64 bit machine, the range of actually mapped addresses is small relative to all the possible values that can fit into 64 bits. Second, from an attacker's perspective, the values corresponding to mapped memory are extremely difficult to predict, and the values binary-equivalent to large allocations are impossible to predict, even with access to the source. If you think you can still do it, all of *.golang.org and golang.org are running Go on Appengine, with the source code being freely available. This is your opportunity to get a back door into Google's servers.
    - jedbrown 15y ago
      If you make a huge allocation (many pages), isn't the Go runtime very likely to call malloc()? For large allocations, malloc() is going to get you a bunch of fresh pages and you will generally get the address of the start of a page. The offset of the pointer within the page is then likely to be deterministic, so you probably only need one unit of pointer-equivalent data per page. If you have enabled huge pages (e.g. 2MB, not uncommon), then you have already soaked up 21 bits of the 48 bits of address space that are actually used by x86-64 implementations, leaving only 27 bits for a collision. The stack grows down from 2^46 and typical heap values on x86-64 are still well within 32 bits. Finally, a collision need not be frequent to be a serious DOS concern.
      - 4ad 15y ago
        The Go runtime does not call malloc(3) for heap, it reserves address space at known high locations (over 2^32) with mmap(2) using the MAP_FIXED flag, and it does so in 16GB increments (or is it just one 16GB allocation? can't remember). I won't comment on the DOS concern until I've investigated further.
  - ianlancetaylor 15y ago
    You need more than that. You need a server that regularly allocates large amounts of memory and then leaves them unreferenced so that the garbage collector can collect them. Then you also need the program to store data that you control, and to also keep references to that data--after all, if that data is collected, then the faux-pointers no longer pin the other allocations. Overall this does not sound like a common allocation pattern for servers.
smanek 15y ago
Yep, that's basically it. It sounds crazy (and it is!) - but it often works reasonably well in practice. SBCL (one of the most performant Common Lisp implementations) has an 'imprecise gc' that works the same way - and I've seen reasonably heavily stressed processes with uptimes in the weeks/months. Remember, even a few years ago (before fastthread, etc) a reasonably loaded Ruby on Rails app couldn't stay up for more than ~10 minutes w/o memory leaks forcing a restart (DHH said 37Signals as doing ~400 restarts/day per process, IIRC) because the runtime was such a piece of crap. Yet, many people still used it to solve real problems and make real money. At least Go's memory leaks are much slower than Ruby's ;-)
- eblume 15y ago
  That's really interesting - is there a place where I can read more about these early Rails performance issues?
  - smanek 15y ago
    There was some back-and-forth between Zed Shaw and DHH that I remember being interested by at the time. Zed deleted all his posts - but here are some places to get started: - Shaw's opening volley: http://web.archive.org/web/20080103072111/http://www.zedshaw.com/rants/rails_is_a_ghetto.html http://web.archive.org/web/20080103072111/http://www.zedshaw... [lots of stupid personal flames - but his technical points are consistent with what I remember from the time] - DHH's response: http://david.heinemeierhansson.com/posts/31-myth-2-rails-is-expected-to-crash-400-timesday http://david.heinemeierhansson.com/posts/31-myth-2-rails-is-... - I can't find a copy of shaw's actual response, but here's the relevant HN thread: http://news.ycombinator.com/item?id=364659 http://news.ycombinator.com/item?id=364659 The salient point (quoting Zed): """ Now, DHH tells me that he’s got 400 restarts a mother fucking day. That’s 1 restart about ever 4 minutes bitches. These restarts went away after I exposed bugs in the GC and Threads which Mentalguy fixed with fastthread (like a Ninja, Mentalguy is awesome). If anyone had known Rails was that unstable they would have laughed in his face. Think about it further, this means that the creator of Rails in his flagship products could not keep them running for longer than 4 minutes on average. Repeat that to yourself. “He couldn’t keep his own servers running for longer than 4 minutes on average.” """ I've never been much of Ruby/Rails guy - but my understanding is that everyone had to restart Rails a few times an hour cause the memory leaks were so bad in those days.

krakensden 15y ago

The heap AND the data section. Although apparently, gcc-go doesn't do that for the data section.

jedbrown 15y ago
That would seem to also present a DOS vector (even on 64 bit) if a user can get the program to store data (of any type, e.g. char or floating point) that happen to be binary-equivalent to pointers to large allocations.
- agentS 15y ago
  This is realistically not an issue. First, on a 64 bit machine, the range of actually mapped addresses is small relative to all the possible values that can fit into 64 bits. Second, from an attacker's perspective, the values corresponding to mapped memory are extremely difficult to predict, and the values binary-equivalent to large allocations are impossible to predict, even with access to the source. If you think you can still do it, all of *.golang.org and golang.org are running Go on Appengine, with the source code being freely available. This is your opportunity to get a back door into Google's servers.
  - jedbrown 15y ago
    If you make a huge allocation (many pages), isn't the Go runtime very likely to call malloc()? For large allocations, malloc() is going to get you a bunch of fresh pages and you will generally get the address of the start of a page. The offset of the pointer within the page is then likely to be deterministic, so you probably only need one unit of pointer-equivalent data per page. If you have enabled huge pages (e.g. 2MB, not uncommon), then you have already soaked up 21 bits of the 48 bits of address space that are actually used by x86-64 implementations, leaving only 27 bits for a collision. The stack grows down from 2^46 and typical heap values on x86-64 are still well within 32 bits. Finally, a collision need not be frequent to be a serious DOS concern.
    - 4ad 15y ago
      The Go runtime does not call malloc(3) for heap, it reserves address space at known high locations (over 2^32) with mmap(2) using the MAP_FIXED flag, and it does so in 16GB increments (or is it just one 16GB allocation? can't remember). I won't comment on the DOS concern until I've investigated further.
- ianlancetaylor 15y ago
  You need more than that. You need a server that regularly allocates large amounts of memory and then leaves them unreferenced so that the garbage collector can collect them. Then you also need the program to store data that you control, and to also keep references to that data--after all, if that data is collected, then the faux-pointers no longer pin the other allocations. Overall this does not sound like a common allocation pattern for servers.

smanek 15y ago

Yep, that's basically it. It sounds crazy (and it is!) - but it often works reasonably well in practice. SBCL (one of the most performant Common Lisp implementations) has an 'imprecise gc' that works the same way - and I've seen reasonably heavily stressed processes with uptimes in the weeks/months. Remember, even a few years ago (before fastthread, etc) a reasonably loaded Ruby on Rails app couldn't stay up for more than ~10 minutes w/o memory leaks forcing a restart (DHH said 37Signals as doing ~400 restarts/day per process, IIRC) because the runtime was such a piece of crap. Yet, many people still used it to solve real problems and make real money. At least Go's memory leaks are much slower than Ruby's ;-)

eblume 15y ago
That's really interesting - is there a place where I can read more about these early Rails performance issues?
- smanek 15y ago
  There was some back-and-forth between Zed Shaw and DHH that I remember being interested by at the time. Zed deleted all his posts - but here are some places to get started: - Shaw's opening volley: http://web.archive.org/web/20080103072111/http://www.zedshaw.com/rants/rails_is_a_ghetto.html http://web.archive.org/web/20080103072111/http://www.zedshaw... [lots of stupid personal flames - but his technical points are consistent with what I remember from the time] - DHH's response: http://david.heinemeierhansson.com/posts/31-myth-2-rails-is-expected-to-crash-400-timesday http://david.heinemeierhansson.com/posts/31-myth-2-rails-is-... - I can't find a copy of shaw's actual response, but here's the relevant HN thread: http://news.ycombinator.com/item?id=364659 http://news.ycombinator.com/item?id=364659 The salient point (quoting Zed): """ Now, DHH tells me that he’s got 400 restarts a mother fucking day. That’s 1 restart about ever 4 minutes bitches. These restarts went away after I exposed bugs in the GC and Threads which Mentalguy fixed with fastthread (like a Ninja, Mentalguy is awesome). If anyone had known Rails was that unstable they would have laughed in his face. Think about it further, this means that the creator of Rails in his flagship products could not keep them running for longer than 4 minutes on average. Repeat that to yourself. “He couldn’t keep his own servers running for longer than 4 minutes on average.” """ I've never been much of Ruby/Rails guy - but my understanding is that everyone had to restart Rails a few times an hour cause the memory leaks were so bad in those days.