3 ms·

Go doesn't scan big []byte/strings, so the GC isn't a problem.

chetanahuja 13y ago
Brad could you elaborate on this statement please. I tried a few google searches to find documentation on that behaviour but failed.
- twotwotwo 13y ago
  Roughly: a GC has to look at the pointers in use to see what objects are still reachable. But it doesn't have to look at the contents of strings, byte slices, int arrays, etc., because, by definition, those don't contain pointers. For dl.google.com, the first user of groupcache, it sounds like the Go runtime was working with relatively few, relatively large chunks of memory (think 2MB chunks of a Chrome binary). Since that doesn't have so many pointer-containing objects, just big chunks of inert data that don't need scanned, it's easyish. On the other hand, something like throwaway2424's case could be a GC stress test--many gigs of RAM holding a network of small, pointer-filled objects. Really, the tl;dr may be to prototype/measure if you need to know how a GC will do. Assumptions are easy; data is hard.
  - chetanahuja 13y ago
    Ahh got it. Also, the reason Brad specifically mentions big arrays of bytes/strings is that they don't look like pointers. Soo.... I'm guessing you still want to stay away from huge arrays of ints (or whatever type has the same size as pointers on your platform). Right ? And as for measuring... it's not cheap (developer time wise) to build a realistic enough model of your application and put realistic enough loads on it for long periods of time to test out each of your hypotheses (and GC theories do require long periods of load). So you have to narrow the space of choices informed by past experiences and sometimes by gathering semi-reliable folk lore. I'm currently engaging in the latter activity :-)
    - ablankst 13y ago
      In Go, and other type-safe languages, the compiler usually knows exactly which ints are being used as pointers. This is the difference between "precise" and "conservative" GC and is one of the reasons that comparing GC performance in C++ to GC performance in other languages is difficult.
      - chetanahuja 13y ago
        I was under the impression (again, folk-lore, mailing lists etc.) that Go had a conservative collector. I can't find an official documentation link that'll tell me if it is or not at the moment.
        twotwotwo 13y ago
        This neat StackOverflow answer appears to be by the 'atom' fellow who contributed a patch to make the collector more precise (patch review = https://codereview.appspot.com/7307086/ https://codereview.appspot.com/7307086/): http://stackoverflow.com/questions/7823725/what-kind-of-garbage-collection-does-go-use http://stackoverflow.com/questions/7823725/what-kind-of-garb... Atom also says 1.0's was more conservative, but, as Brad also said, still didn't scan "objects such as []byte" (meaning all plain-old-data arrays? who knows). The Go 1.1 Release Notes mention the collector becoming more precise, which was a particular issue on 32-bit because big heaps could span a lot of the address space. You can see the GC source itself doing some per-type switching: https://code.google.com/p/go/source/browse/src/pkg/runtime/mgc0.c https://code.google.com/p/go/source/browse/src/pkg/runtime/m... At some point, this sort of discussion probably gets you less useful info per unit effort than just playing with a Go distribution, trying out whatever toy programs you find interesting.
    - twotwotwo 13y ago
      ++ablankst that byte isn't special here. Brad singled out []byte/string because those are the types stored in groupcache (see ByteView in the groupcache code for his sneaky tricks to make them look the same to other code). And yes, unfair to ask someone to test every guess they have. But if you do want to know a bit more about what effect GC and Go would have, experimenting with toy programs isn't a bad way to get a feel.

Brad could you elaborate on this statement please. I tried a few google searches to find documentation on that behaviour but failed.

twotwotwo 13y ago
Roughly: a GC has to look at the pointers in use to see what objects are still reachable. But it doesn't have to look at the contents of strings, byte slices, int arrays, etc., because, by definition, those don't contain pointers. For dl.google.com, the first user of groupcache, it sounds like the Go runtime was working with relatively few, relatively large chunks of memory (think 2MB chunks of a Chrome binary). Since that doesn't have so many pointer-containing objects, just big chunks of inert data that don't need scanned, it's easyish. On the other hand, something like throwaway2424's case could be a GC stress test--many gigs of RAM holding a network of small, pointer-filled objects. Really, the tl;dr may be to prototype/measure if you need to know how a GC will do. Assumptions are easy; data is hard.
- chetanahuja 13y ago
  Ahh got it. Also, the reason Brad specifically mentions big arrays of bytes/strings is that they don't look like pointers. Soo.... I'm guessing you still want to stay away from huge arrays of ints (or whatever type has the same size as pointers on your platform). Right ? And as for measuring... it's not cheap (developer time wise) to build a realistic enough model of your application and put realistic enough loads on it for long periods of time to test out each of your hypotheses (and GC theories do require long periods of load). So you have to narrow the space of choices informed by past experiences and sometimes by gathering semi-reliable folk lore. I'm currently engaging in the latter activity :-)
  - ablankst 13y ago
    In Go, and other type-safe languages, the compiler usually knows exactly which ints are being used as pointers. This is the difference between "precise" and "conservative" GC and is one of the reasons that comparing GC performance in C++ to GC performance in other languages is difficult.
    - chetanahuja 13y ago
      I was under the impression (again, folk-lore, mailing lists etc.) that Go had a conservative collector. I can't find an official documentation link that'll tell me if it is or not at the moment.
      - twotwotwo 13y ago
        This neat StackOverflow answer appears to be by the 'atom' fellow who contributed a patch to make the collector more precise (patch review = https://codereview.appspot.com/7307086/ https://codereview.appspot.com/7307086/): http://stackoverflow.com/questions/7823725/what-kind-of-garbage-collection-does-go-use http://stackoverflow.com/questions/7823725/what-kind-of-garb... Atom also says 1.0's was more conservative, but, as Brad also said, still didn't scan "objects such as []byte" (meaning all plain-old-data arrays? who knows). The Go 1.1 Release Notes mention the collector becoming more precise, which was a particular issue on 32-bit because big heaps could span a lot of the address space. You can see the GC source itself doing some per-type switching: https://code.google.com/p/go/source/browse/src/pkg/runtime/mgc0.c https://code.google.com/p/go/source/browse/src/pkg/runtime/m... At some point, this sort of discussion probably gets you less useful info per unit effort than just playing with a Go distribution, trying out whatever toy programs you find interesting.
  - twotwotwo 13y ago
    ++ablankst that byte isn't special here. Brad singled out []byte/string because those are the types stored in groupcache (see ByteView in the groupcache code for his sneaky tricks to make them look the same to other code). And yes, unfair to ask someone to test every guess they have. But if you do want to know a bit more about what effect GC and Go would have, experimenting with toy programs isn't a bad way to get a feel.