8 ms·

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced m

by rsc 5y ago

This article is full of misinformation. Just a few representative things: - The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint, by letting the OS demand-page this critical table that is used any time a stack must be walked (in particular, during garbage collection). See https://golang.org/s/go12symtab https://golang.org/s/go12symtab for details. - We (the Go team) did not “recompress” pclntab in Go 1.15. We did not remove pclntab in Go 1.16. Nor do we have plans to do either. Consequently, we never claimed “pclntab has been reduced to zero”, which is presented in the article as if a direct quote. - If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not. - The big table seems to claim that a 40 MB Go 1.8 binary has grown to a 289 MB Go 1.16 binary. That’s certainly not the case. More is changing from line to line in that table than the Go version. Overall, the claim of “dark bytes” or “non-useful bytes” strikes me as similar to the claims of “junk DNA”. They’re not dark or non-useful. It turns out that having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space, which we’ve worked over time at reducing. But the dynamic possibilities in reflection and interface assertions mean that fewer bytes can be dropped than you’d hope. We track binary size work in https://golang.org/issue/6853 https://golang.org/issue/6853. An unfortunate article.

rsc 5y ago
An easily obtained apples-to-apples¹ table: $ for i in $(seq 3 16); do curl -sLo go1.$i.tgz https://golang.org/dl/go1.$i.linux-amd64.tar.gz tar xzf go1.$i.tgz go/bin/gofmt size=$(ls -l go/bin/gofmt | awk '{print $5}') strip go/bin/gofmt size2=$(ls -l go/bin/gofmt | awk '{print $5}') echo go1.$i $size $size2 done go1.3 3496520 2528664 go1.4² 14398336 13139184 go1.5 3937888 2765696 go1.6 3894568 2725376 go1.7 3036195 1913704 go1.8 3481554 2326760 go1.9 3257829 2190792 go1.10 3477807 2166536 go1.11 3369391 2441288 go1.12 3513529 2506632 go1.13 3543823 2552632 go1.14 3587746 2561208 go1.15 3501176 2432248 go1.16 3448663 2443736 $ Size fluctuates from release to release, but the overall trendline is flat: Go 1.16 binaries are roughly where Go 1.3 binaries were. At the moment, it looks like Go 1.17 binaries will get a bit smaller thanks to the new register ABI making executable code smaller (and faster). ¹ Well, not completely. The gofmt code itself was changing from release to release, but not much. Most of the binary is the libraries and runtime, though, so it's still accurate for trends. ² Turns out we shipped the go 1.4 gofmt binary built with the race detector enabled! Oops.
- knz42 5y ago
  Strip removes the symbol tables and DWARF information. But still, the sum of all the bytes advertised in symbol tables for the non-DWARF data does not sum up to the stripped size. What's the remainder about? I am reminded of how early versions of MSWord were embedding pages of heap space in save files that were not relevant to the document being saved, just because it made the saving algorithm simpler. For all we know, the go linker could be embedding random data.
  - rsc 5y ago
    > For all we know, the go linker could be embedding random data. I do know, and it is not.
    - knz42 5y ago
      > But still, the sum of all the bytes advertised in symbol tables for the non-DWARF data does not sum up to the stripped size. What's the remainder about? if you do know, then pray, what is the answer to this question?
      - rsc 5y ago
        Maybe another day I will take the time to write a full-length blog post examining the bytes in a Go binary. Today I have other work planned and still intend to do it. My points today are only that: 1. Go binary size has not gotten dramatically better or worse in any particular release and is mostly unchanged since Go 1.3. 2. Many claimed facts in the blog post are incorrect. 3. The linker is not "embedding random data" into Go binaries as you conjectured in the comment above. Stepping back a level, you don't seem to be interested in engaging in good faith at all. I'm not going to reply to any more of your comments.
        hobbified 5y ago
        It's a mystery, not a lynch mob. Everyone reading is interested in knowing "huh, what is this stuff then?"
knz42 5y ago
As explained in another comment below, it's a good thing when all the byte usage is represented in the ELF section headers or the symbol tables. The DWARF data is currently so represented, and so was pclntab prior to 1.16. Today, the DWARF data is still present; the symbol for pclntab has an advertized size of zero, and the amount of data not represented in the symbol table and ELF section headers has grown. > If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not. Probably not if all of it was removed at once. It could be that just 5% of it is useful and removing all of it would produce a non-working binary. What does the experiment prove exactly? Look you can claim that "having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space" but without clear evidence of how much space that is, with proper accounting of byte usage, this claim is non-falsifiable and thus of doubtful value.
- rsc 5y ago
  > Look you can claim that "having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space" but without clear evidence of how much space that is, with proper accounting of byte usage, this claim is non-falsifiable and thus of doubtful value. I respectfully disagree. I believe there is value in pointing out the false claims in a long blog post even when there isn't time to write a full-length rebuttal.
- erincandescent 5y ago
  > As explained in another comment below, it's a good thing when all the byte usage is represented in the ELF section headers or the symbol tables. It's an ELF binary; all that's relevant are the program/segment headers and dynamic table. The section headers and non-dynamic symbol table are just there for orthogonality and to make the debugger's life a little easier (and the debugger would much prefer DWARF data and doesn't care about the section headers, tbh)
- arp242 5y ago
  > Look you can claim that "having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space" but without clear evidence of how much space that is, with proper accounting of byte usage, this claim is non-falsifiable and thus of doubtful value. The article made the claim that 70% of space is wasted "dark bytes". The article should prove the claim, which it did not. It's an extraordinary claim that really requires more evidence than just an off-hand napkin calculation and talk about mysterious "dark bytes". It takes very little time to write up something that's wrong. It takes much more time to write a detailed rebuttal. What you're doing here is pretty much the same trick quacks, young-earth creationists, and purveyors of all sorts of pseudo-scientific claims pull whenever they're challenged. Any fool can claim the earth is 6,000 years old. Proving that it's actually several billions years old requires deep insight in multiple branches of science. People stopped doing this after a while as it's so much effort and pointless as they're not listening anyway, so now they pull the "aha, you didn't falsify my claim about this or that bullshit I pulled out of my ass therefore I am right" zinger and think they look Very Smart And Scientific™. But that's not how it works. Also just outright disbelieving people like this is rather rude. You're coming off really badly here and your comment has the strong implication that Russ is outright lying. Yikes!
knz42 5y ago
> The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint [...] yes this is acknowledged in the OP > We (the Go team) did not “recompress” pclntab in Go 1.15. There's now less redundancy in the funcdata, so in my book less redundancy = more compression. > We did not remove pclntab in Go 1.16. Correct; it is not "removed"; instead the advertised size of the symbol has been reduced to zero. Maybe the data is still there, but it's not accounted any more. Changed in the text. (The correction is already present in the original version of the analysis, and the cockroach labs copy should be updated soon) > we never claimed “pclntab has been reduced to zero”, which is presented in the article as if a direct quote. Correct, there was indeed no such claim. Removed the quotes and rephrased that paragraph. > If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not. 1) the phrase "non-useful" was a mistake. There is no definite proof it is non-useful, as you pointed out. Corrected in the text. 2) see discussion below - the demonstration is more complicated than that, as removing 100 bytes where just 1 byte is necessary will break the executable in the same way as removing 100 necessary bytes. I think the proper next step here is to acknowledge that the problem is not "usefulness" but rather accountability. > The big table seems to claim that a 40 MB Go 1.8 binary has grown to a 289 MB Go 1.16 binary. That’s certainly not the case. More is changing from line to line in that table than the Go version. Correct. Added a note to emphasize this fact. > Overall, the claim of “dark bytes” or “non-useful bytes” strikes me as similar to the claims of “junk DNA”. They’re not dark or non-useful. Let's forget about "non-useful", this was a mistake and will be corrected. The word 'dark' is still relevant however. The adequate comparison is not "junk DNA", but instead "dark silicon": https://ieeexplore.ieee.org/abstract/document/6307773 https://ieeexplore.ieee.org/abstract/document/6307773 We're talking about a general % of executable size that's necessary for a smaller % of use cases in program function. I'm all for trade-offs, but IMHO they should be transparent.
- returningfory2 5y ago
  In this case, what is the point the blog post is trying to make? The title of the post is "Go Executable Files Are Still Getting Larger". Upon further reading and conversation here it seems this is possibly not true, nor what the post is about. If we believe Russ's comments, Go executable sizes haven't increased much in general. Perhaps the reason you're seeing increases in Cockroach DB is because you keep writing more code for Cockroach DB? Now the point has shifted to this notion of "dark bytes". So the article is about ... how the way you previously diagnosed the contents of binaries doesn't work anymore? That's fine and legitimate, but it seems like the point is over-extrapolated to become a criticism of the Go team.
rsc 5y ago
Another thing I noticed in the revised blog post on a second skim, regarding this claim: > Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. That 70% includes the ELF/DWARF metadata that is easily removed from the binary using strip. It's true that the DWARF info in particular has gotten larger in recent releases, because we've included more information to make debuggers work better. I don't think it has grown quite as rapidly as the table indicates - I think some of the rows already have some of the debug metadata removed in "Raw size". Regardless, I would hope that anyone sensitive to networking costs at this level would be shipping around stripped binaries, so the growth in accurate DWARF info should not be relevant to this post at all. That is, the right comparison is to the "Stripped" column in the big table. If you subtract out the strippable overheads and you take the "Dark + pclntab" as an accurate representation of Go-specific overhead (debatable but not today), then the situation has actually improved markedly since Go 1.12, which would have been current in April 2019 when the first post was written. Whereas in Go 1.12 the measured "actual program" was only about 40% of the stripped binary, in Go 1.16 that fraction has risen to closer to 55%. This is a marked-up copy of the table from the dr-knz.net revised post that at time of writing has not yet made it to cockroachlabs.com: https://swtch.com/tmp/cockroach-blog.png https://swtch.com/tmp/cockroach-blog.png I think the numbers in the table may be suspect in other ways, so I am NOT claiming that from Go 1.12 to Go 1.16 there has actually been a 15% reduction in "Go metadata overhead". I honestly don't know one way or the other without spending a lot more time looking into this. But supposing we accept for sake of argument that the byte counts in the table are valid, they do not support the text or the title of the post. In fact, they tell the opposite story: the stripped CockroachDB binary in question has gotten smaller since April 2019, and less of the binary is occupied by what the post calls "non-useful" or "Go-internal" bytes.
- knz42 5y ago
  Thanks Russ for that additional insight. > I would hope that anyone sensitive to networking costs at this level would be shipping around stripped binaries, so the growth in accurate DWARF info should not be relevant to this post at all. Good point. I removed that part from the conclusion. > If you subtract out the strippable overheads and you take the "Dark + pclntab" as an accurate representation of Go-specific overhead [...] then the situation has actually improved markedly since Go 1.12 [...] Whereas in Go 1.12 the measured "actual program" was only about 40% of the stripped binary, in Go 1.16 that fraction has risen to closer to 55%. Ok, that is fair. I will attempt to produce a new version of these tables with this clarification. > the stripped CockroachDB binary in question has gotten smaller since April 2019, and less of the binary is occupied by what the post calls "non-useful" or "Go-internal" bytes. There's an explanation for that, which is that the crdb code was also reduced in that time frame.
hatuio 5y ago
In env without swap, the binary size should/might block relative amount of ram. Might it be possible to stream binaries or to detect junks which could be unloaded like an json parser which is only needed when reading json
- remram 5y ago
  Without swap, you mean without virtual memory or without a swap partition/file? Because even without a swap partition/file, the whole executable will not block physical memory, but will page in/out as needed. And whole sections of it will never be loaded at all.
  - hatuio 5y ago
    Without swap. You have to disable it for k8s
    - remram 5y ago
      So you did mean partition/file, and you are misinformed.
      - hatuio 5y ago
        So it is getting loaded as file and can be swapped out due to this?
        remram 5y ago
        Yes, any mmap'd file will be swapped in on demand, when a page is first accessed, it will not all be copied in physical memory at once. In case of memory pressure, pages will be removed from physical memory ("swapped out"), since they can be loaded back from the file again when needed. Since the executable is mapped read-only, the pages loaded in physical memory can also be shared between multiple instances of the process.
johnisgood 5y ago
Yeah, I do not understand those numbers. I just compiled one of my Go projects with version 1.13.8 and version 1.16.3. The size was 8.3 MB (6.5 MB after stripping), and 8.0 MB (5.8 MB after stripping) respectively.
rsc 5y ago
The article has been updated to avoid calling the bytes "not useful" and to address the narrow, specific points I raised, but it is still generally suspect. Over on Lobsters, zeebo took the time (thanks!) to build the same version of CockroachDB with various historical Go toolchains, varying only the toolchain, and found that if anything the Go executables are getting smaller over time, in some cases significantly so. v1.0 v20.2.0 1.8 58,099,688 n/a 1.9 57,897,616 314,191,032 1.10 57,722,520 313,669,616 1.11 48,961,712 233,170,304 1.12 52,440,168 236,192,600 1.13 50,844,048 214,373,144 1.14 50,527,320 212,699,656 1.15 47,910,360 201,391,416 https://lobste.rs/s/gvtstv/my_go_executable_files_are_still_getting#c_dbfije https://lobste.rs/s/gvtstv/my_go_executable_files_are_still_... The title of the article ("My Go executable files are still getting larger") appears to be literally untrue, at least read as a critique of Go itself. If they are getting larger, it's because new code is being added, not because the Go runtime or compiler is degrading in some way over time.
- knz42 5y ago
  > The title of the article ("My Go executable files are still getting larger") appears to be literally untrue, at least read as a critique of Go itself. If they are getting larger, it's because new code is being added, not because the Go runtime or compiler is degrading in some way over time. Yes this is a fair assessment, although I find it surprising (and enlightening) that you refer to “a critique of Go”. At no moment was the intent to critique Go specifically; the entire analysis is made of observation of the results of combining Go with specific (and varying) amounts of source code. In any case, based on this discussion I have decided to amend the title and emphasize in the conclusion that the absolute size of the code+data for a fixed amount of source code has decreased between go 1.15 and 1.16. edit: This is relevant to this discussion: https://sneak.berlin/20191201/american-communication/ https://sneak.berlin/20191201/american-communication/
  - rsc 5y ago
    In addition to the title (why mention "Go" if it was not a critique of Go?), you also wrote in bold: - "These Go executable files are rather... bloated." - "70% of a couple hundred megabytes are copied around for no good reason" I find it hard to believe that even a European software team would not consider those direct criticisms.
josvazg-yc 5y ago
Indeed. You do NOT write an article talking about "dark bytes" and a 70% "not useful" piece of binary for an Open Source project. Specially one such as Golang where you can easily reach out to the creators at https://groups.google.com/g/golang-dev https://groups.google.com/g/golang-dev to ask for help understanding what your experiment tools can't explain... Eg. """ Hey guys I am doing some digging with "such and such" tools and found that 70% of my binary size has no explanation. BTW this binary is Cockroach DB, an well known program in the Go ecosystem. How can I find out what are those bytes for? Is there any documentation about it, other than the source code itself? Or maybe some tooling that explains it? Or some other set of tools I should be using instead or on top of my current setup? """ I mean... there is no defense possible after writing such an article IMHO. Why didn't you ask first? Having said that, when you deviate from the ubiquitous C standards for calling conventions and binary format layout, that comes with a price. I totally get that Go had to take such path, because Go binaries embed a non trivial runtime dealing, at the very least, with goroutines & GC support. But the price is there nonetheless: As you are not using a standard, you cannot rely on pre-existing tooling or documentation. You have to write your own. It is understandable that writing such binary dissection documentation or tooling (or both) might not be a top priority compared to other more pressing matters for the Go Language. But this kind of thing becomes technical debt and as such, the longer you take to pay it, the more expensive it becomes. For instance: - This happened, and might have damaged the Go Language reputation, at least temporarily for some people. - I am surprised how could the Go team (or community) achieve improvements at all in the binary size work without tools to properly dissect binaries or detailed documentation. I am guessing it relied on the original programmers of those bits to have a very clear idea of the binary layout uploaded in their brains, so they did not need to re-grok all the sources again... with time that becomes more and more difficult, people forget stuff, move to other projects, etc. I think it would be wise to prioritize this somehow. If the Go team cannot take it for now, maybe they can promote it as a good Go community project to be done ASAP, and promise to help the takers with all info and reviews needed. Or maybe the article writers should take up the challenge, to fully redeem themselves ;-p

by rsc 5y ago

rsc 5y ago

An easily obtained apples-to-apples¹ table: $ for i in $(seq 3 16); do curl -sLo go1.$i.tgz https://golang.org/dl/go1.$i.linux-amd64.tar.gz tar xzf go1.$i.tgz go/bin/gofmt size=$(ls -l go/bin/gofmt | awk '{print $5}') strip go/bin/gofmt size2=$(ls -l go/bin/gofmt | awk '{print $5}') echo go1.$i $size $size2 done go1.3 3496520 2528664 go1.4² 14398336 13139184 go1.5 3937888 2765696 go1.6 3894568 2725376 go1.7 3036195 1913704 go1.8 3481554 2326760 go1.9 3257829 2190792 go1.10 3477807 2166536 go1.11 3369391 2441288 go1.12 3513529 2506632 go1.13 3543823 2552632 go1.14 3587746 2561208 go1.15 3501176 2432248 go1.16 3448663 2443736 $ Size fluctuates from release to release, but the overall trendline is flat: Go 1.16 binaries are roughly where Go 1.3 binaries were. At the moment, it looks like Go 1.17 binaries will get a bit smaller thanks to the new register ABI making executable code smaller (and faster). ¹ Well, not completely. The gofmt code itself was changing from release to release, but not much. Most of the binary is the libraries and runtime, though, so it's still accurate for trends. ² Turns out we shipped the go 1.4 gofmt binary built with the race detector enabled! Oops.

knz42 5y ago
Strip removes the symbol tables and DWARF information. But still, the sum of all the bytes advertised in symbol tables for the non-DWARF data does not sum up to the stripped size. What's the remainder about? I am reminded of how early versions of MSWord were embedding pages of heap space in save files that were not relevant to the document being saved, just because it made the saving algorithm simpler. For all we know, the go linker could be embedding random data.
- rsc 5y ago
  > For all we know, the go linker could be embedding random data. I do know, and it is not.
  - knz42 5y ago
    > But still, the sum of all the bytes advertised in symbol tables for the non-DWARF data does not sum up to the stripped size. What's the remainder about? if you do know, then pray, what is the answer to this question?
    - rsc 5y ago
      Maybe another day I will take the time to write a full-length blog post examining the bytes in a Go binary. Today I have other work planned and still intend to do it. My points today are only that: 1. Go binary size has not gotten dramatically better or worse in any particular release and is mostly unchanged since Go 1.3. 2. Many claimed facts in the blog post are incorrect. 3. The linker is not "embedding random data" into Go binaries as you conjectured in the comment above. Stepping back a level, you don't seem to be interested in engaging in good faith at all. I'm not going to reply to any more of your comments.
      - hobbified 5y ago
        It's a mystery, not a lynch mob. Everyone reading is interested in knowing "huh, what is this stuff then?"

knz42 5y ago

As explained in another comment below, it's a good thing when all the byte usage is represented in the ELF section headers or the symbol tables. The DWARF data is currently so represented, and so was pclntab prior to 1.16. Today, the DWARF data is still present; the symbol for pclntab has an advertized size of zero, and the amount of data not represented in the symbol table and ELF section headers has grown. > If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not. Probably not if all of it was removed at once. It could be that just 5% of it is useful and removing all of it would produce a non-working binary. What does the experiment prove exactly? Look you can claim that "having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space" but without clear evidence of how much space that is, with proper accounting of byte usage, this claim is non-falsifiable and thus of doubtful value.

rsc 5y ago
> Look you can claim that "having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space" but without clear evidence of how much space that is, with proper accounting of byte usage, this claim is non-falsifiable and thus of doubtful value. I respectfully disagree. I believe there is value in pointing out the false claims in a long blog post even when there isn't time to write a full-length rebuttal.
erincandescent 5y ago
> As explained in another comment below, it's a good thing when all the byte usage is represented in the ELF section headers or the symbol tables. It's an ELF binary; all that's relevant are the program/segment headers and dynamic table. The section headers and non-dynamic symbol table are just there for orthogonality and to make the debugger's life a little easier (and the debugger would much prefer DWARF data and doesn't care about the section headers, tbh)
arp242 5y ago
> Look you can claim that "having the necessary metadata for garbage collection and reflection in a statically-compiled language takes up a significant amount of space" but without clear evidence of how much space that is, with proper accounting of byte usage, this claim is non-falsifiable and thus of doubtful value. The article made the claim that 70% of space is wasted "dark bytes". The article should prove the claim, which it did not. It's an extraordinary claim that really requires more evidence than just an off-hand napkin calculation and talk about mysterious "dark bytes". It takes very little time to write up something that's wrong. It takes much more time to write a detailed rebuttal. What you're doing here is pretty much the same trick quacks, young-earth creationists, and purveyors of all sorts of pseudo-scientific claims pull whenever they're challenged. Any fool can claim the earth is 6,000 years old. Proving that it's actually several billions years old requires deep insight in multiple branches of science. People stopped doing this after a while as it's so much effort and pointless as they're not listening anyway, so now they pull the "aha, you didn't falsify my claim about this or that bullshit I pulled out of my ass therefore I am right" zinger and think they look Very Smart And Scientific™. But that's not how it works. Also just outright disbelieving people like this is rather rude. You're coming off really badly here and your comment has the strong implication that Russ is outright lying. Yikes!

knz42 5y ago

> The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint [...] yes this is acknowledged in the OP > We (the Go team) did not “recompress” pclntab in Go 1.15. There's now less redundancy in the funcdata, so in my book less redundancy = more compression. > We did not remove pclntab in Go 1.16. Correct; it is not "removed"; instead the advertised size of the symbol has been reduced to zero. Maybe the data is still there, but it's not accounted any more. Changed in the text. (The correction is already present in the original version of the analysis, and the cockroach labs copy should be updated soon) > we never claimed “pclntab has been reduced to zero”, which is presented in the article as if a direct quote. Correct, there was indeed no such claim. Removed the quotes and rephrased that paragraph. > If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not. 1) the phrase "non-useful" was a mistake. There is no definite proof it is non-useful, as you pointed out. Corrected in the text. 2) see discussion below - the demonstration is more complicated than that, as removing 100 bytes where just 1 byte is necessary will break the executable in the same way as removing 100 necessary bytes. I think the proper next step here is to acknowledge that the problem is not "usefulness" but rather accountability. > The big table seems to claim that a 40 MB Go 1.8 binary has grown to a 289 MB Go 1.16 binary. That’s certainly not the case. More is changing from line to line in that table than the Go version. Correct. Added a note to emphasize this fact. > Overall, the claim of “dark bytes” or “non-useful bytes” strikes me as similar to the claims of “junk DNA”. They’re not dark or non-useful. Let's forget about "non-useful", this was a mistake and will be corrected. The word 'dark' is still relevant however. The adequate comparison is not "junk DNA", but instead "dark silicon": https://ieeexplore.ieee.org/abstract/document/6307773 https://ieeexplore.ieee.org/abstract/document/6307773 We're talking about a general % of executable size that's necessary for a smaller % of use cases in program function. I'm all for trade-offs, but IMHO they should be transparent.

returningfory2 5y ago
In this case, what is the point the blog post is trying to make? The title of the post is "Go Executable Files Are Still Getting Larger". Upon further reading and conversation here it seems this is possibly not true, nor what the post is about. If we believe Russ's comments, Go executable sizes haven't increased much in general. Perhaps the reason you're seeing increases in Cockroach DB is because you keep writing more code for Cockroach DB? Now the point has shifted to this notion of "dark bytes". So the article is about ... how the way you previously diagnosed the contents of binaries doesn't work anymore? That's fine and legitimate, but it seems like the point is over-extrapolated to become a criticism of the Go team.

rsc 5y ago

Another thing I noticed in the revised blog post on a second skim, regarding this claim: > Every time, 70% of a couple hundred megabytes are copied around for no good reason and someone needs to pay ingress/egress networking costs for these file copies. That 70% includes the ELF/DWARF metadata that is easily removed from the binary using strip. It's true that the DWARF info in particular has gotten larger in recent releases, because we've included more information to make debuggers work better. I don't think it has grown quite as rapidly as the table indicates - I think some of the rows already have some of the debug metadata removed in "Raw size". Regardless, I would hope that anyone sensitive to networking costs at this level would be shipping around stripped binaries, so the growth in accurate DWARF info should not be relevant to this post at all. That is, the right comparison is to the "Stripped" column in the big table. If you subtract out the strippable overheads and you take the "Dark + pclntab" as an accurate representation of Go-specific overhead (debatable but not today), then the situation has actually improved markedly since Go 1.12, which would have been current in April 2019 when the first post was written. Whereas in Go 1.12 the measured "actual program" was only about 40% of the stripped binary, in Go 1.16 that fraction has risen to closer to 55%. This is a marked-up copy of the table from the dr-knz.net revised post that at time of writing has not yet made it to cockroachlabs.com: https://swtch.com/tmp/cockroach-blog.png https://swtch.com/tmp/cockroach-blog.png I think the numbers in the table may be suspect in other ways, so I am NOT claiming that from Go 1.12 to Go 1.16 there has actually been a 15% reduction in "Go metadata overhead". I honestly don't know one way or the other without spending a lot more time looking into this. But supposing we accept for sake of argument that the byte counts in the table are valid, they do not support the text or the title of the post. In fact, they tell the opposite story: the stripped CockroachDB binary in question has gotten smaller since April 2019, and less of the binary is occupied by what the post calls "non-useful" or "Go-internal" bytes.

knz42 5y ago
Thanks Russ for that additional insight. > I would hope that anyone sensitive to networking costs at this level would be shipping around stripped binaries, so the growth in accurate DWARF info should not be relevant to this post at all. Good point. I removed that part from the conclusion. > If you subtract out the strippable overheads and you take the "Dark + pclntab" as an accurate representation of Go-specific overhead [...] then the situation has actually improved markedly since Go 1.12 [...] Whereas in Go 1.12 the measured "actual program" was only about 40% of the stripped binary, in Go 1.16 that fraction has risen to closer to 55%. Ok, that is fair. I will attempt to produce a new version of these tables with this clarification. > the stripped CockroachDB binary in question has gotten smaller since April 2019, and less of the binary is occupied by what the post calls "non-useful" or "Go-internal" bytes. There's an explanation for that, which is that the crdb code was also reduced in that time frame.

hatuio 5y ago

In env without swap, the binary size should/might block relative amount of ram. Might it be possible to stream binaries or to detect junks which could be unloaded like an json parser which is only needed when reading json

remram 5y ago
Without swap, you mean without virtual memory or without a swap partition/file? Because even without a swap partition/file, the whole executable will not block physical memory, but will page in/out as needed. And whole sections of it will never be loaded at all.
- hatuio 5y ago
  Without swap. You have to disable it for k8s
  - remram 5y ago
    So you did mean partition/file, and you are misinformed.
    - hatuio 5y ago
      So it is getting loaded as file and can be swapped out due to this?
      - remram 5y ago
        Yes, any mmap'd file will be swapped in on demand, when a page is first accessed, it will not all be copied in physical memory at once. In case of memory pressure, pages will be removed from physical memory ("swapped out"), since they can be loaded back from the file again when needed. Since the executable is mapped read-only, the pages loaded in physical memory can also be shared between multiple instances of the process.

johnisgood 5y ago

Yeah, I do not understand those numbers. I just compiled one of my Go projects with version 1.13.8 and version 1.16.3. The size was 8.3 MB (6.5 MB after stripping), and 8.0 MB (5.8 MB after stripping) respectively.

rsc 5y ago

The article has been updated to avoid calling the bytes "not useful" and to address the narrow, specific points I raised, but it is still generally suspect. Over on Lobsters, zeebo took the time (thanks!) to build the same version of CockroachDB with various historical Go toolchains, varying only the toolchain, and found that if anything the Go executables are getting smaller over time, in some cases significantly so. v1.0 v20.2.0 1.8 58,099,688 n/a 1.9 57,897,616 314,191,032 1.10 57,722,520 313,669,616 1.11 48,961,712 233,170,304 1.12 52,440,168 236,192,600 1.13 50,844,048 214,373,144 1.14 50,527,320 212,699,656 1.15 47,910,360 201,391,416 https://lobste.rs/s/gvtstv/my_go_executable_files_are_still_getting#c_dbfije https://lobste.rs/s/gvtstv/my_go_executable_files_are_still_... The title of the article ("My Go executable files are still getting larger") appears to be literally untrue, at least read as a critique of Go itself. If they are getting larger, it's because new code is being added, not because the Go runtime or compiler is degrading in some way over time.

knz42 5y ago
> The title of the article ("My Go executable files are still getting larger") appears to be literally untrue, at least read as a critique of Go itself. If they are getting larger, it's because new code is being added, not because the Go runtime or compiler is degrading in some way over time. Yes this is a fair assessment, although I find it surprising (and enlightening) that you refer to “a critique of Go”. At no moment was the intent to critique Go specifically; the entire analysis is made of observation of the results of combining Go with specific (and varying) amounts of source code. In any case, based on this discussion I have decided to amend the title and emphasize in the conclusion that the absolute size of the code+data for a fixed amount of source code has decreased between go 1.15 and 1.16. edit: This is relevant to this discussion: https://sneak.berlin/20191201/american-communication/ https://sneak.berlin/20191201/american-communication/
- rsc 5y ago
  In addition to the title (why mention "Go" if it was not a critique of Go?), you also wrote in bold: - "These Go executable files are rather... bloated." - "70% of a couple hundred megabytes are copied around for no good reason" I find it hard to believe that even a European software team would not consider those direct criticisms.

josvazg-yc 5y ago

Indeed. You do NOT write an article talking about "dark bytes" and a 70% "not useful" piece of binary for an Open Source project. Specially one such as Golang where you can easily reach out to the creators at https://groups.google.com/g/golang-dev https://groups.google.com/g/golang-dev to ask for help understanding what your experiment tools can't explain... Eg. """ Hey guys I am doing some digging with "such and such" tools and found that 70% of my binary size has no explanation. BTW this binary is Cockroach DB, an well known program in the Go ecosystem. How can I find out what are those bytes for? Is there any documentation about it, other than the source code itself? Or maybe some tooling that explains it? Or some other set of tools I should be using instead or on top of my current setup? """ I mean... there is no defense possible after writing such an article IMHO. Why didn't you ask first? Having said that, when you deviate from the ubiquitous C standards for calling conventions and binary format layout, that comes with a price. I totally get that Go had to take such path, because Go binaries embed a non trivial runtime dealing, at the very least, with goroutines & GC support. But the price is there nonetheless: As you are not using a standard, you cannot rely on pre-existing tooling or documentation. You have to write your own. It is understandable that writing such binary dissection documentation or tooling (or both) might not be a top priority compared to other more pressing matters for the Go Language. But this kind of thing becomes technical debt and as such, the longer you take to pay it, the more expensive it becomes. For instance: - This happened, and might have damaged the Go Language reputation, at least temporarily for some people. - I am surprised how could the Go team (or community) achieve improvements at all in the binary size work without tools to properly dissect binaries or detailed documentation. I am guessing it relied on the original programmers of those bits to have a very clear idea of the binary layout uploaded in their brains, so they did not need to re-grok all the sources again... with time that becomes more and more difficult, people forget stuff, move to other projects, etc. I think it would be wise to prioritize this somehow. If the Go team cannot take it for now, maybe they can promote it as a good Go community project to be done ASAP, and promise to help the takers with all info and reviews needed. Or maybe the article writers should take up the challenge, to fully redeem themselves ;-p