6 ms·

> The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint [...] yes this is acknowledged in the OP > We (the Go tea

by knz42 5y ago

> The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint [...] yes this is acknowledged in the OP > We (the Go team) did not “recompress” pclntab in Go 1.15. There's now less redundancy in the funcdata, so in my book less redundancy = more compression. > We did not remove pclntab in Go 1.16. Correct; it is not "removed"; instead the advertised size of the symbol has been reduced to zero. Maybe the data is still there, but it's not accounted any more. Changed in the text. (The correction is already present in the original version of the analysis, and the cockroach labs copy should be updated soon) > we never claimed “pclntab has been reduced to zero”, which is presented in the article as if a direct quote. Correct, there was indeed no such claim. Removed the quotes and rephrased that paragraph. > If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not. 1) the phrase "non-useful" was a mistake. There is no definite proof it is non-useful, as you pointed out. Corrected in the text. 2) see discussion below - the demonstration is more complicated than that, as removing 100 bytes where just 1 byte is necessary will break the executable in the same way as removing 100 necessary bytes. I think the proper next step here is to acknowledge that the problem is not "usefulness" but rather accountability. > The big table seems to claim that a 40 MB Go 1.8 binary has grown to a 289 MB Go 1.16 binary. That’s certainly not the case. More is changing from line to line in that table than the Go version. Correct. Added a note to emphasize this fact. > Overall, the claim of “dark bytes” or “non-useful bytes” strikes me as similar to the claims of “junk DNA”. They’re not dark or non-useful. Let's forget about "non-useful", this was a mistake and will be corrected. The word 'dark' is still relevant however. The adequate comparison is not "junk DNA", but instead "dark silicon": https://ieeexplore.ieee.org/abstract/document/6307773 https://ieeexplore.ieee.org/abstract/document/6307773 We're talking about a general % of executable size that's necessary for a smaller % of use cases in program function. I'm all for trade-offs, but IMHO they should be transparent.

returningfory2 5y ago
In this case, what is the point the blog post is trying to make? The title of the post is "Go Executable Files Are Still Getting Larger". Upon further reading and conversation here it seems this is possibly not true, nor what the post is about. If we believe Russ's comments, Go executable sizes haven't increased much in general. Perhaps the reason you're seeing increases in Cockroach DB is because you keep writing more code for Cockroach DB? Now the point has shifted to this notion of "dark bytes". So the article is about ... how the way you previously diagnosed the contents of binaries doesn't work anymore? That's fine and legitimate, but it seems like the point is over-extrapolated to become a criticism of the Go team.
- knz42 5y ago
  > Go executable sizes haven't increased much in general. Russ's example was just the "gofmt" program. > Perhaps the reason you're seeing increases in Cockroach DB is because you keep writing more code for Cockroach DB? If that was the only reason, then the % overhead would remain constant-ish. But it is increasing. So there is a non-linear factor for _some_ go programs (like cockroachdb) and it's still unclear what that factor is.
  - mort96 5y ago
    It's not clear that the overhead is due to Go itself producing bigger binaries over time though. If you recompiled all the different CockroachDB versions with Go 1.8 (if that was feasible), it's quite probable that the tables you would end up with would look fairly similar to the ones you're actually showing. If there is superlinear growth in binary sizes as the project grows – for example, if some part is O(n^2) in the number of interfaces – then that's certainly interesting. If you demonstrated that such superlinear growth is happening, and wrote an article based on that, people wouldn't be so critical. If Go binaries are getting bigger because Go produces bigger binaries for the same source code over time, then that's also interesting. If you demonstrated that Go binaries are getting more and more bloated over time for the same source code, and wrote an article based on that, people wouldn't be critical. But as it is, you kind of just complained that CockroachDB is getting bigger, tried to blame it partly on the Go compiler producing more bloated code over time, partly on a mystical "dark area" which you don't understand, you mentioned superlinear growth only in the comment section, and you didn't actually gather data or do experiments to prove or disprove any of the things you're claiming as a cause. That's why people are complaining.
    - knz42 5y ago
      > tried to blame it partly on the Go compiler producing more bloated code over time Where? The argument is _precisely_ that the growth is occurring in non-code areas. > partly on a mystical "dark area" which you don't understand The _observation_ is that the growth is happening in an area of the file that's not accounted for in the symtable. That's what makes it "dark". It's not mythical: it's _there_ and you can observe it just as well as anyone else. > you mentioned superlinear growth only in the comment section it's in the reported measurements. > and you didn't actually gather data or do experiments to prove or disprove any of the things you're claiming as a cause The analysis is stating observations and reporting that the size is increasingly due to non-accounted data. That observation is substantiated by measurements. There's no claim of cause in the text!
mort96 5y ago
The term "dark silicon" refers to silicon that's not (currently) in use, such as when you have a workload which only exercises one arithmetic unit of one core even if you have 8 cores with 4 arithmetic units each (only one arithmetic unit is "lit up", 63 are "dark"). There's no reason to believe that what you're calling "dark bytes" isn't actively used while executing the program.
- knz42 5y ago
  Dark silicon is not used all the time - that's the key point. In the same way, the functab data in Go is not used all the time either, only when generating stack traces. Also since that original article from 2011 was published, the phrase "dark silicon" was extended to designate silicon IP which is not directly necessary for a given target application but is embedded for the occasional use by a niche in the same market. In the same way, all languages (not just Go) embed constructs in generated code that are there only to handle edge cases in certain applications/deployments, and are inactive for 90%+ of executions.

by knz42 5y ago

returningfory2 5y ago

In this case, what is the point the blog post is trying to make? The title of the post is "Go Executable Files Are Still Getting Larger". Upon further reading and conversation here it seems this is possibly not true, nor what the post is about. If we believe Russ's comments, Go executable sizes haven't increased much in general. Perhaps the reason you're seeing increases in Cockroach DB is because you keep writing more code for Cockroach DB? Now the point has shifted to this notion of "dark bytes". So the article is about ... how the way you previously diagnosed the contents of binaries doesn't work anymore? That's fine and legitimate, but it seems like the point is over-extrapolated to become a criticism of the Go team.

knz42 5y ago
> Go executable sizes haven't increased much in general. Russ's example was just the "gofmt" program. > Perhaps the reason you're seeing increases in Cockroach DB is because you keep writing more code for Cockroach DB? If that was the only reason, then the % overhead would remain constant-ish. But it is increasing. So there is a non-linear factor for _some_ go programs (like cockroachdb) and it's still unclear what that factor is.
- mort96 5y ago
  It's not clear that the overhead is due to Go itself producing bigger binaries over time though. If you recompiled all the different CockroachDB versions with Go 1.8 (if that was feasible), it's quite probable that the tables you would end up with would look fairly similar to the ones you're actually showing. If there is superlinear growth in binary sizes as the project grows – for example, if some part is O(n^2) in the number of interfaces – then that's certainly interesting. If you demonstrated that such superlinear growth is happening, and wrote an article based on that, people wouldn't be so critical. If Go binaries are getting bigger because Go produces bigger binaries for the same source code over time, then that's also interesting. If you demonstrated that Go binaries are getting more and more bloated over time for the same source code, and wrote an article based on that, people wouldn't be critical. But as it is, you kind of just complained that CockroachDB is getting bigger, tried to blame it partly on the Go compiler producing more bloated code over time, partly on a mystical "dark area" which you don't understand, you mentioned superlinear growth only in the comment section, and you didn't actually gather data or do experiments to prove or disprove any of the things you're claiming as a cause. That's why people are complaining.
  - knz42 5y ago
    > tried to blame it partly on the Go compiler producing more bloated code over time Where? The argument is _precisely_ that the growth is occurring in non-code areas. > partly on a mystical "dark area" which you don't understand The _observation_ is that the growth is happening in an area of the file that's not accounted for in the symtable. That's what makes it "dark". It's not mythical: it's _there_ and you can observe it just as well as anyone else. > you mentioned superlinear growth only in the comment section it's in the reported measurements. > and you didn't actually gather data or do experiments to prove or disprove any of the things you're claiming as a cause The analysis is stating observations and reporting that the size is increasingly due to non-accounted data. That observation is substantiated by measurements. There's no claim of cause in the text!

mort96 5y ago

The term "dark silicon" refers to silicon that's not (currently) in use, such as when you have a workload which only exercises one arithmetic unit of one core even if you have 8 cores with 4 arithmetic units each (only one arithmetic unit is "lit up", 63 are "dark"). There's no reason to believe that what you're calling "dark bytes" isn't actively used while executing the program.

knz42 5y ago
Dark silicon is not used all the time - that's the key point. In the same way, the functab data in Go is not used all the time either, only when generating stack traces. Also since that original article from 2011 was published, the phrase "dark silicon" was extended to designate silicon IP which is not directly necessary for a given target application but is embedded for the occasional use by a niche in the same market. In the same way, all languages (not just Go) embed constructs in generated code that are there only to handle edge cases in certain applications/deployments, and are inactive for 90%+ of executions.