11 ms·
Go has added Valgrind support
- defraudbah 1y agolooks very promising, one of the biggest issue in golang for me is profiling and constant memory leaks/pressure. Not sure if there is an alternative of what people use now
- felixge 1y agoI'd love to hear more! What kind of profiling issues are you running into? I'm assuming the inuse memory profiles are sometimes not good enough to track down leaks since they only show the allocation stack traces? Have you tried goref [1]?. What kind of memory pressure issues are you dealing with? [1] https://github.com/cloudwego/goref https://github.com/cloudwego/goref Disclaimer: I work on continuous profiling for Datadog and contribute to the profiling features in the runtime.
- tsimionescu 1y agoI for one am still mystified how it's possible that a GC language can't expose the GC roots in a memory profile. I've lost so many hours of my life manually trying to figure out what might be keeping some objects live, information the GC figures out every single time it runs...
- felixge 1y agoDo you think the GC roots alone (goroutine stacks with goroutine id, package globals) would be enough? I think in many cases you'd want the reference chains. The GC could certainly keep track of those, but at the expense of making things slower. My colleagues Nick and Daniel prototyped this at some point [1]. Alternatively the tracing of reference chains can be done on heap dumps, but it requires maintaining a partial replica of the GC in user space, see goref [2] for that approach. So it's not entirely trivial, but rest assured that it's definitely being considered by the Go project. You can see some discussions related to it here [3]. Disclaimer: I contribute to the Go runtime as part of my job at Datadog. I can't speak on behalf of the Go team. [1] https://go-review.googlesource.com/c/go/+/552736 https://go-review.googlesource.com/c/go/+/552736 [2] https://github.com/cloudwego/goref/blob/main/docs/principle.md https://github.com/cloudwego/goref/blob/main/docs/principle.... [3] https://github.com/golang/go/issues/57175 https://github.com/golang/go/issues/57175
- defraudbah 1y agono, haven't heard of goref yet but will give it a shot! usually I go with pprof, like basic stuff and it helps. I would NOT say memory leak is the biggest or most common issue I see, however as time goes and services become more complicated what I often see in the metrics is how RAM gets eaten and does not get freed as time goes, so the app eats more and more memory as time goes and only restart helps. It's hard to call it memory leak in "original meaning of memory leak" but the memory does not get cleaned up because the choices I made and I want to understand how to make it better. Thanks for the tool!
- prerok 1y agoSorry if this is a basic question but are you setting the GOMEMLIMIT? Also, are you running the code in a container? In K8s?
- pjmlp 1y agoIdeally, they would have learnt from other languages, and offered explicit control over what goes into the stack instead of relying into escape analysis alone. As it is, the only way to currently handle that is with " -gcflags -m=3" or using something like VSCode Go plugin, via "ui.codelenses" and "ui.diagnostic.annotations" configurations.
- johncolanduoni 1y agoI sometimes dream of a GCed language with a non-escaping pointer type. However to make it really useful (i.e. let you put it inside other non-escaping structs) you need something on the scale of the Rust borrow checker, which means adding a lot of complexity.
- SkySkimmer 1y agohttps://oxcaml.org/documentation/stack-allocation/intro/ https://oxcaml.org/documentation/stack-allocation/intro/ ?
- pjmlp 1y agoWhich is why several GC languages (the CS meaning of GC), are rather going down the path to keeping their approach to automatic resource management, plus type systems improvements for low level high performance code when needed. So you only go down into the complexity of affine types, linear types, effects, formal proofs, dependent types, if really needed, after spending time reasoning with a profiler. Now, this does not need to be so complex, since languages like Interlisp-D and Cedar at Xerox, that many GC languages have offered value types and explicit stack allocation. That alone is already good enough for most scenarios, provided people actually spend some time thinking about how to design their data structures instead of placing everything into the heap.
- Thaxll 1y agopprof is pretty good, what do you need?
- defraudbah 1y agoyes, that's what I use, just wonder if there are alternatives. I am not sure how valgrind compares to it or the goref tool mentioned above, just asking around, does not hurt.
- Thaxll 1y agoAlternative to solve what problem? pprof is very powerful, it's not missing much.
- styluss 1y agoPprof doesn't tell you if something was leaked aka still around. I fixed a leak recently because of misuse of a slice with code like slice = append(slice[1:], newElement) I only figured it out by looking at the pprof heap endpoint output and noticed there were multiple duplicate entries.
- 0x696C6961 1y agoHow are you getting "constant memory leaks" in a GC'd language?
- sim7c00 1y agoit's not hard. GC lets shit leak until it decided to clean it up... do you think they will enable Valgrind if there's no leaks?
- chrsig 1y agovalgrind finds sooooo many more problems than just memory leaks uninitialized memory, illegal writes, etc... There's a lot of good stuff that could be discovered.
- sim7c00 1y agonot to mention cachegrind, callgrind and other things it bundles. sorry, i guess when i say leaks i mean a bit more broad stuff :'). my own words are a bit leaky hah still doesnt mean i am wrong. GC doesnt clean up memory when its released but when it wants to, effectively offering opportunities to get that data after a program dont need it anymore. until some point in time u can usually not specify, just hint at. that in light of things like bad memory ordering between threads etc..can have nasty bugs... (raii has similar bugs but since its more determenistic you can program your way around lot of it more easily and reliably)
- johncolanduoni 1y agoGolang has a feature that I love in general but that makes it very easy to keep unintended allocations around. If you have a struct with a simple int field, and you store that somewhere as an *int, the entire struct and anything it points to will be kept alive. This is super useful for short-lived pointers, and super dangerous for long-lived pointers. Most other widely used GCed languages don’t allow the use of arbitrary interior pointers (though most GCs can actually handle them at the register level).
- 1y ago
- sethammons 1y ago> constant memory leaks/pressure In Go, never launch a goroutine that you don't know exactly how it will be cleaned up.
- amelius 1y agoIt only works if every package tests with it. Otherwise the relevant warnings get swamped by a huge amount by irrelevant warnings. This is why running Valgrind on Python code does not work.
- jzwinck 1y agoIf that were true it would also apply to C and C++. I have used Valgrind with Python + Boost C++ hybrid programs and it worked fine after spending an hour making a suppressions file.
- giancarlostoro 1y ago> it worked fine after spending an hour making a suppressions file. So you are confirming the problem, but treating it as if ignoring it is the solution for all?
- sauercrowd 1y agoit's a rejection of the thesis that it "does not work". It does, but it requires investing into a suppression file.
- pjmlp 1y ago> Instead of adding the Valgrind headers to the tree, and using cgo to call the various Valgrind client request macros, we just add an assembly function which emits the necessary instructions to trigger client requests. Love that they have taken this route, this is the way bootstraped toolchains should be, minimal building blocks and everything else on the language itself.
- giancarlostoro 1y agoI am still curious, had they not gone this route, and avoided the other two routes mentioned, what could they have done to make this process as simple as the rest of Go tends to be, and nearly as performant? I guess this is an ongoing question to be solved at a future date.
- pjmlp 1y agoIt would be another scenario to use as ammunition for "see you can't implement a language toolchain without using C", usually voiced by folks without background in compiler design, and understanding that most of the time that is a decision that spurs out of convenience and nothing else. Assembly isn't that hard, those of us that grown around 8 bit home computers were writing Z80 and 6502 Assembly aged 10 - 12 years old, while having fun cracking games and setting the roots of Demoscene.
- deleted 1y ago[deleted]
- thechao 1y agoOh. There was a comment to your comment saying that kids learning assembly was easy and — I guess? — implying that adults-learning-assembly is hard. I teach adults assembly on an irregular basis. Adults-learning-assembly is hard because adults are rational animals who (correctly) assume I'm an idiot for insisting on assembly. Once I explain the long-term benefits for our exceedingly specific use case, they pick up assembly in a few hours. Assembly isn't hard. Assembly is annoying because it takes absolutely gobsmacking amounts of assembly to do anything.
- 0x696C6961 1y agoThis is only useful for cgo correct?
- kevincox 1y agoI presume it is also useful if you are using the unsafe APIs as well to mess with pointers and do raw memory reads.
- 9rx 1y agoIt's really for crypto. https://news.ycombinator.com/item?id=45348445 https://news.ycombinator.com/item?id=45348445 But maybe others will find a way to use it. Who knows?
- pbd 1y ago[flagged]
- preisschild 1y agoAnd yet it's mature enough to be used for highly critical software such as Kubernetes...
- giancarlostoro 1y agoGood enough to be used by every major cloud provider, every time you download Google Chrome, and Android SDKs you pull from a standard library Go based HTTP server.
- frollogaston 1y agoHow do you know the Chrome downloads server specifically is written in Go?
- c2xlZXB5Cg1 1y agohttps://go.dev/talks/2013/oscon-dl.slide#1 https://go.dev/talks/2013/oscon-dl.slide#1
- worksonmine 1y ago> Next up: maybe Go will discover gdb integration and we can debug like it's 1999 That's already possible and documented[1]. I don't understand if you're sarcastic though, what's wrong with GDB? I use it in vim :termdebug and I wish all languages had native support for it. [1]: https://go.dev/doc/gdb https://go.dev/doc/gdb
- alias_neo 1y agoC is also half a century old. Perhaps Valgrind wasn't a high priority because the newer language also has newer tools? Nothing wrong with adding tried and tested tools later if people want them. Did you have a need for Valgrind in Go that wasn't served by any other tools until now?
- rwmj 1y agoValgrind is a hidden super-power. In much of the software I write, there's 'make check' which runs the test cases, and 'make check-valgrind' that runs the same test cases under valgrind. The latter is only used on developer machines. It often reveals memory leaks or other subtle memory bugs.
- stingraycharles 1y agoSomewhat yes, but as soon as you enter the world of multi-threading (which Go does a lot), the abstraction doesn’t work anymore: as I understand it (or rather, understood: last time I really spent a lot of time digging into it with C++ code was a while ago) it uses its own scheduler, and as such, a lot of subtle real world issues that would arise due to concurrency / race conditions / etc do not pop up in valgrind. And the performance penalty in general is very heavy. Having said that, it saved my ass a lot of times, and I’m very grateful that it exists.
- ben-schaaf 1y agoIME Helgrind does an great job finding concurrency issues.
- cozzyd 1y agoYes though last I tried to use it it sadly didn't support openmp. Maybe that's fixed now (that was a while ago) (I think it was possible to use on openmp if you compiled your compiler with special options)
- hedora 1y agotsan from LLVM works a bit better in my experience. I still like valgrind in general though!
- rwmj 1y agoFor fuzzing we don't use valgrind, but use Clang + ASan instead. All these tools have their niches.
- starboyy 1y agooh man. you came at the right time.
- paulf38 1y agoThat's quite nice. There is a small risk that the client request mechanism might change . The headers don't change much - mostly when a new platform gets added. Go is only targeting amd64 and arm64. This isn't so much about leaks. The most important thing that this will enable is correct analysis of uninitialised memory. Without annotation memory that gets recycled will not be correctly poisoned. I imagine that it will also be useful for the other tools (except cachegrind and callgrind).
- deleted 1y ago[deleted]
- suobset 1y agoNot even a Go user, and yet this is one of the best things I have read today morning. Valgrind is possibly one of the most powerful tools I have in my belt!!
- DarkNova6 1y agoWould you mind to elaborate? I don't program in C but it sounds interesting.
- paulf38 1y agoC is the language that benefits the most from tools like Valgrind. It's just so easy in C to write code with memory faults. Memcheck (the main tool) has shortcomings (very slow, does not detect all kinds of errors). Its strongest point is that it does not need an instrumented build. That can be particularly important if you have issues in 3rd party libraries that you can't build. Its other strong point is that it checks for both addressability and initialisedness at the same time. My favourite feature is using GDB with Valgrind+vgdb. That allows you to see what memory is addressable and/or initialised from within GDB.
- olivia-banks 1y agoI love Valgrind, but since my main development machine is an M3, I don’t get to use it nearly as much as I would like.
- jasonjmcghee 1y agohttps://github.com/LouisBrunner/valgrind-macos https://github.com/LouisBrunner/valgrind-macos
- paulf38 1y agoIf any macOS experts could help with this port it would be most welcome. Apple have been making big changes that keep breaking things and Valgrind has not kept up. Louis Brunner has done an amazing job more or less single handedly managed to keep the basic flow working.
- DishyDev 1y agoVery cool. Should flush out a few bugs. I'd be interested to know why Valgrind vs the Clang AddressSanitizer and MemorySaniziter. These normally find more types of errors (like use-after-return) and I find it significantly faster than Valgrind.
- tasn 1y agoGo doesn't use clang/llvm, so they can't use these tools.
- pjmlp 1y agoTinyGo does, but it is also behind in language support.
- yxhuvud 1y agoValgrind also does stuff like memory tracking and memory-profiling, so this is great also from a performance tracking point of view.
- acidx 1y agoGo has had its own version of msan and asan for years at this point.
- 1718627440 1y agoValgrind is way faster and can be attached to a running program.
- acidx 1y agoPrograms running under any Valgrind tool will be executed using a CPU emulator, making it quite a bit slower than, say, running the instrumented binaries as required by sanitizers; it's often an order of magnitude slower, but could be very well be close to two orders of magnitude slower in some cases. This also means that it just can't be attached to any running program, because, well, it's emulating a whole CPU to track everything it can. (Valgrind using a CPU emulator allows for a lot of interesting things, such as also emulating cache behavior and whatnot; it may be slow and have other drawbacks -- it has to be updated every time the instruction set adds a new instruction for instance -- but it's able to do things that aren't usually possible otherwise precisely because it has a CPU emulator!)
- bracewel 1y agoAuthor of the linked CL here: we added this mostly so that we could abuse the memory initialization tracking to test the constant-time-ness of crypto code (similar to what BoringSSL does, proposed by agl around fifteen years ago: https://www.imperialviolet.org/2010/04/01/ctgrind.html https://www.imperialviolet.org/2010/04/01/ctgrind.html), which is an annoyingly hard property to test. We're hoping that there are also a bunch of other interesting side-effects of enabling the usage of Valgrind for Go, in particular seeing how we can use it to track how the runtime handles memory (hopefully correctly!) edit: also strong disclaimer that this support is still somewhat experimental. I am not 100% confident we are properly instrumenting everything, and it's likely there are still some errant warnings that don't fully make sense.
- chrsig 1y agow.r.t. your edit: Is there anything the community at large can do to aid your efforts?
- on_the_beach 1y agoThis is super cool. Hopefully it will flush out other issues in Go too. But I wonder why its not trivial to throw a bunch of different inputs at your cyphering functions and measure that the execution times are all within an epsilon tolerance? I mean, you want to show constant time of your crypto functions, why not just directly measure the time under lots of inputs? (and maybe background Garbage Collection and OS noise) and see how constant they are directly? Also some CPUs have a counter for conditional branches (that the rr debuger leverages), and you could sample that before and after and make sure the number of conditional branches does not change between decrypts -- as that AGL post mentions branching being the same is important for constant time. Finally, it would also seem trivial to track the first 10 decrypts, take their maximum time add a small extra few nanoseconds tolerance, and pad every following decrypt with a few nanoseconds (executing noops) to force constant time when it is varying. And you could add an assert that anything over that established upper bound crashes the program since it is violating the constant time property. I suppose the real difficulty is if the OS deschedules your execution and throws off your timing check...
- 1y ago
- tasn 1y agoThis feels more like a failure than a win. Don't get me wrong, I love Valgrind, and have been using it extensively in my past life as a C developer. Though the fact that Go needs Valgrind feels like a failure of the language or the ecosystem. I've been doing Rust for ~6 years now, and haven't had to reach for Valgrind even once (I think a team member may have use it once). I realize that's probably because of cgo, and maybe it's in-fact a step forward, but I can help but feel like it is a step backwards.
- pjmlp 1y agoDepends on how much unsafe you actually happen to write, use unsafe crates, or link into C and C++ libraries. I also seldom need something like this in Java, .NET or node, until a dependency makes it otherwise.
- tasn 1y agoFor sure, that's why I said it's possibly due to the ecosystem. We link against two C libraries other than libc: OpenSSL and librdkafka. Though they are both abstracted away with solid Rust bindings so for us, so as a consumer of these libs it hasn't been a problem (I guess it may be a problem for the people developing them). I guess maybe the failure is not the addition of it (as it's useful for people writing the bindings), but rather how happy everyone on the thread is (which means it's more useful than it should be due to a failure with the ecosystem).
- pjmlp 1y agoEven though there is the whole CGO is not Go meme, it certainly makes it rather easy to write C and C++ code directly on a Go project, thus I imagine some folks reach rather easy to it. Which I can relate to, when doing stuff that is Windows only , I rather make use of C++/CLI than getting P/Invoke declarations correctly.
- 9rx 1y ago> but rather how happy everyone on the thread is (which means it's more useful than it should be due to a failure with the ecosystem). More likely Go users are just happy in general. The Rust users always come across as being incredibly grumpy for some reason, which may be why that happiness — or what would be considered normalcy in any other venue — seems to stand out so much in comparison. > We link against [...] OpenSSL Which is kind of funny as Valgrind support was added specifically for the crypto package to help test tricky constant-time cases. You think that the failure of the ecosystem is that Go has built high-quality crypto support in rather than relying on the Heartbleed library instead...? That is certainly an interesting take.
- chrsig 1y agoI'm glad to see rsc still actively involved. And commenting on commit messages. The older I get the more I value commit messages. It's too easy to just leave a message like "adding valgrind support", which isn't very useful to future readers doing archaeology.
- pstuart 1y agorsc is a rock star! I believe his focus now is on using AI to manage issues and PRs and such -- I'm sure it will bear copious fruit.
- holyknight 1y agodamn, i remember using valgrind when writing C in university a long time ago.