12 ms·
> After digging through the Go source code, we learned that Go will force a garbage collection run every 2 minutes at minimum. In other words, if garbage collec
by flafla2 7y ago
> After digging through the Go source code, we learned that Go will force a garbage collection run every 2 minutes at minimum. In other words, if garbage collection has not run for 2 minutes, regardless of heap growth, go will still force a garbage collection.
> We figured we could tune the garbage collector to happen more often in order to prevent large spikes, so we implemented an endpoint on the service to change the garbage collector GC Percent on the fly. Unfortunately, no matter how we configured the GC percent nothing changed. How could that be? It turns out, it was because we were not allocating memory quickly enough for it to force garbage collection to happen more often.
As someone not too familiar with GC design, this seems like an absurd hack. That this 2-minute hardcoded limitation is not even configurable comes across as amateurish even. I have no experience with Go -- do people simply live with this and not talk about it?
- JBReefer 7y agoGo always feels like an amateur language to me, I’ve given up on it. This feels right in line - similar to the hardcode GitHub magic.
- reificator 7y agoI could be wrong, but I don't believe there is "hardcoded[d] GitHub magic". IIRC I have used GitLab and Bitbucket and self-hosted Gitea instances the same exact way, and I'm fairly sure there was an hg repo in one of those. Don't recall doing anything out of the ordinary compared to how I would use a github URL.
- heinrich5991 7y agoThere are a couple of hosting services hardcoded in Go. I believe it was about splitting the URL into the actual URL and the branch name.
- Nullabillity 7y agohttps://github.com/golang/go/blob/e6ebbe0d20fe877b111cf4ccf8349cba129d6d3a/src/cmd/go/internal/get/vcs.go#L1022 https://github.com/golang/go/blob/e6ebbe0d20fe877b111cf4ccf8... Ouch, Go never ceases to amaze. The Bitbucket case[0] is even more crazy, calling out to the Bitbucket API to figure out which VCS to use. It has a special case for private repositories, but seems to hard-code cloning over HTTPS. If only we had some kind of universal way to identify resources, that told you how to access it... [0]: https://github.com/golang/go/blob/e6ebbe0d20fe877b111cf4ccf8349cba129d6d3a/src/cmd/go/internal/get/vcs.go#L1113 https://github.com/golang/go/blob/e6ebbe0d20fe877b111cf4ccf8...
- reificator 7y agoThanks for the reference to prove me wrong. Wow, that's sad. I'm glad it works seamlessly, don't get me wrong, but I was assuming I could chalk it up to defacto standards between the various vendors here.
- deleted 7y ago[deleted]
- ptrincr 7y agoYou are able to disable GC with: GOGC=off As someone mentions below. More details here: https://golang.org/pkg/runtime/ https://golang.org/pkg/runtime/
- singron 7y agoKeeping GC off for a long running service might become problematic. Also, the steady state might have few allocations, but startup may produce a lot of garbage that you might want to evict. I've never done this, but you can also turn GC off at runtime with SetGCPercent(-1). I think with that, you could turn off GC after startup, then turn it back on at desired intervals (e.g. once an hour or after X cache misses). It's definitely risky though. E.g. if there is a hiccup with the database backend, the client library might suddenly produce more garbage than normal, and all instances might OOM near the same time. When they all restart with cold caches, they might hammer the database again and cause the issue to repeat.
- ignoramous 7y ago> ...all instances might OOM near the same time. CloudFront, for this reason, allocates heterogeneous fleets in its PoPs which have diff RAM sizes and CPUs [0], and even different software versions [1]. > When they all restart with cold caches, they might hammer the database again and cause the issue to repeat. Reminds me of the DynamoDB outage of 2015 that essentially took out us-east-1 [2]. Also, ELB had a similar outage due to unending backlog of work [3]. Someone must write a book on design patterns for distributed system outages or something? [0] https://youtube.com/watch?v=pq6_Bd24Jsw&t=50m40s https://youtube.com/watch?v=pq6_Bd24Jsw&t=50m40s [1] https://youtube.com/watch?v=n8qQGLJeUYAt=39m0s https://youtube.com/watch?v=n8qQGLJeUYAt=39m0s [2] https://aws.amazon.com/message/5467D2/ https://aws.amazon.com/message/5467D2/ [3] https://aws.amazon.com/message/67457/ https://aws.amazon.com/message/67457/
- singron 7y agoGoogle's SRE book covers some of this (if you aren't cheekily referring to that). E.g. chapters 21 and 22 are "Handling Overload" and "Addressing Cascading Failures". The SRE book also covers mitigation by operators (e.g. manually setting traffic to 0 at load balancer and ramping back up, manually increasing capacity), but it also talks about engineering the service in the first place. This is definitely a familiar problem if you rely on caches for throughput (I think caches are most often introduced for latency, but eventually the service is rescaled to traffic and unintentionally needs the cache for throughput). You can e.g. pre-warm caches before accepting requests or load-shed. Load-shedding is really good and more general than pre-warming, so it's probably a great idea to deploy throughout the service anyway. You can also load-shed on the client, so servers don't even have to accept, shed, then close a bunch of connections. The more general pattern to load-shedding is to make sure you handle a subset of the requests well instead of degrading all requests equally. E.g. processing incoming requests FIFO means that as queue sizes grow, all requests become slower. Using LIFO will allow some requests to be just as fast and the rest will timeout.
- _bxg1 7y agoIt does sound like Discord's case was fairly extraordinary in terms of the degree of the spike: > We kept digging and learned the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire LRU cache in order to determine if the memory was truly free from references. So maybe this is one of those things that just doesn't come up in most cases? Maybe most services also generate enough garbage that that 2-minute maximum doesn't really come into play?
- spullara 7y agoHeap caches that keep things longer than a GC cycle are terrible under GC unless you have a collector in the new style like ZGC, Azul or Shenandoah.
- deleted 7y ago[deleted]
- sitkack 7y agoSystems with poor GC and the need to keep data for lifetimes greater than a request should have an easy to use off heap mechanism to prevent these problems. Often something like Redis is used as a shared cache that is invisible to the garbage collector, there is a natural key with a weak reference (by name) into a KV store. One could embed a KV store into an application that the GC can't scan into.
- spullara 7y ago100%. In Java, you would often use OpenHFT's ChronicleMap for now and hopefully inline classes/records in Java 16 or so.
- cgh 7y agoEhcache has an efficient off-heap store: https://github.com/Terracotta-OSS/offheap-store/ https://github.com/Terracotta-OSS/offheap-store/ Doesn't Go have something like this available? It's an obvious thing for garbage-collected languages to have.
- ericflo 7y agoIt comes from a desire to run in the exact opposite direction as the JVM, which has options for every conceivable parameter. Go has gone through a lot of effort to keep the number of configurable GC parameters to 1.
- erik_seaberg 7y agoAnyone who pushes the limits of a machine needs tuning options. If you can't turn knobs you have to keep rewriting code until you happen to get the same effect.
- ericflo 7y agoThere's definitely a happy medium. One setting may indeed be too few, but JVM's many options ends in mass cargo-cult copypasta, often leading to really bad configurations.
- deleted 7y ago[deleted]
- Polyisoprene 7y agoHaven’t really seen anyone trying to use JVM options to get performance benefits without benchmarks for their specific use case the last 10 years or so.
- tomc1985 7y agoYeAh BuT wE dOnT wAnT tO hAvE tO tEsT eVeRy oPtIoN!!! - lazy devs and product managers, everywhere
- hombre_fatal 7y agoThis was the first time I've seen that annoying cAsE meme on HN and I pray it's the last. It is a lazy way to make your point, hoping your meme-case does all the work for you so that you don't have to say anything substantial. Or do you think it adds to the discussion?
- dickeytk 7y agoI think for most applications (especially the common use-case of migrating a scripting web monolith to a go service), people just aren't hitting performance issues with GC. Discord being a notable exception. If these issues were more common, there would be more configuration available. [EDIT] to downvoters: I'm not saying it's not an issue worth addressing (and it may have already been since they were on 1.9), I was just answering the question of "why this might happen"
- biomcgary 7y agoOr, in the case of latency, just wait a few months because the Go team obsesses about latency (no surprise from a Google supported language). Discord's comparison is using Go1.9. Their problem may well have been addressed in Go1.12. See https://golang.org/doc/go1.12#runtime https://golang.org/doc/go1.12#runtime.
- Nyra 7y agoFunnily enough, something similar happened at Twitch regarding their API front end written in Go: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
- mrpotato 7y agoInteresting, they went a totally different route. > The ballast in our application is a large allocation of memory that provides stability to the heap. > As noted earlier, the GC will trigger every time the heap size doubles. The heap size is the total size of allocations on the heap. Therefore, if a ballast of 10 GiB is allocated, the next GC will only trigger when the heap size grows to 20 GiB. At that point, there will be roughly 10 GiB of ballast + 10 GiB of other allocations.
- firethief 7y agoWow, that puts Discord's "absurd hack" into perspective! I feel like the moral here is a corollary to that law where people will depend on any observable behavior of the implementation: people will use any available means to tune important performance parameters; so you might as well expose an API directly, because doing so actually results in less dependence on your implementation details than if people resort to ceremonial magic.
- tick_tock_tick 7y agoI mean if you read Twitch's hack they intentionally did it in code so they didn't need to tune the GC parameter. They wanted to avoid all environment config.
- firethief 7y agoI missed that part. I thought they would use a parameter if it were available, because they said this: > For those interested, there is a proposal to add a target heap size flag to the GC which will hopefully make its way into the Go runtime soon. What's wrong with the existing parameter? I'm sure they aren't going this far to avoid all environment config without a good reason, but any good reason would be a flaw in some part of their stack.
- nickserv 7y agoThis is in line with Go's philosophy, they try to keep the language as simple as possible. Sometimes it means an easy thing in most other languages is difficult or tiresome to do in Go. Sometimes it means hard-coded values/decisions you can't change (only tabs anyone?). But overall this makes for a language that's very easy to learn, where code from project to project and team to team is very similar and quick to understand. Like anything, it all depends on your needs. We've found it suits ours quite well, and migrating from a Ruby code base has been a breath of fresh air for the team. But we don't have the same performance requirements as Discord.
- andai 7y agoOfftopic but what are you missing when you have to use tabs instead of spaces? I can understand different indentation preferences but I can change the indentation width per tab in my editor. And then everyone can read the code with the indentation they prefer, while the file stays the same.
- nickserv 7y agoIt's just an example of something that the Go team took a decision on, and won't allow you to change. I mean, even Python lets you choose. I don't really have a problem with it however, even if I do prefer spaces.
- giancarlostoro 7y agoPython chose spaces a la PEP8 by the way.
- dragonwriter 7y agoPEP8 isn't a language requirement, but a style guide. There are tools to enforce style on Python, but the language itself does not.
- giancarlostoro 7y ago
- abraxas 7y agoSeems like Go is more suitable for the “spin up, spin down, never let the GC run” kind of scenario that is being pushed by products like AWS Lambda and other function as a service frameworks.
- _ph_ 7y agoWhy do you think it is? Go has a really great gc which mostly runs in parallel to your program with gc stops only in the doman of less than milliseconds. Discord ran into a corner case where they did not create enough garbage to trigger gc cycles, but had a performance impact due to scheduled gc cycles for returning memory to the OS (which they wouldn't need to do either).
- deleted 7y ago[deleted]
- abraxas 7y agoBecause many services eventually become performance bottlenecked either via accumulation of users or accumulation of features. In either case eventually performance becomes very critical.
- _ph_ 7y agoSure, but that doesn't make Go unsuitable for those tasks on a fundamental basis. Go is very high performance. Whether Go or another language is the best match very much depends on the problem at hand and the especial requirements. Even in the described case they might have tweaked the GC to fit their bill.
- abraxas 7y agoGC pauses aside can Go match the performance of Rust when coded properly? Would sorting an array of structs in Go be in the same ballpark as sorting the same sized array of structures in Rust? I don't know a whole lot about how Go manages the heap under the covers.
- recuter 7y agoIf you want to force it you can call "runtime.GC()" but that's almost always a step in the wrong direction. It is worth it to read and understand: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...
- _ph_ 7y agoWith recent Go releases, GC pauses have become neglible for most applications. So this should not get into your way. However, it can easily tweaked, if needed. There is runtime.ForceGCPeriod, which is a pointer to the forcegcperiod variable. A Go program, which really needs to change this, can do it, but most programs shouldn't require this. Also, it is almost trivial to edit the Go sources (they are included in the distribution) and rebuild it, which usually takes just a minute. So Go is really suited for your own experiments - especially, as Go is implemented in Go.
- nemo1618 7y agoruntime.ForceGCPeriod is only exported in testing, so you wouldn't be able to use it in production. But as you said, the distribution could easily be modified to fit their needs.
- _ph_ 7y agoThanks, didn't catch that this is for testing only.
- calcifer 7y ago> especially, as Go is implemented in Go. Well, parts of it. You can't implement "make" or "new" in Go yourself, for example.
- _ph_ 7y agoYou have to distinguish between the features available to a Go program as the user writes it and the implementation of the language. The immplementation is completely written in Go (plus a bit of low-level assembly). Even if the internals of e.g. the GC are not visible to a Go program, the GC itself is implemented in Go and thus easily readeable and hackeable for experienced Go programmers. And you can quickly rebuild the whole Go stack.
- calcifer 7y ago> You have to distinguish between the features available to a Go program as the user writes it and the implementation of the language. I do, I'm just objecting to "Go is implemented in Go".
- tedunangst 7y agoTypically a GC runtime will do a collection when you allocate memory, probably when the heap size is 2x the size after the last collection. But this doesn't free memory when the process doesn't allocate memory. The goal is to return unused memory back to the operating system so it's available for other purposes. (You allocate a ton of memory, calculate some result, write the result to a file, and drop references to the memory. When will it be freed?)