3 ms·

> A good portion of production outages are likely related to cascading failures due to too long GC pauses, and a good portion of developer time is spent testing

> A good portion of production outages are likely related to cascading failures due to too long GC pauses, and a good portion of developer time is spent testing and tuning GC parameters After 14 years in JVM dev in areas where latency and reliability are business critical, I disagree. Yes, excessive GC stop the world pauses can cause latency spikes, and excessive GC time is bad, and yes, when a new GC algorithm is released that you think might offer improvements, you test it thoroughly to determine if it's better or worse for your workload. But a "good portion" of outages and developer time? Nope. Most outages occur for the same old boring reasons - someone smashed the DB with an update that hits a pathological case and deadlocks processes using the same table, a DC caught fire, someone committed code with a very bad logical bug, someone considered a guru heard that gRPC was cool and used it without adequate code review and didn't understand that gRPC's load balancing defaults to pick first, etc. etc. The outages caused by GC were very very few. Outages caused by screw-ups or lack of understanding of subtleties of a piece of tech, as common as they are in every other field of development. Then there's the question of what outages GCed languages _don't_ suffer. I've never had to debug corrupted memory, or how a use after free bug let people exfiltrate data.

zinekeller 5y ago
> I've never had to debug corrupted memory You're lucky! When OpenJDK was still closed-sourced Hotspot from Sun, we have chased bugs that Sun confirmed was a defect on how Hotspot handle memory (and this is on a ECC'd system of course), although these days I can't remind of anything remotely related. > or how a use after free bug let people exfiltrate data. Technically you're just outsourcing it :)
- Karrot_Kream 5y ago
  > You're lucky! When OpenJDK was still closed-sourced Hotspot from Sun, we have chased bugs that Sun confirmed was a defect on how Hotspot handle memory (and this is on a ECC'd system of course), although these days I can't remind of anything remotely related. I mean sure. I remember having similar issues with early (< 2.3) Python builds as well. But in the last decade of my career, only a handful of outages were caused by Java GC issues. Most of them happened for a myriad of other architectural reasons.
- EdwardDiego 5y ago
  Yeah, have only ever hit one or two JVM bugs in very rare circumstances - which we usually fixed by upgrading. > Technically you're just outsourcing it :) Haha, very true. Luckily, to developers who are far better at that stuff than the average bear. The recent log4j rigmarole is a great example of what I was describing in JVM dev though - no complicated memory issues involved, definitely not GC related, just developers making decisions using technologies that had very subtle footguns they didn't understand (the capacity to load arbitrary code via LDAP was, AFAIK, very poorly known, if not forgotten, until Log4Shell).
zepolen 5y ago
> After 14 years in JVM dev in areas where latency and reliability are business critical What sort of industry/use cases are we talking here? There is business critical and mission critical and if your experience is in network applications as your next paragraph seems to imply then no offence, but you have never worked with critical systems where an nondeterministic GC pause can send billions worth of metal into the sun or kill people.
- EdwardDiego 5y ago
  Um, how did you derive from this conversation that the "outages" in question were about space missions failing? Curious, and a tad confused.
  - zepolen 5y ago
    The parent of this thread being is about systems languages and how GC languages are rarely the right tool. Then your comment about having experience that the GC doesn't really matter in critical environments, which I personally find not to be true at all but am interested in which domain your comment is based on. So please answer the original question.

zinekeller 5y ago

> I've never had to debug corrupted memory You're lucky! When OpenJDK was still closed-sourced Hotspot from Sun, we have chased bugs that Sun confirmed was a defect on how Hotspot handle memory (and this is on a ECC'd system of course), although these days I can't remind of anything remotely related. > or how a use after free bug let people exfiltrate data. Technically you're just outsourcing it :)

Karrot_Kream 5y ago
> You're lucky! When OpenJDK was still closed-sourced Hotspot from Sun, we have chased bugs that Sun confirmed was a defect on how Hotspot handle memory (and this is on a ECC'd system of course), although these days I can't remind of anything remotely related. I mean sure. I remember having similar issues with early (< 2.3) Python builds as well. But in the last decade of my career, only a handful of outages were caused by Java GC issues. Most of them happened for a myriad of other architectural reasons.
EdwardDiego 5y ago
Yeah, have only ever hit one or two JVM bugs in very rare circumstances - which we usually fixed by upgrading. > Technically you're just outsourcing it :) Haha, very true. Luckily, to developers who are far better at that stuff than the average bear. The recent log4j rigmarole is a great example of what I was describing in JVM dev though - no complicated memory issues involved, definitely not GC related, just developers making decisions using technologies that had very subtle footguns they didn't understand (the capacity to load arbitrary code via LDAP was, AFAIK, very poorly known, if not forgotten, until Log4Shell).

zepolen 5y ago

> After 14 years in JVM dev in areas where latency and reliability are business critical What sort of industry/use cases are we talking here? There is business critical and mission critical and if your experience is in network applications as your next paragraph seems to imply then no offence, but you have never worked with critical systems where an nondeterministic GC pause can send billions worth of metal into the sun or kill people.

EdwardDiego 5y ago
Um, how did you derive from this conversation that the "outages" in question were about space missions failing? Curious, and a tad confused.
- zepolen 5y ago
  The parent of this thread being is about systems languages and how GC languages are rarely the right tool. Then your comment about having experience that the GC doesn't really matter in critical environments, which I personally find not to be true at all but am interested in which domain your comment is based on. So please answer the original question.