8 ms·

Finding Java Thread Leaks with JDK Flight Recorder and a Bit of SQL

197 points by gunnarmorling 4y ago

marginalia_nu 4y ago
Say what you will about Java, but its tooling is not lacking.
- deleted 4y ago
  [deleted]
- nih0 4y ago
  i miss the tooling so much, theres not many programming languages that can compete, maybe c# with the jetbrains suite
  - pjmlp 4y ago
    .NET with Visual Studio Enterprise. There is enough of .NET that Rider still doesn't do.
    - whoisthemachine 4y ago
      Any examples? The only one I've encountered in the last few years is editing .net core WinForms UIs with the visual editor. In every other use case of mine, Rider has outclassed Visual Studio.
      - pjmlp 4y ago
        WPF, Blend, hot code reloading, repl, mixed language debugging with C++/CLI, C++/CX, C++/WinRT, SharePoint, Dynamics, SQL Server SP, Architecture tooling, coverage and automatic unit tests, COM interop tooling, Azure workloads integrations, some of the examples that come to my mind. Also, when already paid for MSDN license which includes VS, why pay for Rider?
        whoisthemachine 4y ago
        Rider has a great WPF editor (again, I would say it often surpasses VS in terms of code completion), and is capable of hot code reloading (for server code - it doesn't work with WPF AFAIK). I'm also not sure what you mean by "REPL" as I tend to think of that as a language feature, not an IDE feature (most read-eval-print-loops are done through a terminal, which Rider has). Rider also has coverage and automatic unit tests (and it's sibling product Resharper had it for years before Visual Studio added it). "Architecture tooling" could be a quite broad category, but as far as where I've used Jetbrains products in this sense, they can automatically build class diagrams and the like from your code, and even export them to PlantUML if you want. Jetbrains also offers an IDE that can handle C++ just fine - https://www.jetbrains.com/clion/ https://www.jetbrains.com/clion/. I don't know how well it handles WinRT, but I'll admit I've never had the need to develop or deploy a WinRT application. I do know that they've started building in MAUI integration [2]. I'll confess I don't use Dynamics, or know what SQL Server SP is, but Rider has great Database tooling built in. I've never had a problem connecting to MS SQL Server, SQLite, Postgres, or MySQL databases with it. > Also, when already paid for MSDN license which includes VS, why pay for Rider? I guess this question is aggressively asking why someone would choose Rider if they already have Visual Studio Enterprise, and beyond it's great tooling, the main benefits for me are the great code completion, the snappiness of the UI (you can use the editor before it has loaded the AST), and how rare crashes are with it (and it has seemingly gotten even rarer in the last few months, I can't remember it crashing). Beyond that, if you have to choose between the two, an Enterprise MSDN license is $6k/yr [0], and a commercial Rider license + the other Jetbrains C# tools is $470/yr [1]. The former is usually a difficult question with your manager and/or purchasing, while the latter hardly raises an eyebrow. [0] https://www.microsoft.com/en-us/d/visual-studio-enterprise-subscription/dg7gmgf0dst4?activetab=pivot%3Aoverviewtab https://www.microsoft.com/en-us/d/visual-studio-enterprise-s... [1] https://www.jetbrains.com/rider/buy/#commercial https://www.jetbrains.com/rider/buy/#commercial [2] https://blog.jetbrains.com/dotnet/2022/08/02/rider-2022-2-released/#EarlypreviewofMAUIsupport https://blog.jetbrains.com/dotnet/2022/08/02/rider-2022-2-re...
        easton 4y ago
        > The former is usually a difficult question with your manager and/or purchasing, while the latter hardly raises an eyebrow. The one thing is that everyone on your team probably uses Visual Studio or has a license if you’re a .NET shop, that may not be true of Rider or ReSharper. Your manager is already convinced of the need for VS. (I just bought my own JetBrains Toolbox because I had a coupon, but my company will pay for ReSharper if we want it)
      - sebazzz 4y ago
        Rider does have parallel stacks, but does not have a function to show tasks. Some deadlocks are much more difficult to debug if you can't show the tasks.
- cbm-vic-20 4y ago
  One thing I love about the JVM is that the tooling can be used in production if necessary- generating thread dumps, JFR recordings, monitoring heap usage, etc., all of this is invaluable when diagnosing problems.
  - marginalia_nu 4y ago
    Yeah, if needs must, you can even attach a remote debugger to a process i prod. Pretty crazy.
    - zxexz 4y ago
      Yes! I’ve had to do this at a previous company, while leadership was demoing at a conference, there was a bug that would crash the system. Connected remotely through a debugger and listened for the condition, modifying state to keep the crash from happening. Can’t remember the details, but it was exciting.
- ActorNightly 4y ago
  JDB is pretty terrible tbh. Granted most of its use is indirectly through IDES, but still. Also, having tooling to address/fix the many issues Java has isn't really that good in the first place.
pron 4y ago
JFR is one of my favourite Java features. Like eBPF but with more events baked into the runtime and easier to use. Gunnar has used it in some cool ways. Not just https://github.com/moditect/jfr-analytics https://github.com/moditect/jfr-analytics, which he showcases in this post, but also https://github.com/moditect/jfrunit https://github.com/moditect/jfrunit
- pjmlp 4y ago
  And if my history recollection is right, originally born on J/Rockit JVM.
  - pron 4y ago
    Correct.
- gunnarmorling 4y ago
  JFR is just awesome. Thanks a lot for the nice words, Ron! Means a lot to me, coming from you.
TYMorningCoffee 4y ago
> and the stack trace of the parent thread when starting the child thread That's incredibly useful. We have a strict policy of all threads and pools being named, but sometimes default named threads sneak in making it difficult to figure out where the thread is used for.
nomemory 4y ago
Java is not the most appealing programming language, but you've got to love the JVM, the standard library, all the libraries and frameworks, plus the tooling around the ecosystem.
- smallerfish 4y ago
  Hence, Kotlin.
  - pjmlp 4y ago
    So when are we getting Kotlin Virtual Machine, designed for Kotlin?
    - MattPalmer1086 4y ago
      Does the JVM impose heavy constraints on Kotlin? I ask because the JVM is an amazing bit of technology that has been battle tested for decades. Would be hard to beat I would think.
      - vips7L 4y ago
        One of the major issues that Kotlin faces is that it’s hard for them to change the VM to provide a feature they want while Java and the OpenJDK developers have the freedom to either change the VM or the language to provide something.
        MattPalmer1086 4y ago
        Sounds like Kotlin and other JVM languages need greater representation in OpenJDK then, if that would be possible.
        tadfisher 4y ago
        I don't think it's too burdensome, because they can emulate features until the JVM gets support. I can think of several: - Data classes can become JVM records with the @JvmRecord annotation and some additional type constraints. - Inline classes will certainly become Valhalla value types (hence the required @JvmInline annotation). - Nullable types are supported with compile-time checks and additional runtime checks inserted by the compiler at time-of-use, and if Valhalla adds a nullability marker to the bytecode, Kotlin can support it. - Coroutines/structured-concurrency already uses Executors under the hood, so it's likely that we'll have Loom thread support without any changes to the coroutines APIs (though the standard dispatchers will need to use virtual threads themselves). Coroutines themselves are CPS-transforms of ordinary code and don't need VM support. - Tail-call elimination is supported by transforming to trampolines, something which the JVM will probably never support until Java needs it.
exabrial 4y ago
I need to learn to use jfr apparently. Just something I’ve known existed and never used. We attach a hawtio.io jvm agent to every single prod jvm we run and query the agent using telegraph. (If you don’t do this by the way, you should!) I noticed in the GUI portion of hawtio, it slows you to start/stop a jfr recording, but I’ve never messed with it
Scarbutt 4y ago
HN being very anti Java, why does such many Java articles make it to the front page?
- mqus 4y ago
  Maybe the commenters are "anti" and the voters aren't (more comments being also a detriment to getting to the frontpage)? But I don't see any real "anti" comments, only valid criticism and I say that as a Java dev myself. It does have its wrinkles. But for it's age and popularity it's still a pretty good language.
- ncann 4y ago
  That's not my experience at all, if anything I feel like HN is quite fond of Java. The anti Java crowd is mostly on Reddit instead where it's the subject of memes and jokes and the like.
  - lenkite 4y ago
    Except /r/java where Java is God
    - arnold_palmur 4y ago
      I think it's safe to assume that as a general rule on /r/<X>, <X> is god.
    - re-thc 4y ago
      There are lots of people in /r/java that defy God :p
- exabrial 4y ago
  A lot of it is misdirected hate. In the first dot.com boom, a lot of people worked for tech companies, and Microsoft was the Google|Apple back then. Their corporate culture was dog-eat-dog nonsense, but it trickled down through the industry... and of course at the time Java was super super popular, even though _it really sucked_ back then (not so much now, it's quite awesome). One of the most _hated_ positions was "The Architect" who usually rose to the top via floating on top of a bubble of attrition; so rather than being promoted because you're good at leadership or a great person that's fun to be around or a technically savvy and loved to teach... these architects were ignorant egotistical maniacal dictators that got promoted merely because everyone else quit. They turned static analysis tools into weapons of war. They made arbitrary standards that really didn't work for anyone, rather than trying to work for someone. Everything had to be over-engineered (sound familiar? If not, look at today's SPAs) with AbstractFactoryProducerMethodProducerImplInterfaces. I worked at a lot of these companies, and it just wasn't a fun time to be in software development. Rather than people seeing the reality that Poor Leadership and zero charisma as the root cause, somehow people arrived at Java was the core problem. A lot of the things were good in their intention; for instance, having the whole company use tabs instead of spaces (or vice versa) and having a universal code formatter have the potential to increase communication. But instead, it was just executed poorly at nearly every shop and the vast majority of the problems got turned into Dilbert comics. But yeah, Java.
sgt 4y ago
For a split second there, I thought a post-airline accident flight recorder had some Java heap dumps that showed major issues and thread leaks.
tmd83 4y ago
JFR is nice but I can't seem to find issues as easily with the new JMC as I used to 5.5 or before. I'm not sure what I'm doing wrong though.

197 points by gunnarmorling 4y ago

marginalia_nu 4y ago

Say what you will about Java, but its tooling is not lacking.

deleted 4y ago
[deleted]
nih0 4y ago
i miss the tooling so much, theres not many programming languages that can compete, maybe c# with the jetbrains suite
- pjmlp 4y ago
  .NET with Visual Studio Enterprise. There is enough of .NET that Rider still doesn't do.
  - whoisthemachine 4y ago
    Any examples? The only one I've encountered in the last few years is editing .net core WinForms UIs with the visual editor. In every other use case of mine, Rider has outclassed Visual Studio.
    - pjmlp 4y ago
      WPF, Blend, hot code reloading, repl, mixed language debugging with C++/CLI, C++/CX, C++/WinRT, SharePoint, Dynamics, SQL Server SP, Architecture tooling, coverage and automatic unit tests, COM interop tooling, Azure workloads integrations, some of the examples that come to my mind. Also, when already paid for MSDN license which includes VS, why pay for Rider?
      - whoisthemachine 4y ago
        Rider has a great WPF editor (again, I would say it often surpasses VS in terms of code completion), and is capable of hot code reloading (for server code - it doesn't work with WPF AFAIK). I'm also not sure what you mean by "REPL" as I tend to think of that as a language feature, not an IDE feature (most read-eval-print-loops are done through a terminal, which Rider has). Rider also has coverage and automatic unit tests (and it's sibling product Resharper had it for years before Visual Studio added it). "Architecture tooling" could be a quite broad category, but as far as where I've used Jetbrains products in this sense, they can automatically build class diagrams and the like from your code, and even export them to PlantUML if you want. Jetbrains also offers an IDE that can handle C++ just fine - https://www.jetbrains.com/clion/ https://www.jetbrains.com/clion/. I don't know how well it handles WinRT, but I'll admit I've never had the need to develop or deploy a WinRT application. I do know that they've started building in MAUI integration [2]. I'll confess I don't use Dynamics, or know what SQL Server SP is, but Rider has great Database tooling built in. I've never had a problem connecting to MS SQL Server, SQLite, Postgres, or MySQL databases with it. > Also, when already paid for MSDN license which includes VS, why pay for Rider? I guess this question is aggressively asking why someone would choose Rider if they already have Visual Studio Enterprise, and beyond it's great tooling, the main benefits for me are the great code completion, the snappiness of the UI (you can use the editor before it has loaded the AST), and how rare crashes are with it (and it has seemingly gotten even rarer in the last few months, I can't remember it crashing). Beyond that, if you have to choose between the two, an Enterprise MSDN license is $6k/yr [0], and a commercial Rider license + the other Jetbrains C# tools is $470/yr [1]. The former is usually a difficult question with your manager and/or purchasing, while the latter hardly raises an eyebrow. [0] https://www.microsoft.com/en-us/d/visual-studio-enterprise-subscription/dg7gmgf0dst4?activetab=pivot%3Aoverviewtab https://www.microsoft.com/en-us/d/visual-studio-enterprise-s... [1] https://www.jetbrains.com/rider/buy/#commercial https://www.jetbrains.com/rider/buy/#commercial [2] https://blog.jetbrains.com/dotnet/2022/08/02/rider-2022-2-released/#EarlypreviewofMAUIsupport https://blog.jetbrains.com/dotnet/2022/08/02/rider-2022-2-re...
        easton 4y ago
        > The former is usually a difficult question with your manager and/or purchasing, while the latter hardly raises an eyebrow. The one thing is that everyone on your team probably uses Visual Studio or has a license if you’re a .NET shop, that may not be true of Rider or ReSharper. Your manager is already convinced of the need for VS. (I just bought my own JetBrains Toolbox because I had a coupon, but my company will pay for ReSharper if we want it)
    - sebazzz 4y ago
      Rider does have parallel stacks, but does not have a function to show tasks. Some deadlocks are much more difficult to debug if you can't show the tasks.
cbm-vic-20 4y ago
One thing I love about the JVM is that the tooling can be used in production if necessary- generating thread dumps, JFR recordings, monitoring heap usage, etc., all of this is invaluable when diagnosing problems.
- marginalia_nu 4y ago
  Yeah, if needs must, you can even attach a remote debugger to a process i prod. Pretty crazy.
  - zxexz 4y ago
    Yes! I’ve had to do this at a previous company, while leadership was demoing at a conference, there was a bug that would crash the system. Connected remotely through a debugger and listened for the condition, modifying state to keep the crash from happening. Can’t remember the details, but it was exciting.
ActorNightly 4y ago
JDB is pretty terrible tbh. Granted most of its use is indirectly through IDES, but still. Also, having tooling to address/fix the many issues Java has isn't really that good in the first place.

pron 4y ago

JFR is one of my favourite Java features. Like eBPF but with more events baked into the runtime and easier to use. Gunnar has used it in some cool ways. Not just https://github.com/moditect/jfr-analytics https://github.com/moditect/jfr-analytics, which he showcases in this post, but also https://github.com/moditect/jfrunit https://github.com/moditect/jfrunit

pjmlp 4y ago
And if my history recollection is right, originally born on J/Rockit JVM.
- pron 4y ago
  Correct.
gunnarmorling 4y ago
JFR is just awesome. Thanks a lot for the nice words, Ron! Means a lot to me, coming from you.

TYMorningCoffee 4y ago

> and the stack trace of the parent thread when starting the child thread That's incredibly useful. We have a strict policy of all threads and pools being named, but sometimes default named threads sneak in making it difficult to figure out where the thread is used for.

nomemory 4y ago

Java is not the most appealing programming language, but you've got to love the JVM, the standard library, all the libraries and frameworks, plus the tooling around the ecosystem.

smallerfish 4y ago
Hence, Kotlin.
- pjmlp 4y ago
  So when are we getting Kotlin Virtual Machine, designed for Kotlin?
  - MattPalmer1086 4y ago
    Does the JVM impose heavy constraints on Kotlin? I ask because the JVM is an amazing bit of technology that has been battle tested for decades. Would be hard to beat I would think.
    - vips7L 4y ago
      One of the major issues that Kotlin faces is that it’s hard for them to change the VM to provide a feature they want while Java and the OpenJDK developers have the freedom to either change the VM or the language to provide something.
      - MattPalmer1086 4y ago
        Sounds like Kotlin and other JVM languages need greater representation in OpenJDK then, if that would be possible.
      - tadfisher 4y ago
        I don't think it's too burdensome, because they can emulate features until the JVM gets support. I can think of several: - Data classes can become JVM records with the @JvmRecord annotation and some additional type constraints. - Inline classes will certainly become Valhalla value types (hence the required @JvmInline annotation). - Nullable types are supported with compile-time checks and additional runtime checks inserted by the compiler at time-of-use, and if Valhalla adds a nullability marker to the bytecode, Kotlin can support it. - Coroutines/structured-concurrency already uses Executors under the hood, so it's likely that we'll have Loom thread support without any changes to the coroutines APIs (though the standard dispatchers will need to use virtual threads themselves). Coroutines themselves are CPS-transforms of ordinary code and don't need VM support. - Tail-call elimination is supported by transforming to trampolines, something which the JVM will probably never support until Java needs it.

exabrial 4y ago

I need to learn to use jfr apparently. Just something I’ve known existed and never used. We attach a hawtio.io jvm agent to every single prod jvm we run and query the agent using telegraph. (If you don’t do this by the way, you should!) I noticed in the GUI portion of hawtio, it slows you to start/stop a jfr recording, but I’ve never messed with it

Scarbutt 4y ago

HN being very anti Java, why does such many Java articles make it to the front page?

mqus 4y ago
Maybe the commenters are "anti" and the voters aren't (more comments being also a detriment to getting to the frontpage)? But I don't see any real "anti" comments, only valid criticism and I say that as a Java dev myself. It does have its wrinkles. But for it's age and popularity it's still a pretty good language.
ncann 4y ago
That's not my experience at all, if anything I feel like HN is quite fond of Java. The anti Java crowd is mostly on Reddit instead where it's the subject of memes and jokes and the like.
- lenkite 4y ago
  Except /r/java where Java is God
  - arnold_palmur 4y ago
    I think it's safe to assume that as a general rule on /r/<X>, <X> is god.
  - re-thc 4y ago
    There are lots of people in /r/java that defy God :p
exabrial 4y ago
A lot of it is misdirected hate. In the first dot.com boom, a lot of people worked for tech companies, and Microsoft was the Google|Apple back then. Their corporate culture was dog-eat-dog nonsense, but it trickled down through the industry... and of course at the time Java was super super popular, even though _it really sucked_ back then (not so much now, it's quite awesome). One of the most _hated_ positions was "The Architect" who usually rose to the top via floating on top of a bubble of attrition; so rather than being promoted because you're good at leadership or a great person that's fun to be around or a technically savvy and loved to teach... these architects were ignorant egotistical maniacal dictators that got promoted merely because everyone else quit. They turned static analysis tools into weapons of war. They made arbitrary standards that really didn't work for anyone, rather than trying to work for someone. Everything had to be over-engineered (sound familiar? If not, look at today's SPAs) with AbstractFactoryProducerMethodProducerImplInterfaces. I worked at a lot of these companies, and it just wasn't a fun time to be in software development. Rather than people seeing the reality that Poor Leadership and zero charisma as the root cause, somehow people arrived at Java was the core problem. A lot of the things were good in their intention; for instance, having the whole company use tabs instead of spaces (or vice versa) and having a universal code formatter have the potential to increase communication. But instead, it was just executed poorly at nearly every shop and the vast majority of the problems got turned into Dilbert comics. But yeah, Java.

sgt 4y ago

For a split second there, I thought a post-airline accident flight recorder had some Java heap dumps that showed major issues and thread leaks.

tmd83 4y ago

JFR is nice but I can't seem to find issues as easily with the new JMC as I used to 5.5 or before. I'm not sure what I'm doing wrong though.