4 ms·

I concur. These results would have been better if they'd been generated using the native GC logging format and then run through an analysis tool. Similarly, th

by dpratt 11y ago

I concur. These results would have been better if they'd been generated using the native GC logging format and then run through an analysis tool. Similarly, the article mentions that the synthetic benchmark was run for only a minute. I am heavily suspicious of this - any results you get here will be useless until HotSpot has had a bit of time to optimize the running code. I heavily suspect that the G1 throughput would have been significantly higher if the benchmark had been allowed to run longer.

iheartmemcache 11y ago
Not only did he not give the Hotspot time to warm up but we have no idea what the overhead of the Dropwizard stack is (i.e., if its heavily IO dependent that's going to obviously shift things), what Application<T> looks like, what the emitted code is. If it's anything like Clj, the overhead of the package runtime is not insignificant. The native GC logging is way better than using log4j (which is great for logging, but abymsal in situations like this for obvious reasons...). He's on OSX, if he wanted to conduct this with even the slightest bit of rigor he would have used Dtrace and probed a few application runs with appropriately placed probes. Like it's already been mentioned, he should be using whatever the JDK (or ideally JMH, IME) gives him to bench rather than that log4j derivative, because that could potentially introduce disk seek variance between the two GCs (i.e one could be simply buffering, then capturing stdout's FD, and flushing -- rather than having 60 very significant potential 'get FD, (potentially block until you can acquire a mutex on it; then potentially wait to seek out) , write, flush/sync' while the other one might have just done it once (as I have no idea about the the library he's using to log, nor have I used an OS X file system in production (unless it uses UFS1 in which case I when I was 13 or 14 for a bit). He's writing to a tty so that's probably not going to impact performance too much I'd imagine, but if you've real-time cat'd a 200 meg file out to a tty, you'll see that the benchmark is actually output to the screen rather than the throughput of the disk (again, not sure why, probably has a lot to do with the X11/Xorg frame-buffer being slammed and/or the bloat of the term you're using (you'd absolutely see variance between something minimal like xterm + sh, vs gnome-terminal + zsh)). Anyways, yeah, he should have read some Oracle docs to minimize the number of independent variables between his two cases to make this even moderately definitive. (Also I'm baffled why he even put in any IO what so ever if the whole point was to bench the 4 different modes of GC...). Anyways, instead of tearing apart his post by pointing out the complete lack of rigor and experimental design foresight he exhibited, something fun to do is to take the Sun JVM, IBM J9, OpenJDK, Graal (or something along the lines of that) and Zulu (if you can scare up a Zulu license you're in for a pretty interesting treat). It's amazing how one IL can yield so many different performance results.
- hyperpape 11y ago
  I don't see anything about log4j, he's using the JVM built in GC Logging, and can you explain the dtrace reference? Wouldn't think it's what you'd use for Java GC analysis. Is this some kind of odd sarcasm?
  - iheartmemcache 11y ago
    Absolutely nothing sarcastic about it. RE: Log4j. I haven't written Java professionally in almost a decade, so my mind substitude "something similar to log4j" in. Slf4j, log4j 2.0, both libs serve (as I understand it, am I horribly off base here?) the same purpose -- your kernel will be forced to make a syscall to write, in this case to his stdout, introducing unnecessary variance when your goal is to contrast two states of GC. RE: Dtrace is a fantastic tool for making granular benchmarks. Again, theres nothing sarcastic about it. As I previously mentioned here JMH would be ideal, confirmed by Oracle[1] as the method to use. (Read the second section, starting with a few paragraphs up from Listing 6.) He basically gives a prescriptive method by which one can use the same tool I mentioned to combat the same issues I brought up, especially for situations like this (micro-benching). DTrace would take it one step further (again, assuming you're skilled enough to use it appropriately) -- you'd end up getting effectively a caliber of measurements on par with Intel PIN. I suppose drastic overkill for this situation, but no, there was nothing sarcastic (nor incorrect, as far as I can tell) about anything I said. Edit: As I re-read my response to him, I'm baffled as how you can even think this was some sort of sarcastic jab. The dude made an awful post, and I tried to diplomatically raise a few of _many_ issues with his method. I guess I was being a bit of a dick to him by just hammering him away. He's probably just some 19 year old and I should have been constructive instead by spending 15 seconds to search for one of the plethora of sources available on how to properly benchmark the variance of different styles of garbage collection on stack-based VMs. I suppose my foul mood and an easy blog post as a target will yield make for poor manners. Apologies, OP [1] http://www.oracle.com/technetwork/articles/java/architect-benchmarking-2266277.html http://www.oracle.com/technetwork/articles/java/architect-be...
    - hyperpape 11y ago

by dpratt 11y ago

iheartmemcache 11y ago

Not only did he not give the Hotspot time to warm up but we have no idea what the overhead of the Dropwizard stack is (i.e., if its heavily IO dependent that's going to obviously shift things), what Application<T> looks like, what the emitted code is. If it's anything like Clj, the overhead of the package runtime is not insignificant. The native GC logging is way better than using log4j (which is great for logging, but abymsal in situations like this for obvious reasons...). He's on OSX, if he wanted to conduct this with even the slightest bit of rigor he would have used Dtrace and probed a few application runs with appropriately placed probes. Like it's already been mentioned, he should be using whatever the JDK (or ideally JMH, IME) gives him to bench rather than that log4j derivative, because that could potentially introduce disk seek variance between the two GCs (i.e one could be simply buffering, then capturing stdout's FD, and flushing -- rather than having 60 very significant potential 'get FD, (potentially block until you can acquire a mutex on it; then potentially wait to seek out) , write, flush/sync' while the other one might have just done it once (as I have no idea about the the library he's using to log, nor have I used an OS X file system in production (unless it uses UFS1 in which case I when I was 13 or 14 for a bit). He's writing to a tty so that's probably not going to impact performance too much I'd imagine, but if you've real-time cat'd a 200 meg file out to a tty, you'll see that the benchmark is actually output to the screen rather than the throughput of the disk (again, not sure why, probably has a lot to do with the X11/Xorg frame-buffer being slammed and/or the bloat of the term you're using (you'd absolutely see variance between something minimal like xterm + sh, vs gnome-terminal + zsh)). Anyways, yeah, he should have read some Oracle docs to minimize the number of independent variables between his two cases to make this even moderately definitive. (Also I'm baffled why he even put in any IO what so ever if the whole point was to bench the 4 different modes of GC...). Anyways, instead of tearing apart his post by pointing out the complete lack of rigor and experimental design foresight he exhibited, something fun to do is to take the Sun JVM, IBM J9, OpenJDK, Graal (or something along the lines of that) and Zulu (if you can scare up a Zulu license you're in for a pretty interesting treat). It's amazing how one IL can yield so many different performance results.

hyperpape 11y ago
I don't see anything about log4j, he's using the JVM built in GC Logging, and can you explain the dtrace reference? Wouldn't think it's what you'd use for Java GC analysis. Is this some kind of odd sarcasm?
- iheartmemcache 11y ago
  Absolutely nothing sarcastic about it. RE: Log4j. I haven't written Java professionally in almost a decade, so my mind substitude "something similar to log4j" in. Slf4j, log4j 2.0, both libs serve (as I understand it, am I horribly off base here?) the same purpose -- your kernel will be forced to make a syscall to write, in this case to his stdout, introducing unnecessary variance when your goal is to contrast two states of GC. RE: Dtrace is a fantastic tool for making granular benchmarks. Again, theres nothing sarcastic about it. As I previously mentioned here JMH would be ideal, confirmed by Oracle[1] as the method to use. (Read the second section, starting with a few paragraphs up from Listing 6.) He basically gives a prescriptive method by which one can use the same tool I mentioned to combat the same issues I brought up, especially for situations like this (micro-benching). DTrace would take it one step further (again, assuming you're skilled enough to use it appropriately) -- you'd end up getting effectively a caliber of measurements on par with Intel PIN. I suppose drastic overkill for this situation, but no, there was nothing sarcastic (nor incorrect, as far as I can tell) about anything I said. Edit: As I re-read my response to him, I'm baffled as how you can even think this was some sort of sarcastic jab. The dude made an awful post, and I tried to diplomatically raise a few of _many_ issues with his method. I guess I was being a bit of a dick to him by just hammering him away. He's probably just some 19 year old and I should have been constructive instead by spending 15 seconds to search for one of the plethora of sources available on how to properly benchmark the variance of different styles of garbage collection on stack-based VMs. I suppose my foul mood and an easy blog post as a target will yield make for poor manners. Apologies, OP [1] http://www.oracle.com/technetwork/articles/java/architect-benchmarking-2266277.html http://www.oracle.com/technetwork/articles/java/architect-be...
  - hyperpape 11y ago