4 ms·

Superduper Slow Jar Command (embarassing bug in the jar code)

28 points by ks 17y ago

jganetsk 17y ago
I can't believe that no one's discovered this yet. Maybe this is a manifestation of the Python paradox. The kind of people using the jar command (Java developers) are the kind of people who just accept the limitations of their systems without questioning.
- patio11 17y ago
  The kind of people using the jar command (Java developers) are the kind of people who just accept the limitations of their systems without questioning. I've been a Java programmer for, yikes, is it ten years now? Of that decade, perhaps 45 minutes has been spent waiting for jar to finish. Tell me, does it sound like a productive use of my time to go hunting for an optimization to cut 30 minutes off of that? If I were to wake up in the morning one day and say "You know, instead of implementing features customers actually want, I think today I'm going to make fast things faster", I'd start by targeting things which happen frequently, take a lot of time, and are on the critical path for my users. JAR is oh for three. For my business, I have to run it once per release candidate, or about eight times a year, for a total cost of two minutes. Even at my day job, since you can restart development servers without re-jarring everything, we only have to re-jar once per every restart of the production servers, a process which is rate limited by collecting approval on paper for the restart anyway.
  - jganetsk 17y ago
    I totally agree in that I don't expect anyone to get up in the morning and make their mission-of-the-day the optimization of jar. I am just surprised that no one has noticed jar behaving strangely. As a developer, I am frequently on the lookout for programs that hit 100% CPU utilization. This is a very useful indicator. I have top running all the time. I don't run jar frequently... but if I did and it did hit 100% CPU, I would be curious indeed. I have been at a new job for 2 weeks, using Java EE. My IDE and my application use tons of memory, and I am often waiting for it to respond. Why is it paused? My coworkers just say "garbage collection". But they don't know if that is, in fact, what's happening. If you see jar running at 100%, do you think "oh it's just garbage collection", or "a bug, perhaps?" Considering the fact that, there are millions of Java developers out there, that open source development has proven true the mantra "with enough eyes, all bugs are shallow", and that this jar bug has potentially been around for years... I am just surprised. Also, as much as I would love to extrapolate meaning from your single anecdote, I suspect that some people out there are running jar a lot. In fact, the blog post's writer mentions that he was trying to address actual complaints of the user base. It is a "well-known" fact that jar is slow.
    - ankhmoop 17y ago
      I can't believe that no one's discovered this yet. Maybe this is a manifestation of the Python paradox. The kind of people using the jar command (Java developers) are the kind of people who just accept the limitations of their systems without questioning. Your argument is pretentious, arrogant, and ultimately clueless. For small builds, jar is sufficiently fast as to not raise any red flags. For large builds, jar overhead is small enough as to be easily lost in the noise. The build should have been profiled, and this is an embarrassing bug, but it's absolutely, completely ridiculous to generalize this to the fact that Java developers could be the "kind of people who just accept the limitations of their systems without questioning".
      - fsniper 17y ago
        Thinking of how broadly java and java build tools are used, cumulative losses of world is "remarkable" in economical and ecological terms.
    - gojomo 17y ago
      Teams that run jar all the time probably do so in an asynchronous auto-build. It's already a process that runs 'many minutes' and you check back on later. So the delay is already mitigated, and the benefit less than the raw speedup would suggest. (And if lightning-fast builds were project-critical, even after this fix you'd switch to native zip.) Still, I'm glad this is fixed and wouldn't mind if Sun or other big Java vendors had a few interns/students just roaming legacy code looking for cheap wins like this.
  - Daniel_Newby 17y ago
    I'd say that is a big honking "yes" to jganetsk's question. The acceptance of the build system limitations is so uncritical that you don't even try to build more than a few times a year. Which means that you don't do regression localization by doing an automated binary search of the total revision history of the complete product. ("Oh, but touching the complete product is rate limited by paperwork." [sob]) Having the build system hand you the exact delta that introduced a subtle bug once in your product's lifetime would pay for rather a lot of build optimization.
    - pohl 17y ago
      A jar is a packaging mechanism. It is not required step in most builds that I've encountered. Nor is it required to run (test) the code. Java classes can also be linked into a CLASSPATH from an unpackaged directory. Infrequent packaging does not imply infrequent building and/or testing.
- nradov 17y ago
  Most jar creation is done on automated build servers that aren't closely monitored. I can't remember the last time I ran the command line tool to create a jar file on my own workstation. So it's not surprising that no one noticed.
- gojomo 17y ago
  Another factor: this tool is part of the sun.* packages, for which source has not historically been available (even though other java.* system libraries source has long been available). So even now, with my local jdk1.6.0_10 package, I can't easily browse the JAR code with the problem. Had Sun revealed all source sooner people outside Sun might've discovered this years ago.
iigs 17y ago
Sorry to be naive about java, but isn't .jar just .zip with a manifest and a different extension? Even if the goal is to make jar "100% pure java", which I don't recall it being, why wouldn't they transcribe a relatively complete implementation of .zip from the get-go? A relatively complete, solid, performant implementation of what is probably the world's most common compression algorithm seems like exactly the kind of thing that belongs in Java's enormous library system. Maybe typical jar duties wouldn't tax it but it seems like someone somewhere would use it for something big and these kinds of issues would have shaken out long ago.
- nradov 17y ago
  The JSE library has had zip support since version 1.1. http://java.sun.com/javase/6/docs/api/java/util/zip/package-summary.html http://java.sun.com/javase/6/docs/api/java/util/zip/package-... The jar performance defect described in that article is apparently in the application, not the library.
Hexstream 17y ago
I don't know if there are comparable Windows and Mac equivalents, but with Ubuntu I have this nice "Gnome system monitor" that shows me CPU, Memory, Network, Load Average and Disk IO usage % in real-time at all times. I can't imagine living without it. There's no way I couldn't notice something taking 100% CPU for a long time while not writing to disk when it ought to start writing to disk immediately. It's part of the reason I notice and hate flash stuff that takes 100% CPU for no reason at all. (but I'd notice anyway because my fan gets noisier real fast)
- halo 17y ago
  GNOME System Monitor is largely a clone of Windows Task Manager, which is accessible via the shortcut Ctrl+Shift+Esc (or via the Ctrl+Alt+Del menu) on any Windows system. Mac OS X also has Activity Monitor which serves a similar purpose.
  - Hexstream 17y ago
    I'm not sure we're talking about the same thing. I'm talking about a real-time monitor of all of CPU, RAM, Bandwidth, Load average and Disk I/O (you can show even more or less) that sits in the menu bar. Windows only shows CPU like that, and it works really badly because if your CPU spikes to 100% it momentarily stops updating. There's simply no comparison between the two.
- brown 17y ago
  I highly recommend Process Explorer (http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx http://technet.microsoft.com/en-us/sysinternals/bb896653.asp...). It is like the default Windows Task Manager on steroids. It is the tool most preferred by Microsoft devs. It is very lightweight and powerful. I have long hoped for a Process Explorer clone for linux (maybe a side project for someone with lots of free time...).
  - moe 17y ago
    I have long hoped for a Process Explorer clone for linux It's called 'top'.
tezza 17y ago
Yuck! I've been using ant scripts which jar up dependencies for years. Other people will be even more impacted than me, like shops that use continuous builds ala CruiseControl. I always wondered why it was so slow, I thought my OS was having to page the files back into memory inefficiently (too little ram cache). Or my disk was too slow. SSD write was so slow that builds weren't improved, so this will be nice to compensate. It sucks that I've been wasting that much time, but it is great that I'm about to see a real speed up.

28 points by ks 17y ago

I can't believe that no one's discovered this yet. Maybe this is a manifestation of the Python paradox. The kind of people using the jar command (Java developers) are the kind of people who just accept the limitations of their systems without questioning.

patio11 17y ago
The kind of people using the jar command (Java developers) are the kind of people who just accept the limitations of their systems without questioning. I've been a Java programmer for, yikes, is it ten years now? Of that decade, perhaps 45 minutes has been spent waiting for jar to finish. Tell me, does it sound like a productive use of my time to go hunting for an optimization to cut 30 minutes off of that? If I were to wake up in the morning one day and say "You know, instead of implementing features customers actually want, I think today I'm going to make fast things faster", I'd start by targeting things which happen frequently, take a lot of time, and are on the critical path for my users. JAR is oh for three. For my business, I have to run it once per release candidate, or about eight times a year, for a total cost of two minutes. Even at my day job, since you can restart development servers without re-jarring everything, we only have to re-jar once per every restart of the production servers, a process which is rate limited by collecting approval on paper for the restart anyway.
- jganetsk 17y ago
  I totally agree in that I don't expect anyone to get up in the morning and make their mission-of-the-day the optimization of jar. I am just surprised that no one has noticed jar behaving strangely. As a developer, I am frequently on the lookout for programs that hit 100% CPU utilization. This is a very useful indicator. I have top running all the time. I don't run jar frequently... but if I did and it did hit 100% CPU, I would be curious indeed. I have been at a new job for 2 weeks, using Java EE. My IDE and my application use tons of memory, and I am often waiting for it to respond. Why is it paused? My coworkers just say "garbage collection". But they don't know if that is, in fact, what's happening. If you see jar running at 100%, do you think "oh it's just garbage collection", or "a bug, perhaps?" Considering the fact that, there are millions of Java developers out there, that open source development has proven true the mantra "with enough eyes, all bugs are shallow", and that this jar bug has potentially been around for years... I am just surprised. Also, as much as I would love to extrapolate meaning from your single anecdote, I suspect that some people out there are running jar a lot. In fact, the blog post's writer mentions that he was trying to address actual complaints of the user base. It is a "well-known" fact that jar is slow.
  - ankhmoop 17y ago
    I can't believe that no one's discovered this yet. Maybe this is a manifestation of the Python paradox. The kind of people using the jar command (Java developers) are the kind of people who just accept the limitations of their systems without questioning. Your argument is pretentious, arrogant, and ultimately clueless. For small builds, jar is sufficiently fast as to not raise any red flags. For large builds, jar overhead is small enough as to be easily lost in the noise. The build should have been profiled, and this is an embarrassing bug, but it's absolutely, completely ridiculous to generalize this to the fact that Java developers could be the "kind of people who just accept the limitations of their systems without questioning".
    - fsniper 17y ago
      Thinking of how broadly java and java build tools are used, cumulative losses of world is "remarkable" in economical and ecological terms.
  - gojomo 17y ago
    Teams that run jar all the time probably do so in an asynchronous auto-build. It's already a process that runs 'many minutes' and you check back on later. So the delay is already mitigated, and the benefit less than the raw speedup would suggest. (And if lightning-fast builds were project-critical, even after this fix you'd switch to native zip.) Still, I'm glad this is fixed and wouldn't mind if Sun or other big Java vendors had a few interns/students just roaming legacy code looking for cheap wins like this.
- Daniel_Newby 17y ago
  I'd say that is a big honking "yes" to jganetsk's question. The acceptance of the build system limitations is so uncritical that you don't even try to build more than a few times a year. Which means that you don't do regression localization by doing an automated binary search of the total revision history of the complete product. ("Oh, but touching the complete product is rate limited by paperwork." [sob]) Having the build system hand you the exact delta that introduced a subtle bug once in your product's lifetime would pay for rather a lot of build optimization.
  - pohl 17y ago
    A jar is a packaging mechanism. It is not required step in most builds that I've encountered. Nor is it required to run (test) the code. Java classes can also be linked into a CLASSPATH from an unpackaged directory. Infrequent packaging does not imply infrequent building and/or testing.
nradov 17y ago
Most jar creation is done on automated build servers that aren't closely monitored. I can't remember the last time I ran the command line tool to create a jar file on my own workstation. So it's not surprising that no one noticed.
gojomo 17y ago
Another factor: this tool is part of the sun.* packages, for which source has not historically been available (even though other java.* system libraries source has long been available). So even now, with my local jdk1.6.0_10 package, I can't easily browse the JAR code with the problem. Had Sun revealed all source sooner people outside Sun might've discovered this years ago.

iigs 17y ago

Sorry to be naive about java, but isn't .jar just .zip with a manifest and a different extension? Even if the goal is to make jar "100% pure java", which I don't recall it being, why wouldn't they transcribe a relatively complete implementation of .zip from the get-go? A relatively complete, solid, performant implementation of what is probably the world's most common compression algorithm seems like exactly the kind of thing that belongs in Java's enormous library system. Maybe typical jar duties wouldn't tax it but it seems like someone somewhere would use it for something big and these kinds of issues would have shaken out long ago.

nradov 17y ago
The JSE library has had zip support since version 1.1. http://java.sun.com/javase/6/docs/api/java/util/zip/package-summary.html http://java.sun.com/javase/6/docs/api/java/util/zip/package-... The jar performance defect described in that article is apparently in the application, not the library.

Hexstream 17y ago

I don't know if there are comparable Windows and Mac equivalents, but with Ubuntu I have this nice "Gnome system monitor" that shows me CPU, Memory, Network, Load Average and Disk IO usage % in real-time at all times. I can't imagine living without it. There's no way I couldn't notice something taking 100% CPU for a long time while not writing to disk when it ought to start writing to disk immediately. It's part of the reason I notice and hate flash stuff that takes 100% CPU for no reason at all. (but I'd notice anyway because my fan gets noisier real fast)

halo 17y ago
GNOME System Monitor is largely a clone of Windows Task Manager, which is accessible via the shortcut Ctrl+Shift+Esc (or via the Ctrl+Alt+Del menu) on any Windows system. Mac OS X also has Activity Monitor which serves a similar purpose.
- Hexstream 17y ago
  I'm not sure we're talking about the same thing. I'm talking about a real-time monitor of all of CPU, RAM, Bandwidth, Load average and Disk I/O (you can show even more or less) that sits in the menu bar. Windows only shows CPU like that, and it works really badly because if your CPU spikes to 100% it momentarily stops updating. There's simply no comparison between the two.
brown 17y ago
I highly recommend Process Explorer (http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx http://technet.microsoft.com/en-us/sysinternals/bb896653.asp...). It is like the default Windows Task Manager on steroids. It is the tool most preferred by Microsoft devs. It is very lightweight and powerful. I have long hoped for a Process Explorer clone for linux (maybe a side project for someone with lots of free time...).
- moe 17y ago
  I have long hoped for a Process Explorer clone for linux It's called 'top'.

tezza 17y ago

Yuck! I've been using ant scripts which jar up dependencies for years. Other people will be even more impacted than me, like shops that use continuous builds ala CruiseControl. I always wondered why it was so slow, I thought my OS was having to page the files back into memory inefficiently (too little ram cache). Or my disk was too slow. SSD write was so slow that builds weren't improved, so this will be nice to compensate. It sucks that I've been wasting that much time, but it is great that I'm about to see a real speed up.