3 ms·

> How do you compile a Java library that uses native code? Python/Node/Ruby worlds know how, but OpenJDK ignores the question. Because you compile it like any

by pron 3y ago

> How do you compile a Java library that uses native code? Python/Node/Ruby worlds know how, but OpenJDK ignores the question. Because you compile it like any other Java library. > How do you ship a Java library that uses native code? Python/Node/Ruby worlds know, but OpenJDK ignores the question. It doesn't. jmod files and the jmod tool were added in JDK 9 precisely for that (and other things, too). > If even the well funded OpenJDK developers working in Java can't outcompete the poorly funded OpenSSL devs working in an unproductive language, then what chance do other Java library devs have? None at all! People aren't really rewriting OpenSSL in Python or JS, either, but high-level languages are many times more popular than low-level ones [1], and you're misreading the normal behaviour of a fragmented market, when it's worth it to compete and over what. I don't think any language today has a healthier library ecosystem than Java. That's not to say it's as good as wish it would be, but no one seems to do it better. There is, however, a problem with build tools not supporting newer JDK features (that are coming faster now than before) and we'll need to do something about that. [1]: https://www.devjobsscanner.com/blog/top-8-most-demanded-programming-languages/ https://www.devjobsscanner.com/blog/top-8-most-demanded-prog...

geodel 3y ago
> There is, however, a problem with build tools not supporting newer JDK features This is important point. From my experience in Enterprise IT the build pipeline is roughly at JDK-8 level in terms of generating deployable artifacts. I have contorted by maven build config to invoke jlink on builder nodes to make standalone artifacts that won't need JDK on target machines. But it is not optimal for few reasons 1) I have to manually track what JDK modules I need to include in build. 2) Its hard to get new JDK installed on build machines for ones, forget the dot upgrades to keep it latest bug fix level. Their reasoning is build machine is just to invoke maven build. Only target machines need latest patched JDKs. 3) From JDK-8 they are now directly jumping to make docker/kubernetetes builds which are fine but totally different direction if I am just looking to make no-fuss standalone Java application to run on a Linux VM. As one can see some of these are organizational issues not necessarily technical one. But as single developer with no power to force IT infra teams on supporting alternate workflows I am basically stuck with JDK-8 workflows. What is missing in setup is somehow to tell maven I want to generate a standalone Java app using jlink for linux ver.N target so gather up all modules / java binary and other tools required based on information in this pom file without relying on JDK version of builder . This should be able to run builder machine so it remains integrated with current build pipleline.
pron 3y ago
P.S. Oracle's Java Platform Group (JPG), the group behind OpenJDK, had nothing to do with Jipher, but the number of people who contributed to OpenSSL in the past year alone is roughly the same number as the people working on Project Loom, and rewriting OpenSSL from scratch would have obviously required more people than those required for OpenSSL maintenance, so it would have been a project roughly the size of Loom. If it were to actually do the job, it would also require a commitment to long-term maintenance (something that's often missing from various rewrite attempts). I don't think it's necessarily the best use of resources, at least by us, and I don't think it's sad at all that we're investing our resources in things with better bang-for-the-buck. I think that projects like Loom and Panama and Leyden have a much better ROI than an OpenSSL rewrite. Also, most native libraries work perfectly fine on virtual threads and with no degradation in performance. The only thing to watch out for is frequent blocking in native code.
- nvm0n2 3y ago
  But Java already has a cryptography stack (including TLS) in pure Java shipped as part of the JDK which has been maintained for years. So manpower isn't the issue, the problem is that Java code can't be easily used by anything else. The awkwardness of JNI cuts in both directions. It could be solved if there was a way to generate a shared library from Java code with an auto-generated reasonable-ish C API. Then people could take the Java SSL code (+bouncy castle or whatever) and implement the OpenSSL API. Ditto for lots of other libs. But as no such thing exists, everyone gathers around the C impl because that way you get the most stakeholders.
  - pron 3y ago
    > The awkwardness of JNI cuts in both directions. On the consumer side FFM solves that; not awkward anymore! On the host side, well, even Go libraries (and I mention Go because I think Go is considered by some as being lower level than Java for reasons passing understanding) aren't easily shareable by other languages. > It could be solved if there was a way to generate a shared library from Java code with an auto-generated reasonable-ish C API. But why is that a problem worth spending our time on considering that C programmers probably won't appreciate hosting a JVM for that, and that Java is more popular than C as it is? No language is trying to control everything these days. The anomalous and short era of single language dominance is over, at least for a while. Java isn't the one dominant language, but neither is any other. Spending effort to go after a relatively small market share seems like a miscalculation. The market of popular open-source C libraries is both very important and minuscule. It would be like saying that iOS should focus its resources on going after the home-brew Raspberry Pi market. > But as no such thing exists, everyone gathers around the C impl because that way you get the most stakeholders. Why is that a problem, though? Libraries that are shared across languages and runtimes are very important, but their number is also very small. Big and foundational are two very different things in software, and we're in an era of specialisation, not consolidation. Don't confuse "a problem" with "a problem worth solving".
sitta 3y ago
>> How do you ship a Java library that uses native code? Python/Node/Ruby worlds know, but OpenJDK ignores the question. >It doesn't. jmod files and the jmod tool were added in JDK 9 precisely for that (and other things, too). I have never seen a lib package native dependencies using jmod. It's not even clear to me how this would be done. Everyone I'm familiar with bundles binaries inside the jar. Can you point to an example? I would love to learn a better way of doing this.
- pron 3y ago
  That's what JavaFX does (https://openjfx.io/openjfx-docs/#modular https://openjfx.io/openjfx-docs/#modular), but the problem is that Maven and Gradle don't support that well, which is why few libraries do that. The idea is that an application is best deployed with a custom runtime created with jlink (well, all Java runtime these days are created by jlink and so every Java program uses jlink whether it knows it or not, but few use it directly because, well, Maven and Gradle don't support jlink well, and so few applications use jlink well), and jmod files are jlink's input from which it generates the image. Anyway, build tools aside, a library can be distributed as a jmod file, consuming it with jlink places the native libraries in the right place, and that's about it.
  - deleted 3y ago
    [deleted]
  - nvm0n2 3y ago
    JavaFX doesn't use jmods, not really. Try following the tutorials for getting started and see for yourself: you end up using custom JavaFX build system plugins that download jar versions of the modules, not jmods. Also some alternative JDK vendors pre-ship JavaFX in their JDK spins to avoid people having to use jlink. I've worked with jlink extensively. The problems are greater than just Maven and Gradle, which at any rate both have quite sophisticated plugins for working with it. There was just a major error at the requirements analysis phase in how this part of Java works. Problems: 1. You can't put jmods on the module path, even though it would be easily implemented. They can only be used as inputs to jlink. But this is incompatible with how the JVM world works (not just Maven and Gradle), as it is expected to be able to download libraries and place them on the class/module path in combination with a generic runtime in order to use them. Changing that would mean every project to get its own sub-JDK created at build time, which would explode build times and disk space usage (jlink is very slow). So nobody does it, and no build system supports it. 2. When native code gets involved they are platform specific, even though (again) this could have been easily avoided. Even if JVMs supported jmods just like jars, no build system properly supports non-portable jars because the whole point of a jar is that it's portable. It has to be hacked around with custom plugins, which is a very high effort and non-scalable approach. 3. jlink doesn't understand the classpath at all. It thinks every jar/jmod is a module, but in reality very few libraries are usable on the module path (some look usable until you try it and discover their module metadata is broken). So you can't actually produce a standalone app directory with a bundled JVM using jlink because every real app uses non-modularized JARs. You end up needing custom launchers and scripts that try to work out what modules are needed using jlink. 4. The goal of the module system was to increase reliability, but it actually made it lower because there are some common cases where the module system doesn't detect that some theoretically optional modules are actually needed, even in the case of a fully modularized app. For example apps that try to use jlink directly are prone to experiencing randomly broken HTTP connections and (on desktop) mysterious accessibility failures, caused by critical features being optional and loaded dynamically, so they have to be forced into the image using special jlink flags. This is a manhole sized trap to fall into and isn't documented anywhere, there are no warnings printed. You are just expected to ship broken apps, find out the hard way what went wrong and then fix it by hand. At some point the JDK developers need to stop pointing the finger at build tool providers when it comes to Jigsaw adoption. It's not like Maven and Gradle are uniquely deviant. There are other build systems used in the JVM world and not one of them works the way the OpenJDK devs would like. They could contribute patches to change these things upstream, or develop their own build system, but have never done it so we end up with a world where there are ways to distribute libraries that work fine if you pretend people never moved beyond 'make' and checking jars into version control.

by pron 3y ago

geodel 3y ago

> There is, however, a problem with build tools not supporting newer JDK features This is important point. From my experience in Enterprise IT the build pipeline is roughly at JDK-8 level in terms of generating deployable artifacts. I have contorted by maven build config to invoke jlink on builder nodes to make standalone artifacts that won't need JDK on target machines. But it is not optimal for few reasons 1) I have to manually track what JDK modules I need to include in build. 2) Its hard to get new JDK installed on build machines for ones, forget the dot upgrades to keep it latest bug fix level. Their reasoning is build machine is just to invoke maven build. Only target machines need latest patched JDKs. 3) From JDK-8 they are now directly jumping to make docker/kubernetetes builds which are fine but totally different direction if I am just looking to make no-fuss standalone Java application to run on a Linux VM. As one can see some of these are organizational issues not necessarily technical one. But as single developer with no power to force IT infra teams on supporting alternate workflows I am basically stuck with JDK-8 workflows. What is missing in setup is somehow to tell maven I want to generate a standalone Java app using jlink for linux ver.N target so gather up all modules / java binary and other tools required based on information in this pom file without relying on JDK version of builder . This should be able to run builder machine so it remains integrated with current build pipleline.

pron 3y ago

P.S. Oracle's Java Platform Group (JPG), the group behind OpenJDK, had nothing to do with Jipher, but the number of people who contributed to OpenSSL in the past year alone is roughly the same number as the people working on Project Loom, and rewriting OpenSSL from scratch would have obviously required more people than those required for OpenSSL maintenance, so it would have been a project roughly the size of Loom. If it were to actually do the job, it would also require a commitment to long-term maintenance (something that's often missing from various rewrite attempts). I don't think it's necessarily the best use of resources, at least by us, and I don't think it's sad at all that we're investing our resources in things with better bang-for-the-buck. I think that projects like Loom and Panama and Leyden have a much better ROI than an OpenSSL rewrite. Also, most native libraries work perfectly fine on virtual threads and with no degradation in performance. The only thing to watch out for is frequent blocking in native code.

nvm0n2 3y ago
But Java already has a cryptography stack (including TLS) in pure Java shipped as part of the JDK which has been maintained for years. So manpower isn't the issue, the problem is that Java code can't be easily used by anything else. The awkwardness of JNI cuts in both directions. It could be solved if there was a way to generate a shared library from Java code with an auto-generated reasonable-ish C API. Then people could take the Java SSL code (+bouncy castle or whatever) and implement the OpenSSL API. Ditto for lots of other libs. But as no such thing exists, everyone gathers around the C impl because that way you get the most stakeholders.
- pron 3y ago
  > The awkwardness of JNI cuts in both directions. On the consumer side FFM solves that; not awkward anymore! On the host side, well, even Go libraries (and I mention Go because I think Go is considered by some as being lower level than Java for reasons passing understanding) aren't easily shareable by other languages. > It could be solved if there was a way to generate a shared library from Java code with an auto-generated reasonable-ish C API. But why is that a problem worth spending our time on considering that C programmers probably won't appreciate hosting a JVM for that, and that Java is more popular than C as it is? No language is trying to control everything these days. The anomalous and short era of single language dominance is over, at least for a while. Java isn't the one dominant language, but neither is any other. Spending effort to go after a relatively small market share seems like a miscalculation. The market of popular open-source C libraries is both very important and minuscule. It would be like saying that iOS should focus its resources on going after the home-brew Raspberry Pi market. > But as no such thing exists, everyone gathers around the C impl because that way you get the most stakeholders. Why is that a problem, though? Libraries that are shared across languages and runtimes are very important, but their number is also very small. Big and foundational are two very different things in software, and we're in an era of specialisation, not consolidation. Don't confuse "a problem" with "a problem worth solving".

sitta 3y ago

>> How do you ship a Java library that uses native code? Python/Node/Ruby worlds know, but OpenJDK ignores the question. >It doesn't. jmod files and the jmod tool were added in JDK 9 precisely for that (and other things, too). I have never seen a lib package native dependencies using jmod. It's not even clear to me how this would be done. Everyone I'm familiar with bundles binaries inside the jar. Can you point to an example? I would love to learn a better way of doing this.

pron 3y ago
That's what JavaFX does (https://openjfx.io/openjfx-docs/#modular https://openjfx.io/openjfx-docs/#modular), but the problem is that Maven and Gradle don't support that well, which is why few libraries do that. The idea is that an application is best deployed with a custom runtime created with jlink (well, all Java runtime these days are created by jlink and so every Java program uses jlink whether it knows it or not, but few use it directly because, well, Maven and Gradle don't support jlink well, and so few applications use jlink well), and jmod files are jlink's input from which it generates the image. Anyway, build tools aside, a library can be distributed as a jmod file, consuming it with jlink places the native libraries in the right place, and that's about it.
- deleted 3y ago
  [deleted]
- nvm0n2 3y ago
  JavaFX doesn't use jmods, not really. Try following the tutorials for getting started and see for yourself: you end up using custom JavaFX build system plugins that download jar versions of the modules, not jmods. Also some alternative JDK vendors pre-ship JavaFX in their JDK spins to avoid people having to use jlink. I've worked with jlink extensively. The problems are greater than just Maven and Gradle, which at any rate both have quite sophisticated plugins for working with it. There was just a major error at the requirements analysis phase in how this part of Java works. Problems: 1. You can't put jmods on the module path, even though it would be easily implemented. They can only be used as inputs to jlink. But this is incompatible with how the JVM world works (not just Maven and Gradle), as it is expected to be able to download libraries and place them on the class/module path in combination with a generic runtime in order to use them. Changing that would mean every project to get its own sub-JDK created at build time, which would explode build times and disk space usage (jlink is very slow). So nobody does it, and no build system supports it. 2. When native code gets involved they are platform specific, even though (again) this could have been easily avoided. Even if JVMs supported jmods just like jars, no build system properly supports non-portable jars because the whole point of a jar is that it's portable. It has to be hacked around with custom plugins, which is a very high effort and non-scalable approach. 3. jlink doesn't understand the classpath at all. It thinks every jar/jmod is a module, but in reality very few libraries are usable on the module path (some look usable until you try it and discover their module metadata is broken). So you can't actually produce a standalone app directory with a bundled JVM using jlink because every real app uses non-modularized JARs. You end up needing custom launchers and scripts that try to work out what modules are needed using jlink. 4. The goal of the module system was to increase reliability, but it actually made it lower because there are some common cases where the module system doesn't detect that some theoretically optional modules are actually needed, even in the case of a fully modularized app. For example apps that try to use jlink directly are prone to experiencing randomly broken HTTP connections and (on desktop) mysterious accessibility failures, caused by critical features being optional and loaded dynamically, so they have to be forced into the image using special jlink flags. This is a manhole sized trap to fall into and isn't documented anywhere, there are no warnings printed. You are just expected to ship broken apps, find out the hard way what went wrong and then fix it by hand. At some point the JDK developers need to stop pointing the finger at build tool providers when it comes to Jigsaw adoption. It's not like Maven and Gradle are uniquely deviant. There are other build systems used in the JVM world and not one of them works the way the OpenJDK devs would like. They could contribute patches to change these things upstream, or develop their own build system, but have never done it so we end up with a world where there are ways to distribute libraries that work fine if you pretend people never moved beyond 'make' and checking jars into version control.