4 ms·

JEP 457: Class-File API for Parsing, generating, transforming Java classfiles

102 points by carimura 3y ago

spreiti 3y ago
I highly recommend you to watch this video by Brian Goetz if you want to learn more about this JEP. https://youtu.be/pcg-E_qyMOI https://youtu.be/pcg-E_qyMOI
- shellac 3y ago
  And as a companion piece I really enjoyed Paul Sandoz's 'Code Reflection' talk https://www.youtube.com/watch?v=xbk9_6XA_IY https://www.youtube.com/watch?v=xbk9_6XA_IY, which considers how java code might understand java code. The sort of thing that enable pushing functions to SQL or GPUs for example. Edit to add: slides here https://cr.openjdk.org/~psandoz/conferences/2023-JVMLS/Code-Reflection-JVMLS-23-08-07.pdf https://cr.openjdk.org/~psandoz/conferences/2023-JVMLS/Code-...
  - bafe 3y ago
    Interesting, but I don't see how operating on the code model is going to be any easier than working on the AST. It seems much more hard to reason about many transformations as compared to manipulating the AST. I do hope they take inspiration from scala and rust macros
    - shellac 3y ago
      (Late reply, sorry) Well currently you can't get at the AST, only (with some hairy code) bytecode. (The use cases are post-compilation) But suppose this work did enable that. What Paul is saying is that there are intermediate representations between bytecode and AST that are more helpful for these GPU / SQL / whatever runtime compilers. Representations that capture dataflows, for example. Chance are these compilers would transform an AST to something like this anyway. See https://mlir.llvm.org/ https://mlir.llvm.org/ which is referenced in that talk.
      - bafe 3y ago
        Thanks, that makes sense. I think I approached more from the perspective of a library writer that doesn't want to delve into the depth of language to implement transformations/macros. However it does seem reasonable to have a variety of abstractions to be able to express these transformations in a way closer to the target language
- waynesonfire 3y ago
  At round 8:50, Goetz talks about the design, in essence the quote is, "We designed it as a functional library because functionally inspired libraries are successful at meeting these goals." Ironic.
  - za3faran 3y ago
    Why ironic? Java is a multi paradigm language (though obviously not at the same level as say C++). It has been getting "functional" features for a while now, including streaming libraries, pattern matching, immutable records, etc.
    - kagakuninja 3y ago
      Because they have in the past declared "Java is not a functional language" and were hostile to the use of monads in the standard library. They are adding nice features, but the language will never be as clean and unified as Scala.
      - jmaker 3y ago
        Monads don’t go by the name, they must satisfy the axioms. How many people do it right? Even in Haskell lots of folks forget to check the axioms. Also, monads are hard to compose. I think there’s no reason to be like Cats or Arrow. It’s all already there, the mental shift is minimal in my opinion.
        kagakuninja 3y ago
        "Monads don't compose" just means you should choose a single monad for most of your functions; in Scala that is usually Future, Either, Try, IO or ZIO. There are conversion methods between the most common monads. Scala monads are fantastic IMO; Either-based programming is simple and eliminates tons of boilerplate. So is Cats, although the learning curve is steep.
    - jmaker 3y ago
      Unfortunately with many gotchas to be aware of all the time…
wpollock 3y ago
Will this enable lombok officially?
- cogman10 3y ago
  Lombok's headaches are because they are touching Java compiler internals to accomplish their magic. This won't fix that. Lombok wouldn't be nearly as troubled if it was just doing simple bytecode manipulation. If lombok wants to stop the pain, then they'll need to stop reaching into internal APIs. They'll possibly need to remove a few features (like some of the `private final` work they are doing). In other words, you can expect lombok to be a headache for years to come. (Maybe consider not using it? That'd be swell. Speaking as someone that curses lombok everytime jdk updates roll around.)
  - the-smug-one 3y ago
    Why isn't Lombok just written as a pre-javac pass? javac(lombok(srccode)). Just gotta keep that parser alive and up to date, which open source IDEs also must so they're probably available.
    - tmccrary55 3y ago
      It's easier (read more reliable) to manipulate bytecode or other internal representation than the source.
      - aardvark179 3y ago
        Manipulating the AST would be fine, but Lombok pretends to be an annotation processor (which can generate new classes, but not alter the semantics of the class being processed). They could create lombokc and crack open the internals of the Java compiler as much as they like, but this would mean admitting they are really Java, and they don’t seem to want to accept that.
- exabrial 3y ago
  Lombok is a compiler extension, where this is an API for the JVM. So two very different things. Think of Lombok/compiler extensions as #include directives in C... things that happen at compile time. This API is for writing programs at runtime, past the compile phase.
cogman10 3y ago
This is the sort of thing that will have a LARGE impact on the ecosystem for updating from one JDK to the next. One of the big headaches we see with moving up JDK version is bytecode generation/manipulation libraries choking on newer versions of the JDK. It's generally a simple update, but sometimes it's not (for example, when someone is shading asm). Having bytecode generation as part of the JDK will mean all libraries, including asm, can migrate to that and benefit from a perpetually supported API that updates as the JDK does. This is probably 80% of the headaches I've experienced going from the likes of Java 11 to 17 and 17 to 21.
- deleted 3y ago
  [deleted]
exabrial 3y ago
This is pretty exciting... I've used them all libraries at this point in my career: CGLib, ASM, BCEL, ByteBuddy, Javassist, etc... each has its pluses and minuses. I've designed everything from profiling agents, to systems that pack decimals into EBCDIC and invoke COBOL programs on big IBM iron, to lightweight JIT compilers, all using these libraries. > In 2002, the visitor approach used by ASM seemed clever I couldn't agree more. The visitor pattern was very hard to explain/justify back then, and still difficult to explain to newbie programmers just entering the profession. Looking at the examples, I think this is going to be an official replacement for ASM, meaning it's going to be pretty low level. The use of streams pretty straightforward. If anyone from the JEP is reading this: I have two pieces of feedback! First, take some inspiration from the way CDI Portable Extensions work. This is probably the most delightful extension API I've ever used. The @Observe callbacks are super simple to explain to people and it's really easy to write extensions for the framework. Next, I wouldn't ignore the need for a higher-level API akin to ByteBuddy or Javassist. Sometimes I just want to write an interpreter or intercept a method call and thats it. For example in my Junit/Mockito extension https://github.com/exabrial/mockito-object-injection https://github.com/exabrial/mockito-object-injection I need to intercept a call to the class under test in order to lazily inject dependencies at the last possible moment. While I certainly could do this with ASM, Javassist makes this fairly simple with it's MethodHandler api. Side note, it's a damn shame we don't have a mobile operating system that is JVM native :/ All this cool APIs simply never reach a huge number of devices.
- vips7L 3y ago
  I just want to share my support of CDI extensions. They just make sense!
- xxs 3y ago
  >First, take some inspiration from the way CDI Portable Extensions work It's way too late, the API is effectively locked in preview mode. While I am not a fan of ASM's visitor pattern, explaining it to anyone would be the least of my concerns, ASM requires pretty extensive knowledge on class structure, method signatures, and most of the byte code instructions. Whoever ventures in byte code editing mode should be able to read the byte code directly.
lemming 3y ago
I wonder if this will be available as a standalone library for older JDKs, it looks very nice.
saagarjha 3y ago
I’ve always wondered why the APIs for reflection in Java were always so low-level. If you look at, say, Objective-C (a language with a very similar model of Java) almost nobody drops down to assembly to patch things, because the language is expressive enough that you don’t need to do this basically ever. Why not add the same to Java?

102 points by carimura 3y ago

I highly recommend you to watch this video by Brian Goetz if you want to learn more about this JEP. https://youtu.be/pcg-E_qyMOI https://youtu.be/pcg-E_qyMOI

shellac 3y ago
And as a companion piece I really enjoyed Paul Sandoz's 'Code Reflection' talk https://www.youtube.com/watch?v=xbk9_6XA_IY https://www.youtube.com/watch?v=xbk9_6XA_IY, which considers how java code might understand java code. The sort of thing that enable pushing functions to SQL or GPUs for example. Edit to add: slides here https://cr.openjdk.org/~psandoz/conferences/2023-JVMLS/Code-Reflection-JVMLS-23-08-07.pdf https://cr.openjdk.org/~psandoz/conferences/2023-JVMLS/Code-...
- bafe 3y ago
  Interesting, but I don't see how operating on the code model is going to be any easier than working on the AST. It seems much more hard to reason about many transformations as compared to manipulating the AST. I do hope they take inspiration from scala and rust macros
  - shellac 3y ago
    (Late reply, sorry) Well currently you can't get at the AST, only (with some hairy code) bytecode. (The use cases are post-compilation) But suppose this work did enable that. What Paul is saying is that there are intermediate representations between bytecode and AST that are more helpful for these GPU / SQL / whatever runtime compilers. Representations that capture dataflows, for example. Chance are these compilers would transform an AST to something like this anyway. See https://mlir.llvm.org/ https://mlir.llvm.org/ which is referenced in that talk.
    - bafe 3y ago
      Thanks, that makes sense. I think I approached more from the perspective of a library writer that doesn't want to delve into the depth of language to implement transformations/macros. However it does seem reasonable to have a variety of abstractions to be able to express these transformations in a way closer to the target language
waynesonfire 3y ago
At round 8:50, Goetz talks about the design, in essence the quote is, "We designed it as a functional library because functionally inspired libraries are successful at meeting these goals." Ironic.
- za3faran 3y ago
  Why ironic? Java is a multi paradigm language (though obviously not at the same level as say C++). It has been getting "functional" features for a while now, including streaming libraries, pattern matching, immutable records, etc.
  - kagakuninja 3y ago
    Because they have in the past declared "Java is not a functional language" and were hostile to the use of monads in the standard library. They are adding nice features, but the language will never be as clean and unified as Scala.
    - jmaker 3y ago
      Monads don’t go by the name, they must satisfy the axioms. How many people do it right? Even in Haskell lots of folks forget to check the axioms. Also, monads are hard to compose. I think there’s no reason to be like Cats or Arrow. It’s all already there, the mental shift is minimal in my opinion.
      - kagakuninja 3y ago
        "Monads don't compose" just means you should choose a single monad for most of your functions; in Scala that is usually Future, Either, Try, IO or ZIO. There are conversion methods between the most common monads. Scala monads are fantastic IMO; Either-based programming is simple and eliminates tons of boilerplate. So is Cats, although the learning curve is steep.
  - jmaker 3y ago
    Unfortunately with many gotchas to be aware of all the time…

wpollock 3y ago

Will this enable lombok officially?

cogman10 3y ago
Lombok's headaches are because they are touching Java compiler internals to accomplish their magic. This won't fix that. Lombok wouldn't be nearly as troubled if it was just doing simple bytecode manipulation. If lombok wants to stop the pain, then they'll need to stop reaching into internal APIs. They'll possibly need to remove a few features (like some of the `private final` work they are doing). In other words, you can expect lombok to be a headache for years to come. (Maybe consider not using it? That'd be swell. Speaking as someone that curses lombok everytime jdk updates roll around.)
- the-smug-one 3y ago
  Why isn't Lombok just written as a pre-javac pass? javac(lombok(srccode)). Just gotta keep that parser alive and up to date, which open source IDEs also must so they're probably available.
  - tmccrary55 3y ago
    It's easier (read more reliable) to manipulate bytecode or other internal representation than the source.
    - aardvark179 3y ago
      Manipulating the AST would be fine, but Lombok pretends to be an annotation processor (which can generate new classes, but not alter the semantics of the class being processed). They could create lombokc and crack open the internals of the Java compiler as much as they like, but this would mean admitting they are really Java, and they don’t seem to want to accept that.
exabrial 3y ago
Lombok is a compiler extension, where this is an API for the JVM. So two very different things. Think of Lombok/compiler extensions as #include directives in C... things that happen at compile time. This API is for writing programs at runtime, past the compile phase.

cogman10 3y ago

This is the sort of thing that will have a LARGE impact on the ecosystem for updating from one JDK to the next. One of the big headaches we see with moving up JDK version is bytecode generation/manipulation libraries choking on newer versions of the JDK. It's generally a simple update, but sometimes it's not (for example, when someone is shading asm). Having bytecode generation as part of the JDK will mean all libraries, including asm, can migrate to that and benefit from a perpetually supported API that updates as the JDK does. This is probably 80% of the headaches I've experienced going from the likes of Java 11 to 17 and 17 to 21.

deleted 3y ago
[deleted]

exabrial 3y ago

This is pretty exciting... I've used them all libraries at this point in my career: CGLib, ASM, BCEL, ByteBuddy, Javassist, etc... each has its pluses and minuses. I've designed everything from profiling agents, to systems that pack decimals into EBCDIC and invoke COBOL programs on big IBM iron, to lightweight JIT compilers, all using these libraries. > In 2002, the visitor approach used by ASM seemed clever I couldn't agree more. The visitor pattern was very hard to explain/justify back then, and still difficult to explain to newbie programmers just entering the profession. Looking at the examples, I think this is going to be an official replacement for ASM, meaning it's going to be pretty low level. The use of streams pretty straightforward. If anyone from the JEP is reading this: I have two pieces of feedback! First, take some inspiration from the way CDI Portable Extensions work. This is probably the most delightful extension API I've ever used. The @Observe callbacks are super simple to explain to people and it's really easy to write extensions for the framework. Next, I wouldn't ignore the need for a higher-level API akin to ByteBuddy or Javassist. Sometimes I just want to write an interpreter or intercept a method call and thats it. For example in my Junit/Mockito extension https://github.com/exabrial/mockito-object-injection https://github.com/exabrial/mockito-object-injection I need to intercept a call to the class under test in order to lazily inject dependencies at the last possible moment. While I certainly could do this with ASM, Javassist makes this fairly simple with it's MethodHandler api. Side note, it's a damn shame we don't have a mobile operating system that is JVM native :/ All this cool APIs simply never reach a huge number of devices.

vips7L 3y ago
I just want to share my support of CDI extensions. They just make sense!
xxs 3y ago
>First, take some inspiration from the way CDI Portable Extensions work It's way too late, the API is effectively locked in preview mode. While I am not a fan of ASM's visitor pattern, explaining it to anyone would be the least of my concerns, ASM requires pretty extensive knowledge on class structure, method signatures, and most of the byte code instructions. Whoever ventures in byte code editing mode should be able to read the byte code directly.

lemming 3y ago

I wonder if this will be available as a standalone library for older JDKs, it looks very nice.

saagarjha 3y ago

I’ve always wondered why the APIs for reflection in Java were always so low-level. If you look at, say, Objective-C (a language with a very similar model of Java) almost nobody drops down to assembly to patch things, because the language is expressive enough that you don’t need to do this basically ever. Why not add the same to Java?