5 ms·

Manifold: Java Type-Safe Metaprogramming, Structural Typing, Extension Methods

126 points by jeanlucbernard 7y ago

throwaway_x13zd 7y ago
Manifold is an amazing piece of engineering. Scott McKinney, the author, used to work on Gosu but mostly focuses on this now. He added GraphQL support a few weeks ago: https://github.com/manifold-systems/manifold-sample-graphql-app https://github.com/manifold-systems/manifold-sample-graphql-... You can just drop graphql files in your resources directory and start coding against them as type-safe interfaces immediately, no code-gen required. The @Jailbreak annotation is awesome too.
- mindcrime 7y ago
  Looks pretty cool. Reminds me a lot of the ways in which Groovy improves Java.
  - vorg 7y ago
    Apache Groovy really improved on Beanshell (by adding closures, in 2003), which is what improved on Java (by adding dynamic typing, in late 1990's).
der_Fidelis 7y ago
At that point, why not just use another jvm language with these features like Kotlin?
- iSnow 7y ago
  Ever tried to "sell" an alternative language in BigCo Enterprise? Where I work, Javascript is still a front-end only language and just tolerated as a toolchain for compiling LESS, ECMAScript etc.
- sisu2019 7y ago
  the main feature of this allows you to transparently use structured data as types in your code like type providers in .net. I love kotlin but it can't do anything remotely like that.
  - jillesvangurp 7y ago
    Yep, I'm wondering if this actually works in Kotlin as well. I'm thinking it may probably play nice with this as is since Kotlin can interact with Java classes. However, the compiler plugin might be a problem. If so, this would be an awesome thing to add to Kotlin.
- james_s_tayler 7y ago
  at that point why not use C#?
  - tigershark 7y ago
    C# doesn’t support type providers, only f# supports them.
- based2 7y ago
  or Scala https://www.scala-lang.org/ https://www.scala-lang.org/
  - marcinzm 7y ago
    Scala comes with a lot of overhead in terms of features (and breaking changes in versions) and is still missing the meta-programming aspect of this library. In fact, scala has now deprecated all it's previous attempts at meta-programming and is still trying to work out it's next attempt (which is I believe black-box only so can't do what this library does).
- _old_dude_ 7y ago
  Scott McKinney, the guy behind that library, is also the guy behind Gosu [1]. Selling a new language is hard, selling a library is easier, even if this one has a hook deep in the code of javac. [1] https://en.wikipedia.org/wiki/Gosu_(programming_language) https://en.wikipedia.org/wiki/Gosu_(programming_language)
chii 7y ago
Removing checked exceptions - what a dangerous feature!
- mckinney 7y ago
  It's optional. It seems scary, but checked exceptions are a feature exclusive to the Java compiler -- the JVM does not enforce them. This is how JVM languages like Kotlin and others only use unchecked exceptions, and also how Manifold gets away with muting them as a compile-time feature.
  - _old_dude_ 7y ago
    It should not be optional :) Checked exceptions is a stupid feature because it makes an implementation detail, the fact that a code raises an exception, part of the type of a method. Scala, C# or Kotlin have no checked exception and it doesn't make them less safe.
    - bondolo 7y ago
      No less safe until it the code is running in production on a Tuesday at 3AM and a library pukes up an exception it has never thrown before and your code doesn't handle it. I prefer exceptions to be declared, just like types. Having exceptions declared gives me more information about the operation of the code I am calling and that is useful. To treat exception declarations just as annoyance seems like lazy punting.
      - WkndTriathlete 7y ago
        Likely you don't care which exception is thrown, only that one is. In that case, it's overkill to add the exceptions to the type signature. You'd be far likely to be better off using IO<A> from Kotlin or IO[_] from Scala to model computations with exceptions as side-effects. Besides, any non-trivial method that calls any set of libraries/frameworks would need to declare tens or hundreds of checked exceptions in your scenario. And that's just the catchable exceptions. What are you going to do about any (subclass of) Error?
sisu2019 7y ago
This is what I wanted all my life! I wonder what kind of dark magic makes this go and how this will come back to bite me if I use it?
- mckinney 7y ago
  Manifold hooks into the Java compiler via the javac plugin API. This is similar to the way annotation processors work, but without the overhead of additional passes (or rounds) in the compiler. The plugin route also provides access to earlier stages of the compiler, for example this is how Manifold's string interpolation feature works. The overall details are a bit deep and dark for a comment, however ;)
  - brabel 7y ago
    I've played with the compiler's API before and found it incredibly difficult to work with... and completely undocumented... how did you manage to get through that? Great job!
    - mckinney 7y ago
      Thank you! After I got past the initial shock with the javac codebase (it's an unusual design) I came to appreciate it. Given it's age and the number of revisions it has undergone, it holds up pretty well. But, yeah, building Manifold required a bit of determination.
brabel 7y ago
I thought I knew all about Java, but this thing makes me feel like I haven't got a clue! How can it achieve this stuff by just being in the classpath?? No Java agent can be involved, no codegen during the build?! This shouldn't be possible, or??
- brabel 7y ago
  Ok just checked this deeper. The [docs](http://manifold.systems/docs.html http://manifold.systems/docs.html) actually show this is a javac plugin (I didn't know about the `-Xplugin` flag, but you ALSO need to pass that to javac for Manifold to work)... you also need the `tools.jar` file (which includes the JavaCompiler) on the runtime classpath... so suddenly, there's no magic... it's a bit misleading to say this does not require a build step, it clearly does, but as it integrates with javac itself, it "feels" seamless...
  - mckinney 7y ago
    There's a significant difference between a code gen build step and what Manifold is doing with javac. Resource files are virtual source files, this represents a leap in terms of dev productivity.
    - brabel 7y ago
      Yes, I agree... just wished they made it clearer how they achieve that!
      - jillesvangurp 7y ago
        The documentation is a bit handwavy on how it works indeed. Just drop a jar file in the code doesn't quite tell the whole story. However, I'm impressed with how amazingly awesome this stuff is. A lot of the stuff it does is already in Kotlin but being able to use json and property files without boiler plate code is nice. I'd like to be able to do that in Kotlin as well.
  - mckinney 7y ago
stewbrew 7y ago
How is the user experience with that if you don't use Intellij? It seems to require explicit support by the IDE, which is why it doesn't seem to work with eclipse.
- mckinney 7y ago
  (Hi, I'm the author of Manifold) Manifold works equally well with Java 8 - 12. While no IDE is required, it is designed to fully leverage Java's static type system, which heavily benefits from the static analysis provided by a topnotch IDE like IntelliJ IDEA. Code completion, deterministic refactoring, usage searching, feature navigation, incremental compilation, etc. -- suddenly these productive features are now available to any resource connected with Manifold e.g, GraphQL, JSON Schema, Javascript, etc. Re Intellij v. Eclipse. I'm just one guy banging on a project; I only have time to build and maintain one IDE plugin and right now my focus is on IntelliJ. I'd like to start working on an Eclipse plugin at some point... or trick someone else into doing the work ;)
isharamet 7y ago
FYI: Manifold is a pretty popular Clojure library by Zach Tellman. https://github.com/ztellman/manifold https://github.com/ztellman/manifold
orbifold 7y ago
First they came for the particles (electron, gluon, atom) now they are coming for mathematical constructs and give them mundane everyday meaning .
- iSnow 7y ago
  Or car parts, who can tell...
iSnow 7y ago
Wow, this looks like a god-send. The @Jailbreak annotation would give mid-2000's Java coders a heart-attack :) The only contentious point I find is that they are basically replicating Project Lombok's @SneakyThrows annotation via "Xplugin:Manifold stringsexceptions" or ModelMapper in their Structural Typing. Furthermore it seems it should be structured in sub-projects, eg. the String-templates and ManTL in one lib, Structural Typing in another so we can mix & match. But overall, very impressive!
- mckinney 7y ago
  Thanks! The 'exceptions' option is quite a bit different from SneakyThrows. The 'exceptions' option completely mutes checked exceptions in your project -- you don't have to declare a throws clause or catch a checked exception anywhere in your code, thus with the option enabled checked exceptions behave exactly as unchecked exceptions. This behavior is identical to exception treatment in other JVM languages such as Scala, Kotlin, Ceylon, etc. Regarding sub-projects on extensions, I've been meaning to do that. The project is already divided into separate targets e.g., manifold-graphql, manifold-json, etc. But manifold-ext has probably grown too big for one target.
  - iSnow 7y ago
    My god, I just read a bit more about the Extension Manifold, and that is crazy good. I am doing both Java and JS, and I started to like duck-typing and extending prototypes so much in some situations. Now, Extensions give me something similar in Java? I have the highest respect for your work, and no idea how you are doing that more or less on your own. I am just a bit sad your upcoming SQL/DDL extensions are going to kill one of my side-projects :) https://github.com/iSnow/sqlWebservice https://github.com/iSnow/sqlWebservice - if you can use a SQL schema as a type provider, you are leaps and bounds ahead of what I am doing there. So, hats off, great job.
eatonphil 7y ago
If you like this, check out Project Lombok [0] as well. It gives you var/val for local type-inferenced variables, getter/setter code generation, and others to help eliminate boilerplate. [0] https://projectlombok.org https://projectlombok.org
rb808 7y ago
Its interesting with the other thread about Java EE dying and Oracle politics. The ecosystem around Java and JVM is huge, so much cool stuff going on. Definitely a great platform.
kitd 7y ago
This feels a bit like Project Lombok: adding cool features to standard Java. I like the Extensions option. It works very much like D's UFCS mechanism.

Manifold is an amazing piece of engineering. Scott McKinney, the author, used to work on Gosu but mostly focuses on this now. He added GraphQL support a few weeks ago: https://github.com/manifold-systems/manifold-sample-graphql-app https://github.com/manifold-systems/manifold-sample-graphql-... You can just drop graphql files in your resources directory and start coding against them as type-safe interfaces immediately, no code-gen required. The @Jailbreak annotation is awesome too.

mindcrime 7y ago
Looks pretty cool. Reminds me a lot of the ways in which Groovy improves Java.
- vorg 7y ago
  Apache Groovy really improved on Beanshell (by adding closures, in 2003), which is what improved on Java (by adding dynamic typing, in late 1990's).

der_Fidelis 7y ago

At that point, why not just use another jvm language with these features like Kotlin?

iSnow 7y ago
Ever tried to "sell" an alternative language in BigCo Enterprise? Where I work, Javascript is still a front-end only language and just tolerated as a toolchain for compiling LESS, ECMAScript etc.
sisu2019 7y ago
the main feature of this allows you to transparently use structured data as types in your code like type providers in .net. I love kotlin but it can't do anything remotely like that.
- jillesvangurp 7y ago
  Yep, I'm wondering if this actually works in Kotlin as well. I'm thinking it may probably play nice with this as is since Kotlin can interact with Java classes. However, the compiler plugin might be a problem. If so, this would be an awesome thing to add to Kotlin.
james_s_tayler 7y ago
at that point why not use C#?
- tigershark 7y ago
  C# doesn’t support type providers, only f# supports them.
based2 7y ago
or Scala https://www.scala-lang.org/ https://www.scala-lang.org/
- marcinzm 7y ago
  Scala comes with a lot of overhead in terms of features (and breaking changes in versions) and is still missing the meta-programming aspect of this library. In fact, scala has now deprecated all it's previous attempts at meta-programming and is still trying to work out it's next attempt (which is I believe black-box only so can't do what this library does).
_old_dude_ 7y ago
Scott McKinney, the guy behind that library, is also the guy behind Gosu [1]. Selling a new language is hard, selling a library is easier, even if this one has a hook deep in the code of javac. [1] https://en.wikipedia.org/wiki/Gosu_(programming_language) https://en.wikipedia.org/wiki/Gosu_(programming_language)

chii 7y ago

Removing checked exceptions - what a dangerous feature!

mckinney 7y ago
It's optional. It seems scary, but checked exceptions are a feature exclusive to the Java compiler -- the JVM does not enforce them. This is how JVM languages like Kotlin and others only use unchecked exceptions, and also how Manifold gets away with muting them as a compile-time feature.
- _old_dude_ 7y ago
  It should not be optional :) Checked exceptions is a stupid feature because it makes an implementation detail, the fact that a code raises an exception, part of the type of a method. Scala, C# or Kotlin have no checked exception and it doesn't make them less safe.
  - bondolo 7y ago
    No less safe until it the code is running in production on a Tuesday at 3AM and a library pukes up an exception it has never thrown before and your code doesn't handle it. I prefer exceptions to be declared, just like types. Having exceptions declared gives me more information about the operation of the code I am calling and that is useful. To treat exception declarations just as annoyance seems like lazy punting.
    - WkndTriathlete 7y ago
      Likely you don't care which exception is thrown, only that one is. In that case, it's overkill to add the exceptions to the type signature. You'd be far likely to be better off using IO<A> from Kotlin or IO[_] from Scala to model computations with exceptions as side-effects. Besides, any non-trivial method that calls any set of libraries/frameworks would need to declare tens or hundreds of checked exceptions in your scenario. And that's just the catchable exceptions. What are you going to do about any (subclass of) Error?

sisu2019 7y ago

This is what I wanted all my life! I wonder what kind of dark magic makes this go and how this will come back to bite me if I use it?

mckinney 7y ago
Manifold hooks into the Java compiler via the javac plugin API. This is similar to the way annotation processors work, but without the overhead of additional passes (or rounds) in the compiler. The plugin route also provides access to earlier stages of the compiler, for example this is how Manifold's string interpolation feature works. The overall details are a bit deep and dark for a comment, however ;)
- brabel 7y ago
  I've played with the compiler's API before and found it incredibly difficult to work with... and completely undocumented... how did you manage to get through that? Great job!
  - mckinney 7y ago
    Thank you! After I got past the initial shock with the javac codebase (it's an unusual design) I came to appreciate it. Given it's age and the number of revisions it has undergone, it holds up pretty well. But, yeah, building Manifold required a bit of determination.

brabel 7y ago

I thought I knew all about Java, but this thing makes me feel like I haven't got a clue! How can it achieve this stuff by just being in the classpath?? No Java agent can be involved, no codegen during the build?! This shouldn't be possible, or??

brabel 7y ago
Ok just checked this deeper. The [docs](http://manifold.systems/docs.html http://manifold.systems/docs.html) actually show this is a javac plugin (I didn't know about the `-Xplugin` flag, but you ALSO need to pass that to javac for Manifold to work)... you also need the `tools.jar` file (which includes the JavaCompiler) on the runtime classpath... so suddenly, there's no magic... it's a bit misleading to say this does not require a build step, it clearly does, but as it integrates with javac itself, it "feels" seamless...
- mckinney 7y ago
  There's a significant difference between a code gen build step and what Manifold is doing with javac. Resource files are virtual source files, this represents a leap in terms of dev productivity.
  - brabel 7y ago
    Yes, I agree... just wished they made it clearer how they achieve that!
    - jillesvangurp 7y ago
      The documentation is a bit handwavy on how it works indeed. Just drop a jar file in the code doesn't quite tell the whole story. However, I'm impressed with how amazingly awesome this stuff is. A lot of the stuff it does is already in Kotlin but being able to use json and property files without boiler plate code is nice. I'd like to be able to do that in Kotlin as well.
- mckinney 7y ago

stewbrew 7y ago

How is the user experience with that if you don't use Intellij? It seems to require explicit support by the IDE, which is why it doesn't seem to work with eclipse.

mckinney 7y ago
(Hi, I'm the author of Manifold) Manifold works equally well with Java 8 - 12. While no IDE is required, it is designed to fully leverage Java's static type system, which heavily benefits from the static analysis provided by a topnotch IDE like IntelliJ IDEA. Code completion, deterministic refactoring, usage searching, feature navigation, incremental compilation, etc. -- suddenly these productive features are now available to any resource connected with Manifold e.g, GraphQL, JSON Schema, Javascript, etc. Re Intellij v. Eclipse. I'm just one guy banging on a project; I only have time to build and maintain one IDE plugin and right now my focus is on IntelliJ. I'd like to start working on an Eclipse plugin at some point... or trick someone else into doing the work ;)

isharamet 7y ago

FYI: Manifold is a pretty popular Clojure library by Zach Tellman. https://github.com/ztellman/manifold https://github.com/ztellman/manifold

orbifold 7y ago

First they came for the particles (electron, gluon, atom) now they are coming for mathematical constructs and give them mundane everyday meaning .

iSnow 7y ago
Or car parts, who can tell...

iSnow 7y ago

Wow, this looks like a god-send. The @Jailbreak annotation would give mid-2000's Java coders a heart-attack :) The only contentious point I find is that they are basically replicating Project Lombok's @SneakyThrows annotation via "Xplugin:Manifold stringsexceptions" or ModelMapper in their Structural Typing. Furthermore it seems it should be structured in sub-projects, eg. the String-templates and ManTL in one lib, Structural Typing in another so we can mix & match. But overall, very impressive!

mckinney 7y ago
Thanks! The 'exceptions' option is quite a bit different from SneakyThrows. The 'exceptions' option completely mutes checked exceptions in your project -- you don't have to declare a throws clause or catch a checked exception anywhere in your code, thus with the option enabled checked exceptions behave exactly as unchecked exceptions. This behavior is identical to exception treatment in other JVM languages such as Scala, Kotlin, Ceylon, etc. Regarding sub-projects on extensions, I've been meaning to do that. The project is already divided into separate targets e.g., manifold-graphql, manifold-json, etc. But manifold-ext has probably grown too big for one target.
- iSnow 7y ago
  My god, I just read a bit more about the Extension Manifold, and that is crazy good. I am doing both Java and JS, and I started to like duck-typing and extending prototypes so much in some situations. Now, Extensions give me something similar in Java? I have the highest respect for your work, and no idea how you are doing that more or less on your own. I am just a bit sad your upcoming SQL/DDL extensions are going to kill one of my side-projects :) https://github.com/iSnow/sqlWebservice https://github.com/iSnow/sqlWebservice - if you can use a SQL schema as a type provider, you are leaps and bounds ahead of what I am doing there. So, hats off, great job.

eatonphil 7y ago

If you like this, check out Project Lombok [0] as well. It gives you var/val for local type-inferenced variables, getter/setter code generation, and others to help eliminate boilerplate. [0] https://projectlombok.org https://projectlombok.org

rb808 7y ago

Its interesting with the other thread about Java EE dying and Oracle politics. The ecosystem around Java and JVM is huge, so much cool stuff going on. Definitely a great platform.

kitd 7y ago

This feels a bit like Project Lombok: adding cool features to standard Java. I like the Extensions option. It works very much like D's UFCS mechanism.