8 ms·

Typed, modular macros for OCaml

134 points by otini 10y ago

wcrichton 10y ago
This is awesome, and a big step forward for OCaml. Only criticism is that it seems difficult/awkward that you need different functions/values for each phase. In other staged macro systems (Scala/LMS, Rust/Compiler Plugins) you can refer to the same functions in any phase, and you can also splice values from the staging phase into the runtime phase without explicit use of a function like Expr.of_int.
- a0 10y ago
  I also agree that there seems to be too much separation between the phases. I wonder if this is a design choice or a design limitation. MetaOCaml also requires explicit lifting/unlifting of values.
  - otini 10y ago
    There are certainly some design choices here. I want to emphasize that in the few projects where I've used macros (e.g. https://github.com/OlivierNicole/macros-examples https://github.com/OlivierNicole/macros-examples), I found that these translating functions were scarcely needed and I never had to pass a non-standard type between phases. You may want to read this comment which I find relevant: https://www.reddit.com/r/ocaml/comments/5e8slg/bringing_typed_modular_macros_to_ocaml/dac9j0t/ https://www.reddit.com/r/ocaml/comments/5e8slg/bringing_type...
- otini 10y ago
  This sounds very handy and I definitely need to learn more about LMS and Compiler Plugins. I wonder then what are the limitations on the values you can pass automatically across phases, since there must be (you cannot transfer e.g. a file handle from compile-time to run-time).
  - kagebe 10y ago
    As far as I understand, for LMS: Those types T for which one has implemented needed operations for the Rep[T] type. Shameless plug: For another comparison of MetaOCaml, LMS, Terra and AnyDSL (our approach) under the lens of staging/partial evaluation see our paper from GPCE'15: http://www.cdl.uni-saarland.de/papers/gpce15.pdf http://www.cdl.uni-saarland.de/papers/gpce15.pdf
    - rubenfiszel 10y ago
      Hello, I work on lms and we are preparing a new version where Rep is implemented as a typeclass instead of a monad. It's much more pleasant because you manipulate IR.Int and scala.Int. So with the proper import you actually handle IR.Int so you're staged code is no different than a normal code.
- yallop 10y ago
  There are some difficulties with moving arbitrary values between phases: it's easy to move an int or a list, but what about a mutable reference or a closure? However, I don't think this will ultimately be a problem in practice, for two reasons. First, global values can be used in different phases via "module lifting". Second, there's a separate proposal for adding overloading to OCaml in the form of modular implicits: https://www.cl.cam.ac.uk/~jdy22/papers/modular-implicits.pdf https://www.cl.cam.ac.uk/~jdy22/papers/modular-implicits.pdf Modular implicits will make it possible to use a single overload function ('lift', say) in place of a family of functions 'Expr.of_int', 'Expr.of_float', etc., which will make things much less awkward. And it's only a small step from there to having 'lift' called implicitly/automatically at appropriate points. Here's a message from an earlier discussion with a few more details: https://sympa.inria.fr/sympa/arc/caml-list/2015-05/msg00032.html https://sympa.inria.fr/sympa/arc/caml-list/2015-05/msg00032....
  - naasking 10y ago
    > There are some difficulties with moving arbitrary values between phases: it's easy to move an int or a list, but what about a mutable reference or a closure? Wasn't this already answered by "Closing the Stage" [1]? [1] http://lambda-the-ultimate.org/node/2575 http://lambda-the-ultimate.org/node/2575
    - yallop 10y ago
      "Closing the Stage" is about a different interaction between staging and closures: that (with some care) staging constructs can be elaborated into closures in an unstaged calculus. The problem with moving arbitrary values between phases with macros is that values can refer to bits of the runtime environment that cease to exist when compilation is finished.
      - naasking 10y ago
        > that (with some care) staging constructs can be elaborated into closures in an unstaged calculus. Exactly, which seems to provide the necessary semantics for references that you mentioned. Clarifying staging semantics for difficult abstractions like refs by elaboration into well understood closure semantics was the point of the paper.
- nemoniac 10y ago
  Hasn't this matter been addressed years ago? http://www.cs.utah.edu/plt/publications/macromod.pdf http://www.cs.utah.edu/plt/publications/macromod.pdf
Ericson2314 10y ago
This seems well done, the sort of design I'm pushing for Rust and Haskell. The money question is: does it work with cross compiling? This is a proper phased design so it should, but that doesn't mean it doesn't.
- otini 10y ago
  Well what macros only ever do is generate OCaml ASTs, so the should work whatever the compilation target is.
  - Ericson2314 10y ago
    Things like the ^ quoting get more complex. You need to quite the target's version of the module.
    - otini 10y ago
      Oh I see what you mean. For now, we have made the choice to compile static code to OCaml bytecode, whatever the compilation target is. While this enables the use of macros regardless of the target (e.g. it works in `ocamlopt`, the native compiler), it does make it necessary to compile a module to bytecode if you want to lift it. It's not a big deal with an adapted build system, but a distant future we might support native compilation of macros on some architectures.
coldcode 10y ago
Having never tried OCaml, what advantages does it have over other languages? Two languages I don't know seem more interesting, Clojure and Rust. Generally I work in Swift these days.
- the_af 10y ago
  Over other languages in general? That's hard to say about any language :) OCaml is a language in the ML family, is backed by INRIA and is the main language of Jane Street Capital (so it definitely has "real world" use!), which also contribute a lot to it. OCaml is pretty good for functional programming, and unlike Clojure, it's statically typed. If you like static typing -- which I personally think is the way to go -- that's a huge reason to prefer it over Clojure. You may also check out Haskell if you go that way. An interesting case study about going to OCaml from another language (Python, in this case) was posted to HN many times already: http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-retrospective/ http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-ret.... I recommend you read it if you want to learn more about why would one choose this language. Why not choose a language in the ML family? Well, if not having C-like syntax is a deal breaker for you, then OCaml is going to frustrate you. A real programmer shouldn't let syntax stop them from learning new languages, though :) Ultimately one has to try doing something with this kind of language, even if it's just a toy project. Just reading about it won't enlighten you much, and you will end up focusing on its syntax instead and deciding it looks too unfamiliar.
  - owl57 10y ago
    > Well, if not having C-like syntax is a deal breaker for you, then OCaml is going to frustrate you. There is a new alternative Reason syntax (https://facebook.github.io/reason/mlCompared.html https://facebook.github.io/reason/mlCompared.html) which replaces some of the unusual syntax with braces and semicolons. Though it still probably looks less C-like than Swift. And I haven't tried writing anything in it, so can't say for sure if it really feels less weird than standard syntax on a "real program".
- int_19h 10y ago
  IMO, the single most interesting thing about OCaml is its object model. On one hand, you have a fairly conventional setup with classes, methods, inheritance, generics etc. On the other hand, the entire system is structurally typed, with pervasive type inference. Recycling bits from my earlier comment on the subject... # let f obj x y = (obj#m x)#n y;; val f : < m : 'a -> < n : 'b -> 'c; .. >; .. > -> 'a -> 'b -> 'c = <fun> Here I defined a function taking 3 arguments, called method m on obj, passing x as argument, then called method n on the result of that, passing y as argument. Note the inferred type signature for this: the first argument is of type <m : 'a -> <n : 'b -> 'c; ..>; ..> - i.e. any object that has a method named m (and possibly some other unspecified members - that's what ".." means), with said method m having a signature that allows us to pass some 'a, and returning another object of type <n : 'b -> 'c; ..> - i.e. something with a method named n that accepts 'b and returns 'c. 'a and 'b' are then used as types of x and y, respectively, and 'c is the the result of f.
int_19h 10y ago
Love the design! Maybe not the simplest to use, and fairly verbose at times... but it's very clear what's going on at any given point - and given how hairy macros can get in production code, I think emphasizing clarity is the right approach.
patrec 10y ago
What's the story for error messages? I read it's inspired by racket; the big thing about racket is that from what I hear it seems to be the only language that has good support for syntactic extension without sacrificing error reporting.
- otini 10y ago
  Since macros aren't just a plugin but are directly baked in the compiler code, error messages can be anything we want… what are you suggesting?
  - patrec 10y ago
    Let's take your static printf as a concrete example. Here's what a not so great error message looks like (ocaml's built in Printf): utop # Printf.sprintf "Some int: %d another int: %d" 1 "2";; Error: This expression has type string but an expression was expected of type int Here's what a good error message looks like: clang -Wall printf.c printf.c:6:52: warning: format specifies type 'int' but the argument has type 'char *' [-Wformat] printf ("Some int: %d some other int: %d\n", 1, "2"); ~~ ^~~ %s Notice how it also highlights the mismatched part in the string-based DSL.
    - otini 10y ago
      The second message is arguably clearer, but I don't think anyone would be really hindered by the first one, especially given that, unlike in C, application of printf have an explicit type: # Printf.sprintf "some int: %d another int: %d";; - : int -> int -> string = <fun> I think a general tendency in statically typed languages is that we like to write stuff using only the type system, rather than adding code to the compiler. Same for macros: we will rather have them proved correct by the typechecker than make them raise errors (also it is possible, through exceptions). But maybe this is detrimental to clarity of errors in some cases. I'd be interested to see how Racket handles errors.
      - 10y ago
lacampbell 10y ago
Can someone tell me why I would want statically typed macros? Why not just simply type check after macro expansion phase? Seems a lot simpler and more intuitive. It strongly feels like a case of everything looks like a nail when you really like hammers (static typing), but I am willing to be proven wrong.
- sordina 10y ago
  There may be a bunch of other reasons, but the one that springs to mind is localisation of errors. You'd ideally like to know about any mistakes you've made in your macro before you run your macro. And when you do learn about a mistake you'd like the error to describe where the mistake is in macro-land rather than generated-code land.
  - junke 10y ago
    Why just type errors? You might want to trace any error that occurs at runtime back to the original source code anyway.
    - sordina 10y ago
      I agree, you may wish to do that but I was answering the question "What is the utility of typed-templates", not "contrast type and runtime error tradeoffs".
- adamnemecek 10y ago
  Look at the C++ templates, I think that some of the clusterfuck that is C++ template compiler errors would go away if templates were type aware.
- otini 10y ago
  Firstly, if your macro does not typecheck, it means that executing it might result in a segfault or something similar, since macros are regular OCaml functions. Secondly, if macro typechecks, then it is guaranteed to expand without errors, and I think that's a nice guarantee to have.

134 points by otini 10y ago

This is awesome, and a big step forward for OCaml. Only criticism is that it seems difficult/awkward that you need different functions/values for each phase. In other staged macro systems (Scala/LMS, Rust/Compiler Plugins) you can refer to the same functions in any phase, and you can also splice values from the staging phase into the runtime phase without explicit use of a function like Expr.of_int.

a0 10y ago
I also agree that there seems to be too much separation between the phases. I wonder if this is a design choice or a design limitation. MetaOCaml also requires explicit lifting/unlifting of values.
- otini 10y ago
  There are certainly some design choices here. I want to emphasize that in the few projects where I've used macros (e.g. https://github.com/OlivierNicole/macros-examples https://github.com/OlivierNicole/macros-examples), I found that these translating functions were scarcely needed and I never had to pass a non-standard type between phases. You may want to read this comment which I find relevant: https://www.reddit.com/r/ocaml/comments/5e8slg/bringing_typed_modular_macros_to_ocaml/dac9j0t/ https://www.reddit.com/r/ocaml/comments/5e8slg/bringing_type...
otini 10y ago
This sounds very handy and I definitely need to learn more about LMS and Compiler Plugins. I wonder then what are the limitations on the values you can pass automatically across phases, since there must be (you cannot transfer e.g. a file handle from compile-time to run-time).
- kagebe 10y ago
  As far as I understand, for LMS: Those types T for which one has implemented needed operations for the Rep[T] type. Shameless plug: For another comparison of MetaOCaml, LMS, Terra and AnyDSL (our approach) under the lens of staging/partial evaluation see our paper from GPCE'15: http://www.cdl.uni-saarland.de/papers/gpce15.pdf http://www.cdl.uni-saarland.de/papers/gpce15.pdf
  - rubenfiszel 10y ago
    Hello, I work on lms and we are preparing a new version where Rep is implemented as a typeclass instead of a monad. It's much more pleasant because you manipulate IR.Int and scala.Int. So with the proper import you actually handle IR.Int so you're staged code is no different than a normal code.
yallop 10y ago
There are some difficulties with moving arbitrary values between phases: it's easy to move an int or a list, but what about a mutable reference or a closure? However, I don't think this will ultimately be a problem in practice, for two reasons. First, global values can be used in different phases via "module lifting". Second, there's a separate proposal for adding overloading to OCaml in the form of modular implicits: https://www.cl.cam.ac.uk/~jdy22/papers/modular-implicits.pdf https://www.cl.cam.ac.uk/~jdy22/papers/modular-implicits.pdf Modular implicits will make it possible to use a single overload function ('lift', say) in place of a family of functions 'Expr.of_int', 'Expr.of_float', etc., which will make things much less awkward. And it's only a small step from there to having 'lift' called implicitly/automatically at appropriate points. Here's a message from an earlier discussion with a few more details: https://sympa.inria.fr/sympa/arc/caml-list/2015-05/msg00032.html https://sympa.inria.fr/sympa/arc/caml-list/2015-05/msg00032....
- naasking 10y ago
  > There are some difficulties with moving arbitrary values between phases: it's easy to move an int or a list, but what about a mutable reference or a closure? Wasn't this already answered by "Closing the Stage" [1]? [1] http://lambda-the-ultimate.org/node/2575 http://lambda-the-ultimate.org/node/2575
  - yallop 10y ago
    "Closing the Stage" is about a different interaction between staging and closures: that (with some care) staging constructs can be elaborated into closures in an unstaged calculus. The problem with moving arbitrary values between phases with macros is that values can refer to bits of the runtime environment that cease to exist when compilation is finished.
    - naasking 10y ago
      > that (with some care) staging constructs can be elaborated into closures in an unstaged calculus. Exactly, which seems to provide the necessary semantics for references that you mentioned. Clarifying staging semantics for difficult abstractions like refs by elaboration into well understood closure semantics was the point of the paper.
nemoniac 10y ago
Hasn't this matter been addressed years ago? http://www.cs.utah.edu/plt/publications/macromod.pdf http://www.cs.utah.edu/plt/publications/macromod.pdf

Ericson2314 10y ago

This seems well done, the sort of design I'm pushing for Rust and Haskell. The money question is: does it work with cross compiling? This is a proper phased design so it should, but that doesn't mean it doesn't.

otini 10y ago
Well what macros only ever do is generate OCaml ASTs, so the should work whatever the compilation target is.
- Ericson2314 10y ago
  Things like the ^ quoting get more complex. You need to quite the target's version of the module.
  - otini 10y ago
    Oh I see what you mean. For now, we have made the choice to compile static code to OCaml bytecode, whatever the compilation target is. While this enables the use of macros regardless of the target (e.g. it works in `ocamlopt`, the native compiler), it does make it necessary to compile a module to bytecode if you want to lift it. It's not a big deal with an adapted build system, but a distant future we might support native compilation of macros on some architectures.

coldcode 10y ago

Having never tried OCaml, what advantages does it have over other languages? Two languages I don't know seem more interesting, Clojure and Rust. Generally I work in Swift these days.

the_af 10y ago
Over other languages in general? That's hard to say about any language :) OCaml is a language in the ML family, is backed by INRIA and is the main language of Jane Street Capital (so it definitely has "real world" use!), which also contribute a lot to it. OCaml is pretty good for functional programming, and unlike Clojure, it's statically typed. If you like static typing -- which I personally think is the way to go -- that's a huge reason to prefer it over Clojure. You may also check out Haskell if you go that way. An interesting case study about going to OCaml from another language (Python, in this case) was posted to HN many times already: http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-retrospective/ http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-ret.... I recommend you read it if you want to learn more about why would one choose this language. Why not choose a language in the ML family? Well, if not having C-like syntax is a deal breaker for you, then OCaml is going to frustrate you. A real programmer shouldn't let syntax stop them from learning new languages, though :) Ultimately one has to try doing something with this kind of language, even if it's just a toy project. Just reading about it won't enlighten you much, and you will end up focusing on its syntax instead and deciding it looks too unfamiliar.
- owl57 10y ago
  > Well, if not having C-like syntax is a deal breaker for you, then OCaml is going to frustrate you. There is a new alternative Reason syntax (https://facebook.github.io/reason/mlCompared.html https://facebook.github.io/reason/mlCompared.html) which replaces some of the unusual syntax with braces and semicolons. Though it still probably looks less C-like than Swift. And I haven't tried writing anything in it, so can't say for sure if it really feels less weird than standard syntax on a "real program".
int_19h 10y ago
IMO, the single most interesting thing about OCaml is its object model. On one hand, you have a fairly conventional setup with classes, methods, inheritance, generics etc. On the other hand, the entire system is structurally typed, with pervasive type inference. Recycling bits from my earlier comment on the subject... # let f obj x y = (obj#m x)#n y;; val f : < m : 'a -> < n : 'b -> 'c; .. >; .. > -> 'a -> 'b -> 'c = <fun> Here I defined a function taking 3 arguments, called method m on obj, passing x as argument, then called method n on the result of that, passing y as argument. Note the inferred type signature for this: the first argument is of type <m : 'a -> <n : 'b -> 'c; ..>; ..> - i.e. any object that has a method named m (and possibly some other unspecified members - that's what ".." means), with said method m having a signature that allows us to pass some 'a, and returning another object of type <n : 'b -> 'c; ..> - i.e. something with a method named n that accepts 'b and returns 'c. 'a and 'b' are then used as types of x and y, respectively, and 'c is the the result of f.

int_19h 10y ago

Love the design! Maybe not the simplest to use, and fairly verbose at times... but it's very clear what's going on at any given point - and given how hairy macros can get in production code, I think emphasizing clarity is the right approach.

patrec 10y ago

What's the story for error messages? I read it's inspired by racket; the big thing about racket is that from what I hear it seems to be the only language that has good support for syntactic extension without sacrificing error reporting.

otini 10y ago
Since macros aren't just a plugin but are directly baked in the compiler code, error messages can be anything we want… what are you suggesting?
- patrec 10y ago
  Let's take your static printf as a concrete example. Here's what a not so great error message looks like (ocaml's built in Printf): utop # Printf.sprintf "Some int: %d another int: %d" 1 "2";; Error: This expression has type string but an expression was expected of type int Here's what a good error message looks like: clang -Wall printf.c printf.c:6:52: warning: format specifies type 'int' but the argument has type 'char *' [-Wformat] printf ("Some int: %d some other int: %d\n", 1, "2"); ~~ ^~~ %s Notice how it also highlights the mismatched part in the string-based DSL.
  - otini 10y ago
    The second message is arguably clearer, but I don't think anyone would be really hindered by the first one, especially given that, unlike in C, application of printf have an explicit type: # Printf.sprintf "some int: %d another int: %d";; - : int -> int -> string = <fun> I think a general tendency in statically typed languages is that we like to write stuff using only the type system, rather than adding code to the compiler. Same for macros: we will rather have them proved correct by the typechecker than make them raise errors (also it is possible, through exceptions). But maybe this is detrimental to clarity of errors in some cases. I'd be interested to see how Racket handles errors.
    - 10y ago

lacampbell 10y ago

Can someone tell me why I would want statically typed macros? Why not just simply type check after macro expansion phase? Seems a lot simpler and more intuitive. It strongly feels like a case of everything looks like a nail when you really like hammers (static typing), but I am willing to be proven wrong.

sordina 10y ago
There may be a bunch of other reasons, but the one that springs to mind is localisation of errors. You'd ideally like to know about any mistakes you've made in your macro before you run your macro. And when you do learn about a mistake you'd like the error to describe where the mistake is in macro-land rather than generated-code land.
- junke 10y ago
  Why just type errors? You might want to trace any error that occurs at runtime back to the original source code anyway.
  - sordina 10y ago
    I agree, you may wish to do that but I was answering the question "What is the utility of typed-templates", not "contrast type and runtime error tradeoffs".
adamnemecek 10y ago
Look at the C++ templates, I think that some of the clusterfuck that is C++ template compiler errors would go away if templates were type aware.
otini 10y ago
Firstly, if your macro does not typecheck, it means that executing it might result in a segfault or something similar, since macros are regular OCaml functions. Secondly, if macro typechecks, then it is guaranteed to expand without errors, and I think that's a nice guarantee to have.