6 ms·

Extension Points, or how OCaml is becoming more like Lisp

104 points by edwintorok 12y ago

edwintorok 12y ago
It would be interesting to see how the speed of -ppx extensions compares to camlp4. If camlp4 is run in bytecode mode it can be quite slow, I looked at 'top' while running 'opam install core' at some point, and most of the time was taken by camlp4. Since -ppx extensions are supposed to run natively (without requiring natdynlink support even) I would expect them to be faster. Has anyone done any comparisons?
- avsm 12y ago
  There are pros and cons to not using dynlink, since it requires another fork (although this can be avoided using a custom compiler driver). Avoiding bytecode in the compilation toolchain is really the main benefit though. This was originally due to the fact that only the bytecode backend supported dynamic loading (which is used by camlp4 to load the syntax extension modules). When native dynamic code loading was added to OCaml circa 2009 or so, the build systems around camlp4 never really took up the change to switch to the much faster native code version. All the ppx extensions emerging now are at least guaranteed to have a native code version on all the major supported architectures (x86, ARM, Sparc, PPC)
obamacaress 12y ago
been writing OCaml since years. Def good news
pjmlp 12y ago
Interesting, seem similar to annotations/attributes. Also some similarity with AST handling in other languages. Now I have something to play with on the upcoming weekend. :)
p4bl0 12y ago
I'm not convinced by the example given at the end of the article. For this precise very simple use case it might be ok, but what about more complicated syntax extensions?
- avsm 12y ago
  Camlp4 is still very much around and maintained out of tree, and will work in the next release of OCaml (https://github.com/ocaml/camlp4 https://github.com/ocaml/camlp4). Extension points are intended to be simpler and more easily integrated into editors for the common case, but you can still use camlp4 for extensions that genuinely extend the OCaml grammar.
wes-exp 12y ago
As languages improve, they keep moving towards Lisp — a story that has been playing out for decades. There is a reason Lisp is called "the final language".
- yminsky 12y ago
  I think Lisp and Scheme have excellent macro systems, which are eminently worth stealing. I don't think OCaml will evolve into them on the type-system level though...
  - hajile 12y ago
    Shen lisp has the world's only turing complete type system (making it even more advanced than Haskell's system).
    - emiljbs 12y ago
      Do you really want a turing complete type system? Being turing complete does not necessarily mean more powerful for real world applications.
    - read 12y ago
      Thank you for Shen Lisp, I wasn't aware of it. The downside of the phrase Turing complete is that there isn't a practical limitation that instantly comes to mind when you hear that phrase. You don't hear people say "Oh no, that wouldn't be Turing complete". Complete or not, how does it affect a real application?
      - pjc50 12y ago
        For a type system, being Turing complete is a disadvantage: you can't prove that your typechecker will always terminate.
        chriswarbo 12y ago
        The result is even stronger: we can prove that it will sometimes not terminate! Still that's not too bad, since type checking is always conservative: if a program passes, we know it's correct(ly typed). If it doesn't pass, we don't gain any knowledge: it may be incorrect, or it may be correct in a way which the type checker's limited algorithm cannot determine (ie. a Goedel sentence). I'm very concerned whether my programs terminate/coterminate, since having to kill them part-way-through could cause corruption and other nastiness. Whether the type-checker terminates or not I don't really care about; I can just kill it after a certain timeout and keep fiddling with my code until it passes, just like any other type error. Of course, a timeout removes Turing-completeness, but that timeout is under my control at the commandline, rather than being an inherent property of the algorithm.
lispm 12y ago
One thing that's not nice in typical Lisp macros: for really complex transformations one needs a code walker. Unfortunately that part has a few problems: no standard code walker, complex api, different ways to access crucial information about the code in various implementations, ...
- rwmj 12y ago
  OCaml has a type-safe AST walker (two actually - one is part of camlp4 and the other called ocaml-ast-analyze).
a-nikolaev 12y ago
Cool. Language extensions are easier to write. Honestly, never had the guts to really learn camlp4 beyond an introductory tutorial, so I am happy to hear that this new approach is easier and also more practical. There is a link from the article with a more complete example of how the extension points really work: http://whitequark.org/blog/2014/04/16/a-guide-to-extension-points-in-ocaml/ http://whitequark.org/blog/2014/04/16/a-guide-to-extension-p... OCaml is great because it is doing a lot of things right. Hope, this new feature will become more widely used than camlp4, and become a stable feature of the language.

104 points by edwintorok 12y ago

It would be interesting to see how the speed of -ppx extensions compares to camlp4. If camlp4 is run in bytecode mode it can be quite slow, I looked at 'top' while running 'opam install core' at some point, and most of the time was taken by camlp4. Since -ppx extensions are supposed to run natively (without requiring natdynlink support even) I would expect them to be faster. Has anyone done any comparisons?

avsm 12y ago
There are pros and cons to not using dynlink, since it requires another fork (although this can be avoided using a custom compiler driver). Avoiding bytecode in the compilation toolchain is really the main benefit though. This was originally due to the fact that only the bytecode backend supported dynamic loading (which is used by camlp4 to load the syntax extension modules). When native dynamic code loading was added to OCaml circa 2009 or so, the build systems around camlp4 never really took up the change to switch to the much faster native code version. All the ppx extensions emerging now are at least guaranteed to have a native code version on all the major supported architectures (x86, ARM, Sparc, PPC)

obamacaress 12y ago

been writing OCaml since years. Def good news

pjmlp 12y ago

Interesting, seem similar to annotations/attributes. Also some similarity with AST handling in other languages. Now I have something to play with on the upcoming weekend. :)

p4bl0 12y ago

I'm not convinced by the example given at the end of the article. For this precise very simple use case it might be ok, but what about more complicated syntax extensions?

avsm 12y ago
Camlp4 is still very much around and maintained out of tree, and will work in the next release of OCaml (https://github.com/ocaml/camlp4 https://github.com/ocaml/camlp4). Extension points are intended to be simpler and more easily integrated into editors for the common case, but you can still use camlp4 for extensions that genuinely extend the OCaml grammar.

wes-exp 12y ago

As languages improve, they keep moving towards Lisp — a story that has been playing out for decades. There is a reason Lisp is called "the final language".

yminsky 12y ago
I think Lisp and Scheme have excellent macro systems, which are eminently worth stealing. I don't think OCaml will evolve into them on the type-system level though...
- hajile 12y ago
  Shen lisp has the world's only turing complete type system (making it even more advanced than Haskell's system).
  - emiljbs 12y ago
    Do you really want a turing complete type system? Being turing complete does not necessarily mean more powerful for real world applications.
  - read 12y ago
    Thank you for Shen Lisp, I wasn't aware of it. The downside of the phrase Turing complete is that there isn't a practical limitation that instantly comes to mind when you hear that phrase. You don't hear people say "Oh no, that wouldn't be Turing complete". Complete or not, how does it affect a real application?
    - pjc50 12y ago
      For a type system, being Turing complete is a disadvantage: you can't prove that your typechecker will always terminate.
      - chriswarbo 12y ago
        The result is even stronger: we can prove that it will sometimes not terminate! Still that's not too bad, since type checking is always conservative: if a program passes, we know it's correct(ly typed). If it doesn't pass, we don't gain any knowledge: it may be incorrect, or it may be correct in a way which the type checker's limited algorithm cannot determine (ie. a Goedel sentence). I'm very concerned whether my programs terminate/coterminate, since having to kill them part-way-through could cause corruption and other nastiness. Whether the type-checker terminates or not I don't really care about; I can just kill it after a certain timeout and keep fiddling with my code until it passes, just like any other type error. Of course, a timeout removes Turing-completeness, but that timeout is under my control at the commandline, rather than being an inherent property of the algorithm.

lispm 12y ago

One thing that's not nice in typical Lisp macros: for really complex transformations one needs a code walker. Unfortunately that part has a few problems: no standard code walker, complex api, different ways to access crucial information about the code in various implementations, ...

rwmj 12y ago
OCaml has a type-safe AST walker (two actually - one is part of camlp4 and the other called ocaml-ast-analyze).

a-nikolaev 12y ago

Cool. Language extensions are easier to write. Honestly, never had the guts to really learn camlp4 beyond an introductory tutorial, so I am happy to hear that this new approach is easier and also more practical. There is a link from the article with a more complete example of how the extension points really work: http://whitequark.org/blog/2014/04/16/a-guide-to-extension-points-in-ocaml/ http://whitequark.org/blog/2014/04/16/a-guide-to-extension-p... OCaml is great because it is doing a lot of things right. Hope, this new feature will become more widely used than camlp4, and become a stable feature of the language.