24 ms·

The road to OCaml 5.0

166 points by wczekalski 5y ago

short_sells_poo 5y ago
I run a hedge fund. On any given day I hear a large number of complaints from the technologists that complex python systems are difficult to look after and we should use something else instead. There's some Rust being used, but there's little chance to get a quant to use Rust to do research because research is an exploratory process and the last thing one wants is a language that requires a lot of thought about lifetimes etc. How is the python-ocaml interop story? To be clear, any language that does not have first-class interop with python is basically dead in the water (at least for our case).
- SquishyPanda23 5y ago
  Some of the Python FFI tools are listed here: https://ocamlverse.github.io/content/ffi.html https://ocamlverse.github.io/content/ffi.html. But clicking through to GitHub, the repos haven't been updated in a while.
  - thingification 5y ago
    https://github.com/thierry-martinez/pyml https://github.com/thierry-martinez/pyml -- 2 days ago
    - SquishyPanda23 5y ago
      I wasn't counting changes to project metadata like gitignore
- jstimpfle 5y ago
  > last thing one wants is a language that requires a lot of thought about lifetimes etc. I challenge you: A lack of understanding about the data lifetimes in a program means lack of understanding about the data. Not saying you can't have a lot of short-lived data items that you don't want to manage one-by-one. I'm saying that for the vast majority of data items, one should be able to give a reasonably well defined lifetime upper bound. So a good solution is to make a few boxes that group items by lifetime. And from time to time, throw the outdated boxes away. And of the few items that don't have such an upper bound at creation time, many can be created in a special box that allows migrating boxes later when required.
  - chrisseaton 5y ago
    > A lack of understanding about the data lifetimes in a program means lack of understanding about the data. But this argument can extend forever. Is your program precisely dependently typed? If not is that a lack of understanding about the nature of the data as well and should you challenge yourself to fix that? You have to trade-off how much you specify things with how valuable it is to get the result more quickly.
    - jstimpfle 5y ago
      What you say is true. I only brought up "boxes" because the concept is still not widely known.
    - deleted 5y ago
      [deleted]
  - jenny91 5y ago
    You don't have to challenge the person you're responding to. You have to challenge their quants. And they're not going to want to add that into the million other things they're thinking about while doing research in a Jupyter notebook or something. You're just not going to get this buy-in from people who want to use a tool to get their work done.
  - short_sells_poo 5y ago
    Thanks, but I think we may be talking cross purpose here. 99% of the research code ends up being thrown away (well, archived). Not because it's bad code necessarily, but because the idea that was being prototyped is a dead end. This means it's paramount that the language you use has to be as low friction and interactive as possible. Imagine you are trying to establish whether there's a relationship between timeseries X and timeseries Y. You just want a tool that allows you to quickly calculate some summary statistics of these timeseries, clean them, convince yourself that they behave according to your expectations and then run some form of regression. Nowhere in this process do you care about lifetimes. It's literally irrelevant. In fact, as long as all your work fits into memory, you don't even care about memory management. Your objective is to answer the primary question, everything else is a costly distraction. The 1% of ideas that ends up being worthwhile is what gets productionized and needs to be robust. But obviously rewriting everything from language A to radically different language B adds it's own headaches.
- bhy 5y ago
  Python type checking (type annotation, mypy) should at least partially solve the problem of maintaining complex Python systems. Though it doesn't help with performance.
  - pharmakom 5y ago
    The larger problem in my view is that big Python systems tend to follow OOP design since functional programming patterns do not work well in Python. So you start with something minimal and simple inside a script or notebook, but quickly it evolves into something more like a Java code-base. Typing does help, agreed.
    - nerdponx 5y ago
      I strongly suggest the Attrs library for cutting down the boilerplate of making small "data classes": https://attrs.org/ https://attrs.org/ With type annotations, you can move away from "inheritance OO" to logicless "data classes" and functions that operate on them.
    - awild 5y ago
      Java has good to great FP, making most objects immutable is also trivial. Mypy is good to get strict typing enforced but, but I still prefer the native and slightly wonky typesystem to a tacked on one that builds on trust.
      - dunefox 5y ago
        > Java has good to great FP Java does absolutely not have "good to great FP" support. It's an imperative and OO programming language that recently got lambdas, no more, no less.
        Zababa 5y ago
        There's sealed classes, records, pattern matching, optionals. It's getting there.
- deleted 5y ago
  [deleted]
- hajile 5y ago
  If your language doesn't worry about lifetimes, they don't go away. It just means you have to worry about them yourself instead. Sometimes that is great. Other times, that will be very hard and error-prone.
  - typon 5y ago
    An extremely simple thing like having two objects stored in a struct where one object has a reference to the other is a Herculean task in Rust. This is not a language designed for prototyping...
    - carlmr 5y ago
      You can easily avoid such constructs in most cases though.
      - typon 5y ago
        I've allocated both these things on the heap and Rust simply won't let me store them both together. I don't know about you but this is an extremely common pattern in almost all languages, you just don't think about it in the gc languages and in C++ storing pointers is no issue. The popularity of crates like rental also shows that it's not as easily avoidable as you suggest.
        jjnoakes 5y ago
        Can you post a simple code example? It is hard to imagine what the difficulty is.
        typon 5y ago
        You can read through this SO question and the top response does a good job explaining the possible solutions: https://stackoverflow.com/questions/32300132/why-cant-i-store-a-value-and-a-reference-to-that-value-in-the-same-struct https://stackoverflow.com/questions/32300132/why-cant-i-stor...
- nerdponx 5y ago
  Have you looked into Julia, Nim, Clojure, or even Common Lisp? I'm not sure bout Python interop with CL, but Nim and Clojure seems to have some kind of beta-grade interop, and there's a solid interop story in Julia. And all of those languages have some of their own "native" data analysis and scientific computing toolkits (Julia having more than "some", of course). That said, complicated Python systems can be improved a lot by adding type annotations. That's more of a solution for web servers and other "easily type-able" applications. Typing support for scientific computing isn't quite there yet. So it depends on what kinds of systems are the complicated ones.
  - lgessler 5y ago
    Link to the interop lib for Clojure you're referring to for people who don't know it: https://github.com/clj-python/libpython-clj https://github.com/clj-python/libpython-clj Really a remarkable feat of engineering. Here's its author giving a talk: https://www.youtube.com/watch?v=vQPW16_jixs https://www.youtube.com/watch?v=vQPW16_jixs
    - adenozine 5y ago
      I was thinking of linking this as well. Clj-python is just such a fascinating junction. I don't care so much for clojure, though I'm continuously impressed by the ecosystem and the productivity of it's experts. Very cool stuff. Core.logic and the like opened a huge door and similar ideas have exhausted my free time for several years now.
  - dunefox 5y ago
    Julias interop with Python is excellent: https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl (also with R, see RCall.jl). It's just not statically typed, so the original problem is not solved - albeit Julia being the better language for scientific purposes. CL could also be great language-wise (https://digikar99.github.io/py4cl2/ https://digikar99.github.io/py4cl2/, https://github.com/snmsts/burgled-batteries3 https://github.com/snmsts/burgled-batteries3) but I don't know how good the interop is in reality since I haven't tried it.
  - 5y ago
- ajoseps 5y ago
  I personally haven't used it but Jane Street heavily uses OCaml and has written a blog post on this: https://blog.janestreet.com/using-python-and-ocaml-in-the-same-jupyter-notebook/ https://blog.janestreet.com/using-python-and-ocaml-in-the-sa...
- mtoner23 5y ago
  Its not the language its the people, instagram is almost entirely run on python, if they can so can you. https://instagram-engineering.com/tagged/python https://instagram-engineering.com/tagged/python
  - nv-vn 5y ago
    This is a terribly misinformed take. If you throw enough resources at Python then sure, you can probably get adequate throughput. The problem is that in finance a lot of problems require you to think about latency, which is a total non-starter for Python
    - mtoner23 5y ago
      If it were a total non starter than why would their entire company be using it?
      - short_sells_poo 5y ago
        I have no doubts that we are able to handle our requirements with python, but if there's a better way, does it not make sense to investigate? A skilled carpenter can undoubtedly use a hammer instead of a screwdriver. This doesn't mean that I should insist they use a hammer when a screwdriver would do a better job.
      - nv-vn 5y ago
        Sounds like they're already using Rust where performance matters and are looking to switch
      - upbeat_general 5y ago
        Instagram has much lower latency requirements…
        carnitine 5y ago
        I believe they’re referring to the hedge fund. Lower latency requirement is a confusing way of phrasing it as lower latency is a higher bar. But generally most hedge funds do not need to operate at the speed Instagram does.
- typon 5y ago
  I would write principled Python with strict coding standards. Make type annotations mandatory and turn up pylint or flake8 to maximum warnings. It really helps avoid a bunch of silly mistakes, while still providing a way out for doing crazy stuff that Python is good at.
- rkangel 5y ago
  I think Elixir would be interesting for your usecase. It's a dynamic, garbage collected language. It's easy to pick up and get going with. As a functional programming language there isn't a lot to learn in the way of language constructs, and you don't even have to do the 'wrestling with the type system' thing that you have to do in compiled functional languages like OCaml or Haskell (like you do in Rust). Its processing 'horsepower' is probably comparable to Python, but it's much better for building low latency things if you want to run something in a bit more of a production use case. This is also improving due to the recent addition of a JIT. The addition of NX is making Elixir an increasingly interesting place to do ML - write Elixir, have it run on GPU etc. See https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-available https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-... Python integration is probably best done using the Erlang 'port' system - running Python as a managed process and communicating with it using messages over stdin/stdout. I use it for C interop and it works well (and fits well with the Elixir/Erlang process model). It's not difficult to roll your own in Python e.g. https://github.com/fujimisakari/erlang-port-with-python/blob/master/erlang_port/protocol.py/ https://github.com/fujimisakari/erlang-port-with-python/blob... or look at something like http://erlport.org/ http://erlport.org/
  - short_sells_poo 5y ago
    Thank you! So this looks interesting but it seems like there's no easy way to share numpy arrays? The main use case for a language other than python is a more robust codebase but also performance. We need to be able to efficiently ship lots of large arrays between the languages and the Rust-Python interop supports zero copy arrays for example.
    - pdimitar 5y ago
      Elixir and Rust are very good friends, so to speak. Writing a library in Rust that you can use from Elixir is only slightly worse than trivially easy. But I agree somebody has to put the work. I've made a good career with Elixir but I still don't think it's a good fit for a hedge fund. IMO invest in Rust.
    - rkangel 5y ago
- philzook 5y ago
  There is an actively developed python to ocaml interop library for purposes quite similar to yours. I have seen demos where ocaml and python are used within the same jupyter notebook https://signalsandthreads.com/python-ocaml-and-machine-learning/ https://signalsandthreads.com/python-ocaml-and-machine-learn... https://github.com/thierry-martinez/pyml https://github.com/thierry-martinez/pyml
  - short_sells_poo 5y ago
    Thank you, I'll pass this on. An important feature is zero copy arrays, which seem to be supported.
Rickasaurus 5y ago
Dang, it's finally happening. I've been waiting for 10+ years.
ubertaco 5y ago
>Hopefully, OCaml 5.0 will then be released between March and April 2022. Just to call out expectation-setting here in the comments: yes, the MVP of multicore will ship in OCaml 5.0, but OCaml 5.0 will ship no sooner than March 2022 (and very likely some point later, based on how challenging it appears to be to integrate the large-scale changes for multicore).
Iv 5y ago
<Obi Wan's voice> "OCaml... That's a name I haven't heard in a long time..."
adultSwim 5y ago
That's great. It's a terrific language. Python let's me start programming quickly. ML let's me finish quickly.
Jenz 5y ago
A lighthearted but very true remark: OCaml is a wonderful language.
davesnx 5y ago
This is a great explanation about concurrency and parallelism and where multicore fits, FYI https://discuss.ocaml.org/t/multicore-ocaml-vs-thread/5838/12 https://discuss.ocaml.org/t/multicore-ocaml-vs-thread/5838/1...
cardanome 5y ago
I really hope to see more interest in OCaml in the future. It is probably one of the most underrated programming languages. The perfect marriage between state of the art functional programming and pragmatism. A great static and strong type system. Solid performance and an insanely fast compiler. Also compiles to JS if you need that. Multicore support will make it quite perfect. Only thing that is holding it back more than that and the reason I have not done many projects with it, is it weirdly fragmented ecosystem. Having to decide which standard library to use is a pain but you can cope with that. Tooling is getting there but stuff like automatic code formatting solutions are still pretty immature (and have really weird defaults). Frontend there is that ReasonML/Reason/ReScript thing that Facebook it trying to do. It offers an alternative syntax but nearly nobody uses it because they changed the name and I think also the syntax three times already. So it is all a mess. Don't let that stop you though. There are some pretty solid mature libraries in OCaml and if need be interop story with C and other languages is solid.
- grumpyprole 5y ago
  If they can also ship algebraic effects (and even better, typed algebraic effects), then I think it will push the language back firmly into "state of the art". This will mean it continues to get the attention it deserves. I'm excited about algebraic effects, I think they are much more intuitive than monads (and don't require code to be rewritten).
  - octachron 5y ago
    The current plan is to have runtime support for effects in 5.0, but without a syntax nor an effect type system. Those two will come later in the 5.x branch. The aim is to decouple the switch to the multicore runtime from the design of the typed effect system. In the interim period, effect handlers would be exposed through an experimental module (exposed through an experimental module (see https://discuss.ocaml.org/t/multicore-ocaml-september-2021-effect-handlers-will-be-in-ocaml-5-0 https://discuss.ocaml.org/t/multicore-ocaml-september-2021-e...) to allow early experimentation.
    - 5y ago
rawoke083600 5y ago
My favourite quote about OCaml: "Never have I took so long, to write so little code, that does so much" OCaml can be a big learning curve, but I urge you to push through. The syntax might not be everyone's cup of team, but you get used to it quickly.
- girzel 5y ago
  I really wanted to settle on OCaml as the "real programming language" that I would learn for any "serious programming" I had to do. I couldn't make it stick (in part because I don't actually do any "serious programming") precisely because of the syntax. There's too little of it! OCaml seems to take a "you don't need syntax except when you need syntax" approach, which I found very destabilizing. One of the major online OCaml tutorials said something like "If it doesn't work the way you expect, try adding parentheses", and I thought "Oh hell no. In a Lisp I know exactly how many parentheses I need: all of them". I prefer not having to think about it, and letting the parentheses become invisible to me. But otherwise I have a deep and irrational fondness for the language, and still wish I'd been able to make it stick.
  - Zababa 5y ago
    > One of the major online OCaml tutorials said something like "If it doesn't work the way you expect, try adding parentheses" That sounds like what I did with C++ with * and & when I didn't understood them. Do you think it's a lack of exprience/comprehension on your part, or that some parts of the syntax are fundamentally flawed?
    - girzel 5y ago
      Well anything can be chalked up to lack of experience -- I'm sure if I was programming OCaml every day it wouldn't be an issue. Nor would I try to categorically claim the syntax is flawed! But so much of the syntax, particularly around function calls, is simply a long row of whitespace-separated tokens, and I feel like my brain has to do extra work to parse what's what, and figure out associativity, and constantly remember what the ~ and the ? are doing. This[1] section of the tutorial makes perfect sense when you read it, but that doesn't mean you can easily scan a long function call with a lot of arguments, and instantly see what's happening. The block-level syntax is great. But it got to the point where if I didn't write/read OCaml for a few days in a row, I forgot how it worked. And that's simply calling a function, nothing esoteric. [1]: https://ocaml.org/learn/tutorials/labels.html#When-and-when-not-to-use-and https://ocaml.org/learn/tutorials/labels.html#When-and-when-...
AzzieElbab 5y ago
I write a lot of Scala for living, Ocaml looks a bit outdated to me. Having said that, Ocaml compiler is one of the greatest miracles in PL when it comes to speed vs complexity of the language. Scala/Haskell/TS are not even close. I hear Ocaml's runtime performance is not too shabby either
- bobbylarrybobby 5y ago
  Out of curiosity, what problems does Haskell's compiler(s) (I think it's really just GHC these days?) face that OCaml's doesn't?
  - AzzieElbab 5y ago
    speed. Ocaml compiler is probably as fast as Go one
    - pjmlp 5y ago
      GHC has many backends and there is GHCi as well. No need to always go through the slowest path.
      - Zababa 5y ago
        The regular backend is the fastest according to their docs: https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/codegens.html https://downloads.haskell.org/~ghc/latest/docs/html/users_gu.... Having an interactive toplevel isn't a substitute for fast compilation (and OCaml has both).
- Zababa 5y ago
  What do you find outdated about it? > Having said that, Ocaml compiler is one of the greatest miracles in PL when it comes to speed vs complexity of the language. Scala/Haskell/TS are not even close. Someone will probably come correct me but what I've heard is that the compilation speed partially comes from the Pascal/Modula-3 influence, since Niklaus Wirth took compilation time into account when designing programming languages. From what I understand, OCaml doesn't allow circular dependencies outside of a single file, and that helps. Go doesn't allow them too, and is also known for its compilation speed.
bigjimslade 5y ago
I really wanted to like OCaml, still do. I gave it a good shot a couple of years ago, wrote a few basic programs and loved it. But it to me seemed packaged like many languages in the days of yore, when a language shipped simply as a compiler, and nothing more. The way of the world today to me seems to be a compiler, together with a complete standard library and consistent packaging system. My experience with OCaml was thwarted repeatedly by a byzantine exploration process of packages depending on other packages, which required other packaging systems. Once I reached that point where it felt like I was spending more time figuring out the complex ecosystem, rather than writing code, I rapidly lost interest. And perhaps such a point comes in exploring any new language. But it came much too early for me in OCaml. I had so much more I wanted to learn, but couldn't. I am hopeful for the new release. Thank you for your efforts, OCaml team.
- thingification 5y ago
  Not sure when you tried it, but as a newcomer I have the impression packaging has got a lot better in the past few years. I didn't have the experience you described (not yet anyway!).
- yawaramin 5y ago
  A couple of years ago, opam was the recommended package manager and dune was the recommended build system–just as today. The opam package index was also searchable for libraries. The OCaml website may have been slightly less clear about these things than it is now, but I think a reasonable user would have been able to find them, especially if they went to the forum and asked. People would have gladly answered questions.
  - Zababa 5y ago
    I don't think that's totally fair. The Up and Running page of the OCaml website (https://ocaml.org/learn/tutorials/up_and_running.html https://ocaml.org/learn/tutorials/up_and_running.html) was added during 2020. Before that it lacked a straighforward introduction on what you need and how to install it. Node, Go and Rust all come with the package manager, and Rust even comes with a way of managing the different Rust versions. The essential part are here, and everything works well, but for new users it lacks polishing. You can argue that it would take a lot of time for a community that is a bit short on manpower, and that's true. But in the end the experience isn't as good as with other ecosystems.

166 points by wczekalski 5y ago

I run a hedge fund. On any given day I hear a large number of complaints from the technologists that complex python systems are difficult to look after and we should use something else instead. There's some Rust being used, but there's little chance to get a quant to use Rust to do research because research is an exploratory process and the last thing one wants is a language that requires a lot of thought about lifetimes etc. How is the python-ocaml interop story? To be clear, any language that does not have first-class interop with python is basically dead in the water (at least for our case).

SquishyPanda23 5y ago
Some of the Python FFI tools are listed here: https://ocamlverse.github.io/content/ffi.html https://ocamlverse.github.io/content/ffi.html. But clicking through to GitHub, the repos haven't been updated in a while.
- thingification 5y ago
  https://github.com/thierry-martinez/pyml https://github.com/thierry-martinez/pyml -- 2 days ago
  - SquishyPanda23 5y ago
    I wasn't counting changes to project metadata like gitignore
jstimpfle 5y ago
> last thing one wants is a language that requires a lot of thought about lifetimes etc. I challenge you: A lack of understanding about the data lifetimes in a program means lack of understanding about the data. Not saying you can't have a lot of short-lived data items that you don't want to manage one-by-one. I'm saying that for the vast majority of data items, one should be able to give a reasonably well defined lifetime upper bound. So a good solution is to make a few boxes that group items by lifetime. And from time to time, throw the outdated boxes away. And of the few items that don't have such an upper bound at creation time, many can be created in a special box that allows migrating boxes later when required.
- chrisseaton 5y ago
  > A lack of understanding about the data lifetimes in a program means lack of understanding about the data. But this argument can extend forever. Is your program precisely dependently typed? If not is that a lack of understanding about the nature of the data as well and should you challenge yourself to fix that? You have to trade-off how much you specify things with how valuable it is to get the result more quickly.
  - jstimpfle 5y ago
    What you say is true. I only brought up "boxes" because the concept is still not widely known.
  - deleted 5y ago
    [deleted]
- jenny91 5y ago
  You don't have to challenge the person you're responding to. You have to challenge their quants. And they're not going to want to add that into the million other things they're thinking about while doing research in a Jupyter notebook or something. You're just not going to get this buy-in from people who want to use a tool to get their work done.
- short_sells_poo 5y ago
  Thanks, but I think we may be talking cross purpose here. 99% of the research code ends up being thrown away (well, archived). Not because it's bad code necessarily, but because the idea that was being prototyped is a dead end. This means it's paramount that the language you use has to be as low friction and interactive as possible. Imagine you are trying to establish whether there's a relationship between timeseries X and timeseries Y. You just want a tool that allows you to quickly calculate some summary statistics of these timeseries, clean them, convince yourself that they behave according to your expectations and then run some form of regression. Nowhere in this process do you care about lifetimes. It's literally irrelevant. In fact, as long as all your work fits into memory, you don't even care about memory management. Your objective is to answer the primary question, everything else is a costly distraction. The 1% of ideas that ends up being worthwhile is what gets productionized and needs to be robust. But obviously rewriting everything from language A to radically different language B adds it's own headaches.
bhy 5y ago
Python type checking (type annotation, mypy) should at least partially solve the problem of maintaining complex Python systems. Though it doesn't help with performance.
- pharmakom 5y ago
  The larger problem in my view is that big Python systems tend to follow OOP design since functional programming patterns do not work well in Python. So you start with something minimal and simple inside a script or notebook, but quickly it evolves into something more like a Java code-base. Typing does help, agreed.
  - nerdponx 5y ago
    I strongly suggest the Attrs library for cutting down the boilerplate of making small "data classes": https://attrs.org/ https://attrs.org/ With type annotations, you can move away from "inheritance OO" to logicless "data classes" and functions that operate on them.
  - awild 5y ago
    Java has good to great FP, making most objects immutable is also trivial. Mypy is good to get strict typing enforced but, but I still prefer the native and slightly wonky typesystem to a tacked on one that builds on trust.
    - dunefox 5y ago
      > Java has good to great FP Java does absolutely not have "good to great FP" support. It's an imperative and OO programming language that recently got lambdas, no more, no less.
      - Zababa 5y ago
        There's sealed classes, records, pattern matching, optionals. It's getting there.
deleted 5y ago
[deleted]
hajile 5y ago
If your language doesn't worry about lifetimes, they don't go away. It just means you have to worry about them yourself instead. Sometimes that is great. Other times, that will be very hard and error-prone.
- typon 5y ago
  An extremely simple thing like having two objects stored in a struct where one object has a reference to the other is a Herculean task in Rust. This is not a language designed for prototyping...
  - carlmr 5y ago
    You can easily avoid such constructs in most cases though.
    - typon 5y ago
      I've allocated both these things on the heap and Rust simply won't let me store them both together. I don't know about you but this is an extremely common pattern in almost all languages, you just don't think about it in the gc languages and in C++ storing pointers is no issue. The popularity of crates like rental also shows that it's not as easily avoidable as you suggest.
      - jjnoakes 5y ago
        Can you post a simple code example? It is hard to imagine what the difficulty is.
        typon 5y ago
        You can read through this SO question and the top response does a good job explaining the possible solutions: https://stackoverflow.com/questions/32300132/why-cant-i-store-a-value-and-a-reference-to-that-value-in-the-same-struct https://stackoverflow.com/questions/32300132/why-cant-i-stor...
nerdponx 5y ago
Have you looked into Julia, Nim, Clojure, or even Common Lisp? I'm not sure bout Python interop with CL, but Nim and Clojure seems to have some kind of beta-grade interop, and there's a solid interop story in Julia. And all of those languages have some of their own "native" data analysis and scientific computing toolkits (Julia having more than "some", of course). That said, complicated Python systems can be improved a lot by adding type annotations. That's more of a solution for web servers and other "easily type-able" applications. Typing support for scientific computing isn't quite there yet. So it depends on what kinds of systems are the complicated ones.
- lgessler 5y ago
  Link to the interop lib for Clojure you're referring to for people who don't know it: https://github.com/clj-python/libpython-clj https://github.com/clj-python/libpython-clj Really a remarkable feat of engineering. Here's its author giving a talk: https://www.youtube.com/watch?v=vQPW16_jixs https://www.youtube.com/watch?v=vQPW16_jixs
  - adenozine 5y ago
    I was thinking of linking this as well. Clj-python is just such a fascinating junction. I don't care so much for clojure, though I'm continuously impressed by the ecosystem and the productivity of it's experts. Very cool stuff. Core.logic and the like opened a huge door and similar ideas have exhausted my free time for several years now.
- dunefox 5y ago
  Julias interop with Python is excellent: https://github.com/JuliaPy/PyCall.jl https://github.com/JuliaPy/PyCall.jl (also with R, see RCall.jl). It's just not statically typed, so the original problem is not solved - albeit Julia being the better language for scientific purposes. CL could also be great language-wise (https://digikar99.github.io/py4cl2/ https://digikar99.github.io/py4cl2/, https://github.com/snmsts/burgled-batteries3 https://github.com/snmsts/burgled-batteries3) but I don't know how good the interop is in reality since I haven't tried it.
- 5y ago
ajoseps 5y ago
I personally haven't used it but Jane Street heavily uses OCaml and has written a blog post on this: https://blog.janestreet.com/using-python-and-ocaml-in-the-same-jupyter-notebook/ https://blog.janestreet.com/using-python-and-ocaml-in-the-sa...
mtoner23 5y ago
Its not the language its the people, instagram is almost entirely run on python, if they can so can you. https://instagram-engineering.com/tagged/python https://instagram-engineering.com/tagged/python
- nv-vn 5y ago
  This is a terribly misinformed take. If you throw enough resources at Python then sure, you can probably get adequate throughput. The problem is that in finance a lot of problems require you to think about latency, which is a total non-starter for Python
  - mtoner23 5y ago
    If it were a total non starter than why would their entire company be using it?
    - short_sells_poo 5y ago
      I have no doubts that we are able to handle our requirements with python, but if there's a better way, does it not make sense to investigate? A skilled carpenter can undoubtedly use a hammer instead of a screwdriver. This doesn't mean that I should insist they use a hammer when a screwdriver would do a better job.
    - nv-vn 5y ago
      Sounds like they're already using Rust where performance matters and are looking to switch
    - upbeat_general 5y ago
      Instagram has much lower latency requirements…
      - carnitine 5y ago
        I believe they’re referring to the hedge fund. Lower latency requirement is a confusing way of phrasing it as lower latency is a higher bar. But generally most hedge funds do not need to operate at the speed Instagram does.
typon 5y ago
I would write principled Python with strict coding standards. Make type annotations mandatory and turn up pylint or flake8 to maximum warnings. It really helps avoid a bunch of silly mistakes, while still providing a way out for doing crazy stuff that Python is good at.
rkangel 5y ago
I think Elixir would be interesting for your usecase. It's a dynamic, garbage collected language. It's easy to pick up and get going with. As a functional programming language there isn't a lot to learn in the way of language constructs, and you don't even have to do the 'wrestling with the type system' thing that you have to do in compiled functional languages like OCaml or Haskell (like you do in Rust). Its processing 'horsepower' is probably comparable to Python, but it's much better for building low latency things if you want to run something in a bit more of a production use case. This is also improving due to the recent addition of a JIT. The addition of NX is making Elixir an increasingly interesting place to do ML - write Elixir, have it run on GPU etc. See https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-available https://dashbit.co/blog/nx-numerical-elixir-is-now-publicly-... Python integration is probably best done using the Erlang 'port' system - running Python as a managed process and communicating with it using messages over stdin/stdout. I use it for C interop and it works well (and fits well with the Elixir/Erlang process model). It's not difficult to roll your own in Python e.g. https://github.com/fujimisakari/erlang-port-with-python/blob/master/erlang_port/protocol.py/ https://github.com/fujimisakari/erlang-port-with-python/blob... or look at something like http://erlport.org/ http://erlport.org/
- short_sells_poo 5y ago
  Thank you! So this looks interesting but it seems like there's no easy way to share numpy arrays? The main use case for a language other than python is a more robust codebase but also performance. We need to be able to efficiently ship lots of large arrays between the languages and the Rust-Python interop supports zero copy arrays for example.
  - pdimitar 5y ago
    Elixir and Rust are very good friends, so to speak. Writing a library in Rust that you can use from Elixir is only slightly worse than trivially easy. But I agree somebody has to put the work. I've made a good career with Elixir but I still don't think it's a good fit for a hedge fund. IMO invest in Rust.
  - rkangel 5y ago
philzook 5y ago
There is an actively developed python to ocaml interop library for purposes quite similar to yours. I have seen demos where ocaml and python are used within the same jupyter notebook https://signalsandthreads.com/python-ocaml-and-machine-learning/ https://signalsandthreads.com/python-ocaml-and-machine-learn... https://github.com/thierry-martinez/pyml https://github.com/thierry-martinez/pyml
- short_sells_poo 5y ago
  Thank you, I'll pass this on. An important feature is zero copy arrays, which seem to be supported.

Rickasaurus 5y ago

Dang, it's finally happening. I've been waiting for 10+ years.

ubertaco 5y ago

>Hopefully, OCaml 5.0 will then be released between March and April 2022. Just to call out expectation-setting here in the comments: yes, the MVP of multicore will ship in OCaml 5.0, but OCaml 5.0 will ship no sooner than March 2022 (and very likely some point later, based on how challenging it appears to be to integrate the large-scale changes for multicore).

Iv 5y ago

<Obi Wan's voice> "OCaml... That's a name I haven't heard in a long time..."

adultSwim 5y ago

That's great. It's a terrific language. Python let's me start programming quickly. ML let's me finish quickly.

Jenz 5y ago

A lighthearted but very true remark: OCaml is a wonderful language.

davesnx 5y ago

This is a great explanation about concurrency and parallelism and where multicore fits, FYI https://discuss.ocaml.org/t/multicore-ocaml-vs-thread/5838/12 https://discuss.ocaml.org/t/multicore-ocaml-vs-thread/5838/1...

cardanome 5y ago

I really hope to see more interest in OCaml in the future. It is probably one of the most underrated programming languages. The perfect marriage between state of the art functional programming and pragmatism. A great static and strong type system. Solid performance and an insanely fast compiler. Also compiles to JS if you need that. Multicore support will make it quite perfect. Only thing that is holding it back more than that and the reason I have not done many projects with it, is it weirdly fragmented ecosystem. Having to decide which standard library to use is a pain but you can cope with that. Tooling is getting there but stuff like automatic code formatting solutions are still pretty immature (and have really weird defaults). Frontend there is that ReasonML/Reason/ReScript thing that Facebook it trying to do. It offers an alternative syntax but nearly nobody uses it because they changed the name and I think also the syntax three times already. So it is all a mess. Don't let that stop you though. There are some pretty solid mature libraries in OCaml and if need be interop story with C and other languages is solid.

grumpyprole 5y ago
If they can also ship algebraic effects (and even better, typed algebraic effects), then I think it will push the language back firmly into "state of the art". This will mean it continues to get the attention it deserves. I'm excited about algebraic effects, I think they are much more intuitive than monads (and don't require code to be rewritten).
- octachron 5y ago
  The current plan is to have runtime support for effects in 5.0, but without a syntax nor an effect type system. Those two will come later in the 5.x branch. The aim is to decouple the switch to the multicore runtime from the design of the typed effect system. In the interim period, effect handlers would be exposed through an experimental module (exposed through an experimental module (see https://discuss.ocaml.org/t/multicore-ocaml-september-2021-effect-handlers-will-be-in-ocaml-5-0 https://discuss.ocaml.org/t/multicore-ocaml-september-2021-e...) to allow early experimentation.
  - 5y ago

rawoke083600 5y ago

My favourite quote about OCaml: "Never have I took so long, to write so little code, that does so much" OCaml can be a big learning curve, but I urge you to push through. The syntax might not be everyone's cup of team, but you get used to it quickly.

girzel 5y ago
I really wanted to settle on OCaml as the "real programming language" that I would learn for any "serious programming" I had to do. I couldn't make it stick (in part because I don't actually do any "serious programming") precisely because of the syntax. There's too little of it! OCaml seems to take a "you don't need syntax except when you need syntax" approach, which I found very destabilizing. One of the major online OCaml tutorials said something like "If it doesn't work the way you expect, try adding parentheses", and I thought "Oh hell no. In a Lisp I know exactly how many parentheses I need: all of them". I prefer not having to think about it, and letting the parentheses become invisible to me. But otherwise I have a deep and irrational fondness for the language, and still wish I'd been able to make it stick.
- Zababa 5y ago
  > One of the major online OCaml tutorials said something like "If it doesn't work the way you expect, try adding parentheses" That sounds like what I did with C++ with * and & when I didn't understood them. Do you think it's a lack of exprience/comprehension on your part, or that some parts of the syntax are fundamentally flawed?
  - girzel 5y ago
    Well anything can be chalked up to lack of experience -- I'm sure if I was programming OCaml every day it wouldn't be an issue. Nor would I try to categorically claim the syntax is flawed! But so much of the syntax, particularly around function calls, is simply a long row of whitespace-separated tokens, and I feel like my brain has to do extra work to parse what's what, and figure out associativity, and constantly remember what the ~ and the ? are doing. This[1] section of the tutorial makes perfect sense when you read it, but that doesn't mean you can easily scan a long function call with a lot of arguments, and instantly see what's happening. The block-level syntax is great. But it got to the point where if I didn't write/read OCaml for a few days in a row, I forgot how it worked. And that's simply calling a function, nothing esoteric. [1]: https://ocaml.org/learn/tutorials/labels.html#When-and-when-not-to-use-and https://ocaml.org/learn/tutorials/labels.html#When-and-when-...

AzzieElbab 5y ago

I write a lot of Scala for living, Ocaml looks a bit outdated to me. Having said that, Ocaml compiler is one of the greatest miracles in PL when it comes to speed vs complexity of the language. Scala/Haskell/TS are not even close. I hear Ocaml's runtime performance is not too shabby either

bobbylarrybobby 5y ago
Out of curiosity, what problems does Haskell's compiler(s) (I think it's really just GHC these days?) face that OCaml's doesn't?
- AzzieElbab 5y ago
  speed. Ocaml compiler is probably as fast as Go one
  - pjmlp 5y ago
    GHC has many backends and there is GHCi as well. No need to always go through the slowest path.
    - Zababa 5y ago
      The regular backend is the fastest according to their docs: https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/codegens.html https://downloads.haskell.org/~ghc/latest/docs/html/users_gu.... Having an interactive toplevel isn't a substitute for fast compilation (and OCaml has both).
Zababa 5y ago
What do you find outdated about it? > Having said that, Ocaml compiler is one of the greatest miracles in PL when it comes to speed vs complexity of the language. Scala/Haskell/TS are not even close. Someone will probably come correct me but what I've heard is that the compilation speed partially comes from the Pascal/Modula-3 influence, since Niklaus Wirth took compilation time into account when designing programming languages. From what I understand, OCaml doesn't allow circular dependencies outside of a single file, and that helps. Go doesn't allow them too, and is also known for its compilation speed.

bigjimslade 5y ago

I really wanted to like OCaml, still do. I gave it a good shot a couple of years ago, wrote a few basic programs and loved it. But it to me seemed packaged like many languages in the days of yore, when a language shipped simply as a compiler, and nothing more. The way of the world today to me seems to be a compiler, together with a complete standard library and consistent packaging system. My experience with OCaml was thwarted repeatedly by a byzantine exploration process of packages depending on other packages, which required other packaging systems. Once I reached that point where it felt like I was spending more time figuring out the complex ecosystem, rather than writing code, I rapidly lost interest. And perhaps such a point comes in exploring any new language. But it came much too early for me in OCaml. I had so much more I wanted to learn, but couldn't. I am hopeful for the new release. Thank you for your efforts, OCaml team.

thingification 5y ago
Not sure when you tried it, but as a newcomer I have the impression packaging has got a lot better in the past few years. I didn't have the experience you described (not yet anyway!).
yawaramin 5y ago
A couple of years ago, opam was the recommended package manager and dune was the recommended build system–just as today. The opam package index was also searchable for libraries. The OCaml website may have been slightly less clear about these things than it is now, but I think a reasonable user would have been able to find them, especially if they went to the forum and asked. People would have gladly answered questions.
- Zababa 5y ago
  I don't think that's totally fair. The Up and Running page of the OCaml website (https://ocaml.org/learn/tutorials/up_and_running.html https://ocaml.org/learn/tutorials/up_and_running.html) was added during 2020. Before that it lacked a straighforward introduction on what you need and how to install it. Node, Go and Rust all come with the package manager, and Rust even comes with a way of managing the different Rust versions. The essential part are here, and everything works well, but for new users it lacks polishing. You can argue that it would take a lot of time for a community that is a bit short on manpower, and that's true. But in the end the experience isn't as good as with other ecosystems.