5 ms·

I took a really hard look at OCaml a year ago, as I was running into performance issues with python. Lack of multicore support made me give up on it. Now that

I took a really hard look at OCaml a year ago, as I was running into performance issues with python. Lack of multicore support made me give up on it. Now that Rust is around and supporting multicore, that's probably where I'll be investing my time. I'd love to hear feedback from people who have used both Rust and OCaml.

e_d_g_a_r 11y ago
What were you doing that you needed multicore?
- dorfsmay 11y ago
  Pivoting hundreds of millions json entries and upload the results to an object store. I ended up using a combination of python threads and processes, which was more work I wanted to do, and still relatively slow.
  - chubot 11y ago
    Why not C or C++? OCaml and Rust aren't going to interface with Python nearly as well. Also, it's trivial to write multithreaded extensions in C or C++ (at least for data processing). You just have to make sure to release the GIL. IMO the trick is to just use the Python C API, and not use any wrappers like SWIG or ctypes. The Python C API is a little odd but it is fairly explicit. Also, it's better to write plain functions rather than classes, but for data processing that is natural anyway. If you need a class, do that part in Python. Writing "one way" extension functions (that don't call back into Python) is quite easy and can give you a huge performance boost. I think people get hung up on Python extensions because they are using TWO unfamiliar languages -- the wrapper language, and C/C++. But if you are just using one additional language, it's not hard to figure out.
    - pcwalton 11y ago
      Why won't Rust interface with a C API?
      - kibwen 11y ago
        Indeed, I think that taking the place of C for writing Python and Ruby extensions will be an enormous opportunity for Rust.
      - chubot 11y ago
        If it has C ABI compatibility, then technically it can. That doesn't mean it's easier to interface to Python than C or C++ though. I'd find it hard to imagine it being easier, and I know it's not simpler. I'll have to watch the video below, but don't you have to write some sort of Rust wrapper for every definition in Python.h? What about macros like Py_INCREF and Py_DECREF? I'd guess that someone has or will eventually do that work, but it's yet another layer of complexity, which can have the same downsides as SWIG. For better or worse, the Python C API is tighlty coupled to the C implementation. That makes it very difficult to be more natural and understandable than C. It's a fairly huge API: https://docs.python.org/2/c-api/index.html https://docs.python.org/2/c-api/index.html People always try to cover it up with "nice" abstractions, but they invariably end up being leaky (e.g. with respect to threads, garbage collection, OS portability, etc.) Another huge can of worms: the build system. With a plain C extension, all I need is a C compiler on the system, and I can just do "python setup.py build". The situation with Windows is also quite messy -- I can't imagine Rust making it better. I have experimented with creating normal .o files from OCaml and linking them with .o files from C/C++. That is a great feature of the OCaml toolchain. Still, I remember the documentation being sparse and I don't even remember how I did it at the moment. FWIW, my main language is Python, but I love OCaml for certain things, and have grown to like C++ as well. Rust seems very interesting to me for its security properties and because it has native threads rather than including a mini-OS in the runtime like Go. My understanding is that Go is hopeless for many kinds of Python extensions, precisely because of the runtime issue, and calling from Go back into Python. (at least this was true a year or 2 ago) I'm always wary of creating more layers than necessary. A system composed of Python and C will necessarily have fewer layers than one composed of Python and Rust, simply because Python is written in C and its interface is defined in C. EDIT: I left out the BIGGEST point -- a dealbreaker. Python does NOT have an ABI. It has an API. AFAIK, that means you have to write a Rust ABI-compatible wrapper for EVERY VERSION of Python. https://www.python.org/dev/peps/pep-0384/ https://www.python.org/dev/peps/pep-0384/ To make a generalization, programmers learn about C APIs before they understand what the ABI is. It's just more concepts that you need to know about to write correct code. So if you just want to speed up an existing Python program, I would still recommend using a simple C or C++ extension. This solves both single-threaded speed issues and give you parallelism with multiple cores, so you will get huge speedups.
    - yellowapple 11y ago
      > OCaml and Rust aren't going to interface with Python nearly as well. I don't know about OCaml, but Rust (supposedly) will interface perfectly fine with anything providing a C API (which includes Python, along with quite a few other scripting languages from that era, like Perl and Ruby). Rust's ABI-compatibility with C is one of the primary selling points. I don't know off the top of my head if anyone's tried writing Python extensions in Rust, but I don't see why it wouldn't be possible to do so with at least as much capability (if not more) as C/C++.
      - jpace121 11y ago
        https://www.youtube.com/watch?v=3CwJ0MH-4MA https://www.youtube.com/watch?v=3CwJ0MH-4MA The talk above was on the front page last week some time and describes one method to write Python extensions in rust.
diginux 11y ago
I have started playing with Rust and have used OCaml on and off, but recently really diving hard into it building a REST-based application from top to bottom in OCaml, including all the infrastructure bits. We are slowly making our stuff open source as it becomes useable: https://github.com/afiniate https://github.com/afiniate In short, OCaml is a mature language that has been used for decades in commercial applications. I feel OCaml is the next progression for the people that got excited about distributed systems via the Erlang path and want more of the safety and reasoning that comes from a strongly/statically-typed language like OCaml. Rust may or may not take off, but I am confident OCaml will remain viable for the foreseeable future, and probably gain slow, but steady popularity as engineers see all the cool things you can do like MirageOS: http://openmirage.org/ http://openmirage.org/
- WaxProlix 11y ago
  Don't forget the .NET OCaml, F#!
  - bunderbunder 11y ago
    F# is a member of the ML family, but I'd really hesitate to call it the .NET OCaml. It would be similarly accurate to say that Objective-C is the Apple C++. They belong to the same family, and they both share a common ancestor that is not object-oriented. But their object systems are very different from each other.
    - yellowapple 11y ago
      > It would be similarly accurate to say that Objective-C is the Apple C++. Well it's not inaccurate. Both of them were designed to make C object-oriented, and they tend to be used for many of the same situations because of that. A more accurate nomenclature would be ".NET's OCaml-equivalent" or "Apple's C++-equivalent" (much like how C# is characterized as ".NET's Java-equivalent). This falls apart with thorough inspection, of course, but it's good enough for tongue-in-cheek comparisons like this.
    - nbevans 11y ago
WaxProlix 11y ago
I'm in the same place sort of, and Golang and Julia seem like the obvious higher-performance transitions from Python. What about Rust makes you consider it so highly?
- yarrel 11y ago
  The Rust hype machine has kicked into high gear. D suddenly seems under-promoted.
- iamd3vil 11y ago
  Take a look at Nim Language. (www.nim-lang.org) A High Performance lang with "Pythonesque" syntax.
  - WaxProlix 11y ago
    I've seen Nim going back to when it was Nimrod, but my impression (correct it here if needed please) of their development cycle isn't especially high. I do like the syntax and the inherent 'threadiness' of it, for lack of a better word.
  - dorfsmay 11y ago
    My understanding is that it does not have multicore support.
- dorfsmay 11y ago
  To be honest the main reasons are: * network effect / hype One of the main reason. It sounds lame, but I can justify to a customer re-writing a project in Rust because they've heard about it and they will be able to hire people who have either used it or at the very least will be interested in learning it. Also, they network/hype means that we are going to see good libraries emerge fairly quickly. * multicore support A year ago I was ready to move to OCaml, bought books, started to learn it, but the multicore situation was worse than python. Today we hear that "there is a good chance it's going to get multicore support". In Rust, it's already there, and it's not and after thought. Rust also happened to be slightly faster for most things according to micro-benchmarks, but we know how reliable those are, and it's we're not talking order of magnitudes here, although it is early and we can hope it gets even better.
saosebastiao 11y ago
I've long admired OCaml, but never used it really. I picked up Rust a while ago and actually made some use of it. I love the language a lot, but it is still too immature and low level for my current situation. I ended up using F# as the compromise, which has been amazing. There are some areas where OCaml is more advanced than F# (functors, the codegen from the optimized compiler, lack of the msbuild barf sandwich, less hacky on non-Windows platforms), but there are also plenty of areas where F# is more advanced than OCaml (computation expressions, code interoperability, real 32 and 64 bit integers, agents, multicore runtime). I would say that if OCaml was ideal for you except for the lack of parallelism, then you should definitely check out F# before you go all the way to Rust. Rust is awesome and for the right use case you should use it, but F# is a lot closer to OCaml than Rust is.
eatonphil 11y ago
I am the author of the major web frameworks for ocaml [0]. On of the nicest things about OCaml is the identical system calls. I built my entire OWebl platform by reading through C servers. On the other hand, Rust (like Erlang) reinvents and wraps a lot of these calls in ways that are not immediately obvious. (Or at least, not AS immediately obvious as they are in OCaml.) This is such a tremendous aid because there are nearly limitless documents and examples of the Unix API. [0] http://github.com/eatonphil/owebl http://github.com/eatonphil/owebl

e_d_g_a_r 11y ago

What were you doing that you needed multicore?

dorfsmay 11y ago
Pivoting hundreds of millions json entries and upload the results to an object store. I ended up using a combination of python threads and processes, which was more work I wanted to do, and still relatively slow.
- chubot 11y ago
  Why not C or C++? OCaml and Rust aren't going to interface with Python nearly as well. Also, it's trivial to write multithreaded extensions in C or C++ (at least for data processing). You just have to make sure to release the GIL. IMO the trick is to just use the Python C API, and not use any wrappers like SWIG or ctypes. The Python C API is a little odd but it is fairly explicit. Also, it's better to write plain functions rather than classes, but for data processing that is natural anyway. If you need a class, do that part in Python. Writing "one way" extension functions (that don't call back into Python) is quite easy and can give you a huge performance boost. I think people get hung up on Python extensions because they are using TWO unfamiliar languages -- the wrapper language, and C/C++. But if you are just using one additional language, it's not hard to figure out.
  - pcwalton 11y ago
    Why won't Rust interface with a C API?
    - kibwen 11y ago
      Indeed, I think that taking the place of C for writing Python and Ruby extensions will be an enormous opportunity for Rust.
    - chubot 11y ago
      If it has C ABI compatibility, then technically it can. That doesn't mean it's easier to interface to Python than C or C++ though. I'd find it hard to imagine it being easier, and I know it's not simpler. I'll have to watch the video below, but don't you have to write some sort of Rust wrapper for every definition in Python.h? What about macros like Py_INCREF and Py_DECREF? I'd guess that someone has or will eventually do that work, but it's yet another layer of complexity, which can have the same downsides as SWIG. For better or worse, the Python C API is tighlty coupled to the C implementation. That makes it very difficult to be more natural and understandable than C. It's a fairly huge API: https://docs.python.org/2/c-api/index.html https://docs.python.org/2/c-api/index.html People always try to cover it up with "nice" abstractions, but they invariably end up being leaky (e.g. with respect to threads, garbage collection, OS portability, etc.) Another huge can of worms: the build system. With a plain C extension, all I need is a C compiler on the system, and I can just do "python setup.py build". The situation with Windows is also quite messy -- I can't imagine Rust making it better. I have experimented with creating normal .o files from OCaml and linking them with .o files from C/C++. That is a great feature of the OCaml toolchain. Still, I remember the documentation being sparse and I don't even remember how I did it at the moment. FWIW, my main language is Python, but I love OCaml for certain things, and have grown to like C++ as well. Rust seems very interesting to me for its security properties and because it has native threads rather than including a mini-OS in the runtime like Go. My understanding is that Go is hopeless for many kinds of Python extensions, precisely because of the runtime issue, and calling from Go back into Python. (at least this was true a year or 2 ago) I'm always wary of creating more layers than necessary. A system composed of Python and C will necessarily have fewer layers than one composed of Python and Rust, simply because Python is written in C and its interface is defined in C. EDIT: I left out the BIGGEST point -- a dealbreaker. Python does NOT have an ABI. It has an API. AFAIK, that means you have to write a Rust ABI-compatible wrapper for EVERY VERSION of Python. https://www.python.org/dev/peps/pep-0384/ https://www.python.org/dev/peps/pep-0384/ To make a generalization, programmers learn about C APIs before they understand what the ABI is. It's just more concepts that you need to know about to write correct code. So if you just want to speed up an existing Python program, I would still recommend using a simple C or C++ extension. This solves both single-threaded speed issues and give you parallelism with multiple cores, so you will get huge speedups.
  - yellowapple 11y ago
    > OCaml and Rust aren't going to interface with Python nearly as well. I don't know about OCaml, but Rust (supposedly) will interface perfectly fine with anything providing a C API (which includes Python, along with quite a few other scripting languages from that era, like Perl and Ruby). Rust's ABI-compatibility with C is one of the primary selling points. I don't know off the top of my head if anyone's tried writing Python extensions in Rust, but I don't see why it wouldn't be possible to do so with at least as much capability (if not more) as C/C++.
    - jpace121 11y ago
      https://www.youtube.com/watch?v=3CwJ0MH-4MA https://www.youtube.com/watch?v=3CwJ0MH-4MA The talk above was on the front page last week some time and describes one method to write Python extensions in rust.

diginux 11y ago

I have started playing with Rust and have used OCaml on and off, but recently really diving hard into it building a REST-based application from top to bottom in OCaml, including all the infrastructure bits. We are slowly making our stuff open source as it becomes useable: https://github.com/afiniate https://github.com/afiniate In short, OCaml is a mature language that has been used for decades in commercial applications. I feel OCaml is the next progression for the people that got excited about distributed systems via the Erlang path and want more of the safety and reasoning that comes from a strongly/statically-typed language like OCaml. Rust may or may not take off, but I am confident OCaml will remain viable for the foreseeable future, and probably gain slow, but steady popularity as engineers see all the cool things you can do like MirageOS: http://openmirage.org/ http://openmirage.org/

WaxProlix 11y ago
Don't forget the .NET OCaml, F#!
- bunderbunder 11y ago
  F# is a member of the ML family, but I'd really hesitate to call it the .NET OCaml. It would be similarly accurate to say that Objective-C is the Apple C++. They belong to the same family, and they both share a common ancestor that is not object-oriented. But their object systems are very different from each other.
  - yellowapple 11y ago
    > It would be similarly accurate to say that Objective-C is the Apple C++. Well it's not inaccurate. Both of them were designed to make C object-oriented, and they tend to be used for many of the same situations because of that. A more accurate nomenclature would be ".NET's OCaml-equivalent" or "Apple's C++-equivalent" (much like how C# is characterized as ".NET's Java-equivalent). This falls apart with thorough inspection, of course, but it's good enough for tongue-in-cheek comparisons like this.
  - nbevans 11y ago

WaxProlix 11y ago

I'm in the same place sort of, and Golang and Julia seem like the obvious higher-performance transitions from Python. What about Rust makes you consider it so highly?

yarrel 11y ago
The Rust hype machine has kicked into high gear. D suddenly seems under-promoted.
iamd3vil 11y ago
Take a look at Nim Language. (www.nim-lang.org) A High Performance lang with "Pythonesque" syntax.
- WaxProlix 11y ago
  I've seen Nim going back to when it was Nimrod, but my impression (correct it here if needed please) of their development cycle isn't especially high. I do like the syntax and the inherent 'threadiness' of it, for lack of a better word.
- dorfsmay 11y ago
  My understanding is that it does not have multicore support.
dorfsmay 11y ago
To be honest the main reasons are: * network effect / hype One of the main reason. It sounds lame, but I can justify to a customer re-writing a project in Rust because they've heard about it and they will be able to hire people who have either used it or at the very least will be interested in learning it. Also, they network/hype means that we are going to see good libraries emerge fairly quickly. * multicore support A year ago I was ready to move to OCaml, bought books, started to learn it, but the multicore situation was worse than python. Today we hear that "there is a good chance it's going to get multicore support". In Rust, it's already there, and it's not and after thought. Rust also happened to be slightly faster for most things according to micro-benchmarks, but we know how reliable those are, and it's we're not talking order of magnitudes here, although it is early and we can hope it gets even better.

saosebastiao 11y ago

I've long admired OCaml, but never used it really. I picked up Rust a while ago and actually made some use of it. I love the language a lot, but it is still too immature and low level for my current situation. I ended up using F# as the compromise, which has been amazing. There are some areas where OCaml is more advanced than F# (functors, the codegen from the optimized compiler, lack of the msbuild barf sandwich, less hacky on non-Windows platforms), but there are also plenty of areas where F# is more advanced than OCaml (computation expressions, code interoperability, real 32 and 64 bit integers, agents, multicore runtime). I would say that if OCaml was ideal for you except for the lack of parallelism, then you should definitely check out F# before you go all the way to Rust. Rust is awesome and for the right use case you should use it, but F# is a lot closer to OCaml than Rust is.

eatonphil 11y ago

I am the author of the major web frameworks for ocaml [0]. On of the nicest things about OCaml is the identical system calls. I built my entire OWebl platform by reading through C servers. On the other hand, Rust (like Erlang) reinvents and wraps a lot of these calls in ways that are not immediately obvious. (Or at least, not AS immediately obvious as they are in OCaml.) This is such a tremendous aid because there are nearly limitless documents and examples of the Unix API. [0] http://github.com/eatonphil/owebl http://github.com/eatonphil/owebl