4 ms·

Frustrations with the OCaml ecosystem while developing a synthesizer library

66 points by stevebox 2y ago

eredengrin 2y ago
Sounds very relatable coming from c++. Managing dependencies is generally not too fun, it's very easy to waste a whole day (or more) fighting with the build system/package manager/linker. I like to think it's the tools but then you see a really smart c++ developer and that excuse starts to sound a little weak. It seems c and c++ attract a relatively high average percentage of developers who are willing to go learn the ins and outs of build systems and how compilers and libraries all interact. I'm curious about the happy path of ocaml vs c++, I'm not sure how much of a happy path there even is in c++. I guess if every dependency uses the same build system as the main project, maybe that's a happy path? Hard to call cmake a happy path though, more so just a tolerable path. Conan sounds nice in theory but my experiences do not match the hopes I had. I remember golang dependencies not being too bad when I tried it, really need to mess around with rust a bit to see what that's like too since I hear so much good about it.
- yawaramin 2y ago
  The article mentions this a bit, but the Happy Path in OCaml is pretty nice. I typically don't run into any of these issues because I'm not trying to use C/C++/Rust libraries in esoteric ways. I've successfully used binding libraries for OpenSSL, libcurl, SQLite, and Postgres. It's fairly simple to get run-of-the-mill C libraries working in OCaml.
- myworkinisgood 2y ago
  The biggest problem in C++ always comes from the fact that the there's no underlying stable ABI to take care of everything (and there cannot be due to performance reasons), and that build options are so important and affect the ABI. Most C++ projects are inherently multi-language system and bring the complexity of all languages to their build systems.
  - pjmlp 2y ago
    Apparently not a lesson taken by all C++ wannabe replacements, with exception of Swift, by completly ignoring ABI stability.
lmm 2y ago
Not really with OCaml so much as with using native libraries from OCaml. Which is an awkward thing in many languages, especially if you want it to be cross-platform as this author does.
_7acn 2y ago
Sounds very relatable coming from F#.
- fire_lake 2y ago
  F# tooling is converging on just using the official .NET cli these days. FAKE, Paket, etc. have some great ideas but seem to work less reliably across a variety of setups.
  - bmitc 2y ago
    I program all my personal projects, and I have been able to completely ignore the existence of FAKE and Paket. The only thing missing from NuGet, for me, is the ability to install a package directly from a GitHub repository instead of being forced to publish a NuGet package. I don't know if that's still true but even Poetry in Python and Hex in Elixir can do that.
rwmj 2y ago
I think the author's problem was using dune at all. It's a very weird and obscure tool, and not necessary, you can just run ocamlopt commands from existing build systems (like autotools, meson etc). I've used OCaml in production for 2 decades without ever needing to use dune, opam etc.
- toolslive 2y ago
  Whenever I ran into trouble (up until ~2019) with dune I would first get it to work with (ocamlfind) ocamlopt, look what the lowest level needed to look like, and then reverse engineer to make dune spit out the same thing. I would hope things have improved since
- cdaringe 2y ago
  I cordially disagree. I do not find dune weird or obscure. It’s formally part of the OCaml platform: https://ocaml.org/platform https://ocaml.org/platform, which I think negates at least the obscure bit. It was “new” to me for a hot moment, and it’s certainly a less common format for config files (s (expressions)). However, what I see is a tool that solves concrete problems that uses a declarative, structured format (not very weird). It also has pretty great docs!
  - rwmj 2y ago
    I have to deal with dune across all the Fedora OCaml packaging (something like 200 different upstreams) and it's quite difficult to deal with. Not the worst build system out there, but makes me a bit sad every time I have to modify something in a project.
  - RandomThoughts3 2y ago
    Dune is part of the platform mostly because it was there and was the tool used and developed by Ocaml biggest corporate user but it’s hard to disagree with it’s weird (s-expressions in a non LISP, really?) and a bit obscure. Esy seems to solve most of these issues at the costs of a tie-in to npm.
  - mattpallissard 2y ago
trpal 2y ago
OCaml itself is built with Makefiles, which just works. Coq has used various new tools over the years and at times has been more difficult to build than 15 years ago. With OCaml-5 the situation seems to have gotten better: At least the infamous caml4p problem was solved, and I "only" had to fix a wrong ocamlfind config file that was installed by default. ocamlbuild is generally good, but Dune and Opam are a black box that is simply often broken. I wish more mature projects switched to Makefiles, which despite all the hate for them are always the most reliable build way. But OCaml-5 is so great that all of this is minor bickering! It is confusing for beginners though.
- aseipp 2y ago
  Coq doesn't seem meaningfully harder to build than in the past but maybe I'm just not in touch. It was always the Gtk bits that added issues for me, not Dune or anything. > which despite all the hate for them are always the most reliable build way. Based on the amount of totally broken Makefiles I find on a regular basis as a package maintainer (logic bugs, recursive make, invalid deps = no parallel builds) suggests to me this is not the case at all. Doesn't even touch on how awful these things are often written, even if they work reliably enough. Insane idioms in the name of one-liners, implicit dependencies on GNU specific features, incredibly inefficient shell pipeline commands, overly large rules that mean many intermediate steps aren't cached, subtly broken rules and invocations based on spaces or non-utf8 chars, etc. I worked on the Glasgow Haskell Compiler and the truth is that the Make-based build process was only reliable because a massive amount of effort was spent on it. I suspect it's probably the same with OCaml, and not because Makefiles are special. There are degrees to it all, but they certainly don't have good syntax (too terse), aren't easy to read (batshit insane escape rules that were fixed by Plan 9 mk in 1991), and make expressing many types of dependencies really awkward by design (I want a dependency on an environment variable?) They're just really easy to get started with and you can write something broken that looks like it works zero effort, honestly.
- c-cube 2y ago
  OCaml might be built with makefiles, but it's also not an easy path at all. Adding a module to the compiler or stdlib is really not straightforward compared to what it'd be with dune (you have to list your new file somewhere, regenerate dependencies, order matters in obscure ways, etc.). To be clear, OCaml is a special project since it also has to solve bootstrapping issues, but still, it's not an argument for "makefiles just work". Dune works extremely well for pure OCaml projects.
aseipp 2y ago
Honestly using Rust recently has taught me that, the only reason C and C++ dependencies in Rust are remotely tolerable (unlike most other languages) is because Rust has an unusually high degree of talented developers, many former C/C++, who are able and willing to mercilessly bludgeon these libraries and their build systems into a square shape, then fit them in the circle-shaped hole we call "build.rs". I guess my point is, yeah all of this sort of stuff sucks, and to the extent it sucks less elsewhere, it's mostly due to the sheer effort consumed by it.

66 points by stevebox 2y ago

Sounds very relatable coming from c++. Managing dependencies is generally not too fun, it's very easy to waste a whole day (or more) fighting with the build system/package manager/linker. I like to think it's the tools but then you see a really smart c++ developer and that excuse starts to sound a little weak. It seems c and c++ attract a relatively high average percentage of developers who are willing to go learn the ins and outs of build systems and how compilers and libraries all interact. I'm curious about the happy path of ocaml vs c++, I'm not sure how much of a happy path there even is in c++. I guess if every dependency uses the same build system as the main project, maybe that's a happy path? Hard to call cmake a happy path though, more so just a tolerable path. Conan sounds nice in theory but my experiences do not match the hopes I had. I remember golang dependencies not being too bad when I tried it, really need to mess around with rust a bit to see what that's like too since I hear so much good about it.

yawaramin 2y ago
The article mentions this a bit, but the Happy Path in OCaml is pretty nice. I typically don't run into any of these issues because I'm not trying to use C/C++/Rust libraries in esoteric ways. I've successfully used binding libraries for OpenSSL, libcurl, SQLite, and Postgres. It's fairly simple to get run-of-the-mill C libraries working in OCaml.
myworkinisgood 2y ago
The biggest problem in C++ always comes from the fact that the there's no underlying stable ABI to take care of everything (and there cannot be due to performance reasons), and that build options are so important and affect the ABI. Most C++ projects are inherently multi-language system and bring the complexity of all languages to their build systems.
- pjmlp 2y ago
  Apparently not a lesson taken by all C++ wannabe replacements, with exception of Swift, by completly ignoring ABI stability.

lmm 2y ago

Not really with OCaml so much as with using native libraries from OCaml. Which is an awkward thing in many languages, especially if you want it to be cross-platform as this author does.

_7acn 2y ago

Sounds very relatable coming from F#.

fire_lake 2y ago
F# tooling is converging on just using the official .NET cli these days. FAKE, Paket, etc. have some great ideas but seem to work less reliably across a variety of setups.
- bmitc 2y ago
  I program all my personal projects, and I have been able to completely ignore the existence of FAKE and Paket. The only thing missing from NuGet, for me, is the ability to install a package directly from a GitHub repository instead of being forced to publish a NuGet package. I don't know if that's still true but even Poetry in Python and Hex in Elixir can do that.

rwmj 2y ago

I think the author's problem was using dune at all. It's a very weird and obscure tool, and not necessary, you can just run ocamlopt commands from existing build systems (like autotools, meson etc). I've used OCaml in production for 2 decades without ever needing to use dune, opam etc.

toolslive 2y ago
Whenever I ran into trouble (up until ~2019) with dune I would first get it to work with (ocamlfind) ocamlopt, look what the lowest level needed to look like, and then reverse engineer to make dune spit out the same thing. I would hope things have improved since
cdaringe 2y ago
I cordially disagree. I do not find dune weird or obscure. It’s formally part of the OCaml platform: https://ocaml.org/platform https://ocaml.org/platform, which I think negates at least the obscure bit. It was “new” to me for a hot moment, and it’s certainly a less common format for config files (s (expressions)). However, what I see is a tool that solves concrete problems that uses a declarative, structured format (not very weird). It also has pretty great docs!
- rwmj 2y ago
  I have to deal with dune across all the Fedora OCaml packaging (something like 200 different upstreams) and it's quite difficult to deal with. Not the worst build system out there, but makes me a bit sad every time I have to modify something in a project.
- RandomThoughts3 2y ago
  Dune is part of the platform mostly because it was there and was the tool used and developed by Ocaml biggest corporate user but it’s hard to disagree with it’s weird (s-expressions in a non LISP, really?) and a bit obscure. Esy seems to solve most of these issues at the costs of a tie-in to npm.
- mattpallissard 2y ago

trpal 2y ago

OCaml itself is built with Makefiles, which just works. Coq has used various new tools over the years and at times has been more difficult to build than 15 years ago. With OCaml-5 the situation seems to have gotten better: At least the infamous caml4p problem was solved, and I "only" had to fix a wrong ocamlfind config file that was installed by default. ocamlbuild is generally good, but Dune and Opam are a black box that is simply often broken. I wish more mature projects switched to Makefiles, which despite all the hate for them are always the most reliable build way. But OCaml-5 is so great that all of this is minor bickering! It is confusing for beginners though.

aseipp 2y ago
Coq doesn't seem meaningfully harder to build than in the past but maybe I'm just not in touch. It was always the Gtk bits that added issues for me, not Dune or anything. > which despite all the hate for them are always the most reliable build way. Based on the amount of totally broken Makefiles I find on a regular basis as a package maintainer (logic bugs, recursive make, invalid deps = no parallel builds) suggests to me this is not the case at all. Doesn't even touch on how awful these things are often written, even if they work reliably enough. Insane idioms in the name of one-liners, implicit dependencies on GNU specific features, incredibly inefficient shell pipeline commands, overly large rules that mean many intermediate steps aren't cached, subtly broken rules and invocations based on spaces or non-utf8 chars, etc. I worked on the Glasgow Haskell Compiler and the truth is that the Make-based build process was only reliable because a massive amount of effort was spent on it. I suspect it's probably the same with OCaml, and not because Makefiles are special. There are degrees to it all, but they certainly don't have good syntax (too terse), aren't easy to read (batshit insane escape rules that were fixed by Plan 9 mk in 1991), and make expressing many types of dependencies really awkward by design (I want a dependency on an environment variable?) They're just really easy to get started with and you can write something broken that looks like it works zero effort, honestly.
c-cube 2y ago
OCaml might be built with makefiles, but it's also not an easy path at all. Adding a module to the compiler or stdlib is really not straightforward compared to what it'd be with dune (you have to list your new file somewhere, regenerate dependencies, order matters in obscure ways, etc.). To be clear, OCaml is a special project since it also has to solve bootstrapping issues, but still, it's not an argument for "makefiles just work". Dune works extremely well for pure OCaml projects.

aseipp 2y ago

Honestly using Rust recently has taught me that, the only reason C and C++ dependencies in Rust are remotely tolerable (unlike most other languages) is because Rust has an unusually high degree of talented developers, many former C/C++, who are able and willing to mercilessly bludgeon these libraries and their build systems into a square shape, then fit them in the circle-shaped hole we call "build.rs". I guess my point is, yeah all of this sort of stuff sucks, and to the extent it sucks less elsewhere, it's mostly due to the sheer effort consumed by it.