4 ms·

To be honest the story about the two closed PRs for dynamic arrays doesn't really inspire contributions :)

kcsrk 1y ago
You are right that the dynamic arrays story does not read like a straightforward “how to inspire contributions.” But part of what I wanted to do in the talk was to show things as they actually unfolded. In OCaml compiler development, there is a very strong emphasis on correctness and long-term stability. That can make contributions, especially to core language features, feel harder than they might in faster-moving ecosystems. The dynamic arrays case is a good illustration. What began as a small PR grew into years of design iterations, debates about representation, performance, and multicore safety, and eventually a couple of thousand lines of code and more than 500 comments before it landed. From one perspective, that looks discouraging. From another, it shows the weight we place on getting things right, because once a feature ships, it is very hard to undo. That tension, between wanting to be open and encouraging contributions but also needing to protect stability, is something I think we should be talking about openly. My hope is that by making the process more visible we can demystify it and help contributors understand not just what happened, but why.
- octachron 1y ago
  A point that I find missing in the timeline for dynamic array is that there have been implementation for dynamic arrays available in libraries for more than twenty years. However, none of the authors of those libraries were really happy with their own implementation because those implementations had to choose between performance, API usability or thread safety. When I closed the student pull request (which was a naive implementation with no unsafe features), it was with the idea that it was unfair to expect a beginner use to solve those issues. The subsequent iterations explored different part of the design space before the final iteration which converged to safely using unsafe language features to reach a new local API optimum.
- sidkshatriya 1y ago
  I think this is the tension in most software. If you want to have excellent and correct software it will take time. And if you want more features with a "fix as you go approach" you will often have huge technical debt and get saddled with poor interfaces, often forever. But, I think OCaml errs too much on the side of getting it right the first time. The result is that state of the art keeps moving far ahead. By the time OCaml "catches up" the field of programming languages has moved far ahead. So OCaml always remains the Jack of all trades and the master of none (IMHO). I like the direction OxCaml is taking. But the problem is that no one has another 10 years to see its learnings get folded back into OCaml. There is a real chance that OxCaml may diverge so much that it becomes impractical to merge it into OCaml. Flambda2 is another great piece of software that may also take a long time to come into OCaml proper. So I feel that things need to be "speeded up" if OCaml has to become a bigger ecosystem. You can see that some big projects are moving away from OCaml -- facebook for instance used to have their python typechecker in OCaml. Their new one, pyrefly is in Rust. This could be an isolated story, no doubt.
  - sidkshatriya 1y ago
    Now OCaml values adding features carefully to the language so that there is no future regret. But being slow and conservative has _not_ minimized regret. The "O" in OCaml i.e. Objects (and Classes) is almost ignored nowadays. Janestreet, a large industrial user seems to be actively against using the "O" part of OCaml. So here we have gotten the worst of both worlds -- a language that is evolving slowly and a language that has large features that are almost soft discouraged. My primary language is Rust and not OCaml (mostly dabble in OCaml) so I may not fully know what I'm talking about when it comes to OCaml.
    - spit2wind 1y ago
      Interesting. I'm still working my way through Correct+Efficient+Beautiful. My takeaway so far has been that Modules _are_ the "O" part of OCaml. I guess there's something more "traditionally" an object? Does that mean there were modules in Caml (or whatever the predecessor was) and it was decided classes might be a good feature to add?
klodolph 1y ago
Maybe what I read here is “this is how contributions go”… Get the API right first. Make sure it’s correct, safe, and useful. Iterate on the performance afterwards. IMO, a lot of contributions should take this shape.
- kcsrk 1y ago
  It is often hard to see the shape of these things before a serious PR attempt is made. Each of the PRs reveals more of the shape of the problem being solved. Hard to skip them in practice, especially for new contributors.
aseipp 1y ago
I think it's just the nature of the beast, in this case. Serious "industrial" implementations of a programming language might stick around for a long time, and breaking things a lot can mar the appeal; getting it right the first time pays off in that case. I think the acceptance threshold can be much lower in other kinds of tooling. "It is what it is", so to speak.
- Quekid5 1y ago
  Add a sane deprecation process and this is much less of an issue -- see e.g. the Java language. Sure, it's not ideal to have multiple implementations of the 'same' data structure (if a better way is found, say)... but at least you aren't stalling everything and causing API interop issues for years and years.

You are right that the dynamic arrays story does not read like a straightforward “how to inspire contributions.” But part of what I wanted to do in the talk was to show things as they actually unfolded. In OCaml compiler development, there is a very strong emphasis on correctness and long-term stability. That can make contributions, especially to core language features, feel harder than they might in faster-moving ecosystems. The dynamic arrays case is a good illustration. What began as a small PR grew into years of design iterations, debates about representation, performance, and multicore safety, and eventually a couple of thousand lines of code and more than 500 comments before it landed. From one perspective, that looks discouraging. From another, it shows the weight we place on getting things right, because once a feature ships, it is very hard to undo. That tension, between wanting to be open and encouraging contributions but also needing to protect stability, is something I think we should be talking about openly. My hope is that by making the process more visible we can demystify it and help contributors understand not just what happened, but why.

octachron 1y ago
A point that I find missing in the timeline for dynamic array is that there have been implementation for dynamic arrays available in libraries for more than twenty years. However, none of the authors of those libraries were really happy with their own implementation because those implementations had to choose between performance, API usability or thread safety. When I closed the student pull request (which was a naive implementation with no unsafe features), it was with the idea that it was unfair to expect a beginner use to solve those issues. The subsequent iterations explored different part of the design space before the final iteration which converged to safely using unsafe language features to reach a new local API optimum.
sidkshatriya 1y ago
I think this is the tension in most software. If you want to have excellent and correct software it will take time. And if you want more features with a "fix as you go approach" you will often have huge technical debt and get saddled with poor interfaces, often forever. But, I think OCaml errs too much on the side of getting it right the first time. The result is that state of the art keeps moving far ahead. By the time OCaml "catches up" the field of programming languages has moved far ahead. So OCaml always remains the Jack of all trades and the master of none (IMHO). I like the direction OxCaml is taking. But the problem is that no one has another 10 years to see its learnings get folded back into OCaml. There is a real chance that OxCaml may diverge so much that it becomes impractical to merge it into OCaml. Flambda2 is another great piece of software that may also take a long time to come into OCaml proper. So I feel that things need to be "speeded up" if OCaml has to become a bigger ecosystem. You can see that some big projects are moving away from OCaml -- facebook for instance used to have their python typechecker in OCaml. Their new one, pyrefly is in Rust. This could be an isolated story, no doubt.
- sidkshatriya 1y ago
  Now OCaml values adding features carefully to the language so that there is no future regret. But being slow and conservative has _not_ minimized regret. The "O" in OCaml i.e. Objects (and Classes) is almost ignored nowadays. Janestreet, a large industrial user seems to be actively against using the "O" part of OCaml. So here we have gotten the worst of both worlds -- a language that is evolving slowly and a language that has large features that are almost soft discouraged. My primary language is Rust and not OCaml (mostly dabble in OCaml) so I may not fully know what I'm talking about when it comes to OCaml.
  - spit2wind 1y ago
    Interesting. I'm still working my way through Correct+Efficient+Beautiful. My takeaway so far has been that Modules _are_ the "O" part of OCaml. I guess there's something more "traditionally" an object? Does that mean there were modules in Caml (or whatever the predecessor was) and it was decided classes might be a good feature to add?

klodolph 1y ago

Maybe what I read here is “this is how contributions go”… Get the API right first. Make sure it’s correct, safe, and useful. Iterate on the performance afterwards. IMO, a lot of contributions should take this shape.

kcsrk 1y ago
It is often hard to see the shape of these things before a serious PR attempt is made. Each of the PRs reveals more of the shape of the problem being solved. Hard to skip them in practice, especially for new contributors.

aseipp 1y ago

I think it's just the nature of the beast, in this case. Serious "industrial" implementations of a programming language might stick around for a long time, and breaking things a lot can mar the appeal; getting it right the first time pays off in that case. I think the acceptance threshold can be much lower in other kinds of tooling. "It is what it is", so to speak.

Quekid5 1y ago
Add a sane deprecation process and this is much less of an issue -- see e.g. the Java language. Sure, it's not ideal to have multiple implementations of the 'same' data structure (if a better way is found, say)... but at least you aren't stalling everything and causing API interop issues for years and years.