17 ms·

PR to Merge Multicore OCaml

388 points by c-cube 5y ago

sadiq 5y ago
As usual, happy to answer any questions.
- pdimitar 5y ago
  Is the previous plan for adding effects in OCaml 5.1 still in place? I am very excited not only for the native parallelism that's coming in 5.0 but also about the effect handlers! I am sure many others are looking to start creating very interesting things with them, e.g. Rust-alike async capabilities or Erlang's preemptive green threads / actors runtime.
  - sadiq 5y ago
    5.00 (the first release with multicore) will include effects! More info: https://discuss.ocaml.org/t/multicore-ocaml-september-2021-effect-handlers-will-be-in-ocaml-5-0/8554 https://discuss.ocaml.org/t/multicore-ocaml-september-2021-e...
    - pdimitar 5y ago
      Oh. I haven't followed lately. Thanks a lot for the updated info! Really looking forward to what the community will build with OCaml 5.00! IMO it will shoot the language straight into the mainstream and I can't wait. OCaml is applicable for like 90% what's out there, including as a Python replacement. And its syntax is often much terser than Rust (although the higher-level typing constructs can be confusing to read). After that, all that's left is a tool like Elixir's mix or Rust's cargo and the language is basically not only in the 21st century but much farther than many others! Looking forward to it.
      - square_usual 5y ago
        > all that's left is a tool like Elixir's mix This so much. Mix is so pleasant to work with, but getting an OCaml project off the ground requires messing with dune + esy if you want consistent and isolated package management. It's a massive pain.
    - jasone 5y ago
      Yes, but note the absence of syntactic support. Effects will be usable for writing libraries and experimentation, but routine use won't be very ergonomic until a later release.
      - simplify 5y ago
        Considering the hard part of effects is implementing them at the language level, deferring on syntax is a good problem to have.
- Zababa 5y ago
  I've searched a bit for this and haven't been able to find a good answer. Let's say that I have m tasks that I want to run concurrently on n cores, for example handling HTTP requests or searching for something in files. I'd also like to not have to manage manually how the tasks are going to be distributed. Basically, have the same experience as when writing Go code. Is there a way to do this in multicore OCaml? I've found the task pool in domainslib [1] but I'm not sure if it's what I'm looking for. [1]: https://github.com/ocaml-multicore/domainslib/blob/master/lib/task.mli https://github.com/ocaml-multicore/domainslib/blob/master/li...
  - vphantom 5y ago
    I think libraries like Lwt will be the ones to offer the level of abstraction which you're describing. I too would like to see the simplicity of goroutines make it into OCaml 5 ASAP.
  - sadiq 5y ago
    You could use eio with an event loop per domain and the domain manager to distribute work to other domains. The restriction at the moment is that the tasks you spin off to other domains can't do asynchronous io. There is work on-going at the moment to bridge or even unify eio (concurrency via effects) and domainslib (nested parallelism via domains and effects) but it's a few months out.
    - Zababa 5y ago
      Thank you for the clear explanation!
    - talex5 5y ago
      To clarify that, there are two systems here: - domainslib schedules all tasks across all cores (like Go). - eio keeps tasks on the same core (and you can use a shared job queue to distribute work between cores if you want). Eio can certainly do async IO on multiple cores. Moving tasks freely between cores has some major downsides - for example every time a Go program wants to access a shared value, it needs to take a mutex (and be careful to avoid deadlocks). Such races can be very hard to debug. I suspect that the extra reliability is often worth the cost of sometimes having unbalanced workloads between cores. We're still investigating how big this effect is. When I worked at Docker, we spent a lot of time dealing with races in the Go code, even though almost nothing in Docker is CPU intensive! For a group of tasks on a single core, you can be sure that e.g. incrementing a counter or scanning an array is an atomic operation. Only blocking operations (such as reading from a file or socket) can allow something else to run. And eio schedules tasks deterministically, so if you test with (deterministic) mocks then the trace of your tests is deterministic too. Eio's own unit-tests are mostly expect-based tests where the test just checks that the trace output matches the recorded trace, for example. The Eio README has more information, plus a getting-started guide: https://github.com/ocaml-multicore/eio/blob/main/README.md https://github.com/ocaml-multicore/eio/blob/main/README.md
- rwmj 5y ago
  I'm slightly concerned if this means changing C extensions (of which we have rather a lot). We only have a few places that use "naked" pointers, which are in any case deprecated, and we should be able to fix those easily enough. Mostly the rest are fairly ordinary extensions that just call C functions and use the usual CAMLparam stuff. We do have a few places that register global roots. And several packages that do callbacks from C back to OCaml. We also use @@noalloc a lot. Is there anything else? Since there's so much code, what should I be grepping for to find code that might be of concern? Edit: Some examples of simple stuff: https://github.com/libguestfs/libguestfs-common/blob/master/mltools/tools_utils-c.c https://github.com/libguestfs/libguestfs-common/blob/master/... https://github.com/libguestfs/libguestfs-common/blob/master/mlpcre/pcre-c.c https://github.com/libguestfs/libguestfs-common/blob/master/... https://github.com/libguestfs/libguestfs-common/blob/master/mlutils/unix_utils-c.c https://github.com/libguestfs/libguestfs-common/blob/master/... More complex stuff: https://gitlab.com/nbdkit/nbdkit/-/tree/master/plugins/ocaml https://gitlab.com/nbdkit/nbdkit/-/tree/master/plugins/ocaml http://oirase.annexia.org/tmp/ocaml/ http://oirase.annexia.org/tmp/ocaml/
  - avsm 5y ago
    You may find the “nnpchecker” configure option in 4.13.0 useful. This will print a detected use of naked pointers to stderr, which can hopefully be triggered by test suites. Noalloc and registering roots should all be the same, as are callbacks (for sequential code, which all existing code will be)
  - sadiq 5y ago
    To echo what avsm said, our intention is to preserve the C API. With the exception of naked pointers, if you follow the rules around the existing C API then your extensions should continue to work in sequential code running on 5.0. We have a scheduled build and test of every package in opam with multicore: http://check.ocamllabs.io:8082/ http://check.ocamllabs.io:8082/ to try to shake out C API incompatibilities and that's proved fruitful. If you do find things that don't work on 5.0, please let us know (and if you can, get it in to opam so we test it automatically!).
    - 5y ago
- jpf0 5y ago
  A bunch of questions: Is multi-threading happening at the level of user-defined functions? How are threads scheduled? What's the underlying method or library for enabling multi-threading (Cilk, openMP, some LWT library...)? To what extent has this level of granularity been tested against other levels of nested parallelism (e.g. SIMD or otherwise parallel operators)? Have you tested performance by OS, and if so, have you noted any necessary OS-level modifications related to thread management? Is this part of a broader roadmap for accelerator integration?
  - sadiq 5y ago
    1. Domains are the unit of parallelism. A domain is essentially an OS thread with a bunch of extra runtime book-keeping data. You can use Domain.spawn (https://github.com/ocaml-multicore/ocaml-multicore/blob/5.00/stdlib/domain.mli#L23 https://github.com/ocaml-multicore/ocaml-multicore/blob/5.00...) to spawn off a new domain which will run the supplied function and terminate when it finishes. This is heavyweight though, domains are expected to be long-running. 2. Domainslib is the library developed alongside multicore to aid users in exploiting parallelism. It supports nested parallelism and is pretty highly optimised (https://github.com/ocaml-multicore/domainslib/pull/29 https://github.com/ocaml-multicore/domainslib/pull/29 for some graphs/numbers). The domainslib repo has some good examples: https://github.com/ocaml-multicore/domainslib/tree/master/test https://github.com/ocaml-multicore/domainslib/tree/master/te... 3. We've not tested against other forms of parallelism. There isn't anything stopping you exploiting SIMD in addition to parallelism from domains. 4. No, we've not compared performance by OS. 5. No plans for the multicore team to look at accelerator integration at the moment.
- jez 5y ago
  This is only glancingly related to the topic of multicore but I figure I’ll venture the question anyways: There were three big reasons pushing me towards other languages (C++, Rust) for a certain class of throughput-focused workflows: 1. Lack of multicore execution 2. Lack of control over how many bytes a datatype is (useful for many things, like ensuring something fits in a machine word or can avoid chasing a pointer in a loop) 3. Difficulty controlling where memory is allocated/copied vs moved (though some of this is less relevant when everything uses reference semantics and mutability is tracked with ref/mutable) I’ll be glad to see (1) fading into history! Do you have any personal tips/anecdotes for memory optimizations in practice, or any suggestions for something to read/follow/search?
  - sadiq 5y ago
    For both 2 and 3 you may find https://github.com/ocaml/RFCs/blob/unboxed-types/rfcs/unboxed-types.md https://github.com/ocaml/RFCs/blob/unboxed-types/rfcs/unboxe... very interesting. I'm hoping that progresses beyond the RFC stage.
2wrist 5y ago
Great stuff!!
michaelmcmillan 5y ago
Finally a PR worthy of Hacktoberfest.
gmfawcett 5y ago
The multicore team has done an excellent job of communicating their plans and progress over the past few years. Innovative research, solid engineering, and a top-notch communications game. Whether or not you care about Ocaml, many other SE projects could learn a lot from studying this team.
- eatYourFood 5y ago
  > could learn a lot from studying this team. How? Where?
  - gmfawcett 5y ago
    Fair question! Here's a good place to start, they've been doing monthly progress reports here since January 2020: https://discuss.ocaml.org/tag/multicore-monthly https://discuss.ocaml.org/tag/multicore-monthly There have also been presentations, videos, academic papers, etc., which should all be documented in the monthly reports.
lgrialn 5y ago
Hurray! My wish for Ocaml is that it could somehow be popular enough to have more… casual users, let’s say. I enjoy playing with it more than any other language I’ve run into, but everyone else falls so deeply into it that I don’t generally follow their discussions very well. (Maybe I’ll fall in deep myself in time, but it remains to be seen.)
- ModernMech 5y ago
  I don't know if you've seen this language yet and I haven't used it, but Grain seems to take a lot of inspiration from Reason. Seems like they are trying to target a more casual audience. https://grain-lang.org https://grain-lang.org
- Ericson2314 5y ago
  Someone needs to get OCaml running on the Go runtime, so it can be an F# for that ecosystem.
  - yawaramin 5y ago
    With all the problems arising from leaky abstractions and trying to adapt to a foreign runtime? Imho there's no point. Especially now that OCaml Multicore is on the way. A much better effort, for anyone interested, would be to vastly simplify the OCaml build and package management story to be more Go-like.
    - Ericson2314 5y ago
      > With all the problems arising from leaky abstractions and trying to adapt to a foreign runtime? Huh? The point is Go might be a lousy language, but I can't think of anything OCaml needs to do that the Go runtime cannot do. This wouldn't be leaky and the FFI could be quite good. > Especially now that OCaml Multicore is on the way. As I wrote in the other reply, this would be a naked ploy for new users, and flexing the language isn't wedded to a single run time. Multicore OCaml isn't just a runtime change, but also demonstration of the new effects indicating that the language can do parallelism without bad concurrency problems. All that language-side work carries over to the other runtime: you get to show off Goroutines done more safely!
noncoml 5y ago
I really didn’t think I’d see the day.
deleted 5y ago
[deleted]
keewee7 5y ago
F# is a ML/OCaml with solid concurrent and parallel programming features and has been multiplatform since Microsoft released .Net Core 1.0 in 2016.
- klibertp 5y ago
  > F# is a ML/OCaml No. F# is an ML dialect, like OCaml. But, you should be aware that the "O" in "OCaml" actually means something. OCaml comes with beautiful object system that's prototypal in nature and structurally typed integrated into the language. F# on the other hand has the object system inherited from .NET, and it's simplistic in comparison. Also, modules and functors. So, no, F# is not OCaml. If you absolutely need to use .NET, and know some ML, you can reach for F#, but its limitations will drive you nuts quickly. F# is for C# programmers - they have "full power of .NET" still accessible, plus a few nice things like HM type inference, immutability by default, and maybe computational expressions.
  - pbiggar 5y ago
    But basically no one uses the object part of OCaml
    - pjmlp 5y ago
      For various levels of no one.
      - rackjack 5y ago
        GTK comes to mind
  - pjmlp 5y ago
    F# is not really for C# programmers, C# programmers just get by the F# features that keep being copied into C#. F# is an ML that is kind of allowed to play on .NET because in a given moment management accepted to integrate it into Visual Studio, and keeps looking for the golden spot that it will take it beyond the VB/C# shadow, always a second thought when the .NET team designs new architecture features only taking VB and C# into consideration. F# twittersphere is its own bubble, not always with nice opinions about Microsoft and the .NET platform it depends on.
pjmlp 5y ago
Great news! Congratulations to everyone involved.
nicolashahn 5y ago
If OCaml gets good tooling and ecosystem I'm totally in. I've heard it's close to Rust but GCed and that sounds extremely enticing.
- nestorD 5y ago
  Tooling got me from Ocaml to F#, I would recommend it if that is a criteria for you.
- c54 5y ago
  I like OCaml a lot but I agree that it suffers from this tooling/ecosystem problem. Even un-fancy languages with good tooling are just going to win, at the end of the day. Go is the main example of this. Rust also has great tooling that's gotten better over time. I think over time Jane Street will (continue to) open source more of its libraries, and hopefully soon will also migrate onto Dune (internally they use jenga[0]). This should mean that the ecosystem within Jane Street more closely matches the external environment and tooling should get better as they push patches upstream. Hopefully. [0] https://discuss.ocaml.org/t/does-jenga-have-more-features-than-dune-is-it-safe-to-assume-that-dune-will-replace-jenga/2578/2 https://discuss.ocaml.org/t/does-jenga-have-more-features-th...
  - wk_end 5y ago
    I mean, maybe. But Core and Async have been open source for a long time, and there's still plenty of people out there using Batteries/Containers/Iter/BOS/Lwt.
- jasone 5y ago
  I started using OCaml in 2017, and had a rough start. But since then Dune (https://dune.build/ https://dune.build/) and the now-excellent LSP implementation have resolved the biggest issues I had. I haven't had such a pleasant development environment since Turbo Pascal! (To be fair, IntelliJ was also pretty good other than the shortcomings of Java.)
  - sideeffffect 5y ago
dang 5y ago
Past related threads: Multicore OCaml: October 2021 - https://news.ycombinator.com/item?id=29238972 https://news.ycombinator.com/item?id=29238972 - Nov 2021 (12 comments) Effective Concurrency with Algebraic Effects in Multicore OCaml - https://news.ycombinator.com/item?id=28838099 https://news.ycombinator.com/item?id=28838099 - Oct 2021 (59 comments) Multicore OCaml: September 2021, effect handlers will be in OCaml 5.0 - https://news.ycombinator.com/item?id=28742033 https://news.ycombinator.com/item?id=28742033 - Oct 2021 (3 comments) Multicore OCaml: September 2021 - Effect handlers will be in OCaml 5.0 - https://news.ycombinator.com/item?id=28719088 https://news.ycombinator.com/item?id=28719088 - Oct 2021 (3 comments) Adapting the OCaml Ecosystem for Multicore OCaml - https://news.ycombinator.com/item?id=28440385 https://news.ycombinator.com/item?id=28440385 - Sept 2021 (1 comment) Adapting the OCaml Ecosystem for Multicore OCaml - https://news.ycombinator.com/item?id=28373155 https://news.ycombinator.com/item?id=28373155 - Aug 2021 (21 comments) Multicore OCaml: July 2021 - https://news.ycombinator.com/item?id=28039219 https://news.ycombinator.com/item?id=28039219 - Aug 2021 (14 comments) Multicore OCaml: May 2021 - https://news.ycombinator.com/item?id=27480678 https://news.ycombinator.com/item?id=27480678 - June 2021 (27 comments) Multicore OCaml: April 2021 - https://news.ycombinator.com/item?id=27140522 https://news.ycombinator.com/item?id=27140522 - May 2021 (89 comments) Multicore OCaml: Feb 2021 with new preprint on Effect Handlers - https://news.ycombinator.com/item?id=26424785 https://news.ycombinator.com/item?id=26424785 - March 2021 (29 comments) Multicore OCaml: October 2020 - https://news.ycombinator.com/item?id=25034538 https://news.ycombinator.com/item?id=25034538 - Nov 2020 (9 comments) Multicore OCaml: September 2020 - https://news.ycombinator.com/item?id=24719124 https://news.ycombinator.com/item?id=24719124 - Oct 2020 (43 comments) Parallel Programming in Multicore OCaml - https://news.ycombinator.com/item?id=23740869 https://news.ycombinator.com/item?id=23740869 - July 2020 (15 comments) Multicore OCaml: May 2020 update - https://news.ycombinator.com/item?id=23380370 https://news.ycombinator.com/item?id=23380370 - June 2020 (17 comments) Multicore OCaml: March 2020 update - https://news.ycombinator.com/item?id=22727975 https://news.ycombinator.com/item?id=22727975 - March 2020 (37 comments) Multicore OCaml: Feb 2020 update - https://news.ycombinator.com/item?id=22443428 https://news.ycombinator.com/item?id=22443428 - Feb 2020 (80 comments) State of Multicore OCaml [pdf] - https://news.ycombinator.com/item?id=17416797 https://news.ycombinator.com/item?id=17416797 - June 2018 (103 comments) OCaml-multicore now at 4.04.2 - https://news.ycombinator.com/item?id=16646181 https://news.ycombinator.com/item?id=16646181 - March 2018 (4 comments) A deep dive into Multicore OCaml garbage collector - https://news.ycombinator.com/item?id=14780159 https://news.ycombinator.com/item?id=14780159 - July 2017 (89 comments) Lock-free programming for the masses - https://news.ycombinator.com/item?id=11907584 https://news.ycombinator.com/item?id=11907584 - June 2016 (29 comments) Lock-free programming for the masses - https://news.ycombinator.com/item?id=11893911 https://news.ycombinator.com/item?id=11893911 - June 2016 (4 comments) OCaml 4.03 will, “if all goes well”, support multicore - https://news.ycombinator.com/item?id=9582980 https://news.ycombinator.com/item?id=9582980 - May 2015 (113 comments) Multicore OCaml - https://news.ycombinator.com/item?id=8003699 https://news.ycombinator.com/item?id=8003699 - July 2014 (1 comment)
deleted 5y ago
[deleted]
rackjack 5y ago
Whoa, earlier than I expected (didn't think we'd see the PR within the year). Exciting!!
ijustboughtit 5y ago
Last time (ca. 1 year ago) I tried learning OCaml, I ended up reading the beta version of Real World OCaml 2nd Ed. IIRC and for some reason all I remember now is that I didn't feel confident regarding the learning material. The RWO-website https://dev.realworldocaml.org https://dev.realworldocaml.org claims that the 2nd. Ed. has been published in Q4 2021, but I can't find it anywhere. Can somebody with experience tell me if RWO 2nd Ed. is the way to go?
- avsm 5y ago
  (Coauthor of RWO here) We are just finishing edits of a few chapters (the tooling, testing and GADT ones), and then it’ll be off to the publishers early in the new year. The online one is therefore pretty up to date. There’s also a thread on the OCaml forums on this topic with more suggestions: https://discuss.ocaml.org/t/how-do-you-stay-productive-in-ocaml/8221/5 https://discuss.ocaml.org/t/how-do-you-stay-productive-in-oc...
  - ijustboughtit 5y ago
    Thanks a bunch!
  - baby 5y ago
    since you're here, what do you think about starting a new OCaml project with esy as best practice today? I was working on https://o1-labs.github.io/ocamlbyexample/ https://o1-labs.github.io/ocamlbyexample/ and I was thinking of starting with that approach as it seems to provide a more modern experience than just opam/dune. (I read your book as introduction to OCaml and still struggled a ton with all the edge-cases of dune and opam. Even today I'm iffy every time I start working on some OCaml code again because I know I'll have to deal with opam/dune issues.)
jabl 5y ago
Is there an ELI5 writeup somewhere about what's so nice about algebraic effects? I skimmed through some preprint on arxiv, and I got the impression that one could implement async/await as well as something like green threads if you select a suitable runtime. But beyond implementing other concurrency abstractions, why should I care?
- yawaramin 5y ago
  Check out https://overreacted.io/algebraic-effects-for-the-rest-of-us/ https://overreacted.io/algebraic-effects-for-the-rest-of-us/
- simplify 5y ago
  The short answer is, algebraic effects allow you to build your own abstractions like async/await. That's extremely powerful.
- classified 5y ago
  https://speakerdeck.com/kayceesrk/effect-handlers-in-multicore-ocaml https://speakerdeck.com/kayceesrk/effect-handlers-in-multico...
- kcsrk 5y ago
  Effect handlers are a foundation of all non-local control flow abstractions. Anything that requires fancy control-flow can be implemented with effect handlers. This includes green threads, async/await, generators (or iterators as some languages call it), but also higher-level applications such as algorithmic differentiation, probabilistic programming, model checking and fuzzing of parallel programs, etc. A few of these applications were discovered only very recently. The aim is that this language primitive will inspire further discoveries for expressing useful abstractions elegantly. Some of these ideas are summarised in this talk: https://m.youtube.com/watch?v=VEhkhxoGJSk https://m.youtube.com/watch?v=VEhkhxoGJSk
VitalyAnkh 5y ago
Oh the PR is so huge. Would this be difficult to review?
- brabel 5y ago
  It's probably a final PR from multiple smaller, reviewed ones, I would guess.
sideeffffect 5y ago
I don't know much about OCaml, only Scala and Cats Effect/ZIO/Monix, but would appreciate if I could get a gist of what Multicore OCaml brings to the table. Is there somebody who's familiar with both worlds and could compare them and explain how Multicore OCaml (and possibly the new effect system) work? Thanks in advance!
- yawaramin 5y ago
  Multicore OCaml works using a thread-safe generational garbage collector. Each domain (i.e. thread) gets its own minor heap, and the major heap is shared. Each heap is GCd independently. The domain-level minor heaps are restricted so different domains can't write to each other's minor heap. And the shared major heap is protected with more bookkeeping. More details in https://kcsrk.info/multicore/gc/2017/07/06/multicore-ocaml-gc/ https://kcsrk.info/multicore/gc/2017/07/06/multicore-ocaml-g... Scala of course just uses the set of GCs that are available on the JVM. OCaml's GC has an advantage in that it's optimized for quickly creating and collecting many small objects–perfect for functional programming. But some advanced JVM GCs probably come close to that. The new effect system is basically 'resumable exceptions'. An effect is declared somewhat similarly to an exception. When the effect is 'thrown', it is handled by the nearest effect handler up the callstack. The difference from an exception is that once (if) it is handled, it actually resumes at the exact position in the code where it left off, not in a continuation created by a closure which is effectively what Scala and every other userland effect system does.
roguas 5y ago
LGTM +1
sharmin123 5y ago
eatonphil 5y ago
Congrats!
Kototama 5y ago
I'm excited. Haskell gets a lot of attention but I have the feeling that OCaml may be better for engineers.
- deleted 5y ago
  [deleted]
- ghostwriter 5y ago
  How can parallel untracked mutations of untracked state be better for engineers?
  - naasking 5y ago
    > How can parallel untracked mutations of untracked state be better for engineers? It basically requires eager evaluation which, believe it or not, is a huge ergonomic win for people wanting to work on practical problems. OCaml also preserves the concise syntax and great type inference you get in Haskell. All in all, a more pragmatic set of tradeoffs I'd say, although it certainly has its own problems.
  - sidkshatriya 5y ago
    There are many paradigms in programming. Each have their strengths. A purely functional approach ala Haskell is not the only way. Based on your comment, it would seem no one should use C/C++. Yet many do. It depends on what you want to achieve, what is your abstraction budget, your performance requirement, legacy code... OCaml offers a pragmatic functional approach to programming. And now you are going to be able to have your OCaml code run in a truly parallel fashion on your multicore CPU. In the future there are plans to add typing to effects though. (There is support for effects currently but its untyped and experimental). When that happens you can track changes to state (which is a kind of "effect") if you want to...
    - ghostwriter 5y ago
      > A purely functional approach ala Haskell is not the only way. that wasn't the OP's argument though, the argument was that OCaml is somehow generally better for engineers. > OCaml offers a pragmatic functional approach to programming. an evaluation of something as pragmatic depends purely on what one whishes to practice. There's no universally objective notion of pragmatism. > And now you are going to be able to have your OCaml code run in a truly parallel fashion on your multicore CPU. no, it won't be able to do that automatically. Your code will have to respect certain invariants to function properly, and you as a developer will have to enforce these invariants with the available tooling at hand. Haskell has purity, guaranteed STM, and `par` labels for that. OCaml doesn't have those and the existing codebases will have to eliminate their thread-unsafe public interfaces first. > When that happens you can track changes to state how are you planning to track state changes without purity?
sidkshatriya 5y ago
Some questions: - Is merging this PR going to be more or less a formality now? I'm assuming subsequent PRs will make further improvements/fix any further bugs discovered. When is the merge to trunk likely to happen? - (Not that it really matters, but I'm curious). Will this PR be merged as-is with nearly 4000 commits or will it be squashed? The history exists in the multicore repo but will the history be brought into the ocaml repo?
- sadiq 5y ago
  1. Before multicore got to this PR stage it went through two phases of detailed review by the core team. A summary of this is on November's Multicore Monthly: https://discuss.ocaml.org/t/multicore-ocaml-november-2021-with-results-of-code-review/8934 https://discuss.ocaml.org/t/multicore-ocaml-november-2021-wi... . The tasks coming out of that which are marked post-MVP will be follow-up PRs before the 5.00 release. 2. That is unclear at the moment. There's a lot of useful history in those commits (which link out to issues and PRs on ocaml-multicore's repo) but at the same time, it also includes a lot of experiments that were ultimately backed out.
  - mst 5y ago
    Sounds like the sort of situation where I prepare a fastforwadable PR and then force git to create a merge commit anyway.

388 points by c-cube 5y ago

sadiq 5y ago

As usual, happy to answer any questions.

pdimitar 5y ago
Is the previous plan for adding effects in OCaml 5.1 still in place? I am very excited not only for the native parallelism that's coming in 5.0 but also about the effect handlers! I am sure many others are looking to start creating very interesting things with them, e.g. Rust-alike async capabilities or Erlang's preemptive green threads / actors runtime.
- sadiq 5y ago
  5.00 (the first release with multicore) will include effects! More info: https://discuss.ocaml.org/t/multicore-ocaml-september-2021-effect-handlers-will-be-in-ocaml-5-0/8554 https://discuss.ocaml.org/t/multicore-ocaml-september-2021-e...
  - pdimitar 5y ago
    Oh. I haven't followed lately. Thanks a lot for the updated info! Really looking forward to what the community will build with OCaml 5.00! IMO it will shoot the language straight into the mainstream and I can't wait. OCaml is applicable for like 90% what's out there, including as a Python replacement. And its syntax is often much terser than Rust (although the higher-level typing constructs can be confusing to read). After that, all that's left is a tool like Elixir's mix or Rust's cargo and the language is basically not only in the 21st century but much farther than many others! Looking forward to it.
    - square_usual 5y ago
      > all that's left is a tool like Elixir's mix This so much. Mix is so pleasant to work with, but getting an OCaml project off the ground requires messing with dune + esy if you want consistent and isolated package management. It's a massive pain.
  - jasone 5y ago
    Yes, but note the absence of syntactic support. Effects will be usable for writing libraries and experimentation, but routine use won't be very ergonomic until a later release.
    - simplify 5y ago
      Considering the hard part of effects is implementing them at the language level, deferring on syntax is a good problem to have.
Zababa 5y ago
I've searched a bit for this and haven't been able to find a good answer. Let's say that I have m tasks that I want to run concurrently on n cores, for example handling HTTP requests or searching for something in files. I'd also like to not have to manage manually how the tasks are going to be distributed. Basically, have the same experience as when writing Go code. Is there a way to do this in multicore OCaml? I've found the task pool in domainslib [1] but I'm not sure if it's what I'm looking for. [1]: https://github.com/ocaml-multicore/domainslib/blob/master/lib/task.mli https://github.com/ocaml-multicore/domainslib/blob/master/li...
- vphantom 5y ago
  I think libraries like Lwt will be the ones to offer the level of abstraction which you're describing. I too would like to see the simplicity of goroutines make it into OCaml 5 ASAP.
- sadiq 5y ago
  You could use eio with an event loop per domain and the domain manager to distribute work to other domains. The restriction at the moment is that the tasks you spin off to other domains can't do asynchronous io. There is work on-going at the moment to bridge or even unify eio (concurrency via effects) and domainslib (nested parallelism via domains and effects) but it's a few months out.
  - Zababa 5y ago
    Thank you for the clear explanation!
  - talex5 5y ago
    To clarify that, there are two systems here: - domainslib schedules all tasks across all cores (like Go). - eio keeps tasks on the same core (and you can use a shared job queue to distribute work between cores if you want). Eio can certainly do async IO on multiple cores. Moving tasks freely between cores has some major downsides - for example every time a Go program wants to access a shared value, it needs to take a mutex (and be careful to avoid deadlocks). Such races can be very hard to debug. I suspect that the extra reliability is often worth the cost of sometimes having unbalanced workloads between cores. We're still investigating how big this effect is. When I worked at Docker, we spent a lot of time dealing with races in the Go code, even though almost nothing in Docker is CPU intensive! For a group of tasks on a single core, you can be sure that e.g. incrementing a counter or scanning an array is an atomic operation. Only blocking operations (such as reading from a file or socket) can allow something else to run. And eio schedules tasks deterministically, so if you test with (deterministic) mocks then the trace of your tests is deterministic too. Eio's own unit-tests are mostly expect-based tests where the test just checks that the trace output matches the recorded trace, for example. The Eio README has more information, plus a getting-started guide: https://github.com/ocaml-multicore/eio/blob/main/README.md https://github.com/ocaml-multicore/eio/blob/main/README.md
rwmj 5y ago
I'm slightly concerned if this means changing C extensions (of which we have rather a lot). We only have a few places that use "naked" pointers, which are in any case deprecated, and we should be able to fix those easily enough. Mostly the rest are fairly ordinary extensions that just call C functions and use the usual CAMLparam stuff. We do have a few places that register global roots. And several packages that do callbacks from C back to OCaml. We also use @@noalloc a lot. Is there anything else? Since there's so much code, what should I be grepping for to find code that might be of concern? Edit: Some examples of simple stuff: https://github.com/libguestfs/libguestfs-common/blob/master/mltools/tools_utils-c.c https://github.com/libguestfs/libguestfs-common/blob/master/... https://github.com/libguestfs/libguestfs-common/blob/master/mlpcre/pcre-c.c https://github.com/libguestfs/libguestfs-common/blob/master/... https://github.com/libguestfs/libguestfs-common/blob/master/mlutils/unix_utils-c.c https://github.com/libguestfs/libguestfs-common/blob/master/... More complex stuff: https://gitlab.com/nbdkit/nbdkit/-/tree/master/plugins/ocaml https://gitlab.com/nbdkit/nbdkit/-/tree/master/plugins/ocaml http://oirase.annexia.org/tmp/ocaml/ http://oirase.annexia.org/tmp/ocaml/
- avsm 5y ago
  You may find the “nnpchecker” configure option in 4.13.0 useful. This will print a detected use of naked pointers to stderr, which can hopefully be triggered by test suites. Noalloc and registering roots should all be the same, as are callbacks (for sequential code, which all existing code will be)
- sadiq 5y ago
  To echo what avsm said, our intention is to preserve the C API. With the exception of naked pointers, if you follow the rules around the existing C API then your extensions should continue to work in sequential code running on 5.0. We have a scheduled build and test of every package in opam with multicore: http://check.ocamllabs.io:8082/ http://check.ocamllabs.io:8082/ to try to shake out C API incompatibilities and that's proved fruitful. If you do find things that don't work on 5.0, please let us know (and if you can, get it in to opam so we test it automatically!).
  - 5y ago
jpf0 5y ago
A bunch of questions: Is multi-threading happening at the level of user-defined functions? How are threads scheduled? What's the underlying method or library for enabling multi-threading (Cilk, openMP, some LWT library...)? To what extent has this level of granularity been tested against other levels of nested parallelism (e.g. SIMD or otherwise parallel operators)? Have you tested performance by OS, and if so, have you noted any necessary OS-level modifications related to thread management? Is this part of a broader roadmap for accelerator integration?
- sadiq 5y ago
  1. Domains are the unit of parallelism. A domain is essentially an OS thread with a bunch of extra runtime book-keeping data. You can use Domain.spawn (https://github.com/ocaml-multicore/ocaml-multicore/blob/5.00/stdlib/domain.mli#L23 https://github.com/ocaml-multicore/ocaml-multicore/blob/5.00...) to spawn off a new domain which will run the supplied function and terminate when it finishes. This is heavyweight though, domains are expected to be long-running. 2. Domainslib is the library developed alongside multicore to aid users in exploiting parallelism. It supports nested parallelism and is pretty highly optimised (https://github.com/ocaml-multicore/domainslib/pull/29 https://github.com/ocaml-multicore/domainslib/pull/29 for some graphs/numbers). The domainslib repo has some good examples: https://github.com/ocaml-multicore/domainslib/tree/master/test https://github.com/ocaml-multicore/domainslib/tree/master/te... 3. We've not tested against other forms of parallelism. There isn't anything stopping you exploiting SIMD in addition to parallelism from domains. 4. No, we've not compared performance by OS. 5. No plans for the multicore team to look at accelerator integration at the moment.
jez 5y ago
This is only glancingly related to the topic of multicore but I figure I’ll venture the question anyways: There were three big reasons pushing me towards other languages (C++, Rust) for a certain class of throughput-focused workflows: 1. Lack of multicore execution 2. Lack of control over how many bytes a datatype is (useful for many things, like ensuring something fits in a machine word or can avoid chasing a pointer in a loop) 3. Difficulty controlling where memory is allocated/copied vs moved (though some of this is less relevant when everything uses reference semantics and mutability is tracked with ref/mutable) I’ll be glad to see (1) fading into history! Do you have any personal tips/anecdotes for memory optimizations in practice, or any suggestions for something to read/follow/search?
- sadiq 5y ago
  For both 2 and 3 you may find https://github.com/ocaml/RFCs/blob/unboxed-types/rfcs/unboxed-types.md https://github.com/ocaml/RFCs/blob/unboxed-types/rfcs/unboxe... very interesting. I'm hoping that progresses beyond the RFC stage.

2wrist 5y ago

Great stuff!!

michaelmcmillan 5y ago

Finally a PR worthy of Hacktoberfest.

gmfawcett 5y ago

The multicore team has done an excellent job of communicating their plans and progress over the past few years. Innovative research, solid engineering, and a top-notch communications game. Whether or not you care about Ocaml, many other SE projects could learn a lot from studying this team.

eatYourFood 5y ago
> could learn a lot from studying this team. How? Where?
- gmfawcett 5y ago
  Fair question! Here's a good place to start, they've been doing monthly progress reports here since January 2020: https://discuss.ocaml.org/tag/multicore-monthly https://discuss.ocaml.org/tag/multicore-monthly There have also been presentations, videos, academic papers, etc., which should all be documented in the monthly reports.

lgrialn 5y ago

Hurray! My wish for Ocaml is that it could somehow be popular enough to have more… casual users, let’s say. I enjoy playing with it more than any other language I’ve run into, but everyone else falls so deeply into it that I don’t generally follow their discussions very well. (Maybe I’ll fall in deep myself in time, but it remains to be seen.)

ModernMech 5y ago
I don't know if you've seen this language yet and I haven't used it, but Grain seems to take a lot of inspiration from Reason. Seems like they are trying to target a more casual audience. https://grain-lang.org https://grain-lang.org
Ericson2314 5y ago
Someone needs to get OCaml running on the Go runtime, so it can be an F# for that ecosystem.
- yawaramin 5y ago
  With all the problems arising from leaky abstractions and trying to adapt to a foreign runtime? Imho there's no point. Especially now that OCaml Multicore is on the way. A much better effort, for anyone interested, would be to vastly simplify the OCaml build and package management story to be more Go-like.
  - Ericson2314 5y ago
    > With all the problems arising from leaky abstractions and trying to adapt to a foreign runtime? Huh? The point is Go might be a lousy language, but I can't think of anything OCaml needs to do that the Go runtime cannot do. This wouldn't be leaky and the FFI could be quite good. > Especially now that OCaml Multicore is on the way. As I wrote in the other reply, this would be a naked ploy for new users, and flexing the language isn't wedded to a single run time. Multicore OCaml isn't just a runtime change, but also demonstration of the new effects indicating that the language can do parallelism without bad concurrency problems. All that language-side work carries over to the other runtime: you get to show off Goroutines done more safely!

noncoml 5y ago

I really didn’t think I’d see the day.

deleted 5y ago

[deleted]

keewee7 5y ago

F# is a ML/OCaml with solid concurrent and parallel programming features and has been multiplatform since Microsoft released .Net Core 1.0 in 2016.

klibertp 5y ago
> F# is a ML/OCaml No. F# is an ML dialect, like OCaml. But, you should be aware that the "O" in "OCaml" actually means something. OCaml comes with beautiful object system that's prototypal in nature and structurally typed integrated into the language. F# on the other hand has the object system inherited from .NET, and it's simplistic in comparison. Also, modules and functors. So, no, F# is not OCaml. If you absolutely need to use .NET, and know some ML, you can reach for F#, but its limitations will drive you nuts quickly. F# is for C# programmers - they have "full power of .NET" still accessible, plus a few nice things like HM type inference, immutability by default, and maybe computational expressions.
- pbiggar 5y ago
  But basically no one uses the object part of OCaml
  - pjmlp 5y ago
    For various levels of no one.
    - rackjack 5y ago
      GTK comes to mind
- pjmlp 5y ago
  F# is not really for C# programmers, C# programmers just get by the F# features that keep being copied into C#. F# is an ML that is kind of allowed to play on .NET because in a given moment management accepted to integrate it into Visual Studio, and keeps looking for the golden spot that it will take it beyond the VB/C# shadow, always a second thought when the .NET team designs new architecture features only taking VB and C# into consideration. F# twittersphere is its own bubble, not always with nice opinions about Microsoft and the .NET platform it depends on.

pjmlp 5y ago

Great news! Congratulations to everyone involved.

nicolashahn 5y ago

If OCaml gets good tooling and ecosystem I'm totally in. I've heard it's close to Rust but GCed and that sounds extremely enticing.

nestorD 5y ago
Tooling got me from Ocaml to F#, I would recommend it if that is a criteria for you.
c54 5y ago
I like OCaml a lot but I agree that it suffers from this tooling/ecosystem problem. Even un-fancy languages with good tooling are just going to win, at the end of the day. Go is the main example of this. Rust also has great tooling that's gotten better over time. I think over time Jane Street will (continue to) open source more of its libraries, and hopefully soon will also migrate onto Dune (internally they use jenga[0]). This should mean that the ecosystem within Jane Street more closely matches the external environment and tooling should get better as they push patches upstream. Hopefully. [0] https://discuss.ocaml.org/t/does-jenga-have-more-features-than-dune-is-it-safe-to-assume-that-dune-will-replace-jenga/2578/2 https://discuss.ocaml.org/t/does-jenga-have-more-features-th...
- wk_end 5y ago
  I mean, maybe. But Core and Async have been open source for a long time, and there's still plenty of people out there using Batteries/Containers/Iter/BOS/Lwt.
jasone 5y ago
I started using OCaml in 2017, and had a rough start. But since then Dune (https://dune.build/ https://dune.build/) and the now-excellent LSP implementation have resolved the biggest issues I had. I haven't had such a pleasant development environment since Turbo Pascal! (To be fair, IntelliJ was also pretty good other than the shortcomings of Java.)
- sideeffffect 5y ago

dang 5y ago

Past related threads: Multicore OCaml: October 2021 - https://news.ycombinator.com/item?id=29238972 https://news.ycombinator.com/item?id=29238972 - Nov 2021 (12 comments) Effective Concurrency with Algebraic Effects in Multicore OCaml - https://news.ycombinator.com/item?id=28838099 https://news.ycombinator.com/item?id=28838099 - Oct 2021 (59 comments) Multicore OCaml: September 2021, effect handlers will be in OCaml 5.0 - https://news.ycombinator.com/item?id=28742033 https://news.ycombinator.com/item?id=28742033 - Oct 2021 (3 comments) Multicore OCaml: September 2021 - Effect handlers will be in OCaml 5.0 - https://news.ycombinator.com/item?id=28719088 https://news.ycombinator.com/item?id=28719088 - Oct 2021 (3 comments) Adapting the OCaml Ecosystem for Multicore OCaml - https://news.ycombinator.com/item?id=28440385 https://news.ycombinator.com/item?id=28440385 - Sept 2021 (1 comment) Adapting the OCaml Ecosystem for Multicore OCaml - https://news.ycombinator.com/item?id=28373155 https://news.ycombinator.com/item?id=28373155 - Aug 2021 (21 comments) Multicore OCaml: July 2021 - https://news.ycombinator.com/item?id=28039219 https://news.ycombinator.com/item?id=28039219 - Aug 2021 (14 comments) Multicore OCaml: May 2021 - https://news.ycombinator.com/item?id=27480678 https://news.ycombinator.com/item?id=27480678 - June 2021 (27 comments) Multicore OCaml: April 2021 - https://news.ycombinator.com/item?id=27140522 https://news.ycombinator.com/item?id=27140522 - May 2021 (89 comments) Multicore OCaml: Feb 2021 with new preprint on Effect Handlers - https://news.ycombinator.com/item?id=26424785 https://news.ycombinator.com/item?id=26424785 - March 2021 (29 comments) Multicore OCaml: October 2020 - https://news.ycombinator.com/item?id=25034538 https://news.ycombinator.com/item?id=25034538 - Nov 2020 (9 comments) Multicore OCaml: September 2020 - https://news.ycombinator.com/item?id=24719124 https://news.ycombinator.com/item?id=24719124 - Oct 2020 (43 comments) Parallel Programming in Multicore OCaml - https://news.ycombinator.com/item?id=23740869 https://news.ycombinator.com/item?id=23740869 - July 2020 (15 comments) Multicore OCaml: May 2020 update - https://news.ycombinator.com/item?id=23380370 https://news.ycombinator.com/item?id=23380370 - June 2020 (17 comments) Multicore OCaml: March 2020 update - https://news.ycombinator.com/item?id=22727975 https://news.ycombinator.com/item?id=22727975 - March 2020 (37 comments) Multicore OCaml: Feb 2020 update - https://news.ycombinator.com/item?id=22443428 https://news.ycombinator.com/item?id=22443428 - Feb 2020 (80 comments) State of Multicore OCaml [pdf] - https://news.ycombinator.com/item?id=17416797 https://news.ycombinator.com/item?id=17416797 - June 2018 (103 comments) OCaml-multicore now at 4.04.2 - https://news.ycombinator.com/item?id=16646181 https://news.ycombinator.com/item?id=16646181 - March 2018 (4 comments) A deep dive into Multicore OCaml garbage collector - https://news.ycombinator.com/item?id=14780159 https://news.ycombinator.com/item?id=14780159 - July 2017 (89 comments) Lock-free programming for the masses - https://news.ycombinator.com/item?id=11907584 https://news.ycombinator.com/item?id=11907584 - June 2016 (29 comments) Lock-free programming for the masses - https://news.ycombinator.com/item?id=11893911 https://news.ycombinator.com/item?id=11893911 - June 2016 (4 comments) OCaml 4.03 will, “if all goes well”, support multicore - https://news.ycombinator.com/item?id=9582980 https://news.ycombinator.com/item?id=9582980 - May 2015 (113 comments) Multicore OCaml - https://news.ycombinator.com/item?id=8003699 https://news.ycombinator.com/item?id=8003699 - July 2014 (1 comment)

deleted 5y ago

[deleted]

rackjack 5y ago

Whoa, earlier than I expected (didn't think we'd see the PR within the year). Exciting!!

ijustboughtit 5y ago

Last time (ca. 1 year ago) I tried learning OCaml, I ended up reading the beta version of Real World OCaml 2nd Ed. IIRC and for some reason all I remember now is that I didn't feel confident regarding the learning material. The RWO-website https://dev.realworldocaml.org https://dev.realworldocaml.org claims that the 2nd. Ed. has been published in Q4 2021, but I can't find it anywhere. Can somebody with experience tell me if RWO 2nd Ed. is the way to go?

avsm 5y ago
(Coauthor of RWO here) We are just finishing edits of a few chapters (the tooling, testing and GADT ones), and then it’ll be off to the publishers early in the new year. The online one is therefore pretty up to date. There’s also a thread on the OCaml forums on this topic with more suggestions: https://discuss.ocaml.org/t/how-do-you-stay-productive-in-ocaml/8221/5 https://discuss.ocaml.org/t/how-do-you-stay-productive-in-oc...
- ijustboughtit 5y ago
  Thanks a bunch!
- baby 5y ago
  since you're here, what do you think about starting a new OCaml project with esy as best practice today? I was working on https://o1-labs.github.io/ocamlbyexample/ https://o1-labs.github.io/ocamlbyexample/ and I was thinking of starting with that approach as it seems to provide a more modern experience than just opam/dune. (I read your book as introduction to OCaml and still struggled a ton with all the edge-cases of dune and opam. Even today I'm iffy every time I start working on some OCaml code again because I know I'll have to deal with opam/dune issues.)

jabl 5y ago

Is there an ELI5 writeup somewhere about what's so nice about algebraic effects? I skimmed through some preprint on arxiv, and I got the impression that one could implement async/await as well as something like green threads if you select a suitable runtime. But beyond implementing other concurrency abstractions, why should I care?

yawaramin 5y ago
Check out https://overreacted.io/algebraic-effects-for-the-rest-of-us/ https://overreacted.io/algebraic-effects-for-the-rest-of-us/
simplify 5y ago
The short answer is, algebraic effects allow you to build your own abstractions like async/await. That's extremely powerful.
classified 5y ago
https://speakerdeck.com/kayceesrk/effect-handlers-in-multicore-ocaml https://speakerdeck.com/kayceesrk/effect-handlers-in-multico...
kcsrk 5y ago
Effect handlers are a foundation of all non-local control flow abstractions. Anything that requires fancy control-flow can be implemented with effect handlers. This includes green threads, async/await, generators (or iterators as some languages call it), but also higher-level applications such as algorithmic differentiation, probabilistic programming, model checking and fuzzing of parallel programs, etc. A few of these applications were discovered only very recently. The aim is that this language primitive will inspire further discoveries for expressing useful abstractions elegantly. Some of these ideas are summarised in this talk: https://m.youtube.com/watch?v=VEhkhxoGJSk https://m.youtube.com/watch?v=VEhkhxoGJSk

VitalyAnkh 5y ago

Oh the PR is so huge. Would this be difficult to review?

brabel 5y ago
It's probably a final PR from multiple smaller, reviewed ones, I would guess.

sideeffffect 5y ago

I don't know much about OCaml, only Scala and Cats Effect/ZIO/Monix, but would appreciate if I could get a gist of what Multicore OCaml brings to the table. Is there somebody who's familiar with both worlds and could compare them and explain how Multicore OCaml (and possibly the new effect system) work? Thanks in advance!

yawaramin 5y ago
Multicore OCaml works using a thread-safe generational garbage collector. Each domain (i.e. thread) gets its own minor heap, and the major heap is shared. Each heap is GCd independently. The domain-level minor heaps are restricted so different domains can't write to each other's minor heap. And the shared major heap is protected with more bookkeeping. More details in https://kcsrk.info/multicore/gc/2017/07/06/multicore-ocaml-gc/ https://kcsrk.info/multicore/gc/2017/07/06/multicore-ocaml-g... Scala of course just uses the set of GCs that are available on the JVM. OCaml's GC has an advantage in that it's optimized for quickly creating and collecting many small objects–perfect for functional programming. But some advanced JVM GCs probably come close to that. The new effect system is basically 'resumable exceptions'. An effect is declared somewhat similarly to an exception. When the effect is 'thrown', it is handled by the nearest effect handler up the callstack. The difference from an exception is that once (if) it is handled, it actually resumes at the exact position in the code where it left off, not in a continuation created by a closure which is effectively what Scala and every other userland effect system does.

roguas 5y ago

LGTM +1

sharmin123 5y ago

eatonphil 5y ago

Congrats!

Kototama 5y ago

I'm excited. Haskell gets a lot of attention but I have the feeling that OCaml may be better for engineers.

deleted 5y ago
[deleted]
ghostwriter 5y ago
How can parallel untracked mutations of untracked state be better for engineers?
- naasking 5y ago
  > How can parallel untracked mutations of untracked state be better for engineers? It basically requires eager evaluation which, believe it or not, is a huge ergonomic win for people wanting to work on practical problems. OCaml also preserves the concise syntax and great type inference you get in Haskell. All in all, a more pragmatic set of tradeoffs I'd say, although it certainly has its own problems.
- sidkshatriya 5y ago
  There are many paradigms in programming. Each have their strengths. A purely functional approach ala Haskell is not the only way. Based on your comment, it would seem no one should use C/C++. Yet many do. It depends on what you want to achieve, what is your abstraction budget, your performance requirement, legacy code... OCaml offers a pragmatic functional approach to programming. And now you are going to be able to have your OCaml code run in a truly parallel fashion on your multicore CPU. In the future there are plans to add typing to effects though. (There is support for effects currently but its untyped and experimental). When that happens you can track changes to state (which is a kind of "effect") if you want to...
  - ghostwriter 5y ago
    > A purely functional approach ala Haskell is not the only way. that wasn't the OP's argument though, the argument was that OCaml is somehow generally better for engineers. > OCaml offers a pragmatic functional approach to programming. an evaluation of something as pragmatic depends purely on what one whishes to practice. There's no universally objective notion of pragmatism. > And now you are going to be able to have your OCaml code run in a truly parallel fashion on your multicore CPU. no, it won't be able to do that automatically. Your code will have to respect certain invariants to function properly, and you as a developer will have to enforce these invariants with the available tooling at hand. Haskell has purity, guaranteed STM, and `par` labels for that. OCaml doesn't have those and the existing codebases will have to eliminate their thread-unsafe public interfaces first. > When that happens you can track changes to state how are you planning to track state changes without purity?

sidkshatriya 5y ago

Some questions: - Is merging this PR going to be more or less a formality now? I'm assuming subsequent PRs will make further improvements/fix any further bugs discovered. When is the merge to trunk likely to happen? - (Not that it really matters, but I'm curious). Will this PR be merged as-is with nearly 4000 commits or will it be squashed? The history exists in the multicore repo but will the history be brought into the ocaml repo?

sadiq 5y ago
1. Before multicore got to this PR stage it went through two phases of detailed review by the core team. A summary of this is on November's Multicore Monthly: https://discuss.ocaml.org/t/multicore-ocaml-november-2021-with-results-of-code-review/8934 https://discuss.ocaml.org/t/multicore-ocaml-november-2021-wi... . The tasks coming out of that which are marked post-MVP will be follow-up PRs before the 5.00 release. 2. That is unclear at the moment. There's a lot of useful history in those commits (which link out to issues and PRs on ocaml-multicore's repo) but at the same time, it also includes a lot of experiments that were ultimately backed out.
- mst 5y ago
  Sounds like the sort of situation where I prepare a fastforwadable PR and then force git to create a merge commit anyway.