8 ms·

Unix system programming in OCaml (2014)

269 points by jxub 8y ago

emmelaich 8y ago
Ocaml, in comparison to other functional statically typed languages has had many successful Unix applications. For a great tour of going to Ocaml (from Python), see Thomas Leonard's blog. A retrospective is here but read the whole lot. http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-retrospective/ http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-ret...
- cbcoutinho 8y ago
  And related hackernews discussion https://news.ycombinator.com/item?id=7858276 https://news.ycombinator.com/item?id=7858276
a0 8y ago
This is such a great book. OCaml’s type system feels like a superpower specially in the context of Unix development which is traditionally done in C.
tntn 8y ago
Only somewhat related,but I've tried to learn ocaml before but have never been able to figure out when to put what kind of delimiters where. Can anyone recommend a resource that explains the usage of the double comma and the like?
- laylomo2 8y ago
  The double semicolon only matters in the REPL (aka the "toplevel"). Off the top of my head, I can think of at least two other situations where punctuation matters: 1) Whenever you have nested match cases: match left with | Some left -> match right with | Some right -> Some (left, right) | None -> None | None -> None The compiler does not use indentation to figure out scoping, so the above gets treated as: match left with | Some left -> match right with | Some right -> Some (left, right) | None -> None | None -> None So you need to surround the inner match with parens match left with | Some left -> ( match right with | Some right -> Some (left, right) | None -> None ) | None -> None 2) If you are doing imperative things inside an if statement: Consider the following function: let cache_file file = if not (file_exists file) then printf "downloading file\n"; download file else printf "file already exists\n" The problem here is that OCaml allows one to create an if statement with no else clause. And since the compiler doesn't use indentation to figure out scoping, then the code is essentially treated like the following: let cache_file file = (if not (file_exists file) then printf "downloading file\n"); download file; else (printf "file already exists\n"); Which is just totally wrong. So in order to do multiple imperative actions within an if block, you should use either parens, or begin/end delimiters (which are treated exactly the same as parens) let cache_file file = if not (file_exists file) then begin printf "downloading file\n"; download file end else printf "file already exists\n"
  - dpwm 8y ago
    There's an interesting thing with the semicolon. I've heard people advocating thinking of the semicolon as a binary operator: (;) : unit -> unit -> unit Of course, the reality is more complicated than this, because it can be used at the end of an expression, where it behaves like a postfix operator of type 'a -> 'a. Let's take your example: match left with | Some left -> match right with | Some right -> Some (left, right) | None -> None | None -> None you can actually do match left with | Some left -> match right with | Some right -> Some (left, right) | None -> None ; ; (* <-- two separate semicolons *) | None -> None This doesn't just work for imperative stuff, like the operator theory of the semicolon, it will work with this example. My understanding is that this is due to the fact that optional semicolons are allowed at the end of statements and close the current expression. So we're closing the first expression (None) and then closing the second expression (the inner match). The double semicolon is something else altogether. Though it is worth noting that this example is straying from idiomatic OCaml and we can avoid this altogether with better matching: match (left, right) with | (Some left, Some right) -> Some (left, right) | _ -> None Both the native optimizing compiler and BuckleScript are able to do what you would expect with this: the matched tuple is optimized away. Your second unfixed example actually causes a syntax error. There is an if expression in OCaml and an if statement. This is particularly unfortunate, because the let cache_file = function | file when file_exists file -> printf "downloading file\n"; download file | _ -> printf "file already exists\n" first semicolon within an if expression turns the whole thing into an if statement, and an if statement can have no else clause. The else clause is then without an if statement, because we've triggered the imperative if statement. This is a syntax error. were we to define an operator let (>>>) a b = a; b then we could replace that semicolon with an >>> and things behave as expected. But it's really that if can be both a statement and an expression. For this reason I prefer using matches in general. let cache_file = function | file when not (file_exists file) -> printf "downloading file\n"; download file | _ -> printf "file already exists\n"
    - theaeolist 8y ago
      OCaml has no guaranteed order of evaluation, which is not what you want with (;). It needs to be part of the language as special syntax.
- thedufer 8y ago
  I can't think of any time a double-comma would be syntactically correct. Double-semicolons, which may be what you're referring to, are never required (although in some situations they will cause syntax errors to appear closer to the actual problem). They can be placed after all top-level value bindings. Probably your best bet is to follow an intro series so you see some examples (https://dev.realworldocaml.org/ https://dev.realworldocaml.org/ is the best at the moment, I believe), although if you're so inclined you could just read the BNF for the language (https://caml.inria.fr/pub/docs/manual-ocaml/language.html https://caml.inria.fr/pub/docs/manual-ocaml/language.html).
- richeyryan 8y ago
  Not a direct answer to your question but you could look at Reason. It is a C style syntax that layers on top of the OCaml semantics. Writing in Reason is writing OCaml. However, the new syntax makes it more approachable for people coming from C style languages and removes idiosyncrasies like the double semicolon. Its still progressing and more focus has been given to getting it up and running in a compile to JS context but it should be equally as capable of systems work, especially as time goes on.
deleted 8y ago
[deleted]
themckman 8y ago
For those interested, some of the underlying libraries that make up Docker for Mac (and, I think, Windows) are written in OCaml (or have components written in OCaml): VPNKit[0], DataKit[1] and HyperKit[2] (qcow2 support is implemented in OCaml). 0: https://github.com/moby/vpnkit https://github.com/moby/vpnkit 1: https://github.com/moby/datakit https://github.com/moby/datakit 2: https://github.com/moby/hyperkit https://github.com/moby/hyperkit
- dfee 8y ago
  To be fair, GitHub attributes 1.5% of hyperkit’s code to ocaml
  - djs55 8y ago
    The 1.5% that's in the hyperkit repo is the shim to the Mirage block layer. The qcow2 implementation that's normally linked in is from https://github.com/mirage/ocaml-qcow https://github.com/mirage/ocaml-qcow .
- pjmlp 8y ago
  They are taken from MirageOS. https://blog.docker.com/2016/05/docker-unikernels-open-source/ https://blog.docker.com/2016/05/docker-unikernels-open-sourc...
rpcope1 8y ago
I think one major turn off, especially for systems programming, was the lack of multicore threading support (i.e. it had no way to get parallelism using threads). Does anyone know if this has changed?
- atombender 8y ago
  Posted a week ago: http://kcsrk.info/slides/mcocaml_gallium.pdf http://kcsrk.info/slides/mcocaml_gallium.pdf Discussion: https://news.ycombinator.com/item?id=17416797 https://news.ycombinator.com/item?id=17416797 In short, expect it merged in mid-to-late 2019, maybe.
- ZirconiumX 8y ago
  While this is an issue, there are cooperative threading libraries (mainly Lwt and Async, though the community prefers Lwt) which can mitigate most of the reasons you'd need multithreading. Hell, because it's only single threaded, you can mostly ignore locking.
- StreamBright 8y ago
  Isn't this project aiming to solve that? http://ocamllabs.io/doc/multicore.html http://ocamllabs.io/doc/multicore.html
- fpoling 8y ago
  One of the fastest web servers is nginx that uses child processes (typically one child per core) and asynchronous IO within the child without any threads. The same model with a parent process that monitor threadless child workers is often used in embedded system. One can and people do exactly the same in OCaml. Surely it requires more code to setup things especially if the child processes need to communicate over shared memory to max the performance, but the big plus is that the resulting system is much more robust. In case of troubles one can just let a child die or even use kill -9 and start a new child. This is not possible with most threading implementations. For example, try to recover from out-of-memory in multi threaded C++/Java/Go etc. application. It is very hard. With threadless children it is almost trivial.
  - 8y ago
testaccount7 8y ago
IIRC in Coders at Work, Brendan Eich talks about hiring a programmer who wrote an OS in OCaml.
ofrzeta 8y ago
For some real world applications take a look at libguestfs http://libguestfs.org http://libguestfs.org, a library that can inspect, mount, edit VM filesystem images. Also includes tools for p2v and v2v migration.
toolslive 8y ago
You really don't want to use the Unix module directly. If you're serious about system programming in OCaml, use Lwt or Async to allow for concurrency.
emersion 8y ago
I'm contributing to a linker written in ~OCaml [1], and now I understand writing systems code in OCaml is really a bad idea. It makes things more complicated when you really don't want them to be. As James Mickens says [2]: >You can’t just place a LISP book on top of an x86 chip and hope that the hardware learns about lambda calculus by osmosis. [1]: https://github.com/rems-project/linksem https://github.com/rems-project/linksem [2]: http://scholar.harvard.edu/files/mickens/files/thenightwatch.pdf http://scholar.harvard.edu/files/mickens/files/thenightwatch...
- phaer 8y ago
  Can you provide an example of the things, OCaml makes more complicated and explain which languages you compare it to? I guess it's quite different in comparison to C, C++ or maybe Rust than to other garbage-collected 'system programming' languages like Go?
  - pasabagi 8y ago
    I think he's getting at the distinction between operational and denotational semantics. Presumably, operational semantics are a better fit for systems programming, because if you need to imagine what the computer is actually doing (Linus's comments about why he likes C comes to mind). Ocaml is more in a denotational direction than C - although to be honest, I don't ultimately understand the distinction, given that C relates to an abstract machine that is not all that similar to most machines today.
    - ernst_klim 8y ago
      But monads are a version of small-stem semantics. And the write linker, so something that operates low-level data, not an actual low-level code, OCaml is a great language for that. Take a look at BAP, it heavily uses OCaml features and implements a beautiful design: https://github.com/BinaryAnalysisPlatform/bap https://github.com/BinaryAnalysisPlatform/bap https://binaryanalysisplatform.github.io/bap/api/master/argot_index.html https://binaryanalysisplatform.github.io/bap/api/master/argo... I'm not convinced that OCaml is unsuitable for such tasks.

269 points by jxub 8y ago

Ocaml, in comparison to other functional statically typed languages has had many successful Unix applications. For a great tour of going to Ocaml (from Python), see Thomas Leonard's blog. A retrospective is here but read the whole lot. http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-retrospective/ http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-ret...

cbcoutinho 8y ago
And related hackernews discussion https://news.ycombinator.com/item?id=7858276 https://news.ycombinator.com/item?id=7858276

a0 8y ago

This is such a great book. OCaml’s type system feels like a superpower specially in the context of Unix development which is traditionally done in C.

tntn 8y ago

Only somewhat related,but I've tried to learn ocaml before but have never been able to figure out when to put what kind of delimiters where. Can anyone recommend a resource that explains the usage of the double comma and the like?

laylomo2 8y ago
The double semicolon only matters in the REPL (aka the "toplevel"). Off the top of my head, I can think of at least two other situations where punctuation matters: 1) Whenever you have nested match cases: match left with | Some left -> match right with | Some right -> Some (left, right) | None -> None | None -> None The compiler does not use indentation to figure out scoping, so the above gets treated as: match left with | Some left -> match right with | Some right -> Some (left, right) | None -> None | None -> None So you need to surround the inner match with parens match left with | Some left -> ( match right with | Some right -> Some (left, right) | None -> None ) | None -> None 2) If you are doing imperative things inside an if statement: Consider the following function: let cache_file file = if not (file_exists file) then printf "downloading file\n"; download file else printf "file already exists\n" The problem here is that OCaml allows one to create an if statement with no else clause. And since the compiler doesn't use indentation to figure out scoping, then the code is essentially treated like the following: let cache_file file = (if not (file_exists file) then printf "downloading file\n"); download file; else (printf "file already exists\n"); Which is just totally wrong. So in order to do multiple imperative actions within an if block, you should use either parens, or begin/end delimiters (which are treated exactly the same as parens) let cache_file file = if not (file_exists file) then begin printf "downloading file\n"; download file end else printf "file already exists\n"
- dpwm 8y ago
  There's an interesting thing with the semicolon. I've heard people advocating thinking of the semicolon as a binary operator: (;) : unit -> unit -> unit Of course, the reality is more complicated than this, because it can be used at the end of an expression, where it behaves like a postfix operator of type 'a -> 'a. Let's take your example: match left with | Some left -> match right with | Some right -> Some (left, right) | None -> None | None -> None you can actually do match left with | Some left -> match right with | Some right -> Some (left, right) | None -> None ; ; (* <-- two separate semicolons *) | None -> None This doesn't just work for imperative stuff, like the operator theory of the semicolon, it will work with this example. My understanding is that this is due to the fact that optional semicolons are allowed at the end of statements and close the current expression. So we're closing the first expression (None) and then closing the second expression (the inner match). The double semicolon is something else altogether. Though it is worth noting that this example is straying from idiomatic OCaml and we can avoid this altogether with better matching: match (left, right) with | (Some left, Some right) -> Some (left, right) | _ -> None Both the native optimizing compiler and BuckleScript are able to do what you would expect with this: the matched tuple is optimized away. Your second unfixed example actually causes a syntax error. There is an if expression in OCaml and an if statement. This is particularly unfortunate, because the let cache_file = function | file when file_exists file -> printf "downloading file\n"; download file | _ -> printf "file already exists\n" first semicolon within an if expression turns the whole thing into an if statement, and an if statement can have no else clause. The else clause is then without an if statement, because we've triggered the imperative if statement. This is a syntax error. were we to define an operator let (>>>) a b = a; b then we could replace that semicolon with an >>> and things behave as expected. But it's really that if can be both a statement and an expression. For this reason I prefer using matches in general. let cache_file = function | file when not (file_exists file) -> printf "downloading file\n"; download file | _ -> printf "file already exists\n"
  - theaeolist 8y ago
    OCaml has no guaranteed order of evaluation, which is not what you want with (;). It needs to be part of the language as special syntax.
thedufer 8y ago
I can't think of any time a double-comma would be syntactically correct. Double-semicolons, which may be what you're referring to, are never required (although in some situations they will cause syntax errors to appear closer to the actual problem). They can be placed after all top-level value bindings. Probably your best bet is to follow an intro series so you see some examples (https://dev.realworldocaml.org/ https://dev.realworldocaml.org/ is the best at the moment, I believe), although if you're so inclined you could just read the BNF for the language (https://caml.inria.fr/pub/docs/manual-ocaml/language.html https://caml.inria.fr/pub/docs/manual-ocaml/language.html).
richeyryan 8y ago
Not a direct answer to your question but you could look at Reason. It is a C style syntax that layers on top of the OCaml semantics. Writing in Reason is writing OCaml. However, the new syntax makes it more approachable for people coming from C style languages and removes idiosyncrasies like the double semicolon. Its still progressing and more focus has been given to getting it up and running in a compile to JS context but it should be equally as capable of systems work, especially as time goes on.

deleted 8y ago

[deleted]

themckman 8y ago

For those interested, some of the underlying libraries that make up Docker for Mac (and, I think, Windows) are written in OCaml (or have components written in OCaml): VPNKit[0], DataKit[1] and HyperKit[2] (qcow2 support is implemented in OCaml). 0: https://github.com/moby/vpnkit https://github.com/moby/vpnkit 1: https://github.com/moby/datakit https://github.com/moby/datakit 2: https://github.com/moby/hyperkit https://github.com/moby/hyperkit

dfee 8y ago
To be fair, GitHub attributes 1.5% of hyperkit’s code to ocaml
- djs55 8y ago
  The 1.5% that's in the hyperkit repo is the shim to the Mirage block layer. The qcow2 implementation that's normally linked in is from https://github.com/mirage/ocaml-qcow https://github.com/mirage/ocaml-qcow .
pjmlp 8y ago
They are taken from MirageOS. https://blog.docker.com/2016/05/docker-unikernels-open-source/ https://blog.docker.com/2016/05/docker-unikernels-open-sourc...

rpcope1 8y ago

I think one major turn off, especially for systems programming, was the lack of multicore threading support (i.e. it had no way to get parallelism using threads). Does anyone know if this has changed?

atombender 8y ago
Posted a week ago: http://kcsrk.info/slides/mcocaml_gallium.pdf http://kcsrk.info/slides/mcocaml_gallium.pdf Discussion: https://news.ycombinator.com/item?id=17416797 https://news.ycombinator.com/item?id=17416797 In short, expect it merged in mid-to-late 2019, maybe.
ZirconiumX 8y ago
While this is an issue, there are cooperative threading libraries (mainly Lwt and Async, though the community prefers Lwt) which can mitigate most of the reasons you'd need multithreading. Hell, because it's only single threaded, you can mostly ignore locking.
StreamBright 8y ago
Isn't this project aiming to solve that? http://ocamllabs.io/doc/multicore.html http://ocamllabs.io/doc/multicore.html
fpoling 8y ago
One of the fastest web servers is nginx that uses child processes (typically one child per core) and asynchronous IO within the child without any threads. The same model with a parent process that monitor threadless child workers is often used in embedded system. One can and people do exactly the same in OCaml. Surely it requires more code to setup things especially if the child processes need to communicate over shared memory to max the performance, but the big plus is that the resulting system is much more robust. In case of troubles one can just let a child die or even use kill -9 and start a new child. This is not possible with most threading implementations. For example, try to recover from out-of-memory in multi threaded C++/Java/Go etc. application. It is very hard. With threadless children it is almost trivial.
- 8y ago

testaccount7 8y ago

IIRC in Coders at Work, Brendan Eich talks about hiring a programmer who wrote an OS in OCaml.

ofrzeta 8y ago

For some real world applications take a look at libguestfs http://libguestfs.org http://libguestfs.org, a library that can inspect, mount, edit VM filesystem images. Also includes tools for p2v and v2v migration.

toolslive 8y ago

You really don't want to use the Unix module directly. If you're serious about system programming in OCaml, use Lwt or Async to allow for concurrency.

emersion 8y ago

I'm contributing to a linker written in ~OCaml [1], and now I understand writing systems code in OCaml is really a bad idea. It makes things more complicated when you really don't want them to be. As James Mickens says [2]: >You can’t just place a LISP book on top of an x86 chip and hope that the hardware learns about lambda calculus by osmosis. [1]: https://github.com/rems-project/linksem https://github.com/rems-project/linksem [2]: http://scholar.harvard.edu/files/mickens/files/thenightwatch.pdf http://scholar.harvard.edu/files/mickens/files/thenightwatch...

phaer 8y ago
Can you provide an example of the things, OCaml makes more complicated and explain which languages you compare it to? I guess it's quite different in comparison to C, C++ or maybe Rust than to other garbage-collected 'system programming' languages like Go?
- pasabagi 8y ago
  I think he's getting at the distinction between operational and denotational semantics. Presumably, operational semantics are a better fit for systems programming, because if you need to imagine what the computer is actually doing (Linus's comments about why he likes C comes to mind). Ocaml is more in a denotational direction than C - although to be honest, I don't ultimately understand the distinction, given that C relates to an abstract machine that is not all that similar to most machines today.
  - ernst_klim 8y ago
    But monads are a version of small-stem semantics. And the write linker, so something that operates low-level data, not an actual low-level code, OCaml is a great language for that. Take a look at BAP, it heavily uses OCaml features and implements a beautiful design: https://github.com/BinaryAnalysisPlatform/bap https://github.com/BinaryAnalysisPlatform/bap https://binaryanalysisplatform.github.io/bap/api/master/argot_index.html https://binaryanalysisplatform.github.io/bap/api/master/argo... I'm not convinced that OCaml is unsuitable for such tasks.