4 ms·

Two answers: scaling through structured distributed systems abstractions is a key aim in Mirage. We deliberately want each VM to be predictable, single vCPU and

by avsm 12y ago

Two answers: scaling through structured distributed systems abstractions is a key aim in Mirage. We deliberately want each VM to be predictable, single vCPU and scale via multiple VMs. It's far more efficient to scale via lots of small VMs that are scheduled independently than having a few big multicpu VMs (which have a lot of overhead since the CPU sync also needs to be virtualised). See our ASPLOS 2013 paper for some simple workloads on this topic. We are building this library to simplify this style of distributed programming: http://openmirage.org/blog/introducing-irmin http://openmirage.org/blog/introducing-irmin Other answer: we will be talking about our multicore ocaml implementation at ICFP in September. I still don't want to see it in Mirage though :-)

wmf 12y ago
It's far more efficient to scale via lots of small VMs that are scheduled independently than having a few big multicpu VMs (which have a lot of overhead since the CPU sync also needs to be virtualised). Are you assuming that vCPUs are scheduled or pinned? It wasn't clear from skimming the paper. I would agree that two levels of scheduling are bad but if that's the problem pinning should fix it. The situation may change if you eliminate the hypervisor. My intuition is that a single multi-threaded process with work-stealing will be faster than separate processes or VMs due to better load balance (see PX vs. 1X dynos) and faster inter-thread communication (if your app has any communication).
- avsm 12y ago
  If you eliminate the hypervisor, you don't have vCPUs at all, so I'm not sure how this is a useful comparison. The question of IPC performance is an interesting one. We've been building up a database of open source results that show wildly diverging results on different architectures. Surf through http://fable.io http://fable.io for it, or read this work in progress: http://anil.recoil.org/papers/drafts/2012-usenix-ipc-draft1.pdf http://anil.recoil.org/papers/drafts/2012-usenix-ipc-draft1.... http://anil.recoil.org/talks/fosdem-io-2012.pdf http://anil.recoil.org/talks/fosdem-io-2012.pdf (fosdem slides) The TL;DR of these numbers is that it's very hard to make firm performance hypotheses about IPC across architectures, NUMA and hypervisors.
616c 12y ago
More on point: So I am understanding correctly message passing will most likely by the paradigm you encourage? And, less on point: > Other answer: we will be talking about our multicore ocaml implementation at ICFP in September. I still don't want to see it in Mirage though :-) Wait what what what? So it is ready for prime-time? (Note, I did not say production.) I remember reading about it on Jane Street, but thought there will still proposals and lots of theoretical and planning wrinkles to iron out before an implementation became reality. I do not want to derail this informative post by you anymore, but do you have a link to more on that topic? And I remember seeing mention of Irmin and not getting how ti fit into your work, asvm. Now seeing it as an answer to my question makes sense. And thanks for your work on Real World Ocaml. I have decided to get back into programming, and some of my co-workers were taking CS classes and systems programming (ironically Harvard Extension School, I can't afford it but good for them) used OCaml. This reminded me to come take a look and I find your book, and the OCaml resurgence fascinating. Maybe one done, I will look at the Haxe compiler and go: "it is not so complicated, I get that now." Only in OCaml could I see people doing such crazy shit.
- amirmc 12y ago
  > ... do you have a link to more on that topic? The following may be of interest [1]. There's been more progress since and we'll be talking about it at OCaml 2014 [2]. [1] http://www.cl.cam.ac.uk/~sd601/multicore.md http://www.cl.cam.ac.uk/~sd601/multicore.md [2] http://ocaml.org/meetings/ocaml/2014/program.html http://ocaml.org/meetings/ocaml/2014/program.html

by avsm 12y ago

wmf 12y ago

It's far more efficient to scale via lots of small VMs that are scheduled independently than having a few big multicpu VMs (which have a lot of overhead since the CPU sync also needs to be virtualised). Are you assuming that vCPUs are scheduled or pinned? It wasn't clear from skimming the paper. I would agree that two levels of scheduling are bad but if that's the problem pinning should fix it. The situation may change if you eliminate the hypervisor. My intuition is that a single multi-threaded process with work-stealing will be faster than separate processes or VMs due to better load balance (see PX vs. 1X dynos) and faster inter-thread communication (if your app has any communication).

avsm 12y ago
If you eliminate the hypervisor, you don't have vCPUs at all, so I'm not sure how this is a useful comparison. The question of IPC performance is an interesting one. We've been building up a database of open source results that show wildly diverging results on different architectures. Surf through http://fable.io http://fable.io for it, or read this work in progress: http://anil.recoil.org/papers/drafts/2012-usenix-ipc-draft1.pdf http://anil.recoil.org/papers/drafts/2012-usenix-ipc-draft1.... http://anil.recoil.org/talks/fosdem-io-2012.pdf http://anil.recoil.org/talks/fosdem-io-2012.pdf (fosdem slides) The TL;DR of these numbers is that it's very hard to make firm performance hypotheses about IPC across architectures, NUMA and hypervisors.

616c 12y ago

More on point: So I am understanding correctly message passing will most likely by the paradigm you encourage? And, less on point: > Other answer: we will be talking about our multicore ocaml implementation at ICFP in September. I still don't want to see it in Mirage though :-) Wait what what what? So it is ready for prime-time? (Note, I did not say production.) I remember reading about it on Jane Street, but thought there will still proposals and lots of theoretical and planning wrinkles to iron out before an implementation became reality. I do not want to derail this informative post by you anymore, but do you have a link to more on that topic? And I remember seeing mention of Irmin and not getting how ti fit into your work, asvm. Now seeing it as an answer to my question makes sense. And thanks for your work on Real World Ocaml. I have decided to get back into programming, and some of my co-workers were taking CS classes and systems programming (ironically Harvard Extension School, I can't afford it but good for them) used OCaml. This reminded me to come take a look and I find your book, and the OCaml resurgence fascinating. Maybe one done, I will look at the Haxe compiler and go: "it is not so complicated, I get that now." Only in OCaml could I see people doing such crazy shit.

amirmc 12y ago
> ... do you have a link to more on that topic? The following may be of interest [1]. There's been more progress since and we'll be talking about it at OCaml 2014 [2]. [1] http://www.cl.cam.ac.uk/~sd601/multicore.md http://www.cl.cam.ac.uk/~sd601/multicore.md [2] http://ocaml.org/meetings/ocaml/2014/program.html http://ocaml.org/meetings/ocaml/2014/program.html