6 ms·

Clive OS: Go applications that can run without an operating system (2016)

162 points by merqurio 9y ago

MrBuddyCasino 9y ago
It would have helped to state their mission goals. This is basically a page that says "look we made this, no underlying stack and lots of CSP", without stating what they hope to achieve or improve versus the status quo. Are there security benefits? Probably. Better performance? Could be. But why did they come up with their solution? What about differences with other Unikernel approaches? I mean its a research lab, not a hobby project.
- carussell 9y ago
  This submission is here because of the link to the intro to OSes book[1] by one of the Clive authors that was posted yesterday. It's worth taking a look at the other stuff referenced on this page page, as well as the author's other links.[2] To answer your question: > Better performance? Could be. That's one aspect. If you follow the link from the Clive page to its predecessor Nix[3], its tagline is "high performance cloud computing is nix". A current (2018) justification for that project can be found in the form of an abstract for a paper with the same name[4]. But then the other part is given away in the abstract for Clive itself: "Clive introduces new file system technology and novel interfaces for the construction of networked system"[5], which is another way of saying that this is blue sky research, which is itself a codeword for we decided to try out a bunch of things purely _because_ they're different and want to see if anything will come out of it. Notice how the screenshot here shows the OS shell is some sort of acme-like[6] made to run in a web browser, for example. To repeat my earlier remark, the author's other pages are well worth a look. For example, don't miss this[7] page chock full of links to research OSes (although many of them are dead). Or there's Picky[8], which looks like what you'd get if you took Pascal/Oberon and more or less did a transliteration so that the grammar gives you a curly brace language instead of a wordy, shouty one with BEGIN, END, etc. 1. https://news.ycombinator.com/item?id=16253193 https://news.ycombinator.com/item?id=16253193 2. http://lsub.org/ls/nix.html http://lsub.org/ls/nix.html 3. http://lsub.org/ls/nix.html http://lsub.org/ls/nix.html 4. http://lsub.org/ls/nixbltjabs.pdf http://lsub.org/ls/nixbltjabs.pdf 5. http://lsub.org/export/clivesys.pdf http://lsub.org/export/clivesys.pdf 6. http://syssoftware.blogspot.com/2016/02/clives-ink-screenshot.html http://syssoftware.blogspot.com/2016/02/clives-ink-screensho... 7. https://lsub.org/who/nemo/os.html https://lsub.org/who/nemo/os.html 8. http://lsub.org/ls/picky.html http://lsub.org/ls/picky.html
  - merqurio 9y ago
    There is a new paper[1] from F.J. Balleteros published on 2017 that serves as a great justification too. - [1] https://jisajournal.springeropen.com/articles/10.1186/s13174-016-0054-8 https://jisajournal.springeropen.com/articles/10.1186/s13174...
m12k 9y ago
For anyone interested in unikernels for other languages than Go, there's a list over on Wikipedia: https://en.wikipedia.org/wiki/Unikernel#Examples https://en.wikipedia.org/wiki/Unikernel#Examples
- pknopf 9y ago
  Why make bootable images, as opposed to docker images? What sorts of applications do you see using this?
  - convolvatron 9y ago
    in one case I boot a blob and it comes up with everything it needs and starts functioning immediately. there is no ambiguity in which code is executing since its exactly those bits that I constructed it with, and no external service dependencies. if you work in a model where you never upgrade servers, just bring up new ones and turn down old ones, doesn't it seem cleaner? obviously they both work.
  - e12e 9y ago
    Hardware level isolation. Or: if you run your docker container as a vm instance; reduced overhead with only the kernel parts you need. Typical docker is just a glorified chroot under a shared Linux kernel. With namespaces it asymptotically approach a bsd jail. But the "easy" way to get some real isolation, is to use hw support for virtualization - but now you need a kernel (software that enable programs to talk via a fixed abstraction to hardware; drivers for real and emulated hw). And you've reinvented 5% of the magic dust that IBM put into mainframes.
ilackarms 9y ago
for anyone interested in running Go apps as Unikernels, I suggest checking out https://github.com/solo-io/unik https://github.com/solo-io/unik disclosure: I'm one of the authors
- equalunique 9y ago
  Java, Node.js, Python 2/3, C/C++, and Go! Bravo!
alex_hitchins 9y ago
Does anyone know of any Unikernerls running out there in the wild? This seems like a very efficient way to run heavy workloads. I wonder what the catch is, and if the benefits are outweighed by what you miss without an OS.
- weberc2 9y ago
  This is all theoretical right now, but I'm more excited about a world where VMs are treated like containers. Application orchestration at the VM level instead of at the container level. Reduced attack surface (no unnecessary OS components). Fewer moving parts = easier to reason about. There are still a lot of kinks to be worked out. For one, what is the debugging story when you can't SSH onto a VM? Or do you compile into your application something analogous to SSH? If so, does that mean you're bringing in much of the "unnecessary baggage" that makes unikernels so compelling in theory? I'm excited, but cautiously so.
  - alex_hitchins 9y ago
    It's certainty an exciting area to keep an eye on. I have not looked at containers too much, I understand the basic theory but I've not had a use case myself to deploy them in production. I am much more familiar and in love with virtual machines however. If applications could be built as a self contained tiny VM's I can see myself using these quite heavily.
    - weberc2 9y ago
      I think the most exciting thing about containers is actually the infrastructure and tooling around them. The technology for managing containers and container images is a lot better than that for managing VMs and VM images. As far as I know, there isn't anything comparable to Docker, Docker registry, Docker swarm, etc for VMs, or at least they are outclassed in terms of power and usability. If we could bring all of the benefits of containers to the VM world (including subsecond boot times), I think that could revolutionize at least cloud computing (if not other forms).
jdmoreira 9y ago
The naming / logo / design is an obvious homage to Clive Sinclair :)
- tomca32 9y ago
  Thank you for pointing this out. I spent so much time with the old ZX Spectrum, but I still totally missed the reference. For context, Clive Sinclair is most famous for the ZX Spectrum computer https://en.wikipedia.org/wiki/ZX_Spectrum https://en.wikipedia.org/wiki/ZX_Spectrum
- Razengan 9y ago
  My very first thought as well, though if it is so, then I wish they should have mentioned that up front. Maybe they don't because of legal reasons? Bleh
kruhft 9y ago
Relevant: Programming Languages as Operating Systems https://www2.ccs.neu.edu/racket/pubs/icfp99-ffkf.pdf https://www2.ccs.neu.edu/racket/pubs/icfp99-ffkf.pdf
- kruhft 9y ago
  Imagine if every language was it's own Interactive Shell with no kernel. Scripting and functional languages already have this, as do C and C++[1]. You would lose language interchangeability on the system until someone wrote compilers for the other language software that they want to support. And who doesn't like to write compilers in their favorite language. [1] https://github.com/root-project/cling https://github.com/root-project/cling
  - pjmlp 9y ago
    No need to imagine, Oberon was quite a nice experience. :)
    - kruhft 9y ago
      For Modula II.
      - kruhft 9y ago
        So were Lisp Machines.
    - kruhft 9y ago
      Now imagine if IncludeOS worked for everything written in C/C++. Instant on the metal interpreters and compilers.
earenndil 9y ago
See also: includeos, http://www.includeos.org/ http://www.includeos.org/

162 points by merqurio 9y ago

It would have helped to state their mission goals. This is basically a page that says "look we made this, no underlying stack and lots of CSP", without stating what they hope to achieve or improve versus the status quo. Are there security benefits? Probably. Better performance? Could be. But why did they come up with their solution? What about differences with other Unikernel approaches? I mean its a research lab, not a hobby project.

carussell 9y ago
This submission is here because of the link to the intro to OSes book[1] by one of the Clive authors that was posted yesterday. It's worth taking a look at the other stuff referenced on this page page, as well as the author's other links.[2] To answer your question: > Better performance? Could be. That's one aspect. If you follow the link from the Clive page to its predecessor Nix[3], its tagline is "high performance cloud computing is nix". A current (2018) justification for that project can be found in the form of an abstract for a paper with the same name[4]. But then the other part is given away in the abstract for Clive itself: "Clive introduces new file system technology and novel interfaces for the construction of networked system"[5], which is another way of saying that this is blue sky research, which is itself a codeword for we decided to try out a bunch of things purely _because_ they're different and want to see if anything will come out of it. Notice how the screenshot here shows the OS shell is some sort of acme-like[6] made to run in a web browser, for example. To repeat my earlier remark, the author's other pages are well worth a look. For example, don't miss this[7] page chock full of links to research OSes (although many of them are dead). Or there's Picky[8], which looks like what you'd get if you took Pascal/Oberon and more or less did a transliteration so that the grammar gives you a curly brace language instead of a wordy, shouty one with BEGIN, END, etc. 1. https://news.ycombinator.com/item?id=16253193 https://news.ycombinator.com/item?id=16253193 2. http://lsub.org/ls/nix.html http://lsub.org/ls/nix.html 3. http://lsub.org/ls/nix.html http://lsub.org/ls/nix.html 4. http://lsub.org/ls/nixbltjabs.pdf http://lsub.org/ls/nixbltjabs.pdf 5. http://lsub.org/export/clivesys.pdf http://lsub.org/export/clivesys.pdf 6. http://syssoftware.blogspot.com/2016/02/clives-ink-screenshot.html http://syssoftware.blogspot.com/2016/02/clives-ink-screensho... 7. https://lsub.org/who/nemo/os.html https://lsub.org/who/nemo/os.html 8. http://lsub.org/ls/picky.html http://lsub.org/ls/picky.html
- merqurio 9y ago
  There is a new paper[1] from F.J. Balleteros published on 2017 that serves as a great justification too. - [1] https://jisajournal.springeropen.com/articles/10.1186/s13174-016-0054-8 https://jisajournal.springeropen.com/articles/10.1186/s13174...

m12k 9y ago

For anyone interested in unikernels for other languages than Go, there's a list over on Wikipedia: https://en.wikipedia.org/wiki/Unikernel#Examples https://en.wikipedia.org/wiki/Unikernel#Examples

pknopf 9y ago
Why make bootable images, as opposed to docker images? What sorts of applications do you see using this?
- convolvatron 9y ago
  in one case I boot a blob and it comes up with everything it needs and starts functioning immediately. there is no ambiguity in which code is executing since its exactly those bits that I constructed it with, and no external service dependencies. if you work in a model where you never upgrade servers, just bring up new ones and turn down old ones, doesn't it seem cleaner? obviously they both work.
- e12e 9y ago
  Hardware level isolation. Or: if you run your docker container as a vm instance; reduced overhead with only the kernel parts you need. Typical docker is just a glorified chroot under a shared Linux kernel. With namespaces it asymptotically approach a bsd jail. But the "easy" way to get some real isolation, is to use hw support for virtualization - but now you need a kernel (software that enable programs to talk via a fixed abstraction to hardware; drivers for real and emulated hw). And you've reinvented 5% of the magic dust that IBM put into mainframes.

ilackarms 9y ago

for anyone interested in running Go apps as Unikernels, I suggest checking out https://github.com/solo-io/unik https://github.com/solo-io/unik disclosure: I'm one of the authors

equalunique 9y ago
Java, Node.js, Python 2/3, C/C++, and Go! Bravo!

alex_hitchins 9y ago

Does anyone know of any Unikernerls running out there in the wild? This seems like a very efficient way to run heavy workloads. I wonder what the catch is, and if the benefits are outweighed by what you miss without an OS.

weberc2 9y ago
This is all theoretical right now, but I'm more excited about a world where VMs are treated like containers. Application orchestration at the VM level instead of at the container level. Reduced attack surface (no unnecessary OS components). Fewer moving parts = easier to reason about. There are still a lot of kinks to be worked out. For one, what is the debugging story when you can't SSH onto a VM? Or do you compile into your application something analogous to SSH? If so, does that mean you're bringing in much of the "unnecessary baggage" that makes unikernels so compelling in theory? I'm excited, but cautiously so.
- alex_hitchins 9y ago
  It's certainty an exciting area to keep an eye on. I have not looked at containers too much, I understand the basic theory but I've not had a use case myself to deploy them in production. I am much more familiar and in love with virtual machines however. If applications could be built as a self contained tiny VM's I can see myself using these quite heavily.
  - weberc2 9y ago
    I think the most exciting thing about containers is actually the infrastructure and tooling around them. The technology for managing containers and container images is a lot better than that for managing VMs and VM images. As far as I know, there isn't anything comparable to Docker, Docker registry, Docker swarm, etc for VMs, or at least they are outclassed in terms of power and usability. If we could bring all of the benefits of containers to the VM world (including subsecond boot times), I think that could revolutionize at least cloud computing (if not other forms).

jdmoreira 9y ago

The naming / logo / design is an obvious homage to Clive Sinclair :)

tomca32 9y ago
Thank you for pointing this out. I spent so much time with the old ZX Spectrum, but I still totally missed the reference. For context, Clive Sinclair is most famous for the ZX Spectrum computer https://en.wikipedia.org/wiki/ZX_Spectrum https://en.wikipedia.org/wiki/ZX_Spectrum
Razengan 9y ago
My very first thought as well, though if it is so, then I wish they should have mentioned that up front. Maybe they don't because of legal reasons? Bleh

kruhft 9y ago

Relevant: Programming Languages as Operating Systems https://www2.ccs.neu.edu/racket/pubs/icfp99-ffkf.pdf https://www2.ccs.neu.edu/racket/pubs/icfp99-ffkf.pdf

kruhft 9y ago
Imagine if every language was it's own Interactive Shell with no kernel. Scripting and functional languages already have this, as do C and C++[1]. You would lose language interchangeability on the system until someone wrote compilers for the other language software that they want to support. And who doesn't like to write compilers in their favorite language. [1] https://github.com/root-project/cling https://github.com/root-project/cling
- pjmlp 9y ago
  No need to imagine, Oberon was quite a nice experience. :)
  - kruhft 9y ago
    For Modula II.
    - kruhft 9y ago
      So were Lisp Machines.
  - kruhft 9y ago
    Now imagine if IncludeOS worked for everything written in C/C++. Instant on the metal interpreters and compilers.

earenndil 9y ago

See also: includeos, http://www.includeos.org/ http://www.includeos.org/