10 ms·

Scriptable Operating Systems with Lua [pdf]

100 points by wormold 12y ago

knz42 12y ago
The article is well-written and makes a good point that scripting languages open new avenues for OS extension. However the question remains whether Lua is the right candidate for the job, being not-so-open yet. Obviously the hacker cultures in NetBSD world and Lua world are still quite different, so I see no official integration in any foreseeable future. A lightweight LISP or Scheme, on the other hand, might be a good candidate for a long-term implementation of the article's claims.
- notacoward 12y ago
  How is Lua "not so open"? Looks like MIT license to me, and I think most people would consider that about as open as it gets. http://www.lua.org/license.html http://www.lua.org/license.html
  - fit2rule 12y ago
    Yes, the MIT License is one of Lua's very, very key features. Those who use Lua and ship it in the millions to paying customers are only able to do so, the way they're doing it, because of the freedoms of the MIT License. Apropos Lua not being 'open', or a 'bad language', as a user of Lua on a full-spectrum approach, I can definitely say "meh!" to all nay-sayers. For every detraction, there is a significant advantage. LuaJIT, cache-coherency, access to high-performance components, well-compartmentalized, due to a simple discipline .. these are the things to love about Lua. But please, anyone not yet familiar with the subject, approach Lua not critically; from the perspective of the progressive, if you haven't grok'ed the way you can drop the VM into anything and pass it byte-code, then please do this first. Grok the VM, guys. Fact is: Lua can be glommed onto anything. Any-thing.
- jonjacky 12y ago
  Schemix was "a Scheme system implemented as a patch to the Linux kernel ... for exploration of the Linux kernel and for rapid, interactive prototyping of Linux drivers and other new kernel features" http://www.abstractnonsense.com/schemix/ http://www.abstractnonsense.com/schemix/ But Schemix has been inactive since 2003.
- theonewolf 12y ago
  Actually I think Lua is a great candidate because of its openness and its similarity to C (predominant systems OS programming language). Lua is open, as their about page (http://www.lua.org/about.html http://www.lua.org/about.html) states: Lua is free Lua is free open-source software, distributed under a very liberal license (the well-known MIT license). It may be used for any purpose, including commercial purposes, at absolutely no cost. Just download it and use it.
  - ufo 12y ago
    I wouldn't say Lua is similar to C as far as syntax and semantics goes (for example, arrays indexes start from 1). That said, Lua is great at working with C - its very easy to call C code from Lua and vice versa.
- justincormack 12y ago
  The Lua module is already in NetBSD, that's pretty official integration. There is also a large overlap between people who use NetBSD and people who use Lua, probably due to the fact that they are both relatively small C based projects that try to do one thing well. Indeed the linked paper is written by two NetBSD commiters, the main author of Lua and another person on the Lua team.
- Quequau 12y ago
  Isn't the whole idea of Guile is that it's a lightweight Lisp intended for Operating System scripting?
  - alricb 12y ago
    Indeed! That's why the latest release version of Guile has a compressed source that takes only 7.2 MB (gzipped).
    - sdegutis 12y ago
      Can't tell if serious.
davidw 12y ago
A while ago, I hacked some forth interpreter (I don't recall which one) into the Linux kernel. It was just fooling around though, and didn't do anything useful or hook into anything interesting. This looks like a much more practical approach to that kind of thing.
_vya7 12y ago
For those who are not familiar with Lua, or who have heard the prevalent "conventional wisdom" about it, here's a quick clarification: A. Lua's source code is really really small and extremely well-commented[1]. B. Lua is super easy to integrate... you can literally just throw Lua's entire source code into your app[2] and it'll compile just fine. C. Lua's C API is super easy to use, and way more thought-out than Python's or Ruby's (no offense to Matz or Guido) -- yes, it is stack-based, but this actually makes its API much simpler and smaller than it would otherwise be. You can probably learn all of it in an afternoon (with the right guide). D. Lua is uncannily similar to JavaScript in semantics, and the syntax[3] is ridiculously simple to learn. Semantically, it has way less magic to learn than either Ruby or Python. E. Lua isn't only for scripting video games. I wish this "conventional wisdom" would just die. Yes, it is good for scripting video games, for the same reason it's excellent for scripting anything. F. Table indices starting at 1 is not hard to reason about, not hard to use, and it's not hard to switch back-and-forth between Lua and languages where indices start at 0. This is a complete myth. [1]: http://www.lua.org/source/5.2/ http://www.lua.org/source/5.2/ [2]: like this: https://github.com/sdegutis/mjolnir/tree/master/Mjolnir/lua https://github.com/sdegutis/mjolnir/tree/master/Mjolnir/lua [3]: here's Lua's EBNF in its entirely, fitting on a single screen for me: http://www.lua.org/manual/5.2/manual.html#9 http://www.lua.org/manual/5.2/manual.html#9
- davidw 12y ago
  All of the above are why antirez picked it for Redis, despite being a fan of Tcl. I've never used Lua, but it strikes me as 'the' thing in highly-embeddable niche these days.
  - sdegutis 12y ago
    Oh thanks for helping me remember the last bullet point: Lua is fine for small embedded environments, but it's also an excellent way to add scripting support to plain old bulky GUI apps like Firefox or Sublime Text or whathaveyou.
- alricb 12y ago
  Re: EBNF and C code, here's the parser for Lua 5.2: http://www.lua.org/source/5.2/lparser.c.html http://www.lua.org/source/5.2/lparser.c.html
jrapdx3 12y ago
A very interesting article. While the idea of using scripting to extend OS functionality has been around for a long time, surprisingly few attempts have been made to implement it. Of the potential uses, an OS built-in webserver would be especially appealing. Many scripting languages provide tcp server/client APIs so it doesn't seem too much of a stretch to embed this functionality in a kernel. For example, Tcl's "socket" core command makes it very easy to set up a tcp server or client. If performance or security is an issue, some interpreted languages can also be compiled to native code. There are Scheme implementations that are embeddable in C and can load interpreted or compiled modules. One thing I didn't completely grasp is the distinction of "embedding" vs. "extending". Are these really disjoint ideas? I found this puzzling: "... Extension scripts are loaded into instances of a kernel-embedded Tcl interpreter that execute in independent system processes [22]. A set of extension bindings exposes to the scripts the necessary resources for extending the kernel. What distinguishes our approach from [micro]Choices is our support of scripting by embedding the language interpreter, in addition to scripting by extending it." [emphasis is mine] On the whole, the idea seems familiar and sensible where sane security policies and safety constraints are applied to actions of the embedded scripting language.
- alricb 12y ago
  Re: extending vs. embedding, it depends on whether your main app is written in Lua, with C bindings (extending), or in C, with the C code calling the Lua interpreter (embedding). See for instance this Python guide: https://docs.python.org/2/extending/ https://docs.python.org/2/extending/
- sdegutis 12y ago
  Yes, there is a difference between embedding and extending. Extending means your app is just a module that lives in the language's interpreter. Embedding means the language interpreter lives in your app. One advocate[-1] of extending prefers it simply because he got bitten by his own poor design. Another advocate[undefined] prefers it simply because he prefers JavaScript and the easiest way to embed it is to write your app as C plugins that hook into V8[NaN]. The Python community[2] will unfailingly tell you to choose extending over embedding, delegating their reasoning to this highly questionable rant[2]. [-1]: https://julien.danjou.info/blog/2011/why-not-lua https://julien.danjou.info/blog/2011/why-not-lua [undefined]: Zhivago in ##c on freenode; please go say hi to him and make sure 2 use abbvs and dont use punctatng or spell thngs rite b4 him [NaN]: http://stackoverflow.com/questions/6334618/experience-embedding-javascript http://stackoverflow.com/questions/6334618/experience-embedd... [2]: aka #python on freenode [2]: http://twistedmatrix.com/users/glyph/rant/extendit.html http://twistedmatrix.com/users/glyph/rant/extendit.html
RexRollman 12y ago
I really like NetBSD. It has a clean design and it was the first Unix-like OS that I really picked up on, right around the v1.5 - v1.6 releases (2001 to 2002). I haven't been able to run it much since then due to owning incompatible hardware, but after seeing how well it runs in emulation, I am considering recycling some PC hardware and running it on top of Microsoft's free Hyper-V Server 2012. The only problem is that Hyper-V Server doesn't support wireless networking and where I want to put it isn't anywhere near a router, so I am unsure how to handle networking.
- gregholmberg 12y ago
  Your system isn't one of these 58 supported platforms? [0] [0] netbsd.org/ports
  - truncate 12y ago
    Maybe he was referring to missing device drivers, not CPU.
    - RexRollman 12y ago
      Exactly.
c3d 12y ago
A lot of the discussion here has been around Lua, but not around what they do with it. I think that putting yet another language and runtime in the OS kernel is a bad idea. It makes the amount of code to validate much bigger. I much prefer the Mill Computing approach to security, where "everybody works the same" including the kernel (see http://millcomputing.com/docs/security/ http://millcomputing.com/docs/security/ if you have not seen it already). That's also the idea behind many microkernels. Although the Mill CPU embeds basic mechanisms in the hardware to make it easier, I believe this could be achieved on modern x86 or ARM as well with reasonable performance. You don't need privileged instructions to deal with packet routing or CPU throttling, you only need controlled memory access to specific regions of memory (e.g. device registers, buffers, etc). I see no reason why the scripts could not do their work from user-space, with kernel-controlled access to the regions of memory they need to operate. And the overhead in doing that is practically zero (basically, it's the cost of the TLB translations, which is paid for every single memory access anyway). Obviously, you also need inter-process and inter-CPU synchro, interrupt handling, etc. But all these are already presented in a virtualised form to user-space. Also, if you want the flexibility of a scripting language, you are willing to forego quite a bit of performance in the process. Is the cost of a user-kernel transition really that relevant in this scenario? And if it is, there are mechanisms to mitigate that cost, e.g. pooling requests, deferring, sending to another CPU, etc. I'm a bit appalled by the way they measure the overhead (Section 5 of the article). They say that their CPU rethrottling code runs in "only" 8 microseconds. Well, if compared to the rescheduling interval, it may seem small, 8 microseconds in kernel code is actually a lot. It means 1/125 of a regular millisecond tick. All this just to change the frequency of the CPU? In short, I agree with the objectives (making the OS more extensible and more flexible), but the way they did it seems dangerous and under-optimal.
- fit2rule 12y ago
  >>I think that putting yet another language and runtime in the OS kernel is a bad idea. I'm of the opposite opinion. My experience with distributions and host-OS/integrated libs+bins, from a product perspective, lends me more towards the new-school idea of putting the Linux kernel in a bare machine with nothing but Lua.. because actually this is already a configuration and approach to embedded computing. Lua is used industriously as a configuration and embedded scripting language; it is one of the small/light/good-enough variants of the gems of the open source world. A full-stack Linux OS with strictly Lua front-end, from embedded to desktop, is an achievable target.. and indeed a worthy goal. There will always be new languages. Always. But what really matters is the usage factor. Putting an extreme case forward as a simplification of the field will always, thus, be an interesting exercise. As we can see with the router-Lua guys, often profitable. Imagine that space encroaching on Android, et al.? There are some who say that a bridging maneuver against the disastrous developer-mindset-killing, "iOS vs. Android" battle, has been waged, and its flag is Lua.

100 points by wormold 12y ago

knz42 12y ago

The article is well-written and makes a good point that scripting languages open new avenues for OS extension. However the question remains whether Lua is the right candidate for the job, being not-so-open yet. Obviously the hacker cultures in NetBSD world and Lua world are still quite different, so I see no official integration in any foreseeable future. A lightweight LISP or Scheme, on the other hand, might be a good candidate for a long-term implementation of the article's claims.

notacoward 12y ago
How is Lua "not so open"? Looks like MIT license to me, and I think most people would consider that about as open as it gets. http://www.lua.org/license.html http://www.lua.org/license.html
- fit2rule 12y ago
  Yes, the MIT License is one of Lua's very, very key features. Those who use Lua and ship it in the millions to paying customers are only able to do so, the way they're doing it, because of the freedoms of the MIT License. Apropos Lua not being 'open', or a 'bad language', as a user of Lua on a full-spectrum approach, I can definitely say "meh!" to all nay-sayers. For every detraction, there is a significant advantage. LuaJIT, cache-coherency, access to high-performance components, well-compartmentalized, due to a simple discipline .. these are the things to love about Lua. But please, anyone not yet familiar with the subject, approach Lua not critically; from the perspective of the progressive, if you haven't grok'ed the way you can drop the VM into anything and pass it byte-code, then please do this first. Grok the VM, guys. Fact is: Lua can be glommed onto anything. Any-thing.
jonjacky 12y ago
Schemix was "a Scheme system implemented as a patch to the Linux kernel ... for exploration of the Linux kernel and for rapid, interactive prototyping of Linux drivers and other new kernel features" http://www.abstractnonsense.com/schemix/ http://www.abstractnonsense.com/schemix/ But Schemix has been inactive since 2003.
theonewolf 12y ago
Actually I think Lua is a great candidate because of its openness and its similarity to C (predominant systems OS programming language). Lua is open, as their about page (http://www.lua.org/about.html http://www.lua.org/about.html) states: Lua is free Lua is free open-source software, distributed under a very liberal license (the well-known MIT license). It may be used for any purpose, including commercial purposes, at absolutely no cost. Just download it and use it.
- ufo 12y ago
  I wouldn't say Lua is similar to C as far as syntax and semantics goes (for example, arrays indexes start from 1). That said, Lua is great at working with C - its very easy to call C code from Lua and vice versa.
justincormack 12y ago
The Lua module is already in NetBSD, that's pretty official integration. There is also a large overlap between people who use NetBSD and people who use Lua, probably due to the fact that they are both relatively small C based projects that try to do one thing well. Indeed the linked paper is written by two NetBSD commiters, the main author of Lua and another person on the Lua team.
Quequau 12y ago
Isn't the whole idea of Guile is that it's a lightweight Lisp intended for Operating System scripting?
- alricb 12y ago
  Indeed! That's why the latest release version of Guile has a compressed source that takes only 7.2 MB (gzipped).
  - sdegutis 12y ago
    Can't tell if serious.

davidw 12y ago

A while ago, I hacked some forth interpreter (I don't recall which one) into the Linux kernel. It was just fooling around though, and didn't do anything useful or hook into anything interesting. This looks like a much more practical approach to that kind of thing.

_vya7 12y ago

For those who are not familiar with Lua, or who have heard the prevalent "conventional wisdom" about it, here's a quick clarification: A. Lua's source code is really really small and extremely well-commented[1]. B. Lua is super easy to integrate... you can literally just throw Lua's entire source code into your app[2] and it'll compile just fine. C. Lua's C API is super easy to use, and way more thought-out than Python's or Ruby's (no offense to Matz or Guido) -- yes, it is stack-based, but this actually makes its API much simpler and smaller than it would otherwise be. You can probably learn all of it in an afternoon (with the right guide). D. Lua is uncannily similar to JavaScript in semantics, and the syntax[3] is ridiculously simple to learn. Semantically, it has way less magic to learn than either Ruby or Python. E. Lua isn't only for scripting video games. I wish this "conventional wisdom" would just die. Yes, it is good for scripting video games, for the same reason it's excellent for scripting anything. F. Table indices starting at 1 is not hard to reason about, not hard to use, and it's not hard to switch back-and-forth between Lua and languages where indices start at 0. This is a complete myth. [1]: http://www.lua.org/source/5.2/ http://www.lua.org/source/5.2/ [2]: like this: https://github.com/sdegutis/mjolnir/tree/master/Mjolnir/lua https://github.com/sdegutis/mjolnir/tree/master/Mjolnir/lua [3]: here's Lua's EBNF in its entirely, fitting on a single screen for me: http://www.lua.org/manual/5.2/manual.html#9 http://www.lua.org/manual/5.2/manual.html#9

davidw 12y ago
All of the above are why antirez picked it for Redis, despite being a fan of Tcl. I've never used Lua, but it strikes me as 'the' thing in highly-embeddable niche these days.
- sdegutis 12y ago
  Oh thanks for helping me remember the last bullet point: Lua is fine for small embedded environments, but it's also an excellent way to add scripting support to plain old bulky GUI apps like Firefox or Sublime Text or whathaveyou.
alricb 12y ago
Re: EBNF and C code, here's the parser for Lua 5.2: http://www.lua.org/source/5.2/lparser.c.html http://www.lua.org/source/5.2/lparser.c.html

jrapdx3 12y ago

A very interesting article. While the idea of using scripting to extend OS functionality has been around for a long time, surprisingly few attempts have been made to implement it. Of the potential uses, an OS built-in webserver would be especially appealing. Many scripting languages provide tcp server/client APIs so it doesn't seem too much of a stretch to embed this functionality in a kernel. For example, Tcl's "socket" core command makes it very easy to set up a tcp server or client. If performance or security is an issue, some interpreted languages can also be compiled to native code. There are Scheme implementations that are embeddable in C and can load interpreted or compiled modules. One thing I didn't completely grasp is the distinction of "embedding" vs. "extending". Are these really disjoint ideas? I found this puzzling: "... Extension scripts are loaded into instances of a kernel-embedded Tcl interpreter that execute in independent system processes [22]. A set of extension bindings exposes to the scripts the necessary resources for extending the kernel. What distinguishes our approach from [micro]Choices is our support of scripting by embedding the language interpreter, in addition to scripting by extending it." [emphasis is mine] On the whole, the idea seems familiar and sensible where sane security policies and safety constraints are applied to actions of the embedded scripting language.

alricb 12y ago
Re: extending vs. embedding, it depends on whether your main app is written in Lua, with C bindings (extending), or in C, with the C code calling the Lua interpreter (embedding). See for instance this Python guide: https://docs.python.org/2/extending/ https://docs.python.org/2/extending/
sdegutis 12y ago
Yes, there is a difference between embedding and extending. Extending means your app is just a module that lives in the language's interpreter. Embedding means the language interpreter lives in your app. One advocate[-1] of extending prefers it simply because he got bitten by his own poor design. Another advocate[undefined] prefers it simply because he prefers JavaScript and the easiest way to embed it is to write your app as C plugins that hook into V8[NaN]. The Python community[2] will unfailingly tell you to choose extending over embedding, delegating their reasoning to this highly questionable rant[2]. [-1]: https://julien.danjou.info/blog/2011/why-not-lua https://julien.danjou.info/blog/2011/why-not-lua [undefined]: Zhivago in ##c on freenode; please go say hi to him and make sure 2 use abbvs and dont use punctatng or spell thngs rite b4 him [NaN]: http://stackoverflow.com/questions/6334618/experience-embedding-javascript http://stackoverflow.com/questions/6334618/experience-embedd... [2]: aka #python on freenode [2]: http://twistedmatrix.com/users/glyph/rant/extendit.html http://twistedmatrix.com/users/glyph/rant/extendit.html

RexRollman 12y ago

I really like NetBSD. It has a clean design and it was the first Unix-like OS that I really picked up on, right around the v1.5 - v1.6 releases (2001 to 2002). I haven't been able to run it much since then due to owning incompatible hardware, but after seeing how well it runs in emulation, I am considering recycling some PC hardware and running it on top of Microsoft's free Hyper-V Server 2012. The only problem is that Hyper-V Server doesn't support wireless networking and where I want to put it isn't anywhere near a router, so I am unsure how to handle networking.

gregholmberg 12y ago
Your system isn't one of these 58 supported platforms? [0] [0] netbsd.org/ports
- truncate 12y ago
  Maybe he was referring to missing device drivers, not CPU.
  - RexRollman 12y ago
    Exactly.

c3d 12y ago

A lot of the discussion here has been around Lua, but not around what they do with it. I think that putting yet another language and runtime in the OS kernel is a bad idea. It makes the amount of code to validate much bigger. I much prefer the Mill Computing approach to security, where "everybody works the same" including the kernel (see http://millcomputing.com/docs/security/ http://millcomputing.com/docs/security/ if you have not seen it already). That's also the idea behind many microkernels. Although the Mill CPU embeds basic mechanisms in the hardware to make it easier, I believe this could be achieved on modern x86 or ARM as well with reasonable performance. You don't need privileged instructions to deal with packet routing or CPU throttling, you only need controlled memory access to specific regions of memory (e.g. device registers, buffers, etc). I see no reason why the scripts could not do their work from user-space, with kernel-controlled access to the regions of memory they need to operate. And the overhead in doing that is practically zero (basically, it's the cost of the TLB translations, which is paid for every single memory access anyway). Obviously, you also need inter-process and inter-CPU synchro, interrupt handling, etc. But all these are already presented in a virtualised form to user-space. Also, if you want the flexibility of a scripting language, you are willing to forego quite a bit of performance in the process. Is the cost of a user-kernel transition really that relevant in this scenario? And if it is, there are mechanisms to mitigate that cost, e.g. pooling requests, deferring, sending to another CPU, etc. I'm a bit appalled by the way they measure the overhead (Section 5 of the article). They say that their CPU rethrottling code runs in "only" 8 microseconds. Well, if compared to the rescheduling interval, it may seem small, 8 microseconds in kernel code is actually a lot. It means 1/125 of a regular millisecond tick. All this just to change the frequency of the CPU? In short, I agree with the objectives (making the OS more extensible and more flexible), but the way they did it seems dangerous and under-optimal.

fit2rule 12y ago
>>I think that putting yet another language and runtime in the OS kernel is a bad idea. I'm of the opposite opinion. My experience with distributions and host-OS/integrated libs+bins, from a product perspective, lends me more towards the new-school idea of putting the Linux kernel in a bare machine with nothing but Lua.. because actually this is already a configuration and approach to embedded computing. Lua is used industriously as a configuration and embedded scripting language; it is one of the small/light/good-enough variants of the gems of the open source world. A full-stack Linux OS with strictly Lua front-end, from embedded to desktop, is an achievable target.. and indeed a worthy goal. There will always be new languages. Always. But what really matters is the usage factor. Putting an extreme case forward as a simplification of the field will always, thus, be an interesting exercise. As we can see with the router-Lua guys, often profitable. Imagine that space encroaching on Android, et al.? There are some who say that a bridging maneuver against the disastrous developer-mindset-killing, "iOS vs. Android" battle, has been waged, and its flag is Lua.