5 ms·

A Lua interpreter in the NetBSD kernel

93 points by gioele 15y ago

udp 15y ago
Is this still being worked on? That page was last updated in 2010.
- iamabhi9 15y ago
  That was a Google Summer of Code Project. The code is now part of the NetBSD kernel and being maintained in the official repository. A NetBSD developer gave a talk on this recently at a conference: http://people.freebsd.org/~pgj/bsd_day(2011)/materials/04_mbalmer_kernel_lua.pdf http://people.freebsd.org/~pgj/bsd_day(2011)/materials/04_mb...
- bch 15y ago
  I can't speak to lunatik, but there is active work bringing Lua into the NetBSD userland[1] (NetBSD -current, which will be tagged NetBSD-6 when it's ready). The notion is that interfaces will be build up around it, allowing scripted access/control of anything from POSIX calls to userland application access (dhcp, gpio[2], for example). [1] http://blog.netbsd.org/tnf/entry/the_lua_scripting_language_in [2] http://cvsweb.netbsd.org/bsdweb.cgi/src/share/examples/lua/?only_with_tag=MAIN
  - joe_the_user 15y ago
    After reading your links, I'm still unclear what the advantage of having lua within the kernel is. Remember, this is different than simply using Lua for installation tasks. The Kernel should expose all needed routines to user-level programs anyway.
    - bch 15y ago
      I agree with you; lunatik is related to the other work in that they're both on NetBSD, and I think Marc Balmer is driving the userspace work, and at least familiar w/ the kernel-space work, so there's cross-pollination of talent. Honestly, I think I would rather see the userspace work out of core (ie: not shipping w/ NetBSD) and just put into pkgsrc (NetBSD version of Ports, or dpkg, etc) as a NetBSD-only package. The kernel work -- I'm just not familiar with deep kernel hacking, so perhaps it's my ignorance, but an interp in-kernel doesn't sound attractive to me. I think I'd rather see kernel access pushed to userspace, like FUSE/PUFFS[1] does for filesystems, or even how procfs allows reading/writing kernel from userspace. Jamming more into the kernel, especially an interpretter makes me shake my head. It could be fun as a personal project, or a Clever Hack, but for a shipping production-ready system... reminds me of when RedHat introduced it's in-kernel http server[2]. All that said -- NetBSD is _still_ the most attractive Unix I'm aware of. I occasionally consider a Linux or FreeBSD, but haven't left yet, since NetBSD 1.6 (c. 2002) [1] http://www.netbsd.org/docs/puffs/ http://www.netbsd.org/docs/puffs/ [2] http://en.wikipedia.org/wiki/TUX_web_server http://en.wikipedia.org/wiki/TUX_web_server
davidw 15y ago
A long while ago I hacked some small forth implementation into the Linux kernel. It didn't really do anything useful though, and was a pretty simplistic implementation. It was kind of fun though.
- agumonkey 15y ago
  Vaguely related, ALSA comes with a lisp interpreter.
  - gioele 15y ago
    The user-space library alsa-lib contains a quite complete lisp interpreter, not the kernel module.
  - kbradero 15y ago
    can you elaborate please ?
toyg 15y ago
Isn't this sort of approach a security nightmare ?
- nonUser 15y ago
  Guess why it's not mainline ;)
- gregholmberg 15y ago
  On a system without mandatory access controls [0], absolutely yes. This is what powers some rootkits. [1] Having a dynamic interpreted language runtime available as just another item in the syscall table [2] would allow arbitrary, unrestricted execution of code that would not even respect resource limitations set in the root user's shell. [0] http://en.wikipedia.org/wiki/Mandatory_access_control http://en.wikipedia.org/wiki/Mandatory_access_control [1] http://en.wikipedia.org/wiki/Rootkit#Kernel-mode http://en.wikipedia.org/wiki/Rootkit#Kernel-mode [2] http://fxr.watson.org/fxr/source/kern/syscalls.master http://fxr.watson.org/fxr/source/kern/syscalls.master
- chalst 15y ago
  Not if it's well-architected. Lua's sandboxes (using setf) allow pretty tight isolation of code. http://lua-users.org/wiki/SandBoxes http://lua-users.org/wiki/SandBoxes There's a slide titled security in Marc Balmer's talk, but I didn't get where he was headed.
- joe_the_user 15y ago
  More to the point, what value can be got from having an interpreter in the kernel? Having the interpreter always available in user space seems more logical.
  - veyron 15y ago
    I think the point is to replace the specialized languages used in frameworks like systemtap and dtruss/dtrace
  - gregholmberg 15y ago
    What value could Lua bring to a Unix kernel? You could do some fast and very sophisticated manipulation of network traffic streams without requiring that each packet be copied to and from user-space. You wouldn't even have to develop much packet-twiddling code. There is a surprisingly large amount of network-specific open source Lua. Wireshark, the most popular network protocol analyzer, has dissectors written in Lua. [0] Nmap, the network mapper and port scanner, has a Lua-based scripting engine to allow session and endpoint property discovery in great detail. [1] The popular intrusion detection tool Snort Security Platform (SnortSP) has included a Lua interpreter since version 3.0. [2] Yes, keeping the interpreter in user-space is safer. I have a hard time imagining sandboxing within the language runtime being able to deal with some of the nastier traffic. Perhaps one prudent thing to do with an in-kernel interpreter would be to use a project-specific sanitizer that could reliably escape tokens and keywords found in live traffic. [0] "Lua Support in Wireshark" http://www.wireshark.org/docs/wsug_html_chunked/wsluarm.html http://www.wireshark.org/docs/wsug_html_chunked/wsluarm.html [1] "Nmap Scripting Engine" http://nmap.org/book/nse.html http://nmap.org/book/nse.html [2] “A Pig(Snort), A Moon (Lua) and one very happy developer (Bill)” http://www.linuxsecurity.com/content/view/128135 http://www.linuxsecurity.com/content/view/128135
rwboyer 15y ago
I wish the resources were thrown at something a little more USEFUL in NetBSD as there are so few to go around. Maybe like ZFS or BTRFS support... I LOVE the BSD's especially NetBSD - what else can make some of my old sentimentally attached hardware work so well. I think I might be the only one running NetBSD as a DOM-0.
- jacques_chester 15y ago
  It's a volunteer organisation and folk are going to volunteer for what interests them, regardless of the stipend.
  - rwboyer 15y ago
    Yea - I know - just wishful thinking by a user/very occasional contributor to the bad minorities.
- Nick_C 15y ago
  > old sentimentally attached hardware work so well I know what you mean. I was running NetBSD on an iMac G4 (the sunflower one) for years and it was great (then the macppc NVidia broke for several months so I switched to OpenBSD). I use it as a photo server sitting on my bookcase. Without wishing to get into a flame war, once you try a BSD, the Linux stuff feels a little...raw.
mappu 15y ago
Very, very cool project, but... using Lua for the kernel's process scheduler? Lunatik is an appropriate name in some regards.

93 points by gioele 15y ago

udp 15y ago

Is this still being worked on? That page was last updated in 2010.

iamabhi9 15y ago
That was a Google Summer of Code Project. The code is now part of the NetBSD kernel and being maintained in the official repository. A NetBSD developer gave a talk on this recently at a conference: http://people.freebsd.org/~pgj/bsd_day(2011)/materials/04_mbalmer_kernel_lua.pdf http://people.freebsd.org/~pgj/bsd_day(2011)/materials/04_mb...
bch 15y ago
I can't speak to lunatik, but there is active work bringing Lua into the NetBSD userland[1] (NetBSD -current, which will be tagged NetBSD-6 when it's ready). The notion is that interfaces will be build up around it, allowing scripted access/control of anything from POSIX calls to userland application access (dhcp, gpio[2], for example). [1] http://blog.netbsd.org/tnf/entry/the_lua_scripting_language_in [2] http://cvsweb.netbsd.org/bsdweb.cgi/src/share/examples/lua/?only_with_tag=MAIN
- joe_the_user 15y ago
  After reading your links, I'm still unclear what the advantage of having lua within the kernel is. Remember, this is different than simply using Lua for installation tasks. The Kernel should expose all needed routines to user-level programs anyway.
  - bch 15y ago
    I agree with you; lunatik is related to the other work in that they're both on NetBSD, and I think Marc Balmer is driving the userspace work, and at least familiar w/ the kernel-space work, so there's cross-pollination of talent. Honestly, I think I would rather see the userspace work out of core (ie: not shipping w/ NetBSD) and just put into pkgsrc (NetBSD version of Ports, or dpkg, etc) as a NetBSD-only package. The kernel work -- I'm just not familiar with deep kernel hacking, so perhaps it's my ignorance, but an interp in-kernel doesn't sound attractive to me. I think I'd rather see kernel access pushed to userspace, like FUSE/PUFFS[1] does for filesystems, or even how procfs allows reading/writing kernel from userspace. Jamming more into the kernel, especially an interpretter makes me shake my head. It could be fun as a personal project, or a Clever Hack, but for a shipping production-ready system... reminds me of when RedHat introduced it's in-kernel http server[2]. All that said -- NetBSD is _still_ the most attractive Unix I'm aware of. I occasionally consider a Linux or FreeBSD, but haven't left yet, since NetBSD 1.6 (c. 2002) [1] http://www.netbsd.org/docs/puffs/ http://www.netbsd.org/docs/puffs/ [2] http://en.wikipedia.org/wiki/TUX_web_server http://en.wikipedia.org/wiki/TUX_web_server

davidw 15y ago

A long while ago I hacked some small forth implementation into the Linux kernel. It didn't really do anything useful though, and was a pretty simplistic implementation. It was kind of fun though.

agumonkey 15y ago
Vaguely related, ALSA comes with a lisp interpreter.
- gioele 15y ago
  The user-space library alsa-lib contains a quite complete lisp interpreter, not the kernel module.
- kbradero 15y ago
  can you elaborate please ?

toyg 15y ago

Isn't this sort of approach a security nightmare ?

nonUser 15y ago
Guess why it's not mainline ;)
gregholmberg 15y ago
On a system without mandatory access controls [0], absolutely yes. This is what powers some rootkits. [1] Having a dynamic interpreted language runtime available as just another item in the syscall table [2] would allow arbitrary, unrestricted execution of code that would not even respect resource limitations set in the root user's shell. [0] http://en.wikipedia.org/wiki/Mandatory_access_control http://en.wikipedia.org/wiki/Mandatory_access_control [1] http://en.wikipedia.org/wiki/Rootkit#Kernel-mode http://en.wikipedia.org/wiki/Rootkit#Kernel-mode [2] http://fxr.watson.org/fxr/source/kern/syscalls.master http://fxr.watson.org/fxr/source/kern/syscalls.master
chalst 15y ago
Not if it's well-architected. Lua's sandboxes (using setf) allow pretty tight isolation of code. http://lua-users.org/wiki/SandBoxes http://lua-users.org/wiki/SandBoxes There's a slide titled security in Marc Balmer's talk, but I didn't get where he was headed.
joe_the_user 15y ago
More to the point, what value can be got from having an interpreter in the kernel? Having the interpreter always available in user space seems more logical.
- veyron 15y ago
  I think the point is to replace the specialized languages used in frameworks like systemtap and dtruss/dtrace
- gregholmberg 15y ago
  What value could Lua bring to a Unix kernel? You could do some fast and very sophisticated manipulation of network traffic streams without requiring that each packet be copied to and from user-space. You wouldn't even have to develop much packet-twiddling code. There is a surprisingly large amount of network-specific open source Lua. Wireshark, the most popular network protocol analyzer, has dissectors written in Lua. [0] Nmap, the network mapper and port scanner, has a Lua-based scripting engine to allow session and endpoint property discovery in great detail. [1] The popular intrusion detection tool Snort Security Platform (SnortSP) has included a Lua interpreter since version 3.0. [2] Yes, keeping the interpreter in user-space is safer. I have a hard time imagining sandboxing within the language runtime being able to deal with some of the nastier traffic. Perhaps one prudent thing to do with an in-kernel interpreter would be to use a project-specific sanitizer that could reliably escape tokens and keywords found in live traffic. [0] "Lua Support in Wireshark" http://www.wireshark.org/docs/wsug_html_chunked/wsluarm.html http://www.wireshark.org/docs/wsug_html_chunked/wsluarm.html [1] "Nmap Scripting Engine" http://nmap.org/book/nse.html http://nmap.org/book/nse.html [2] “A Pig(Snort), A Moon (Lua) and one very happy developer (Bill)” http://www.linuxsecurity.com/content/view/128135 http://www.linuxsecurity.com/content/view/128135

rwboyer 15y ago

I wish the resources were thrown at something a little more USEFUL in NetBSD as there are so few to go around. Maybe like ZFS or BTRFS support... I LOVE the BSD's especially NetBSD - what else can make some of my old sentimentally attached hardware work so well. I think I might be the only one running NetBSD as a DOM-0.

jacques_chester 15y ago
It's a volunteer organisation and folk are going to volunteer for what interests them, regardless of the stipend.
- rwboyer 15y ago
  Yea - I know - just wishful thinking by a user/very occasional contributor to the bad minorities.
Nick_C 15y ago
> old sentimentally attached hardware work so well I know what you mean. I was running NetBSD on an iMac G4 (the sunflower one) for years and it was great (then the macppc NVidia broke for several months so I switched to OpenBSD). I use it as a photo server sitting on my bookcase. Without wishing to get into a flame war, once you try a BSD, the Linux stuff feels a little...raw.

mappu 15y ago

Very, very cool project, but... using Lua for the kernel's process scheduler? Lunatik is an appropriate name in some regards.