8 ms·
Lua support in the NetBSD kernel
- lifeisstillgood 13y agoHoly crapola - they are putting a Lua interpreter in the kernel, and it has access to some limited calls. This is partly driven by "we need device drivers and no one understands C anymore", partly because it's cool, but mostly because Moores law is still alive and well. They are putting a scripting language in a kernel ! Forget write your app in python / perl / Ruby then optimise in C - the raw power argument is going to overwhelm us all. I heard a stat the other day - that a greetings card, the kind that plays a silly tune, has more processing power than all computers on the planet in 1960. And we should expect a similar growth in the next 50 years. Even if that's out by two orders of magnitude just let it sink in. I think we are at a Cambrian explosion period - where the goal is to try out every possible new body shape, as fast as possible and see which ones get the Darwin seal of approval. New organisation forms are possible, some for the first time in human history, new ways of thinking and communicating - it's stuff like this that makes one realise the water around the frog is getting hotter. (Still it's worth remembering that most train companies in 19C England failed and the average return for stockholders was 10% - just because the world will change beyond recognition does not mean industry stocks is a great return)
- recuter 13y agoLua can be sandboxed quite effectively, can you elaborate on your concerns?
- vor_ 13y agoWhat concerns?
- scott_karana 13y agoGP was probably in response to ketralnis' comment. :)
- ketralnis 13y agorecuter's comment already existed when I wrote mine, so it couldn't be in response to me
- robert-wallis 13y agosprintf buffer overflow in the string library https://github.com/LuaDist/lua/commit/52eff16f51750cf47afaa5ac27a8705a1c580859#diff-5fa4a598e8e43f285d6fb9add1a47182R890 https://github.com/LuaDist/lua/commit/52eff16f51750cf47afaa5... This library is not included in NetBSD as far as I can tell: http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/modules/lua/?only_with_tag=MAIN http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/modules/lua/?onl... But it's an indication that it shouldn't be included in a Kernel.
- tedunangst 13y agoYou linked to a 24,000 line diff to point out what exactly?
- robert-wallis 13y agowait for it... the exact section of the code that contains the "sprintf" I mentioned will pop into your screen. Just be patient.
- Sanddancer 13y agoThere were buffer overflows in the Linux kernel in 2012 ( http://www.securityfocus.com/bid/53401 http://www.securityfocus.com/bid/53401 ). Should we not have filesystems in the kernel either?
- pdpi 13y agoWell... No, if you're a proponent of the micro-kernel design.
- robert-wallis 13y agoCertainly the string library, of all places, should be using secure code. I'm sure there are sooo many people running Linux on pre-OSX Mac harddrives formatted with HFS that could be hacked with a buffer overflow. Are you seriously arguing that buffer overflows in the kernel are not a big deal?
- tonyplee 13y agoHaven't they been keep saying that about JAVA - Just Another Vulnerability Announcement? Mac just turn off JAVA completely in user space and NetBSD is putting Lua in kernel? NSA: "I have a dream! Someday, a script language will be in BSD kernel, so I can ping an script into the kernel" :-)
- ketralnis 13y agoKernel programming is an eternal struggle between people trying to pull more into the kernel for performance, and people trying to pull stuff back out for safety and modularity. We've all seen the arguments about the in-kernel HTTP daemons and whatnot. I tend to be on the side that says that more things should be pulled out. > This is partly driven by "we need device drivers and no one understands C anymore" I don't buy this argument. Who are the people * Using NetBSD * having cutting-edge hardware for which drivers don't exist * deciding to write device drivers * that have device specs/docs needed to write them * that don't know/want to use C * that would do it if only they could write it in a scripting language? Do these people exist? I assert that the far more common case is that the device specs, documentation, BIOS, etc. aren't accessible. If that's true, they couldn't write the driver regardless of the language, C is hardly the limiting factor here
- gwu78 13y ago"Do these people exist?" This is spot on. When you hear NetBSD developers arguing about proposals (or commits, ex post facto, like here), there is often mention of "users". But these are mostly fictional. I think it's safe to say most NetBSD users are comfortable working with C. I reckon it's not much different with OpenBSD: most changes they make are for the benefit of the developers themselves, to suit personal preferences, than for the benefit of other "users". (And that's perfectly acceptable, I think.) Of course, these changes can benefit users beyond simply those who make them and those who approve them, but those users are not the ones driving the changes.
- groovy2shoes 13y agoThis is spot on. When you hear NetBSD developers arguing about proposals (or commits, ex post facto, like here), there is often mention of "users". But these are mostly fictional. I think this is true of much open source software. Just for the sake of example, a few years ago the Gnome devs removed some printer config options because they decided the "users" didn't need so many options. It's not that Gnome doesn't have users, but the developers seemed to have some notion of a type of user that didn't really exist. I think more often than not, fictional users are used as justification for the addition of some nice-to-have feature, but of course I'm just speculating.
- fafner 13y agoAre they using regular Lua? Because LuaJIT can actually be quite fast. Snabb Switch is written mostly in Lua and it features Linux userland ethernet device drivers. http://blog.lukego.com/blog/2013/01/03/snabb-switchs-luajit-ethernet-device-driver/ http://blog.lukego.com/blog/2013/01/03/snabb-switchs-luajit-... (edit: link to the code https://github.com/SnabbCo/snabbswitch/tree/master/src/apps/intel https://github.com/SnabbCo/snabbswitch/tree/master/src/apps/...) btw. there once was a project which added a Scheme interpreter to the Linux kernel. Sadly it's dead now: http://www.abstractnonsense.com/schemix/ http://www.abstractnonsense.com/schemix/
- maximilianburke 13y agoI'm not sure they would be well suited together, NetBSD has a huge list of old hardware they support; I don't think Mike Pall will be porting LuaJIT to m68k anytime soon.
- baldfat 13y agoMoore's law will be over by 2025 (We have 12 years or so). Semi-conductor will allow a separation of just 3 - 5 atoms to the conducting part. The belief is that quantum physics will allow the electrons to go through "walls" at that point and the switch will always be closed. I think we will go back to the days of 640k where there is a ceiling and we will just have to refine for "large" projects. Software and not hardware will be Moore's law BUT only for a extreme circumstances. The Golden Age is coming to an end in our life time.
- deleted 13y ago[deleted]
- vor_ 13y ago> Marc Balmer comes across as some sort of Lua zealot. Lua is a great little language but c'mon. He is like a broken record. Why does he want this so badly? Quoting bullet points from his FOSDEM 2013 slides: • Modifying software written in C is hard for users • Give users the power to modify and extend the system • Let users explore the system in an easy way • "Rapid Application Development" approach to driver/kernel development • Modifying the system behavior • Configuration of kernel subsystems
- pppppowerbook 13y agoThose are all terrible reasons to introduce complex code into the kernel. Making kernel development easy won't actually make it easy. The thing that makes it hard isn't C.
- synchronise 13y agoYou can compile NetBSD without kernel Lua support. Literally no reasons for complaint unless this breaks something, which it doesn't.
- voltagex_ 13y agoMaybe this argument is less useful for BSD users, but wouldn't most users be using the stock kernel? So it's up to the whims of the distributors whether to include it or not.
- Sanddancer 13y agoIt's pretty common in the BSD world to build your own kernel. The entire build system, imo, is much more streamlined and well thought out than linux, so relying on the distributor is not needed. Also, one of the missions of NetBSD is to get it working on everything, so as others have stated, being able to poke at the kernel in lua, both for developing drivers, and for pulling state out of the kernel for general debugging, is very useful.
- chipsy 13y agoSome discussion: http://mail-index.netbsd.org/tech-kern/2013/10/06/msg015632.html http://mail-index.netbsd.org/tech-kern/2013/10/06/msg015632....
- jarjoura 13y agoIsn't this similar to Windows wanting to rebuild their entire userland on top of .NET?
- tedunangst 13y agoI would say, no, not at all. Rewriting your entire userland is not similar to adding a script interpreter to be used for writing kernel extensions.
- pjmlp 13y agoActually the best example is OS/400 mainframe system from IBM. The kernel is a mixture of Assembly, C++ and Modula-2, with a JIT. All user space code compiles to bytecode. The kernel JIT compiles the applications on first execution. Afterwards the application is validated against generated native code, and re-JITed again if the bytecode has changed between executions. So Microsoft's attempt was actually nothing new.
- anttiok 13y agoIt's awesome to see this work finally hit the tree. Now, I cannot say I agree with all of the motivations. For example, I do not believe that prototyping should be done in kernel mode at all, nor do I believe that using Lua will shield from incorrect operation or crashes. However, there is already proof that device drivers can be written in Lua (https://github.com/SnabbCo/snabbswitch https://github.com/SnabbCo/snabbswitch), and it's great to see more push in that direction. After all, why use C if you have a working alternative? We just need real world experiments to see if Lua is a working alternative for the kernel.
- morenoh149 13y agoyay. I was just looking at learning lua to do corona development
- ape4 13y agoThe note about the copyright on the diffs seems a bit paranoid or something.
- almk277 13y agoIt's good somebody even tries to break the old myth about C as an essential kernel part. C is _not_ good for drivers: it lacks most of the appropriate abstractions, just as facilities to create them. Once we switch from asm to C - why not go further? Speed matters, of course, but I suppose that even script speed is enough for many drivers (and, just in case, we have luaJIT). Don't sure if lua is a right choice, but it's an interesting experiment - and this is what NetBSD is good for.
- simias 13y ago> C is _not_good for drivers You'll have to back up that assertion with some facts here, there's a whole industry and millions of lines of code that contradict you.
- almk277 13y agoIt's just legacy. C as infrastructure is good, C as a language is bad. There's a whole industry and millions of lines of code in visual basic, but I never met somebody who says that VB is good. Is there a way to say "do this under spinlock"? Is there a way to handle errors not mixing it with main logic? Not wrapping nearly every line with if/goto? Is there a way to create a generic collection? No, linux list.h is not that: it requires manual initialization, it does not work with elements, but with their special parts, it's macroses requires logically unneeded information (such as 'member' in list_for_each_entry), and so on. And this is the best you can do in C. C is too unobvious, too. Google for "wait4 backdoor". C works, of course. It just requires too much boilerplate code and too much programmer resources.
- pjmlp 13y agoC is designed for security exploits from day one. The industry of million lines of code exists, because UNIX and its system programming language, C, spread into the enterprise when the university students brought it from their labs into the enterprise. Thankfully the industry is slowly moving to other languages.
- simias 13y agoAnybody has a link that explains their motivations? There's a lot of speculation in this thread but surely NetBSD being an open community there must have been a public discussion before they merged that in? I write linux drivers for a living and I must say it puzzles be a bit. There are times C falls a bit short (if only by its syntax and idiosyncracies). There are times I wish I could use something like a subset of C++ (for templates and destructors mostly) or maybe Rust these days. But a scripting language? You can already write drivers in userland (using /dev/mem for instance). You can already write device drivers any way you like that way. One of the main limitations is IRQ handling, but do they propose handling IRQs in a scripting language? Maybe I'm too close minded (or just brainwashed by years of C) but it sounds like a recipe for madness. I actually used lua to write a small userland test driver (it was convenient because the lua interpreter is very small and easy to compile and cross-compile). It was quite painful really. There's no builtin bitwise operator (although it's now in the standard library), all integers are stored as double floating point values (floating point in the kernel is hardly ever needed and usually very poorly supported). Using integers and bitwise operator is like all you do in a kernel driver (ok, I'm exaggerating, but not by a lot). I guess they might have changed the syntax a little bit (and use ints instead of doubles for storing integers) but then it means their lua is not compatible with the "real" lua so you can't prototype your code in userland with any stock lua build. I dug up my userland test driver code, here's what the naughty bits look like: function avi_unset(addr, val) local v = assert(devmem.read32(addr)) v = bit32.band(v, bit32.bnot(val)) avi_write(addr, v) end For reference here's what it'd look like in C and its bitwise operators: void avi_unset(unsigned addr, unsigned val) { unsigned v = __raw_readl(addr); v &= ~val; __raw_writel(addr, v); }
- pjmlp 13y agoSymbian, BeOS, Windows and Mac OS X have C++ support for kernel drivers, just as info.
- adamnemecek 13y agoDoes anyone still write drivers for BeOS?