8 ms·
Porting Python to GRUB: “Networking without an operating system”
- sandGorgon 10y agowhy would they not use lua ? which is pretty popular (and pretty much designed) as embedded scripting languages. Not to mention much more lightweight than python.. and has a much smaller footprint. Nginx is a pretty good example of this.
- chungy 10y agoGRUB already has Lua, so I imagine the answer is "because we can"
- sandGorgon 10y agointeresting. I never knew that... i wonder if any additional socket library dependencies were problematic in the lua world.
- Bromskloss 10y agoWhat is it that makes the language suitable for being embedded? Actually, what does it mean to be embedded?
- agumonkey 10y agoLua is extremely small, simpler, and has a jitted version. It's been used extensively as a scripting language in video games so has more mileage in constrained environment. Never used it really, just stating what other people say. And luajit author, Mike Pall, is extremely skilled.
- imtringued 10y agoYou can launch a lua vm with an empty environment table (in other words no standard library). This means only the functions you manually put into the environment table can be used which makes sandboxing extremely easy. I must admit I've only used luaj where interop to java boils down to something like http://stackoverflow.com/a/19629914 http://stackoverflow.com/a/19629914 and you can optionally use reflection for even easier intertop if I recall correctly.
- walkingolof 10y agoLua as mentioned before is very suited to run in an embedded environment because of it small size (both when executing and the actual source code size) and its modules, which makes it trivial to remove everything except the core language (IO support etc). I'm running Lua on a DLX like processor implemented on a fpga, in this context Lua performs the same role as Basic did on many home computers in the 80's, a small IPL is setting up the machine and loading Lua on boot.
- JoshTriplett 10y ago> why would they not use lua ? The target audience for this is BIOS developers and firmware engineers. Several existing development tools in that area already support Python, including Simics and the Intel ITP, so the target audience is used to it. Lua also has several other impedance mismatches with the target audience, such as 1-based array indexing (remember, the target audience writes assembly and C). While people often describe Lua as "easy to embed", personally, I don't find Lua's stack-based C API easy at all; in particular, the references to stack-based indexes that change when you push or pop the stack feels like working with stack-based parameters in assembly without a base pointer, where the offset of a given location can change at any time. I much prefer the Python C API. Finally, Lua would require additional libraries to provide a useful environment, precisely because it can assume so little about its environment. For instance, we'd need to bundle one of the Lua socket libraries in addition to Lua itself, as well as an HTTP library. Python's "batteries included" standard library works nicely here.
- sandGorgon 10y agothank you for that. That explains a lot ! But Lua is already in Grub right ? wouldnt it have been more effective to just extend that anyway ?
- JoshTriplett 10y agoLua isn't part of upstream GRUB; it lives in a separate module, "grub-extras". As far as I can tell, it also wouldn't be easy to extend that with additional functionality or modules without forking it, due to the way GRUB builds out-of-tree modules. Note that this port of Python is almost as old as grub's Lua; we've been working on BITS since ~2009, and it has had Python support since May 2011. For reference, the Python port consists of less than 1000 lines of code, most of which provides compatibility implementations of standard C and POSIX functions that Python expects. (Plus another couple thousand lines implementing additional C Python modules like "_smp", "_acpi", and "_efi", but those are getting smaller as we move more code into Python using ctypes.)
- __b__ 10y ago"... the target audience writes assembly and C)." In C, I make frequent use of "switch" statements. Does CPython have a switch statment?
- detaro 10y agoSpeculation: At least for the given Intel use case, lightweight doesn't sound to important: they can use the entire machine. And given the fairly specialist application, porting a language experts and users are familiar with (or maybe was used in existing tooling) might be better.
- SixSigma 10y agoWhy would they not use C? Or forth, or any number of alternatives. You would get the same question with the names shuffled around whichever one you chose.
- coredog64 10y agoIf you want forth in your BIOS, buy a Sun workstation. /snark:off
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- JoshTriplett 10y agoI'm one of the authors of BITS, and the presenter of the PyCon presentation this article discusses. Happy to answer any questions here.
- ThePhysicist 10y agoDid you modify the CPython binary to run in the BIOS, or did you write a new interpreter from scratch? What was the hardest part of implementing this?
- JoshTriplett 10y agoWe build CPython's source into BITS, which runs as an EFI binary. (Or as a non-EFI image bootable on a BIOS, but that version doesn't include network support.) We have almost no changes to CPython itself, other than a couple of minor fixes to enable some additional calling conventions in the ctypes module. The hardest part of porting Python in general was porting libffi to support the EFI calling convention, which required some very fiddly assembly. That libffi support allows us to call EFI protocols from Python without writing any C code. The hardest part of implementing networking was handling the impedance mismatch between POSIX select and EFI asynchronous calls. POSIX select lets you ask "do any of these sockets have data to read", without actually reading the data; likewise for pending client connections to accept(). EFI expects you to provide a buffer to read the data and a callback for when the data gets read. So, the only way to ask for pending data is to buffer that data for the application to read later. This requires additional copies, limiting performance. (You might have heard of "zero-copy" networking; this is "several-copy" networking.)
- d0mine 10y agoIt sounds like IOCP on Windows. For example, asyncio.ProactorEventLoop and multiprocessing's PipeConnection can use it.
- niftich 10y agoI apologize in advance for my lack of detailed knowledge about low-level IO, but, Would a library like libuv/pyuv that provides an async interface to networking and file IO help in this regard? I understand that your goals were to implement as much of the Python stdlib's APIs so that programs can run unmodified, but could you envision Python applications that are coded against pyuv running on a platform that bridges pyuv calls directly to EFI without the stdlib as an intermediary?
- TempleOSV409 10y agoTempleOS is God's official temple. It will be burned into all Intel chips in the Intel factory in a ROM. God says... still's clubbing temblors blenders personnel's upheaval boost ascend Medan CompuServe Tutankhamen log's mangrove weirdo coroner's homeliest transcending wefts dissent's inspector's Grammy meteorites purrs jargon's Billie's trapped lads coverall's tuckers shrilled Churchill auditions
- vanderZwan 10y agoWas the talk recorded and can we expect it on the Pycon 2016 youtube channel[0] at some point? It's not there right now as far as I can tell. [0] https://www.youtube.com/c/pycon2016/videos https://www.youtube.com/c/pycon2016/videos
- JoshTriplett 10y agoYes: https://www.youtube.com/watch?v=AlkKvetGFSk https://www.youtube.com/watch?v=AlkKvetGFSk The first few minutes of the talk got cut off; the conference organizers have the original footage and plan to re-cut and re-upload it at some point. The talk from PyCon 2015 on the Python port itself is available as well: https://www.youtube.com/watch?v=bYQ_lq5dcvM https://www.youtube.com/watch?v=bYQ_lq5dcvM
- vanderZwan 10y agoHuh, I loaded all the way back to Pycon 2015 videos CTRL+F "Josh" or "Triplett" didn't find anything on my laptop. Strange Anyway, thanks for the links!
- brian_herman 10y agoCould you make a python rumpkernel with this?
- denfromufa 10y agoHow is this different from micropython?
- detaro 10y agoMicropython is a independent implementation of Python with some microcontroller-specific extensions and able to run on smallish microcontrollers, this is a slightly modified "standard" CPython für x86.
- kbradero 10y agois BITS building for ARM ?
- JoshTriplett 10y agoNo. We don't have any need for that ourselves, but if someone wanted to work on that, and produced a clean patch that wasn't filled with #ifdefs in C and ifs in Python, we'd take it. The biggest challenge would be adding equivalents of the assembly that implements SMP support (waking up other CPUs and putting them into a loop ready to run code on request).