4 ms·

Did you modify the CPython binary to run in the BIOS, or did you write a new interpreter from scratch? What was the hardest part of implementing this?

JoshTriplett 10y ago
We build CPython's source into BITS, which runs as an EFI binary. (Or as a non-EFI image bootable on a BIOS, but that version doesn't include network support.) We have almost no changes to CPython itself, other than a couple of minor fixes to enable some additional calling conventions in the ctypes module. The hardest part of porting Python in general was porting libffi to support the EFI calling convention, which required some very fiddly assembly. That libffi support allows us to call EFI protocols from Python without writing any C code. The hardest part of implementing networking was handling the impedance mismatch between POSIX select and EFI asynchronous calls. POSIX select lets you ask "do any of these sockets have data to read", without actually reading the data; likewise for pending client connections to accept(). EFI expects you to provide a buffer to read the data and a callback for when the data gets read. So, the only way to ask for pending data is to buffer that data for the application to read later. This requires additional copies, limiting performance. (You might have heard of "zero-copy" networking; this is "several-copy" networking.)
- d0mine 10y ago
  It sounds like IOCP on Windows. For example, asyncio.ProactorEventLoop and multiprocessing's PipeConnection can use it.
- niftich 10y ago
  I apologize in advance for my lack of detailed knowledge about low-level IO, but, Would a library like libuv/pyuv that provides an async interface to networking and file IO help in this regard? I understand that your goals were to implement as much of the Python stdlib's APIs so that programs can run unmodified, but could you envision Python applications that are coded against pyuv running on a platform that bridges pyuv calls directly to EFI without the stdlib as an intermediary?
  - JoshTriplett 10y ago
    > Would a library like libuv/pyuv that provides an async interface to networking and file IO help in this regard? No, it wouldn't. libuv depends on OS facilities like select, poll, or epoll, and then builds a high-performance event loop on top of them. We needed to implement those facilities themselves. In theory, if we didn't care about Python's standard library modules, we could potentially port libuv to support the asynchronous callback mechanism used by EFI. However, we also wanted to write as much of this code as possible in Python, rather than C, to make it easier to maintain. That's why, for instance, we didn't implement the Berkeley sockets API in C to use the CPython socketmodule.c, but instead implemented our own _socket.py in Python directly on top of the efi module.

JoshTriplett 10y ago

We build CPython's source into BITS, which runs as an EFI binary. (Or as a non-EFI image bootable on a BIOS, but that version doesn't include network support.) We have almost no changes to CPython itself, other than a couple of minor fixes to enable some additional calling conventions in the ctypes module. The hardest part of porting Python in general was porting libffi to support the EFI calling convention, which required some very fiddly assembly. That libffi support allows us to call EFI protocols from Python without writing any C code. The hardest part of implementing networking was handling the impedance mismatch between POSIX select and EFI asynchronous calls. POSIX select lets you ask "do any of these sockets have data to read", without actually reading the data; likewise for pending client connections to accept(). EFI expects you to provide a buffer to read the data and a callback for when the data gets read. So, the only way to ask for pending data is to buffer that data for the application to read later. This requires additional copies, limiting performance. (You might have heard of "zero-copy" networking; this is "several-copy" networking.)

d0mine 10y ago
It sounds like IOCP on Windows. For example, asyncio.ProactorEventLoop and multiprocessing's PipeConnection can use it.
niftich 10y ago
I apologize in advance for my lack of detailed knowledge about low-level IO, but, Would a library like libuv/pyuv that provides an async interface to networking and file IO help in this regard? I understand that your goals were to implement as much of the Python stdlib's APIs so that programs can run unmodified, but could you envision Python applications that are coded against pyuv running on a platform that bridges pyuv calls directly to EFI without the stdlib as an intermediary?
- JoshTriplett 10y ago
  > Would a library like libuv/pyuv that provides an async interface to networking and file IO help in this regard? No, it wouldn't. libuv depends on OS facilities like select, poll, or epoll, and then builds a high-performance event loop on top of them. We needed to implement those facilities themselves. In theory, if we didn't care about Python's standard library modules, we could potentially port libuv to support the asynchronous callback mechanism used by EFI. However, we also wanted to write as much of this code as possible in Python, rather than C, to make it easier to maintain. That's why, for instance, we didn't implement the Berkeley sockets API in C to use the CPython socketmodule.c, but instead implemented our own _socket.py in Python directly on top of the efi module.