5 ms·

Embedding PyPy in a C application

97 points by thedjinn 10y ago

kensai 10y ago
"With only a few lines of code we have written a C program that fires up a PyPy interpreter and runs our Python code as if it was a native C function. Of course I've only shown you the basics here, but the technology is really powerful." So, C speed with Python semantics?
- guipsp 10y ago
  Scripting apis are a possible use too.
  - thedjinn 10y ago
    Yes indeed, and using this approach you can also write Python plugins for other applications if they don't have Python scripting support already.
    - noahdesu 10y ago
      In the article, after the pypy plugin.py step that produced the C library, one could presumably dlopen the library to accomplish a dynamic plugin feature. Is there a way to invoke pypy to JIT this code to avoid invoking pypy and a compiler?
      - thedjinn 10y ago
        The way I understand it is that on first invocation of a function exported in the library this function will spin up a regular PyPy interpreter, which does JIT in exactly the same way as a normal invocation of PyPy would.
- scandinavian 10y ago
  "So, C speed with Python semantics?" That's already what PyPy is, isn't it? (Not really C speed, but the method in the article is just calling the PyPy interpreter, so it's not quite C speed anyway.)
jamesdutc 10y ago
There's interesting overlap between tools like `cffi` and `cython`. This is one of the approaches I am pursuing to polish off something I put together recently: rapid prototyping of LD_PRELOADs in Python. I presented an early approach at PyData London last week using LD_AUDIT and `cython`: https://www.youtube.com/watch?v=L3j2qw9XZCc https://www.youtube.com/watch?v=L3j2qw9XZCc It doesn't seem that amazing to be able to write: @preload def open(pathname: 'const char*', flags: 'int') -> 'int': return __open__(pathname, flags) After all, this is only slightly more convenient syntax than in C. However, imagine how much functionality you could quickly develop with access to the Python standard library. Even using my clumsy `cython` and `LD_AUDIT` approach, I was able to rapidly develop libraries that interpose file-handling library calls and implement: # `less` is used as an example of a libc programme with no concept of Python, git, &c. $ COMMIT_HASH=abc123 gitit less test # interact with files as-of some commit in a git repo via pygit (libgit2) $ runit less test.py # redirect read() so that the contents of the file appears to be the results emitted of having run the file $ zipit less test.gz # redirect read()/write() to transparently handle gzip files Each one of these took about half an hour to throw together, which is substantially faster than I could reliably develop them in pure-C (not to mention how much code I avoid having to write myself by relying on Python's rich stdlib & ecosystem.)
- bartbes 10y ago
  That was a really interesting talk, thanks for sharing!
azag0 10y ago
I used cffi recently to embed CPython into a fairly rigid Fortran program to support dynamic scripting. Was a piece of cake.
- mattip 10y ago
  Can you share the code?
solidsnack9000 10y ago
Does PyPy embedding get around the GIL?
- BuckRogers 10y ago
  No, but if you embed PyPy-STM[0] you get around the GIL. [0]http://doc.pypy.org/en/latest/stm.html http://doc.pypy.org/en/latest/stm.html
curiousgal 10y ago
Noob question, isn't Python already based on C?
- timClicks 10y ago
  The reference Python interpreter is implemented in C, but that doesn't mean that Python equals C. Implementing programming language interpreters is a bigger topic than a HN topic... but in general when things are easier from the programmer's perspective, then the machine needs to do significantly more work to get things right. This is especially true in Python, where 1 + 1 doesn't necessarily mean "add one to one" because the integer literals (1) and the addition operator (+) can be overloaded to mean different things. Even at runtime, it's possible to have the same written code and different semantics. C's semantics are much closer to how the hardware operates. In order to code effectively and efficiently in C, you need to understand much more nitty gritty details about how computers operate.
  - azag0 10y ago
    I know what you wanted to say, but you didn't give the best example. "1 + 1 == 2" will be always true in any Python code since all those symbols are either operators or literals. I believe you cannot change dynamically how the interpreter interprets operators (that is, "+" always calls __add__ and __radd__) or literals.
- azag0 10y ago
  There are other implementation of the Python interpreter than the reference C implementation, notably in RPython (subset of Python), which is called PyPy, Java (Jython) or Cython (this compiles Python statically to C). One could say that Numba is also a very restricted Python interpreter.

97 points by thedjinn 10y ago

"With only a few lines of code we have written a C program that fires up a PyPy interpreter and runs our Python code as if it was a native C function. Of course I've only shown you the basics here, but the technology is really powerful." So, C speed with Python semantics?

guipsp 10y ago
Scripting apis are a possible use too.
- thedjinn 10y ago
  Yes indeed, and using this approach you can also write Python plugins for other applications if they don't have Python scripting support already.
  - noahdesu 10y ago
    In the article, after the pypy plugin.py step that produced the C library, one could presumably dlopen the library to accomplish a dynamic plugin feature. Is there a way to invoke pypy to JIT this code to avoid invoking pypy and a compiler?
    - thedjinn 10y ago
      The way I understand it is that on first invocation of a function exported in the library this function will spin up a regular PyPy interpreter, which does JIT in exactly the same way as a normal invocation of PyPy would.
scandinavian 10y ago
"So, C speed with Python semantics?" That's already what PyPy is, isn't it? (Not really C speed, but the method in the article is just calling the PyPy interpreter, so it's not quite C speed anyway.)

jamesdutc 10y ago

There's interesting overlap between tools like `cffi` and `cython`. This is one of the approaches I am pursuing to polish off something I put together recently: rapid prototyping of LD_PRELOADs in Python. I presented an early approach at PyData London last week using LD_AUDIT and `cython`: https://www.youtube.com/watch?v=L3j2qw9XZCc https://www.youtube.com/watch?v=L3j2qw9XZCc It doesn't seem that amazing to be able to write: @preload def open(pathname: 'const char*', flags: 'int') -> 'int': return __open__(pathname, flags) After all, this is only slightly more convenient syntax than in C. However, imagine how much functionality you could quickly develop with access to the Python standard library. Even using my clumsy `cython` and `LD_AUDIT` approach, I was able to rapidly develop libraries that interpose file-handling library calls and implement: # `less` is used as an example of a libc programme with no concept of Python, git, &c. $ COMMIT_HASH=abc123 gitit less test # interact with files as-of some commit in a git repo via pygit (libgit2) $ runit less test.py # redirect read() so that the contents of the file appears to be the results emitted of having run the file $ zipit less test.gz # redirect read()/write() to transparently handle gzip files Each one of these took about half an hour to throw together, which is substantially faster than I could reliably develop them in pure-C (not to mention how much code I avoid having to write myself by relying on Python's rich stdlib & ecosystem.)

bartbes 10y ago
That was a really interesting talk, thanks for sharing!

azag0 10y ago

I used cffi recently to embed CPython into a fairly rigid Fortran program to support dynamic scripting. Was a piece of cake.

mattip 10y ago
Can you share the code?

solidsnack9000 10y ago

Does PyPy embedding get around the GIL?

BuckRogers 10y ago
No, but if you embed PyPy-STM[0] you get around the GIL. [0]http://doc.pypy.org/en/latest/stm.html http://doc.pypy.org/en/latest/stm.html

curiousgal 10y ago

Noob question, isn't Python already based on C?

timClicks 10y ago
The reference Python interpreter is implemented in C, but that doesn't mean that Python equals C. Implementing programming language interpreters is a bigger topic than a HN topic... but in general when things are easier from the programmer's perspective, then the machine needs to do significantly more work to get things right. This is especially true in Python, where 1 + 1 doesn't necessarily mean "add one to one" because the integer literals (1) and the addition operator (+) can be overloaded to mean different things. Even at runtime, it's possible to have the same written code and different semantics. C's semantics are much closer to how the hardware operates. In order to code effectively and efficiently in C, you need to understand much more nitty gritty details about how computers operate.
- azag0 10y ago
  I know what you wanted to say, but you didn't give the best example. "1 + 1 == 2" will be always true in any Python code since all those symbols are either operators or literals. I believe you cannot change dynamically how the interpreter interprets operators (that is, "+" always calls __add__ and __radd__) or literals.
azag0 10y ago
There are other implementation of the Python interpreter than the reference C implementation, notably in RPython (subset of Python), which is called PyPy, Java (Jython) or Cython (this compiles Python statically to C). One could say that Numba is also a very restricted Python interpreter.