6 ms·

Let's Write an LLVM Specializer for Python

287 points by tenslisi 12y ago

walkamages 12y ago
An excellent article! I had wanted to get back into some python recently after seeing the changes in 3.4, I had also wanted to become more familiar with LLVM, and this does both.
- gtirloni 12y ago
  there is this discussion (flame war?) about python3 not bringing too many benefits. i haven't made my mind yet. could you elaborate what you saw in 3.4 that was nice?
  - gamesbrainiac 12y ago
    I'd like to know too. Are there any LLVM specific enhancements that python 3.4 brings to the table?
    - exDM69 12y ago
      No. AFAIK, there's nothing related to LLVM in core Python. And not in 3.4 changes either.
  - exDM69 12y ago
    Simply put: it's a better language. The whole "discussion" is whether it makes sense to migrate since many parts of the ecosystem (many important libraries and frameworks) have not made the transition. And many distros ship Python 2 by default, Python 3 is optional. Python 3 only is not feasible. To me, the killer feature is better lazy evaluation (generators). In particular, important builtins like map, filter, zip, enumerate, etc are generators, instead of returning lists. This makes it feasible to write things like (process(line) for line in map(str.upper, open('giantfile.txt')) if line.lstrip()[0] != '#') Some of the above can also be done with itertools package in Python 2, but not everything. Python 3.4 changelog is here, it contains e.g. asynchronous io facilities (asyncio module): https://www.python.org/downloads/release/python-342/ https://www.python.org/downloads/release/python-342/ edit: added enumerate() in the example above, for line in open(filename) returns a generator in Python 2.x too. edit2: enumerate is lazy in python2, I replaced it with map(str.upper)
    - halflings 12y ago
      The example you gave works perfectly in Python 2.7 (would also be a generator, and you're not using map filter or else); but I agree: those should've been generators from day 1, especially zip and enumerate since they make more elegant code but often come with a performance overhead in Python 2.7
      - exDM69 12y ago
        Ah, you're absolutely right. for line in open(filename) returns a generator. I added map(str.upper, xxx) there to make it more it make sense. But the point should be obvious, without generators that would potentially consume a lot of memory.
      - danohuiginn 12y ago
        Turning them into generators would have broken a lot of existing code, though, so it's reasonable to leave them until a major version change. Rather, these are the kind of small, obviously-useful changes that should have come immediately in 3.0, giving people some encouragement to switch.
        exDM69 12y ago
        Most of the breaking changes (including making map, zip, filter, etc lazy) were done in Python 3.0.
      - maxerickson 12y ago
        Python 2 enumerate returns an 'enumerate' object that is more or less a light weight wrapper of the sequence that was passed in. Generators provide a convenient syntax to implement that sort of object.
        exDM69 12y ago
        D'oh. I put in a map(), that returns a list.
    - danohuiginn 12y ago
  - mkesper 12y ago
    Saner handling of Unicode, for example.
    - chc 12y ago
      This alone is pretty wonderful. I've only been working in Python a few months and the number of issues I've had to debug in 2.7 that came down to Unicode handling is kind of nuts.
  - baq 12y ago
    much better exceptions and sane unicode are the biggest improvements.
  - ak217 12y ago
    Without listing any of the modules or improvements that are in the standard library in 3 but backported as PyPI modules to 2 (of which there are many), here are the features that I actually use in Python 3: unicode handling that isn't insane, function annotations, async improvements, exception chaining, enums, single-dispatch generics, better SSL support, generator delegation, better int-bytes conversion support, unittest module improvements. The key point is that 2.7 is a language frozen in time, while 3.4+ is continuing to develop and improve. And most of the hand-wringing was before the critical mass of third-party modules was ported to 3.x. https://docs.python.org/3/whatsnew/3.4.html https://docs.python.org/3/whatsnew/3.4.html https://docs.python.org/3/whatsnew/3.3.html https://docs.python.org/3/whatsnew/3.3.html https://docs.python.org/3/whatsnew/3.2.html https://docs.python.org/3/whatsnew/3.2.html https://docs.python.org/3/whatsnew/3.1.html https://docs.python.org/3/whatsnew/3.1.html https://docs.python.org/3/whatsnew/3.0.html https://docs.python.org/3/whatsnew/3.0.html
  - ngoldbaum 12y ago
    My favorite new feature is PEP-442 [0]. Basically, it's now safe to add a __del__ method to a class without worrying about memory leaks caused by reference cycles. [0] https://www.python.org/dev/peps/pep-0442/ https://www.python.org/dev/peps/pep-0442/
    - pekk 12y ago
      Most people should not be writing __del__ methods at all, especially if what they are trying to do is deterministic cleanup.
  - bkeroack 12y ago
    Add to the list: - function annotations (allows runtime type checking via third party modules) - asyncio (not as easy to use as Go's goroutines but still vastly superior to the multiprocessing module)
  - andreasvc 12y ago
    I can't say I disagree with such sentiments, but over time I've ran into a few issues (features and performance improvements) which where only addressed in Python 3. This is reason enough to try working with Python 3.
illumen 12y ago
Very nice article! :) Storing types via traces could be another step for gathering types. As well as using the more advanced static type checking code that is around for python. Now I have something to work through on the weekend. Looking forward to part 2!
- travisoliphant 12y ago
  The numba code-base implements quite a bit of this. We actually moved away from the AST approach and went back to the byte-code approach because the AST approach quickly becomes unwieldy as the number of Visitors that you apply grows. Compile times are also slower. The author is definitely helping people learn about LLVM and how it can be used with Python --- which is great, because this is exactly what Numba is: http://numba.pydata.org http://numba.pydata.org. But, please don't start another "Numba". Just come help us improve the current one.
Erwin 12y ago
Interesting. I currently use Python's AST to convert some nested logical query expression (in a syntax unique to my application) into bytecode executed by a specialized VM (I originally tried using V8 and LuaJit for this but performance wise that was unsuccessful; the project replaced some old Boost::Python C++ code). This article should make it easy to get started attempting an LLVM replacement.
- travisoliphant 12y ago
  Yes, LLVM is a great approach for doing code-gen from arbitrary specialized VMs. And the Python interface to it makes it easy to experiment. We no longer use llvmpy for Numba (we use a simpler interface llvmlite) and so llvmpy could use a maintainer.
deleted 12y ago
[deleted]
jonstewart 12y ago
I really appreciate the length and detail in this blog post. It's comprehensive, not just showing off.
- ch0wn 12y ago
  Stephen Diehl continues to blow my mind on a regular basis. His latest work in progress "Wrote You a Haskell"[0] is also worth keeping an eye on. I've worked through the first couple of chapters and it's fantastic. [0] http://dev.stephendiehl.com/fun/ http://dev.stephendiehl.com/fun/
travisoliphant 12y ago
This is a great tutorial about first-generation Numba. The author learned a lot about LLVM and llvmpy while working with several of our devs. If you are interested in the "Further work" in his article, come join the Numba project.
- tadlan 12y ago
  What is second generation numba? And any plans to branch numba out of pure numeric application s?
wedesoft 12y ago
I did something similar in Ruby but using GCC as a "JIT" compiler for image processing (software [1], thesis [2]). I can really recommend JIT compilation for doing array processing. [1] http://www.wedesoft.de/hornetseye-api/ http://www.wedesoft.de/hornetseye-api/ [2] http://www.wedesoft.de/downloads/thesis_wedekind.pdf http://www.wedesoft.de/downloads/thesis_wedekind.pdf EDIT: In my approach I didn't go through the Ruby AST though. Rather I used the approach of injecting "GCCVariables" which emit C code instead of doing the actual computation.
- chrisseaton 12y ago
  You should submit your thesis to the Ruby Bibliography http://rubybib.org http://rubybib.org
  - wedesoft 12y ago
    Ok, will do. Cheers :)
cyberneticcook 12y ago
Could this be used to ahead-of-time compile Python code ? I'm more interested in getting to a native executable or library.
ericfrederich 12y ago
I love Python but when I use a statically typed language like C, Rust, Go, etc I really feel that it is missing from Python. I'd love to see a new language exactly like Python but compiled and statically typed. Something similar to Cython, but rather than generating a bunch of C code it would target LLVM. Additionally it would be able to generate pure Python code simply by removing any typing syntax.

287 points by tenslisi 12y ago

An excellent article! I had wanted to get back into some python recently after seeing the changes in 3.4, I had also wanted to become more familiar with LLVM, and this does both.

gtirloni 12y ago
there is this discussion (flame war?) about python3 not bringing too many benefits. i haven't made my mind yet. could you elaborate what you saw in 3.4 that was nice?
- gamesbrainiac 12y ago
  I'd like to know too. Are there any LLVM specific enhancements that python 3.4 brings to the table?
  - exDM69 12y ago
    No. AFAIK, there's nothing related to LLVM in core Python. And not in 3.4 changes either.
- exDM69 12y ago
  Simply put: it's a better language. The whole "discussion" is whether it makes sense to migrate since many parts of the ecosystem (many important libraries and frameworks) have not made the transition. And many distros ship Python 2 by default, Python 3 is optional. Python 3 only is not feasible. To me, the killer feature is better lazy evaluation (generators). In particular, important builtins like map, filter, zip, enumerate, etc are generators, instead of returning lists. This makes it feasible to write things like (process(line) for line in map(str.upper, open('giantfile.txt')) if line.lstrip()[0] != '#') Some of the above can also be done with itertools package in Python 2, but not everything. Python 3.4 changelog is here, it contains e.g. asynchronous io facilities (asyncio module): https://www.python.org/downloads/release/python-342/ https://www.python.org/downloads/release/python-342/ edit: added enumerate() in the example above, for line in open(filename) returns a generator in Python 2.x too. edit2: enumerate is lazy in python2, I replaced it with map(str.upper)
  - halflings 12y ago
    The example you gave works perfectly in Python 2.7 (would also be a generator, and you're not using map filter or else); but I agree: those should've been generators from day 1, especially zip and enumerate since they make more elegant code but often come with a performance overhead in Python 2.7
    - exDM69 12y ago
      Ah, you're absolutely right. for line in open(filename) returns a generator. I added map(str.upper, xxx) there to make it more it make sense. But the point should be obvious, without generators that would potentially consume a lot of memory.
    - danohuiginn 12y ago
      Turning them into generators would have broken a lot of existing code, though, so it's reasonable to leave them until a major version change. Rather, these are the kind of small, obviously-useful changes that should have come immediately in 3.0, giving people some encouragement to switch.
      - exDM69 12y ago
        Most of the breaking changes (including making map, zip, filter, etc lazy) were done in Python 3.0.
    - maxerickson 12y ago
      Python 2 enumerate returns an 'enumerate' object that is more or less a light weight wrapper of the sequence that was passed in. Generators provide a convenient syntax to implement that sort of object.
      - exDM69 12y ago
        D'oh. I put in a map(), that returns a list.
  - danohuiginn 12y ago
- mkesper 12y ago
  Saner handling of Unicode, for example.
  - chc 12y ago
    This alone is pretty wonderful. I've only been working in Python a few months and the number of issues I've had to debug in 2.7 that came down to Unicode handling is kind of nuts.
- baq 12y ago
  much better exceptions and sane unicode are the biggest improvements.
- ak217 12y ago
  Without listing any of the modules or improvements that are in the standard library in 3 but backported as PyPI modules to 2 (of which there are many), here are the features that I actually use in Python 3: unicode handling that isn't insane, function annotations, async improvements, exception chaining, enums, single-dispatch generics, better SSL support, generator delegation, better int-bytes conversion support, unittest module improvements. The key point is that 2.7 is a language frozen in time, while 3.4+ is continuing to develop and improve. And most of the hand-wringing was before the critical mass of third-party modules was ported to 3.x. https://docs.python.org/3/whatsnew/3.4.html https://docs.python.org/3/whatsnew/3.4.html https://docs.python.org/3/whatsnew/3.3.html https://docs.python.org/3/whatsnew/3.3.html https://docs.python.org/3/whatsnew/3.2.html https://docs.python.org/3/whatsnew/3.2.html https://docs.python.org/3/whatsnew/3.1.html https://docs.python.org/3/whatsnew/3.1.html https://docs.python.org/3/whatsnew/3.0.html https://docs.python.org/3/whatsnew/3.0.html
- ngoldbaum 12y ago
  My favorite new feature is PEP-442 [0]. Basically, it's now safe to add a __del__ method to a class without worrying about memory leaks caused by reference cycles. [0] https://www.python.org/dev/peps/pep-0442/ https://www.python.org/dev/peps/pep-0442/
  - pekk 12y ago
    Most people should not be writing __del__ methods at all, especially if what they are trying to do is deterministic cleanup.
- bkeroack 12y ago
  Add to the list: - function annotations (allows runtime type checking via third party modules) - asyncio (not as easy to use as Go's goroutines but still vastly superior to the multiprocessing module)
- andreasvc 12y ago
  I can't say I disagree with such sentiments, but over time I've ran into a few issues (features and performance improvements) which where only addressed in Python 3. This is reason enough to try working with Python 3.

illumen 12y ago

Very nice article! :) Storing types via traces could be another step for gathering types. As well as using the more advanced static type checking code that is around for python. Now I have something to work through on the weekend. Looking forward to part 2!

travisoliphant 12y ago
The numba code-base implements quite a bit of this. We actually moved away from the AST approach and went back to the byte-code approach because the AST approach quickly becomes unwieldy as the number of Visitors that you apply grows. Compile times are also slower. The author is definitely helping people learn about LLVM and how it can be used with Python --- which is great, because this is exactly what Numba is: http://numba.pydata.org http://numba.pydata.org. But, please don't start another "Numba". Just come help us improve the current one.

Erwin 12y ago

Interesting. I currently use Python's AST to convert some nested logical query expression (in a syntax unique to my application) into bytecode executed by a specialized VM (I originally tried using V8 and LuaJit for this but performance wise that was unsuccessful; the project replaced some old Boost::Python C++ code). This article should make it easy to get started attempting an LLVM replacement.

travisoliphant 12y ago
Yes, LLVM is a great approach for doing code-gen from arbitrary specialized VMs. And the Python interface to it makes it easy to experiment. We no longer use llvmpy for Numba (we use a simpler interface llvmlite) and so llvmpy could use a maintainer.

deleted 12y ago

[deleted]

jonstewart 12y ago

I really appreciate the length and detail in this blog post. It's comprehensive, not just showing off.

ch0wn 12y ago
Stephen Diehl continues to blow my mind on a regular basis. His latest work in progress "Wrote You a Haskell"[0] is also worth keeping an eye on. I've worked through the first couple of chapters and it's fantastic. [0] http://dev.stephendiehl.com/fun/ http://dev.stephendiehl.com/fun/

travisoliphant 12y ago

This is a great tutorial about first-generation Numba. The author learned a lot about LLVM and llvmpy while working with several of our devs. If you are interested in the "Further work" in his article, come join the Numba project.

tadlan 12y ago
What is second generation numba? And any plans to branch numba out of pure numeric application s?

wedesoft 12y ago

I did something similar in Ruby but using GCC as a "JIT" compiler for image processing (software [1], thesis [2]). I can really recommend JIT compilation for doing array processing. [1] http://www.wedesoft.de/hornetseye-api/ http://www.wedesoft.de/hornetseye-api/ [2] http://www.wedesoft.de/downloads/thesis_wedekind.pdf http://www.wedesoft.de/downloads/thesis_wedekind.pdf EDIT: In my approach I didn't go through the Ruby AST though. Rather I used the approach of injecting "GCCVariables" which emit C code instead of doing the actual computation.

chrisseaton 12y ago
You should submit your thesis to the Ruby Bibliography http://rubybib.org http://rubybib.org
- wedesoft 12y ago
  Ok, will do. Cheers :)

cyberneticcook 12y ago

Could this be used to ahead-of-time compile Python code ? I'm more interested in getting to a native executable or library.

ericfrederich 12y ago

I love Python but when I use a statically typed language like C, Rust, Go, etc I really feel that it is missing from Python. I'd love to see a new language exactly like Python but compiled and statically typed. Something similar to Cython, but rather than generating a bunch of C code it would target LLVM. Additionally it would be able to generate pure Python code simply by removing any typing syntax.