14 ms·

PeachPy: Assembly Code Generation in High-Level Python

116 points by shagunsodhani 11y ago

dang 11y ago
Url changed from http://lambda-the-ultimate.org/node/5251 http://lambda-the-ultimate.org/node/5251, which points to this.
- shagunsodhani 11y ago
  Thanks!
struct 11y ago
I remember coming across ARM's VIXL (https://github.com/armvixl/vixl https://github.com/armvixl/vixl), which is a similar concept but for C++, and just being blown away by how easy it was to twiddle the stack and assemble new functions on the fly - yee haw!
lukego 11y ago
Cool! I have been doing this kind of programming with dynasm [*] and finding it addictive. I wonder if assembler is going to make a come back :). https://github.com/luapower/dynasm/blob/master/dynasm.md https://github.com/luapower/dynasm/blob/master/dynasm.md
- csl 11y ago
  I wonder if assembler is going to make a come back :). I'm certainly seeing a lot of cool projects that make assembly more accessible, not to mention a lot of good guides on the web. As for myself, I've recently made Python ctypes bindings for GNU Lightning, which lets you write architecture agnostic RISC assembly, then JIT to native code on x86, ARM, etc. See https://github.com/cslarsen/lyn/ https://github.com/cslarsen/lyn/ It's pretty light-weight and JITs fast. The point is that it's very fun and addictive to do assembly from a language like Python.
  - dr_zoidberg 11y ago
    It's pretty light-weight and JITs fast. The point is that it's very fun and addictive to do assembly from a language like Python. Because you skip the boring boilerplate, and go straight to the fun parts!
    - lfowles 11y ago
      My biggest problem with assembly was coming up with names for all of the labels :) One of the hardest problems of CS... naming! How does PeachPY handle loops? I'm looking for examples, but will it generate a label for me and then let me specify the contents only? Edit: https://github.com/Maratyszcza/PeachPy/blob/master/examples/make/matmul-opt.py https://github.com/Maratyszcza/PeachPy/blob/master/examples/... ! (Add a bit more, give it some static typing and soon we'll be coding C in Python)
      - pbsd 11y ago
        That's how you unroll a loop, not how you write one. As far as I can tell in PeachPy you can do something similar with anonymous labels: counter = GeneralPurposeRegister64() MOV(counter, 64) with Loop() as loop: ... DEC(counter) JNZ(loop.begin)
        lfowles 11y ago
        Thanks for the clarification, that's still a decent improvement.
  - nickpsecurity 11y ago
    I proposed a while back to replace inline assembler with inline LLVM bytecode. Do the portable code thing in general except accelerate the algorithms with the bytecode. Toolchain produces assembly from that. Probably still have real assembly where it's needed for hardware manipulation, etc but I'd think you'd get quite the performance/optimization improvement with that over C. What you think about LLVM as cross-platform assembly?
- _RPM 11y ago
  > I wonder if assembler is going to make a come back :) Are you trolling? Assembler hasn't gone anywhere nor will it ever.
- agumonkey 11y ago
  A typed assembly would be in order.
  - nickpsecurity 11y ago
    Go get, use and improve them: http://www.cs.cornell.edu/talc/ http://www.cs.cornell.edu/talc/ http://research.microsoft.com/en-us/um/people/nick/coqasm.pdf http://research.microsoft.com/en-us/um/people/nick/coqasm.pd...
- nickpsecurity 11y ago
  It likely won't for mainstream but it's always been popular in niche areas. Here's a few projects using it and pages on it: KolibriOS http://kolibrios.org/en/ http://kolibrios.org/en/ Anything by 2ton and w/ their library https://2ton.com.au/Products/ https://2ton.com.au/Products/ Gibson's guide on Win32 Assembly w/ links https://www.grc.com/smgassembly.htm https://www.grc.com/smgassembly.htm Hyde's AoA and High-level Assembly are cool http://www.plantation-productions.com/Webster/ http://www.plantation-productions.com/Webster/ Or for the elite: 8-bit GUI desktop http://www.symbos.de/ http://www.symbos.de/
  - lfowles 11y ago
    There was a bit of hate for Hyde's HLA last time I posted a link. I'm really not sure why, theoretically it's a great intermediate between C and ASM.
    - nickpsecurity 11y ago
      Same thing happened to me when I posted. The gist of it is they thought you weren't learning real ASM thinking that way. I thought the point was to temporarily ignore more difficult parts of ASM to make learning incremental and easier. And then be able to do same as real assembly coder later w/ ability to make tradeoffs between raw performance, productivity, and easy-to-read code. Overall, a great tool in theory. Also, gave me ideas on how to bootstrap a subversion-resistant system. You start with assembler, build a HLA-like tool from that, easily build a 3GL + straight-to-HLA compiler with that, and then use that to build rest (including optimizations). My concept was inspired by Wirth and Jurg's bitslice -> M-code ASM -> Modula 2 -> OS work in Lilith project. Alternatively, you can go MenuetOS or KolibriOS w/ HLA for above benefits. Use full HLA for rapid addition/testing of new features with raw ASM for optimizing the fast paths. Thoughts on either?
baldfat 11y ago
In the early-mid 1980s there were several projects claiming this high level language to Assembly Code or P-Code (Machine not Microsoft's abandoned p-code). They never really worked out right and due to weird optimizations could be just as fast or 5 times slower.
rmcpherson 11y ago
Great tool. It supports assembly generation for the golang Plan 9 toolchain as well. The author, Marat Dukhan, gave a great talk on this last night at the Atlanta Go Meetup. Slides are at https://docs.google.com/presentation/d/1MYg8PyhEf0oIvZ9YU2panNkVXsKt5UQBl_vGEaCeB1k/edit#slide=id.p https://docs.google.com/presentation/d/1MYg8PyhEf0oIvZ9YU2pa...
ant6n 11y ago
I've been looking for a way to assemble ARM instruction by instruction in python; unfortunately this only seems to do x86.
- ycnews 11y ago
  A brief look at the source reveals ARM support also.
  - Marat_Dukhan 11y ago
    This is a re-write of an older version of PeachPy (you will find it at https://bitbucket.org/MDukhan/peachpy https://bitbucket.org/MDukhan/peachpy), which supported x86-64 and ARM, but didn't generate object files directly. The new PeachPy currently only supports x86-64, albeit there are remnants of old code for ARM.
    - ant6n 11y ago
      I didn't get the example in section "PeachPy as Instruction Encoder" to work for arm on the linked github version, the version currently on pypi, or the older bitbucket version. If the module loads at all, it seems to always complain about lacking function context.
      - Marat_Dukhan 11y ago
        Probably I didn't make it clear enough: - The old version of PeachPy (on BitBucket and PyPI) supports ARM, but can't directly generate machine code. - The new version of PeachPy (on GitHub) can directly encode instructions into machine code, but currently only supports x86-64. The files in peachpy.arm module are merely non-working remnants of the old code.
rthomas6 11y ago
How is an assembly code generator different than a compiler?
- z92 11y ago
  Assembly code then has to be compiled into machine code to create the executable.
  - lfowles 11y ago
    To expand a bit in terrible pseudocode that I sort of remember from Atmel ucs: Assembly: mov 0x1,reg2 # put 1 in register 2 Machine Code: 0xAC12 # Move an immediate value into register 2 Assembly: mov *reg2,reg1 # put contents of address in reg2 in reg1 Machine Code: 0xBF201 # Move an indirect value, offset 0, address in reg2 to reg1 It's a much more straightforward transformation than the one from C->ASM though. Edit: This link might make more sense... the difference is mainly between the mnemonic and the encoding: http://www.mrc.uidaho.edu/mrc/people/jff/digital/MIPSir.html http://www.mrc.uidaho.edu/mrc/people/jff/digital/MIPSir.html You'll also notice there are explicit instructions for immediate ops, but no two assembly schemes seem to share the same level of abstraction.
- Marat_Dukhan 11y ago
  Earlier versions of PeachPy generated assembly source that had to be compiled with NASM or GNU assembler. Now PeachPy can directly generate object files, and it would be fair to call it a compiler, but I like the old name
nickpsecurity 11y ago
This is pretty cool stuff. You get the RAD advantage of Python, probably easy to extend w/ new features, customization of real assembly, and then the assembling part itself. Nice tool. Assembly hasn't been this interesting since Microsoft started coding it in a proof assistant [1]. I like this better as Python is a bit more accessible. ;) [1] http://research.microsoft.com/en-us/um/people/nick/coqasm.pdf http://research.microsoft.com/en-us/um/people/nick/coqasm.pd...
milin 11y ago
Debugging would be a nightmare when you have 300+ line function. https://github.com/Maratyszcza/PeachPy/blob/master/codegen/x86_64.py#L213 https://github.com/Maratyszcza/PeachPy/blob/master/codegen/x...

116 points by shagunsodhani 11y ago

dang 11y ago

Url changed from http://lambda-the-ultimate.org/node/5251 http://lambda-the-ultimate.org/node/5251, which points to this.

shagunsodhani 11y ago
Thanks!

struct 11y ago

I remember coming across ARM's VIXL (https://github.com/armvixl/vixl https://github.com/armvixl/vixl), which is a similar concept but for C++, and just being blown away by how easy it was to twiddle the stack and assemble new functions on the fly - yee haw!

lukego 11y ago

Cool! I have been doing this kind of programming with dynasm [*] and finding it addictive. I wonder if assembler is going to make a come back :). https://github.com/luapower/dynasm/blob/master/dynasm.md https://github.com/luapower/dynasm/blob/master/dynasm.md

csl 11y ago
I wonder if assembler is going to make a come back :). I'm certainly seeing a lot of cool projects that make assembly more accessible, not to mention a lot of good guides on the web. As for myself, I've recently made Python ctypes bindings for GNU Lightning, which lets you write architecture agnostic RISC assembly, then JIT to native code on x86, ARM, etc. See https://github.com/cslarsen/lyn/ https://github.com/cslarsen/lyn/ It's pretty light-weight and JITs fast. The point is that it's very fun and addictive to do assembly from a language like Python.
- dr_zoidberg 11y ago
  It's pretty light-weight and JITs fast. The point is that it's very fun and addictive to do assembly from a language like Python. Because you skip the boring boilerplate, and go straight to the fun parts!
  - lfowles 11y ago
    My biggest problem with assembly was coming up with names for all of the labels :) One of the hardest problems of CS... naming! How does PeachPY handle loops? I'm looking for examples, but will it generate a label for me and then let me specify the contents only? Edit: https://github.com/Maratyszcza/PeachPy/blob/master/examples/make/matmul-opt.py https://github.com/Maratyszcza/PeachPy/blob/master/examples/... ! (Add a bit more, give it some static typing and soon we'll be coding C in Python)
    - pbsd 11y ago
      That's how you unroll a loop, not how you write one. As far as I can tell in PeachPy you can do something similar with anonymous labels: counter = GeneralPurposeRegister64() MOV(counter, 64) with Loop() as loop: ... DEC(counter) JNZ(loop.begin)
      - lfowles 11y ago
        Thanks for the clarification, that's still a decent improvement.
- nickpsecurity 11y ago
  I proposed a while back to replace inline assembler with inline LLVM bytecode. Do the portable code thing in general except accelerate the algorithms with the bytecode. Toolchain produces assembly from that. Probably still have real assembly where it's needed for hardware manipulation, etc but I'd think you'd get quite the performance/optimization improvement with that over C. What you think about LLVM as cross-platform assembly?
_RPM 11y ago
> I wonder if assembler is going to make a come back :) Are you trolling? Assembler hasn't gone anywhere nor will it ever.
agumonkey 11y ago
A typed assembly would be in order.
- nickpsecurity 11y ago
  Go get, use and improve them: http://www.cs.cornell.edu/talc/ http://www.cs.cornell.edu/talc/ http://research.microsoft.com/en-us/um/people/nick/coqasm.pdf http://research.microsoft.com/en-us/um/people/nick/coqasm.pd...
nickpsecurity 11y ago
It likely won't for mainstream but it's always been popular in niche areas. Here's a few projects using it and pages on it: KolibriOS http://kolibrios.org/en/ http://kolibrios.org/en/ Anything by 2ton and w/ their library https://2ton.com.au/Products/ https://2ton.com.au/Products/ Gibson's guide on Win32 Assembly w/ links https://www.grc.com/smgassembly.htm https://www.grc.com/smgassembly.htm Hyde's AoA and High-level Assembly are cool http://www.plantation-productions.com/Webster/ http://www.plantation-productions.com/Webster/ Or for the elite: 8-bit GUI desktop http://www.symbos.de/ http://www.symbos.de/
- lfowles 11y ago
  There was a bit of hate for Hyde's HLA last time I posted a link. I'm really not sure why, theoretically it's a great intermediate between C and ASM.
  - nickpsecurity 11y ago
    Same thing happened to me when I posted. The gist of it is they thought you weren't learning real ASM thinking that way. I thought the point was to temporarily ignore more difficult parts of ASM to make learning incremental and easier. And then be able to do same as real assembly coder later w/ ability to make tradeoffs between raw performance, productivity, and easy-to-read code. Overall, a great tool in theory. Also, gave me ideas on how to bootstrap a subversion-resistant system. You start with assembler, build a HLA-like tool from that, easily build a 3GL + straight-to-HLA compiler with that, and then use that to build rest (including optimizations). My concept was inspired by Wirth and Jurg's bitslice -> M-code ASM -> Modula 2 -> OS work in Lilith project. Alternatively, you can go MenuetOS or KolibriOS w/ HLA for above benefits. Use full HLA for rapid addition/testing of new features with raw ASM for optimizing the fast paths. Thoughts on either?

baldfat 11y ago

In the early-mid 1980s there were several projects claiming this high level language to Assembly Code or P-Code (Machine not Microsoft's abandoned p-code). They never really worked out right and due to weird optimizations could be just as fast or 5 times slower.

rmcpherson 11y ago

Great tool. It supports assembly generation for the golang Plan 9 toolchain as well. The author, Marat Dukhan, gave a great talk on this last night at the Atlanta Go Meetup. Slides are at https://docs.google.com/presentation/d/1MYg8PyhEf0oIvZ9YU2panNkVXsKt5UQBl_vGEaCeB1k/edit#slide=id.p https://docs.google.com/presentation/d/1MYg8PyhEf0oIvZ9YU2pa...

ant6n 11y ago

I've been looking for a way to assemble ARM instruction by instruction in python; unfortunately this only seems to do x86.

ycnews 11y ago
A brief look at the source reveals ARM support also.
- Marat_Dukhan 11y ago
  This is a re-write of an older version of PeachPy (you will find it at https://bitbucket.org/MDukhan/peachpy https://bitbucket.org/MDukhan/peachpy), which supported x86-64 and ARM, but didn't generate object files directly. The new PeachPy currently only supports x86-64, albeit there are remnants of old code for ARM.
  - ant6n 11y ago
    I didn't get the example in section "PeachPy as Instruction Encoder" to work for arm on the linked github version, the version currently on pypi, or the older bitbucket version. If the module loads at all, it seems to always complain about lacking function context.
    - Marat_Dukhan 11y ago
      Probably I didn't make it clear enough: - The old version of PeachPy (on BitBucket and PyPI) supports ARM, but can't directly generate machine code. - The new version of PeachPy (on GitHub) can directly encode instructions into machine code, but currently only supports x86-64. The files in peachpy.arm module are merely non-working remnants of the old code.

rthomas6 11y ago

How is an assembly code generator different than a compiler?

z92 11y ago
Assembly code then has to be compiled into machine code to create the executable.
- lfowles 11y ago
  To expand a bit in terrible pseudocode that I sort of remember from Atmel ucs: Assembly: mov 0x1,reg2 # put 1 in register 2 Machine Code: 0xAC12 # Move an immediate value into register 2 Assembly: mov *reg2,reg1 # put contents of address in reg2 in reg1 Machine Code: 0xBF201 # Move an indirect value, offset 0, address in reg2 to reg1 It's a much more straightforward transformation than the one from C->ASM though. Edit: This link might make more sense... the difference is mainly between the mnemonic and the encoding: http://www.mrc.uidaho.edu/mrc/people/jff/digital/MIPSir.html http://www.mrc.uidaho.edu/mrc/people/jff/digital/MIPSir.html You'll also notice there are explicit instructions for immediate ops, but no two assembly schemes seem to share the same level of abstraction.
Marat_Dukhan 11y ago
Earlier versions of PeachPy generated assembly source that had to be compiled with NASM or GNU assembler. Now PeachPy can directly generate object files, and it would be fair to call it a compiler, but I like the old name

nickpsecurity 11y ago

This is pretty cool stuff. You get the RAD advantage of Python, probably easy to extend w/ new features, customization of real assembly, and then the assembling part itself. Nice tool. Assembly hasn't been this interesting since Microsoft started coding it in a proof assistant [1]. I like this better as Python is a bit more accessible. ;) [1] http://research.microsoft.com/en-us/um/people/nick/coqasm.pdf http://research.microsoft.com/en-us/um/people/nick/coqasm.pd...

milin 11y ago

Debugging would be a nightmare when you have 300+ line function. https://github.com/Maratyszcza/PeachPy/blob/master/codegen/x86_64.py#L213 https://github.com/Maratyszcza/PeachPy/blob/master/codegen/x...