6 ms·

Codon: Python Compiler

83 points by joak 3y ago

debatem1 3y ago
Last I heard Codon was a python-like language rather than actual Python and the author suggests something similar. Headline seems pretty misleading.
- netr0ute 3y ago
  It was only technically python-like, but enough like it that you could conceivably call them the same thing.
  - deleted 3y ago
    [deleted]
  - joak 3y ago
    If it accepts a subset of python it's technically python: any codon-python program is a python program. Not full python but python nonetheless... My understanding: The goal is not to compile existing python code but to write new python code meant to be compiled in order to be faster. Limited use, still useful.
    - debatem1 3y ago
      It is trivially easy to write a codon program which runs in both Codon and Python but behaves differently in a way that is not considered a bug in either language. That means it is not a subset, unless by that you mean that each language has a subset of the other's programs which it can run. In which case C and JavaScript are the same language too.
      - netr0ute 3y ago
        Is there an example of this? I know there's several areas where Cpython has some funky undefined behavior, but those aren't particularly notable because they're by definition implemention dependent.
        selcuka 3y ago
        Isn't fixed width integers as compared to arbitrary width an example of this?
        cozzyd 3y ago
        print(repr(2**64))
        debatem1 3y ago
        The 2**64 example given here is exactly right. You don't need undefined behavior or language lawyering to hit the difference; you just need a 65 element bitset.
  - debatem1 3y ago
    Codon can't even run the whole python standard library. It changes integer semantic from arbitrary width to fixed width. Exaloop's own docs refer to the process of converting code between the two as "porting". Seems like if it isn't pythonic enough to run python code correctly it probably shouldn't be called "python".
voz_ 3y ago
How do you do, fellow python compilers?
- wudangmonk 3y ago
  So many compilers, it sorta reminds me of JS. Could it be that this language too has major flaws? if so why does everyone keep using it only to find out much later that they need to call out to C to do the heavy lifting only to eventually find out again that even that is not fast enough. So they need to compile all the rest of the python parts too. Seems like a rundabout way to use these lower level language without actually acknowledging that fact. An incredible amount of effort has been wasted making Python a viable language. At least the JS people have the excuse that there was no other option.
  - ModernMech 3y ago
    The sad state of affairs is that C is just that hard to use for most people, so the roundabout way is pretty much the only way they can manage.
    - jjnoakes 3y ago
      There's a ton of middle ground though. It just didn't catch on as much. Unfortunately, ecosystem is still king.
  - analog31 3y ago
    Sincere question: I wonder what fraction of Python users bump into these limitations often enough for it to be a problem. I do find Python to be more pleasant than C, though I use them in different domains -- desktop and embedded -- so it's not a direct comparison.
  - nextlevelwizard 3y ago
    I think major contributor is the "Python is slow" meme. Sure it is slow when compared to C or something, but most of the time the time you save by implementing in Python massively out weights any performance gains, but since everyone keep saying that "Python is slow" it pushes some people to go and try to make it fast. Good example of above is this line from you (highlight mine) >An incredible amount of effort has been wasted making Python a *viable* language. What does it even mean for a language to be "viable"? Is fast execution the only criteria? There are good reasons why we aren't writing all software in C or Assembly for maximal potential performance.
nighthawk454 3y ago
Repo for more details: https://github.com/exaloop/codon https://github.com/exaloop/codon > What is Codon? > Codon is a high-performance Python compiler that compiles Python code to native machine code without any runtime overhead. Typical speedups over Python are on the order of 10-100x or more, on a single thread. Codon's performance is typically on par with (and sometimes better than) that of C/C++. Unlike Python, Codon supports native multithreading, which can lead to speedups many times higher still. Codon grew out of the Seq project. > What isn't Codon? > While Codon supports nearly all of Python's syntax, it is not a drop-in replacement, and large codebases might require modifications to be run through the Codon compiler. For example, some of Python's modules are not yet implemented within Codon, and a few of Python's dynamic features are disallowed. The Codon compiler produces detailed error messages to help identify and resolve any incompatibilities. > Codon can be used within larger Python codebases via the @codon.jit decorator. Plain Python functions and libraries can also be called from within Codon via Python interoperability.
fernly 3y ago
Just for reference, * Nuitka[0] "is a Python compiler written in Python. It's fully compatible with Python 2.6, 2.7, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, and 3.11." * Pypy[1] "is a replacement for CPython" with builtin optimizations such as on the fly JIT compiles. * Cython[2] "is an optimising static compiler for both the Python programming language and the extended Cython programming language... makes writing C extensions for Python as easy as Python itself." * Numba[3] "is an open source JIT compiler that translates a subset of Python and NumPy code into fast machine code." * Pyston[4] "is a performance-optimizing JIT for Python, and is drop-in compatible with ... CPython 3.8.12" [0] https://github.com/Nuitka/Nuitka https://github.com/Nuitka/Nuitka [1] https://www.pypy.org/ https://www.pypy.org/ [2] https://cython.org/ https://cython.org/ [3] https://numba.pydata.org/ https://numba.pydata.org/ [4] https://github.com/pyston/pyston https://github.com/pyston/pyston
- Scarbutt 3y ago
  And Cinder
- dragonwriter 3y ago
  While we’re at it: * Mypyc [0] “compiles Python modules to C extensions. It uses standard Python type hints to generate fast code. Mypyc uses mypy to perform type checking and type inference. Mypyc can compile anything from one module to an entire codebase. The mypy project has been using mypyc to compile mypy since 2019, giving it a 4x performance boost over regular Python.” [0] https://github.com/mypyc/mypyc https://github.com/mypyc/mypyc
  - derbOac 3y ago
    ... and also now there's Mojo too (https://www.modular.com/mojo https://www.modular.com/mojo). Not the same but seems relevant.
    - maegul 3y ago
      Yea it’s gotten to the point where it’s a sub-domain of expertise in Python being on top of the various performance extensions/runtimes/hacks. Intuitively, numba and cython still feel like the most interesting/relevant. I recall talk a few years ago about the possibility of writing packages in “numba”. IE, targeting the subset of Python and numpy that numba supports. Occasionally I check in with numba’s releases and they seem to be slowly but surely supporting more of Python such that surely it will start to make sense to talk about the “numba” language. Has anyone got a tighter grip on whether this than I?
kethinov 3y ago
If I understand this correctly, it takes Python code and makes it as fast as C by compiling Python code into binary like compiling C to binary. I want something like this for JavaScript. Given how much stuff is written in JavaScript now, that would have a huge impact on the performance of so many apps.
- Scarbutt 3y ago
  For Javascript, what about that method would make it more performant than Javascript's JITs?
  - kethinov 3y ago
    As I understand it, JavaScript's JIT isn't quite as performant as binary compiled from a low level language like C. I'm presuming that Codon's performance is closer to that than the performance of JavaScript's JIT, but maybe that isn't correct?
antman 3y ago
I will pass. From the license: Terms The Licensor hereby grants you the right to copy, modify, create derivative works, redistribute, and make non-production use of the Licensed Work.
- deleted 3y ago
  [deleted]
- nextlevelwizard 3y ago
  Yeah... I was kind a keeping an eye on the project until I realized that they want to sell you a production license to actually use it.
tekknolagi 3y ago
Their fannkuch benchmark seems to be a bit dishonest. They claim an enormous perf delta on https://exaloop.io/benchmarks.html https://exaloop.io/benchmarks.html but fannkuch uses factorial a lot and they define factorial with a very small (n=20) table: https://github.com/exaloop/codon/blob/fb461371613049539654c11aa52eedeefd2b9a60/stdlib/math.codon#L19 https://github.com/exaloop/codon/blob/fb461371613049539654c1... Disclaimer: I've worked on several Python runtimes and compilers, but I'm not by any means out to get Codon. Just happened across this by accident while looking at their inline LLVM, which is neat.
- Atlas22 3y ago
  Can you elaborate on why using the lookup table makes this objectionable/dishonest? Factorial of 20 is the largest factorial that fits in 64-bit integer. The factorial function you linked is only defined on codon's int type (64-bit signed) so it encompasses the full range of the output datatype.
  - regularfry 3y ago
    It's a targeted micro-optimisation that doesn't say anything about codon's more general-case numeric performance. Unless the other implementations are using the same algorithm it's not comparing like with like.
    - tialaramex 3y ago
      It's not that targeted. Factorial is a reasonable thing to do, optimising for the values that fit in a 64-bit integer is a reasonable thing to do. If I read this in some code, I wouldn't think "Huh, that's a specialised feature, I bet it's attacking a specific benchmark" I would think, "Oh, that makes sense".
    - Atlas22 3y ago
      Looks like cpython does the same thing if for small n. https://github.com/python/cpython/blob/a9c6e0618f26270e2591b3d99ffeef55eea02a33/Modules/mathmodule.c#L2017 https://github.com/python/cpython/blob/a9c6e0618f26270e2591b...
ankur3 3y ago
Why not use 'mojo'!
codr7 3y ago
I've found it useful to embed scripting languages in host languages and remote control them to get the right mix of static/dynamic (fast/slow) code, rather than the other way around (calling host languages from scripting languages). But embedding Python is unfortunately a pretty big deal from what I've seen, and not as ergonomic as it could be since it's not designed to be used that way.
vpastore 3y ago
Every time I read a language describing itself as faster than c I cringe e stop caring
- vore 3y ago
  Why? Depending on the semantics of the language the compiler can make better optimizations than a C compiler could.
jokoon 3y ago
It says "a few python dynamic features are not supported" what are those?

83 points by joak 3y ago

Last I heard Codon was a python-like language rather than actual Python and the author suggests something similar. Headline seems pretty misleading.

netr0ute 3y ago
It was only technically python-like, but enough like it that you could conceivably call them the same thing.
- deleted 3y ago
  [deleted]
- joak 3y ago
  If it accepts a subset of python it's technically python: any codon-python program is a python program. Not full python but python nonetheless... My understanding: The goal is not to compile existing python code but to write new python code meant to be compiled in order to be faster. Limited use, still useful.
  - debatem1 3y ago
    It is trivially easy to write a codon program which runs in both Codon and Python but behaves differently in a way that is not considered a bug in either language. That means it is not a subset, unless by that you mean that each language has a subset of the other's programs which it can run. In which case C and JavaScript are the same language too.
    - netr0ute 3y ago
      Is there an example of this? I know there's several areas where Cpython has some funky undefined behavior, but those aren't particularly notable because they're by definition implemention dependent.
      - selcuka 3y ago
        Isn't fixed width integers as compared to arbitrary width an example of this?
      - cozzyd 3y ago
        print(repr(2**64))
      - debatem1 3y ago
        The 2**64 example given here is exactly right. You don't need undefined behavior or language lawyering to hit the difference; you just need a 65 element bitset.
- debatem1 3y ago
  Codon can't even run the whole python standard library. It changes integer semantic from arbitrary width to fixed width. Exaloop's own docs refer to the process of converting code between the two as "porting". Seems like if it isn't pythonic enough to run python code correctly it probably shouldn't be called "python".

voz_ 3y ago

How do you do, fellow python compilers?

wudangmonk 3y ago
So many compilers, it sorta reminds me of JS. Could it be that this language too has major flaws? if so why does everyone keep using it only to find out much later that they need to call out to C to do the heavy lifting only to eventually find out again that even that is not fast enough. So they need to compile all the rest of the python parts too. Seems like a rundabout way to use these lower level language without actually acknowledging that fact. An incredible amount of effort has been wasted making Python a viable language. At least the JS people have the excuse that there was no other option.
- ModernMech 3y ago
  The sad state of affairs is that C is just that hard to use for most people, so the roundabout way is pretty much the only way they can manage.
  - jjnoakes 3y ago
    There's a ton of middle ground though. It just didn't catch on as much. Unfortunately, ecosystem is still king.
- analog31 3y ago
  Sincere question: I wonder what fraction of Python users bump into these limitations often enough for it to be a problem. I do find Python to be more pleasant than C, though I use them in different domains -- desktop and embedded -- so it's not a direct comparison.
- nextlevelwizard 3y ago
  I think major contributor is the "Python is slow" meme. Sure it is slow when compared to C or something, but most of the time the time you save by implementing in Python massively out weights any performance gains, but since everyone keep saying that "Python is slow" it pushes some people to go and try to make it fast. Good example of above is this line from you (highlight mine) >An incredible amount of effort has been wasted making Python a *viable* language. What does it even mean for a language to be "viable"? Is fast execution the only criteria? There are good reasons why we aren't writing all software in C or Assembly for maximal potential performance.

nighthawk454 3y ago

Repo for more details: https://github.com/exaloop/codon https://github.com/exaloop/codon > What is Codon? > Codon is a high-performance Python compiler that compiles Python code to native machine code without any runtime overhead. Typical speedups over Python are on the order of 10-100x or more, on a single thread. Codon's performance is typically on par with (and sometimes better than) that of C/C++. Unlike Python, Codon supports native multithreading, which can lead to speedups many times higher still. Codon grew out of the Seq project. > What isn't Codon? > While Codon supports nearly all of Python's syntax, it is not a drop-in replacement, and large codebases might require modifications to be run through the Codon compiler. For example, some of Python's modules are not yet implemented within Codon, and a few of Python's dynamic features are disallowed. The Codon compiler produces detailed error messages to help identify and resolve any incompatibilities. > Codon can be used within larger Python codebases via the @codon.jit decorator. Plain Python functions and libraries can also be called from within Codon via Python interoperability.

fernly 3y ago

Just for reference, * Nuitka[0] "is a Python compiler written in Python. It's fully compatible with Python 2.6, 2.7, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, and 3.11." * Pypy[1] "is a replacement for CPython" with builtin optimizations such as on the fly JIT compiles. * Cython[2] "is an optimising static compiler for both the Python programming language and the extended Cython programming language... makes writing C extensions for Python as easy as Python itself." * Numba[3] "is an open source JIT compiler that translates a subset of Python and NumPy code into fast machine code." * Pyston[4] "is a performance-optimizing JIT for Python, and is drop-in compatible with ... CPython 3.8.12" [0] https://github.com/Nuitka/Nuitka https://github.com/Nuitka/Nuitka [1] https://www.pypy.org/ https://www.pypy.org/ [2] https://cython.org/ https://cython.org/ [3] https://numba.pydata.org/ https://numba.pydata.org/ [4] https://github.com/pyston/pyston https://github.com/pyston/pyston

Scarbutt 3y ago
And Cinder
dragonwriter 3y ago
While we’re at it: * Mypyc [0] “compiles Python modules to C extensions. It uses standard Python type hints to generate fast code. Mypyc uses mypy to perform type checking and type inference. Mypyc can compile anything from one module to an entire codebase. The mypy project has been using mypyc to compile mypy since 2019, giving it a 4x performance boost over regular Python.” [0] https://github.com/mypyc/mypyc https://github.com/mypyc/mypyc
- derbOac 3y ago
  ... and also now there's Mojo too (https://www.modular.com/mojo https://www.modular.com/mojo). Not the same but seems relevant.
  - maegul 3y ago
    Yea it’s gotten to the point where it’s a sub-domain of expertise in Python being on top of the various performance extensions/runtimes/hacks. Intuitively, numba and cython still feel like the most interesting/relevant. I recall talk a few years ago about the possibility of writing packages in “numba”. IE, targeting the subset of Python and numpy that numba supports. Occasionally I check in with numba’s releases and they seem to be slowly but surely supporting more of Python such that surely it will start to make sense to talk about the “numba” language. Has anyone got a tighter grip on whether this than I?

kethinov 3y ago

If I understand this correctly, it takes Python code and makes it as fast as C by compiling Python code into binary like compiling C to binary. I want something like this for JavaScript. Given how much stuff is written in JavaScript now, that would have a huge impact on the performance of so many apps.

Scarbutt 3y ago
For Javascript, what about that method would make it more performant than Javascript's JITs?
- kethinov 3y ago
  As I understand it, JavaScript's JIT isn't quite as performant as binary compiled from a low level language like C. I'm presuming that Codon's performance is closer to that than the performance of JavaScript's JIT, but maybe that isn't correct?

antman 3y ago

I will pass. From the license: Terms The Licensor hereby grants you the right to copy, modify, create derivative works, redistribute, and make non-production use of the Licensed Work.

deleted 3y ago
[deleted]
nextlevelwizard 3y ago
Yeah... I was kind a keeping an eye on the project until I realized that they want to sell you a production license to actually use it.

tekknolagi 3y ago

Their fannkuch benchmark seems to be a bit dishonest. They claim an enormous perf delta on https://exaloop.io/benchmarks.html https://exaloop.io/benchmarks.html but fannkuch uses factorial a lot and they define factorial with a very small (n=20) table: https://github.com/exaloop/codon/blob/fb461371613049539654c11aa52eedeefd2b9a60/stdlib/math.codon#L19 https://github.com/exaloop/codon/blob/fb461371613049539654c1... Disclaimer: I've worked on several Python runtimes and compilers, but I'm not by any means out to get Codon. Just happened across this by accident while looking at their inline LLVM, which is neat.

Atlas22 3y ago
Can you elaborate on why using the lookup table makes this objectionable/dishonest? Factorial of 20 is the largest factorial that fits in 64-bit integer. The factorial function you linked is only defined on codon's int type (64-bit signed) so it encompasses the full range of the output datatype.
- regularfry 3y ago
  It's a targeted micro-optimisation that doesn't say anything about codon's more general-case numeric performance. Unless the other implementations are using the same algorithm it's not comparing like with like.
  - tialaramex 3y ago
    It's not that targeted. Factorial is a reasonable thing to do, optimising for the values that fit in a 64-bit integer is a reasonable thing to do. If I read this in some code, I wouldn't think "Huh, that's a specialised feature, I bet it's attacking a specific benchmark" I would think, "Oh, that makes sense".
  - Atlas22 3y ago
    Looks like cpython does the same thing if for small n. https://github.com/python/cpython/blob/a9c6e0618f26270e2591b3d99ffeef55eea02a33/Modules/mathmodule.c#L2017 https://github.com/python/cpython/blob/a9c6e0618f26270e2591b...

ankur3 3y ago

Why not use 'mojo'!

codr7 3y ago

I've found it useful to embed scripting languages in host languages and remote control them to get the right mix of static/dynamic (fast/slow) code, rather than the other way around (calling host languages from scripting languages). But embedding Python is unfortunately a pretty big deal from what I've seen, and not as ergonomic as it could be since it's not designed to be used that way.

vpastore 3y ago

Every time I read a language describing itself as faster than c I cringe e stop caring

vore 3y ago
Why? Depending on the semantics of the language the compiler can make better optimizations than a C compiler could.

jokoon 3y ago

It says "a few python dynamic features are not supported" what are those?