7 ms·

Pallene: A statically compiled companion language for Lua

66 points by wsc981 2y ago

kragen 2y ago
it seems like the raison d'être is performance rather than static correctness checking; i wonder how it compares to luajit? often my crude tests find luajit performance comparable to c compiled by gcc with optimization, but sometimes about half as fast, and presumably on a large enough program with a flat enough profile its performance would converge to slightly worse than simple bytecode interpretation. i'm never sure how much i can rely on luajit's performance, and static compilation would presumably ameliorate that problem. also, static typing would give me compile errors instead of unexplained slowdowns when i do something that frustrates the optimizer the readme says, 'Compared to LuaJIT, Pallene aims to offer more predictable run-time performance.' but seriously luajit is astounding. in https://gitlab.com/kragen/bubbleos/blob/master/yeso/mand.lua https://gitlab.com/kragen/bubbleos/blob/master/yeso/mand.lua there is semantically a function pointer invocation per pixel (pixfunc), but luajit's trace compilation apparently hoists that entirely out of the inner loop of drawing the fractal. conventional compilation of statically typed languages, even ocaml, can't do anything like that; you need something like c++ templates to specialize the inner loop for each possible fractal being drawn
- Rochus 2y ago
  Pallene unfortunately is only a subset of Lua, i.e. you cannot simply take a Lua program and compile it with Pallene. The use case of this language is not obvious to me, since if I have to re-implement the performance-critical parts of the Lua program anyway, I can just as good implement it in C using the Lua C API. Thus the claim of the Pallene authors is a bit misleading. Concerning LuaJIT performance, I have done a lot of measurements over the years (see e.g. https://github.com/rochus-keller/Oberon/blob/master/testcases/Are-we-fast-yet/Are-we-fast-yet_results_linux.pdf https://github.com/rochus-keller/Oberon/blob/master/testcase...) and would rather say that its performance is about factor four worse on average than an equivalent C++ -O2 compiled program. For some exceptional use cases it might perform as good as C, but not in general (one reason is the missing tracing JIT support for closures).
  - kragen 2y ago
    interesting, thank you for sharing your experience! in the example case my pixfunc is in fact not a closure, just a pure function, but it sounds like the performance would collapse if i made it a closure (you can of course compile specialized versions of the inner loop for each fractal in c; 'just' use the preprocessor) it seems plausible to me that it would be more convenient to reimplement a lua function in pallene rather than in c, especially for 'shallow' functions where the code for passing arguments and return values back and forth on the stack to lua is a substantial percentage of the total function code but there's still the question of how good or bad the performance of the code generated by the pallene compiler is! it might be worth rewriting the code in c if gcc or clang produces faster code than pallene does, even if rewriting it in pallene would be easier
    - Rochus 2y ago
      The problem is actually FNEW. If you create your function on top-level and leave it as is then FNEW is only used once. But if you require e.g. nested functions, then each call to FNEW will terminate the trace. My measurement with the Smalltalk and SOM implementations using LuaJIT showed that the benchmark essentially remains in the interpreter most of the time for that reason. > it seems plausible to me that it would be more convenient to reimplement a lua function in pallene rather than in c If just Pallene would be more similar to Lua. But essentially it's a different language which just resembles Lua a bit. So you could use any language compatible with the Lua C API if you don't like C (instead of learning Pallene). You have to rewrite the performance critical parts anyway. Personally I prefer LuaJIT, since everything is Lua, and if you know enough details and do measurements, the performance is no longer a surprise. > how good or bad the performance of the code generated by the pallene compiler is There is no reason to assume that Pallene is faster than LuaJIT if things like the ones mentioned above are considered.
      - kragen 2y ago
        that must have been a huge disappointment when you ran into that problem with som! are you planning to write your own jit compiler, or is there still a way you can leverage luajit?
- efilife 2y ago
  What's the point of saying raison d'être instead of just saying reason? Why did you use a random french phrase here?
  - kragen 2y ago
    it gave me an excuse to use a circumflex
  - brabel 2y ago
    Call the language police!
    - efilife 2y ago
      I just thought it's pointless obfuscation of text, I'm just learning it's apparently a commonly used phrase
      - samatman 2y ago
        You say here that you're not a native English speaker: https://news.ycombinator.com/item?id=41086473 https://news.ycombinator.com/item?id=41086473 Which is fine! Very common on Hacker News. My suggestion is that you approach this kind of question the way you did in that post, not the way you did in this thread. "I'm not a native speaker, why did you say raison d'être there?" would get your question answered without the downvotes. English is a magpie language, and there are dozens to hundreds of undigested phrases borrowed from other languages in it, with more arriving all the time.
        efilife 2y ago
        Yup, it's definitely my fault here. I should have phrased this in a completely different way. Nonetheless, I still think using a foreign phrase is pointless and potentially obfuscates the meaning of a sentence, since we already have english words for that. You are right, my fault for that. English is a dumpster fire sometimes, could have accounted for that
- hugomg 2y ago
  (I'm one of the Pallene authors) For compiled code, Pallene's performance can be comparable to LuaJIT[1]. Pallene might be better in code with unpredictable branches, which don't fit in a single trace. LuaJIT has the edge in purely-interpreted code, and can inline indirect or cross-module function calls. LuaJIT is also more featureful, and has a better FFI story at the moment. [1] https://www.inf.puc-rio.br/~hgualandi/papers/Gualandi-2020-SCP.pdf https://www.inf.puc-rio.br/~hgualandi/papers/Gualandi-2020-S... LuaJIT is really good software, it is hard to beat it as its own game. Pallene tries to get its edge elsewhere: it tracks Lua 5.4, whereas LuaJIT diverged around 5.1 and will stay that way forever. Pallene's implementation is arguably more portable, because it compiles down to C and doesn't need hand-crafted assembly.
  - kragen 2y ago
    this is fantastic, thanks! it seems like you're saying pallene is using a c compiler as its compiler backend? that seems like it could give you a big leg up on difficult optimizations like inlining recursion (compared to writing your own backend, i mean, not specifically compared to luajit) i'll read the paper. aha, it says: > The Pallene compiler generates C source code and uses a conventional C compiler as a backend. This is simple to implement and is also useful for portability. The generated C code for a Pallene module is standard C code that uses the same headers and system calls that the reference Lua does. This means that Pallene code should be portable to any platforms where Lua itself can run. btw, if you're not on a microcontroller, this can even be a feasible thing to do for a jit compiler; running gcc 9.4 on a small program takes about 70 milliseconds on this micropc, 55 milliseconds to compile a small shared library. clang is of course less practical, but older gcc versions were even better, and tcc is even better, taking respectively 10ms and 7ms. you might have to cut down your header files though the paper also has a longer section comparing pallene with luajit, which i will have to read in more detail. thank you very much for linking it! 5.4 vs. 5.1 could be an advantage for either side; minetest, for example, uses 5.1, so luajit is an option and pallene probably isn't
    - sitkack 2y ago
      Afaik there was a prototype Ruby JIT that used the C compiler this way and loaded the resulting code as a shared library. I did this with a Python decorator and ctypes so I could inline and hot reload my C extensions using @c_func def add1(n : int) -> int: """ return n + 1 """ I haven't touched this space in years, I think there are a couple Python libraries that do this now. I did not generate C code at runtime tho.
teamonkey 2y ago
I love Lua but I also prefer static types, so this looks great. I’ve used Luau[1] in the past which has gradual typing, but that’s less useful unless it’s enforced as a coding standard. [1] https://luau-lang.org/ https://luau-lang.org/
- babuskov 2y ago
  IIRC the main selling point of Luau is that you get decent sandboxing by default.
  - teamonkey 2y ago
    It also offers speed improvements and other optimisations specifically for games development, since LuaJIT cannot be used on several target platforms. One of the interesting things about Pallene is that it claims to reduce the overhead between the Lua code and C++, which is hugely important for game dev.
samatman 2y ago
I'd be interested in a compare-and-contrast with Terra: https://terralang.org https://terralang.org The goals and broad approach are quite similar, and I'm sure that the Pallene authors are aware that Terra exists, so I'd love to hear why they decided that a new language was worth pursuing.
- hugomg 2y ago
  tl;dr: For Pallene, Lua is a scripting language. For Terra, Lua is mainly for metaprogramming. The key difference is that Terra cannot directly manipulate Lua data. Although Terra's syntax is similar to Lua, its type systems and data structures are actually much closer to C. If we want to pass Lua table into Terra, we must first convert it into Terra arrays/structs. This cost adds up if the code uses many Lua data structures; in the worst case it can be worse than just using Lua for everything. Passing data between Lua and Pallene should incur no cost. If the programmer rewrites a piece of code from Lua to Pallene, it should hopefully make things faster, and never slow it down. Terra is more focused on metaprogramming. Generating code and executing things at compile-time. Think of C++ templates, but better.
  - samatman 2y ago
    Great answer, thanks! I see the appeal.
conaclos 2y ago
For those wondering what the difference is between Pallene and Teal [1], I found a 2021 comment [2] that gives some clues. [1] https://github.com/teal-language/tl https://github.com/teal-language/tl [2] https://news.ycombinator.com/item?id=26263759 https://news.ycombinator.com/item?id=26263759
breck 2y ago
This documentation is fantastic. I'd love to have Pallene on PLDB.io. If you had a few minutes, here's a post on how to add it: https://pldb.io/blog/addLanguageDemo.html https://pldb.io/blog/addLanguageDemo.html I can also add it myself, but I it's more helpful to have the language creators add their language. (But I'm happy to add if you are too busy, and/or assist with addding)

66 points by wsc981 2y ago

it seems like the raison d'être is performance rather than static correctness checking; i wonder how it compares to luajit? often my crude tests find luajit performance comparable to c compiled by gcc with optimization, but sometimes about half as fast, and presumably on a large enough program with a flat enough profile its performance would converge to slightly worse than simple bytecode interpretation. i'm never sure how much i can rely on luajit's performance, and static compilation would presumably ameliorate that problem. also, static typing would give me compile errors instead of unexplained slowdowns when i do something that frustrates the optimizer the readme says, 'Compared to LuaJIT, Pallene aims to offer more predictable run-time performance.' but seriously luajit is astounding. in https://gitlab.com/kragen/bubbleos/blob/master/yeso/mand.lua https://gitlab.com/kragen/bubbleos/blob/master/yeso/mand.lua there is semantically a function pointer invocation per pixel (pixfunc), but luajit's trace compilation apparently hoists that entirely out of the inner loop of drawing the fractal. conventional compilation of statically typed languages, even ocaml, can't do anything like that; you need something like c++ templates to specialize the inner loop for each possible fractal being drawn

Rochus 2y ago
Pallene unfortunately is only a subset of Lua, i.e. you cannot simply take a Lua program and compile it with Pallene. The use case of this language is not obvious to me, since if I have to re-implement the performance-critical parts of the Lua program anyway, I can just as good implement it in C using the Lua C API. Thus the claim of the Pallene authors is a bit misleading. Concerning LuaJIT performance, I have done a lot of measurements over the years (see e.g. https://github.com/rochus-keller/Oberon/blob/master/testcases/Are-we-fast-yet/Are-we-fast-yet_results_linux.pdf https://github.com/rochus-keller/Oberon/blob/master/testcase...) and would rather say that its performance is about factor four worse on average than an equivalent C++ -O2 compiled program. For some exceptional use cases it might perform as good as C, but not in general (one reason is the missing tracing JIT support for closures).
- kragen 2y ago
  interesting, thank you for sharing your experience! in the example case my pixfunc is in fact not a closure, just a pure function, but it sounds like the performance would collapse if i made it a closure (you can of course compile specialized versions of the inner loop for each fractal in c; 'just' use the preprocessor) it seems plausible to me that it would be more convenient to reimplement a lua function in pallene rather than in c, especially for 'shallow' functions where the code for passing arguments and return values back and forth on the stack to lua is a substantial percentage of the total function code but there's still the question of how good or bad the performance of the code generated by the pallene compiler is! it might be worth rewriting the code in c if gcc or clang produces faster code than pallene does, even if rewriting it in pallene would be easier
  - Rochus 2y ago
    The problem is actually FNEW. If you create your function on top-level and leave it as is then FNEW is only used once. But if you require e.g. nested functions, then each call to FNEW will terminate the trace. My measurement with the Smalltalk and SOM implementations using LuaJIT showed that the benchmark essentially remains in the interpreter most of the time for that reason. > it seems plausible to me that it would be more convenient to reimplement a lua function in pallene rather than in c If just Pallene would be more similar to Lua. But essentially it's a different language which just resembles Lua a bit. So you could use any language compatible with the Lua C API if you don't like C (instead of learning Pallene). You have to rewrite the performance critical parts anyway. Personally I prefer LuaJIT, since everything is Lua, and if you know enough details and do measurements, the performance is no longer a surprise. > how good or bad the performance of the code generated by the pallene compiler is There is no reason to assume that Pallene is faster than LuaJIT if things like the ones mentioned above are considered.
    - kragen 2y ago
      that must have been a huge disappointment when you ran into that problem with som! are you planning to write your own jit compiler, or is there still a way you can leverage luajit?
efilife 2y ago
What's the point of saying raison d'être instead of just saying reason? Why did you use a random french phrase here?
- kragen 2y ago
  it gave me an excuse to use a circumflex
- brabel 2y ago
  Call the language police!
  - efilife 2y ago
    I just thought it's pointless obfuscation of text, I'm just learning it's apparently a commonly used phrase
    - samatman 2y ago
      You say here that you're not a native English speaker: https://news.ycombinator.com/item?id=41086473 https://news.ycombinator.com/item?id=41086473 Which is fine! Very common on Hacker News. My suggestion is that you approach this kind of question the way you did in that post, not the way you did in this thread. "I'm not a native speaker, why did you say raison d'être there?" would get your question answered without the downvotes. English is a magpie language, and there are dozens to hundreds of undigested phrases borrowed from other languages in it, with more arriving all the time.
      - efilife 2y ago
        Yup, it's definitely my fault here. I should have phrased this in a completely different way. Nonetheless, I still think using a foreign phrase is pointless and potentially obfuscates the meaning of a sentence, since we already have english words for that. You are right, my fault for that. English is a dumpster fire sometimes, could have accounted for that
hugomg 2y ago
(I'm one of the Pallene authors) For compiled code, Pallene's performance can be comparable to LuaJIT[1]. Pallene might be better in code with unpredictable branches, which don't fit in a single trace. LuaJIT has the edge in purely-interpreted code, and can inline indirect or cross-module function calls. LuaJIT is also more featureful, and has a better FFI story at the moment. [1] https://www.inf.puc-rio.br/~hgualandi/papers/Gualandi-2020-SCP.pdf https://www.inf.puc-rio.br/~hgualandi/papers/Gualandi-2020-S... LuaJIT is really good software, it is hard to beat it as its own game. Pallene tries to get its edge elsewhere: it tracks Lua 5.4, whereas LuaJIT diverged around 5.1 and will stay that way forever. Pallene's implementation is arguably more portable, because it compiles down to C and doesn't need hand-crafted assembly.
- kragen 2y ago
  this is fantastic, thanks! it seems like you're saying pallene is using a c compiler as its compiler backend? that seems like it could give you a big leg up on difficult optimizations like inlining recursion (compared to writing your own backend, i mean, not specifically compared to luajit) i'll read the paper. aha, it says: > The Pallene compiler generates C source code and uses a conventional C compiler as a backend. This is simple to implement and is also useful for portability. The generated C code for a Pallene module is standard C code that uses the same headers and system calls that the reference Lua does. This means that Pallene code should be portable to any platforms where Lua itself can run. btw, if you're not on a microcontroller, this can even be a feasible thing to do for a jit compiler; running gcc 9.4 on a small program takes about 70 milliseconds on this micropc, 55 milliseconds to compile a small shared library. clang is of course less practical, but older gcc versions were even better, and tcc is even better, taking respectively 10ms and 7ms. you might have to cut down your header files though the paper also has a longer section comparing pallene with luajit, which i will have to read in more detail. thank you very much for linking it! 5.4 vs. 5.1 could be an advantage for either side; minetest, for example, uses 5.1, so luajit is an option and pallene probably isn't
  - sitkack 2y ago
    Afaik there was a prototype Ruby JIT that used the C compiler this way and loaded the resulting code as a shared library. I did this with a Python decorator and ctypes so I could inline and hot reload my C extensions using @c_func def add1(n : int) -> int: """ return n + 1 """ I haven't touched this space in years, I think there are a couple Python libraries that do this now. I did not generate C code at runtime tho.

teamonkey 2y ago

I love Lua but I also prefer static types, so this looks great. I’ve used Luau[1] in the past which has gradual typing, but that’s less useful unless it’s enforced as a coding standard. [1] https://luau-lang.org/ https://luau-lang.org/

babuskov 2y ago
IIRC the main selling point of Luau is that you get decent sandboxing by default.
- teamonkey 2y ago
  It also offers speed improvements and other optimisations specifically for games development, since LuaJIT cannot be used on several target platforms. One of the interesting things about Pallene is that it claims to reduce the overhead between the Lua code and C++, which is hugely important for game dev.

samatman 2y ago

I'd be interested in a compare-and-contrast with Terra: https://terralang.org https://terralang.org The goals and broad approach are quite similar, and I'm sure that the Pallene authors are aware that Terra exists, so I'd love to hear why they decided that a new language was worth pursuing.

hugomg 2y ago
tl;dr: For Pallene, Lua is a scripting language. For Terra, Lua is mainly for metaprogramming. The key difference is that Terra cannot directly manipulate Lua data. Although Terra's syntax is similar to Lua, its type systems and data structures are actually much closer to C. If we want to pass Lua table into Terra, we must first convert it into Terra arrays/structs. This cost adds up if the code uses many Lua data structures; in the worst case it can be worse than just using Lua for everything. Passing data between Lua and Pallene should incur no cost. If the programmer rewrites a piece of code from Lua to Pallene, it should hopefully make things faster, and never slow it down. Terra is more focused on metaprogramming. Generating code and executing things at compile-time. Think of C++ templates, but better.
- samatman 2y ago
  Great answer, thanks! I see the appeal.

conaclos 2y ago

For those wondering what the difference is between Pallene and Teal [1], I found a 2021 comment [2] that gives some clues. [1] https://github.com/teal-language/tl https://github.com/teal-language/tl [2] https://news.ycombinator.com/item?id=26263759 https://news.ycombinator.com/item?id=26263759

breck 2y ago

This documentation is fantastic. I'd love to have Pallene on PLDB.io. If you had a few minutes, here's a post on how to add it: https://pldb.io/blog/addLanguageDemo.html https://pldb.io/blog/addLanguageDemo.html I can also add it myself, but I it's more helpful to have the language creators add their language. (But I'm happy to add if you are too busy, and/or assist with addding)