7 ms·

Fnl: zero-overhead Lisp syntax for Lua

92 points by zeveb 9y ago

peatmoss 9y ago
I don’t have much cause to write Lua, but I can very much support lispy syntax. When we all started coding, we probably used a text editor and thought in terms of editing characters: backspacing, typing, maybe mousing around for bigger changes. Then at some point we graduated to a programmers editor where we got comfortable cutting, pulling, yanking, killing larger units of text: words, lines, paragraphs. With tools like paredit, you can take another evolutionary step to edit the structures of your code. I find this very productive. It feels a little like those code block programming languages for kids. If for no other reason than to enable slurping and barfing code structures in my text editor, this is a win. May all languages someday have a lisp syntax.
- drb91 9y ago
  I'm with you almost all the way.... however, I think what we should learn here is that every language deserves tools as powerful as paredit--that is, ast-aware editing--not that all languages should convert their ast to s-expressions. Imagine not being able to save syntactically incorrect code, or being able to have semantic diffs between domain objects (e.g. functions, classes, strings, types, apis) rather than thinking in terms of characters and lines, or being able to query a code base, like "find me all places where I assign to a variable named 'status' with type under 'error' and list me the names of the functions whose frame the variable belongs to". Paredit is simply where text and ast manipulation converge to the closest point.
  - thesmallestcat 9y ago
    As a fallback. I like LISPs as much as the next guy/gal, but editing the AST by default is too low-level. The importance of paredit is a language smell in my opinion, not something to aspire to.
    - jorams 9y ago
      Can you explain in which direction something better should go? The AST is very much at a higher level than how the vast majority of programming is done, which is by manipulating the text that later gets parsed into an AST. I'm having trouble envisioning what editing a higher level than the AST would look like, without jumping straight to visual programming, which I find very inefficient in every incarnation I've seen. (And we could debate how far above an AST such systems actually are.)
      - briankelly 9y ago
        That seems backwards to me - AST-aware editing is basically embedding more of the compiler into the editor. It seems more valuable to me to be writing in a language that can express higher level concepts with lesser boilerplate, which would eliminate the need (and make far more difficult) for AST aware editing.
        drb91 9y ago
        Why not both?
        cptn_brittish 9y ago
        In regards to lisp a large amount of lisp editors are written in lisp so they do already do exactly that'
        kazinator 9y ago
        You might be surprised how little a Lisp dialect brings to the table specifically for developing the text editor features geared toward editing specifically Lisp. The data structure representing Lisp data (i.e. code) is stripped of relevant text editing attributes, like how it is divided into lines and using what indentation, and, oh those semicolon-delimited comments. Text editors also have to let the user deal with bad syntax. So it's not just a matter of, "oh, we're written in Lisp, so just hand the buffer to the Lisp reader, do something and spit it out again".
        9y ago
  - ZenoArrow 9y ago
    > "Paredit is simply where text and ast manipulation converge to the closest point." What about Parinfer? I don't use it but seems like it's even lower friction than Paredit. https://shaunlebron.github.io/parinfer/ https://shaunlebron.github.io/parinfer/
  - wruza 9y ago
    >not being able to save syntactically incorrect code Going deeper, it would be nice to not being able to enter syntactically incorrect code.
    - kazinator 9y ago
      That's possible, but it requires the machine to enter some things for you. Since ( is syntactically incorrect, it has to be entered together with ). We do see small examples of this in GUI's with strict validation on inputs that is applied for each keystroke. One useful compromise is syntax highlighting that doesn't let you enter syntactically incorrect code without flagging the problem in a loud red color.
      - wruza 9y ago
        Red hints annoy me honestly, when I meet these in real editors. I was thinking of something like grammar walker that completes input to shortest finished constructs. While not semantically correct, but syntactically like yes. pr pr; printf( printf(<?expr?>); Or to not edit code as text at all, but as a tree. Like apple plist editor or windows registry editor, but with convenient shortcuts to add constructs. Xcode can sort of simulate this with templates and placeholders: if␣ if (<?condition?>) { <?body?> }
  - pflanze 9y ago
    S-expressions are not directly an AST: they are a form easily matched to an AST (which could be the Lisp at hands, or some language for which there is some translator/parser/interpreter around, or some language for there is no code in the system to (fully) handle it). This makes the use of s-expressions simple, they can be handled by the development tools without them always having to know what language they represent. If your development tool knows how to handle a particular language, then it can handle the AST for that language, but not any other. You could instruct the tool about the other languages, but that requires ways to let the tool know; in the end this is similar to the static versus dynamic typing discussion: static typing doesn't allow any potentially unsafe data, and a tool working with an AST directly wouldn't allow any potentially invalid programs, which in principle (or theory) is nice but difficult to achieve practically, it presents overheads for the tool implementer and for the user probably alike. And it probably takes away a lot of interactivity: just like reloading modules in a statically typed language (like ML or Haskell) is not as direct as being able to modify values and functions in a running system and having them be called directly. So I think S-expressions are a very pragmatic concept that is going to be difficult to replace without the replacement costing more than it benefits.
    - lispm 9y ago
      That's absolutely right. S-expressions are a syntax for data. It's more related to what a tokenizer does, but with nested lists of tokens - not a flat token stream. It's only loosely related to an AST, since the s-expression represents some form of nesting - but not explicitly what kinds. S-expressions know nothing about the syntax of the program language Lisp (what is the syntax for the LET operator?). If we have a Lisp expression, an AST for it would need to represent specific language constructs - like knowing that LET is a built-in operator and has parts like a list of bindings (where each binding is a symbol or a list of symbol + value form, optionally declarations (with their specific syntax) and a list of body forms. The s-expression does not represent that: in (let ((let 'let)) (list let 'let)) the s-expression does not represent that LET is a built-in operator, a variable name, a symbol, a variable name and a symbol. All we see is five LET symbols and no additional information. Writing Lisp code using s-expressions allows us to use data manipulation functions both on text and on internal Lisp data: reverse, transpose, move, move in, move out, ... We can then also enrich the editor (or other tools) to understand the syntax of Lisp. Example: know what the structure of a LET form is and then allow for example to manipulate the binding list, knowing where it is and what a variable and what a value form is.
  - conistonwater 9y ago
    One thing to consider is that s-expressions make implementing paredit easy. It takes an enormous amount of effort (e.g., libclang in C++) to get close to that without s-expressions. It's as if you think of programming languages as "fixed", and we only get to choose which kinds of tools we develop, but another way of doing it would be to design programming languages in ways that specifically make developing such tools easier. Then s-expressions could well be the go-to solution for that.
- agumonkey 9y ago
  now we need fnl-mode to enjoy lisp-lua repl and paredit in emacs :)
zitterbewegung 9y ago
This could be interesting to integrate support for racket to allow it to interact with Torch. You would just create a small language to compile to lua and then have that run in Torch. https://github.com/torch/torch7 https://github.com/torch/torch7 http://racket-lang.org http://racket-lang.org
- scottlocklin 9y ago
  Back to Lush...
bakpakin 9y ago
Original author here: The largest reason for use of S exprs is making Meta programming easier. While certain things like arithmetic are more difficult with S-exprs, Macros and DSLs are far easier than with C like syntax. Since Lua is actually only a few steps from something like Guile, it takes very little code to translate a scheme like language to Lua without adding too much runtime overhead. The difficult parts of lisp to implement, namely closures, come for free. With LuaJIT as good as it is, I would not be surprised if something like this would outperform guile.
- bjoli 9y ago
  As a guile fanboy, I doubt outperforming guile is very hard (even though it is fast enough for most uses). LuaJIT is most certainly faster than guile, but it isn't a very good comparison. Guile is far from the fastest lisp (or even scheme, for that matter), whereas LuaJIT is probably the best JIT on the planet.
  - bakpakin 9y ago
    That's true. I don't want to criticize Guile or Lisp implementations in general, which have a number of features that Lua can't easily emulate (a numeric tower, true continuations, probably a whole lot more I'm not aware of), but I think Guile and Lua have very similar uses as embedded scripting languages (small, and with fast incremental compilation). Chances are if you using guile you may have considered Lua. As for use-cases for Fnl, I would say it's more applicable for places where one would use Lua or Guile, rather than something like SBCL or any really fast, compiled Lisp. Lua makes a pretty good dynamic language target, though, because of the many great implementations and small language spec.
    - bjoli 9y ago
      I have often asked myself how LuaJIT can be so fast. It runs laps around every other JITed dynamic language, even fast ones like JS-v8. First of all: Mike Pall, but secondly: lua's semantics must lend itself very well to modern optimization techniques. LuaJIT is pretty friggin great and I have embedded it a couple of times with great success :) I do however preferred guile model for multithreading with one instance that an spawn multiple threads, and that can be used safely from multiple threads,but that is just amatter of taste. If you want to spin up multiple instances, guile isn't for you (although you could use multiple threads in a single instance). So whatever floats your boat :) There was some work implementing lua on the guile VM, which would have been cool.
deleted 9y ago
[deleted]
AHTERIX5000 9y ago
This is really interesting! I've been looking for a Lua-like Lisp implementation that would be as easy and clean to integrate, something that would compile with clang/gcc/msvc easily and allow spinning up multiple interpreter instances without having any global state. Only TinyScheme comes quite close I think.

92 points by zeveb 9y ago

I don’t have much cause to write Lua, but I can very much support lispy syntax. When we all started coding, we probably used a text editor and thought in terms of editing characters: backspacing, typing, maybe mousing around for bigger changes. Then at some point we graduated to a programmers editor where we got comfortable cutting, pulling, yanking, killing larger units of text: words, lines, paragraphs. With tools like paredit, you can take another evolutionary step to edit the structures of your code. I find this very productive. It feels a little like those code block programming languages for kids. If for no other reason than to enable slurping and barfing code structures in my text editor, this is a win. May all languages someday have a lisp syntax.

drb91 9y ago
I'm with you almost all the way.... however, I think what we should learn here is that every language deserves tools as powerful as paredit--that is, ast-aware editing--not that all languages should convert their ast to s-expressions. Imagine not being able to save syntactically incorrect code, or being able to have semantic diffs between domain objects (e.g. functions, classes, strings, types, apis) rather than thinking in terms of characters and lines, or being able to query a code base, like "find me all places where I assign to a variable named 'status' with type under 'error' and list me the names of the functions whose frame the variable belongs to". Paredit is simply where text and ast manipulation converge to the closest point.
- thesmallestcat 9y ago
  As a fallback. I like LISPs as much as the next guy/gal, but editing the AST by default is too low-level. The importance of paredit is a language smell in my opinion, not something to aspire to.
  - jorams 9y ago
    Can you explain in which direction something better should go? The AST is very much at a higher level than how the vast majority of programming is done, which is by manipulating the text that later gets parsed into an AST. I'm having trouble envisioning what editing a higher level than the AST would look like, without jumping straight to visual programming, which I find very inefficient in every incarnation I've seen. (And we could debate how far above an AST such systems actually are.)
    - briankelly 9y ago
      That seems backwards to me - AST-aware editing is basically embedding more of the compiler into the editor. It seems more valuable to me to be writing in a language that can express higher level concepts with lesser boilerplate, which would eliminate the need (and make far more difficult) for AST aware editing.
      - drb91 9y ago
        Why not both?
      - cptn_brittish 9y ago
        In regards to lisp a large amount of lisp editors are written in lisp so they do already do exactly that'
        kazinator 9y ago
        You might be surprised how little a Lisp dialect brings to the table specifically for developing the text editor features geared toward editing specifically Lisp. The data structure representing Lisp data (i.e. code) is stripped of relevant text editing attributes, like how it is divided into lines and using what indentation, and, oh those semicolon-delimited comments. Text editors also have to let the user deal with bad syntax. So it's not just a matter of, "oh, we're written in Lisp, so just hand the buffer to the Lisp reader, do something and spit it out again".
      - 9y ago
- ZenoArrow 9y ago
  > "Paredit is simply where text and ast manipulation converge to the closest point." What about Parinfer? I don't use it but seems like it's even lower friction than Paredit. https://shaunlebron.github.io/parinfer/ https://shaunlebron.github.io/parinfer/
- wruza 9y ago
  >not being able to save syntactically incorrect code Going deeper, it would be nice to not being able to enter syntactically incorrect code.
  - kazinator 9y ago
    That's possible, but it requires the machine to enter some things for you. Since ( is syntactically incorrect, it has to be entered together with ). We do see small examples of this in GUI's with strict validation on inputs that is applied for each keystroke. One useful compromise is syntax highlighting that doesn't let you enter syntactically incorrect code without flagging the problem in a loud red color.
    - wruza 9y ago
      Red hints annoy me honestly, when I meet these in real editors. I was thinking of something like grammar walker that completes input to shortest finished constructs. While not semantically correct, but syntactically like yes. pr pr; printf( printf(<?expr?>); Or to not edit code as text at all, but as a tree. Like apple plist editor or windows registry editor, but with convenient shortcuts to add constructs. Xcode can sort of simulate this with templates and placeholders: if␣ if (<?condition?>) { <?body?> }
- pflanze 9y ago
  S-expressions are not directly an AST: they are a form easily matched to an AST (which could be the Lisp at hands, or some language for which there is some translator/parser/interpreter around, or some language for there is no code in the system to (fully) handle it). This makes the use of s-expressions simple, they can be handled by the development tools without them always having to know what language they represent. If your development tool knows how to handle a particular language, then it can handle the AST for that language, but not any other. You could instruct the tool about the other languages, but that requires ways to let the tool know; in the end this is similar to the static versus dynamic typing discussion: static typing doesn't allow any potentially unsafe data, and a tool working with an AST directly wouldn't allow any potentially invalid programs, which in principle (or theory) is nice but difficult to achieve practically, it presents overheads for the tool implementer and for the user probably alike. And it probably takes away a lot of interactivity: just like reloading modules in a statically typed language (like ML or Haskell) is not as direct as being able to modify values and functions in a running system and having them be called directly. So I think S-expressions are a very pragmatic concept that is going to be difficult to replace without the replacement costing more than it benefits.
  - lispm 9y ago
    That's absolutely right. S-expressions are a syntax for data. It's more related to what a tokenizer does, but with nested lists of tokens - not a flat token stream. It's only loosely related to an AST, since the s-expression represents some form of nesting - but not explicitly what kinds. S-expressions know nothing about the syntax of the program language Lisp (what is the syntax for the LET operator?). If we have a Lisp expression, an AST for it would need to represent specific language constructs - like knowing that LET is a built-in operator and has parts like a list of bindings (where each binding is a symbol or a list of symbol + value form, optionally declarations (with their specific syntax) and a list of body forms. The s-expression does not represent that: in (let ((let 'let)) (list let 'let)) the s-expression does not represent that LET is a built-in operator, a variable name, a symbol, a variable name and a symbol. All we see is five LET symbols and no additional information. Writing Lisp code using s-expressions allows us to use data manipulation functions both on text and on internal Lisp data: reverse, transpose, move, move in, move out, ... We can then also enrich the editor (or other tools) to understand the syntax of Lisp. Example: know what the structure of a LET form is and then allow for example to manipulate the binding list, knowing where it is and what a variable and what a value form is.
- conistonwater 9y ago
  One thing to consider is that s-expressions make implementing paredit easy. It takes an enormous amount of effort (e.g., libclang in C++) to get close to that without s-expressions. It's as if you think of programming languages as "fixed", and we only get to choose which kinds of tools we develop, but another way of doing it would be to design programming languages in ways that specifically make developing such tools easier. Then s-expressions could well be the go-to solution for that.
agumonkey 9y ago
now we need fnl-mode to enjoy lisp-lua repl and paredit in emacs :)

zitterbewegung 9y ago

This could be interesting to integrate support for racket to allow it to interact with Torch. You would just create a small language to compile to lua and then have that run in Torch. https://github.com/torch/torch7 https://github.com/torch/torch7 http://racket-lang.org http://racket-lang.org

scottlocklin 9y ago
Back to Lush...

bakpakin 9y ago

Original author here: The largest reason for use of S exprs is making Meta programming easier. While certain things like arithmetic are more difficult with S-exprs, Macros and DSLs are far easier than with C like syntax. Since Lua is actually only a few steps from something like Guile, it takes very little code to translate a scheme like language to Lua without adding too much runtime overhead. The difficult parts of lisp to implement, namely closures, come for free. With LuaJIT as good as it is, I would not be surprised if something like this would outperform guile.

bjoli 9y ago
As a guile fanboy, I doubt outperforming guile is very hard (even though it is fast enough for most uses). LuaJIT is most certainly faster than guile, but it isn't a very good comparison. Guile is far from the fastest lisp (or even scheme, for that matter), whereas LuaJIT is probably the best JIT on the planet.
- bakpakin 9y ago
  That's true. I don't want to criticize Guile or Lisp implementations in general, which have a number of features that Lua can't easily emulate (a numeric tower, true continuations, probably a whole lot more I'm not aware of), but I think Guile and Lua have very similar uses as embedded scripting languages (small, and with fast incremental compilation). Chances are if you using guile you may have considered Lua. As for use-cases for Fnl, I would say it's more applicable for places where one would use Lua or Guile, rather than something like SBCL or any really fast, compiled Lisp. Lua makes a pretty good dynamic language target, though, because of the many great implementations and small language spec.
  - bjoli 9y ago
    I have often asked myself how LuaJIT can be so fast. It runs laps around every other JITed dynamic language, even fast ones like JS-v8. First of all: Mike Pall, but secondly: lua's semantics must lend itself very well to modern optimization techniques. LuaJIT is pretty friggin great and I have embedded it a couple of times with great success :) I do however preferred guile model for multithreading with one instance that an spawn multiple threads, and that can be used safely from multiple threads,but that is just amatter of taste. If you want to spin up multiple instances, guile isn't for you (although you could use multiple threads in a single instance). So whatever floats your boat :) There was some work implementing lua on the guile VM, which would have been cool.

deleted 9y ago

[deleted]

AHTERIX5000 9y ago

This is really interesting! I've been looking for a Lua-like Lisp implementation that would be as easy and clean to integrate, something that would compile with clang/gcc/msvc easily and allow spinning up multiple interpreter instances without having any global state. Only TinyScheme comes quite close I think.