6 ms·

Writing a DSL in Lua (2015)

93 points by flykespice 2y ago

neilv 2y ago
This is a nice example of one way to implement DSLs in a language that doesn't support them well. IIUC, they're using Lua's language's function application syntax, combined with the ability to introduce a different namespace environment for function lookup, and then they're evaluating the tree of function calls at run time. In languages that support doing this at compile time, through compile-time evaluation or compile-time syntax transformation, you can make run time faster. For example of syntax transformation, https://www.neilvandyke.org/racket/html-template/ https://www.neilvandyke.org/racket/html-template/ shows an example of expanded syntax, where it's rendered some of the angle brackets and coalesced that with (escaped) HTML CDATA. If you wanted to extend that, to support chunks of traditional HTML angle-bracket syntax inline with code (without using string literals), like in React, you could define a Racket reader to parse those chunks and transform them to parentheses syntax above.
- wahern 2y ago
  > This is a nice example of one way to implement DSLs in a language that doesn't support them well. It's not a coincidence that Lua supports such syntax for such use cases.
alberth 2y ago
I've always wanted to recreate a Python like syntax for Lua, this might be a good guide for that.
- stdbrouw 2y ago
  Around the time when CoffeeScript was popular, there was also https://moonscript.org/ https://moonscript.org/ for Lua. Made by the very same prolific Leaf who authored this DSL tutorial.
  - isr 2y ago
    Yup, and it's still in active production use (itch.io, for example). There's even a fork of moonscript, called yuescript, which has a speedier parser, and includes macro's. Effectively, it's moonscript-2.0, and it's bundled directly into the dora-ssr game engine (the same dev is involved with both). Of course, the "big dog" in the compile-to-lua space is fennel. The great thing with all of these options is that they don't have a runtime component- it's all lua semantics, all the way down. So you can mix & match from project to project, as your whim takes you - and all the while you're really just gaining more experience in building lua solutions (while not having to fight against some of lua's syntax)
moonlet 2y ago
While I’m surprised to see they aren’t writing about https://www.inf.puc-rio.br/~roberto/lpeg/ https://www.inf.puc-rio.br/~roberto/lpeg/ it looks like they’ve covered LPEG in a previous post, and their chosen method this time around is neat anyway!
dan353hehe 2y ago
I love Lua for being able to do things like this. I was building a bunch of html pages for an htmx frontend and a golang back end, and got really exhausted from using the builtin `html/template` library in golang. I kept trying to build reusable components, and it just didn’t work as I wanted it to. I ended up doing this exact thing as mentioned in this blog post. https://github.com/DBarney/Glua https://github.com/DBarney/Glua Granted this is just for me, and for prototyping. I put it on GitHub so I wouldn’t loose it. Writing html got so much easier this way: local function Component(data) return div{"my name is", data.name} end return function(params) html{ head{title="this is my page"}, body{ h1{"This is my small page I wrote"}, p{ "some content", "more content" }, div{"page has some dynamic values",params.count}, Component{name="daniel"} } } end Edit: formatting
- arccy 2y ago
  you should try https://pkg.go.dev/maragu.dev/gomponents https://pkg.go.dev/maragu.dev/gomponents and do it all in go
dang 2y ago
Related: Writing a DSL in Lua (2015) - https://news.ycombinator.com/item?id=22223542 https://news.ycombinator.com/item?id=22223542 - Feb 2020 (41 comments)
roxolotl 2y ago
Lua is a much cooler and more useful language than I think it, publicly, gets credit for. I know it’s widely used in many industries but so many times I’ve read an article like this and thought that it could be even more widely useful. I’m currently writing a just for me static site generator in Fennel and it’s sad to see just how many powerful packages are abandoned. A lot of it is because they are mostly complete and part of why I like it is lack of churn but a last update on 18 months ago is very different from 10 years. I want to host knitting patterns with my site generator and techniques like this make it super easy to build a little pattern dsl. It’s stuff like that which makes me wish it was more widespread.
- epcoa 2y ago
  > Lua is a much cooler and more useful language than I think it, publicly, gets credit for. I think it’s gets about the right amount of credit. It isn’t obscure by any means, openresty is used quite a bit in odd places, so many games have used it. PyTorch (well what became it) was originally written in it (it was just torch but basically LuaTorch) It doesn’t get more adoption because its warts and limitations really start to show once pushed into more into a “real” language. It doesn’t have the browser monopoly. It’s also quite slow, and LuaJIT doesn’t always magically fix that.
  - Yliaho 2y ago
    > LuaJIT doesn’t always magically fix that. Not to mention there are some significant performance issues on aarch64. Love2D, a popular game framework, opted to not use JIT on macOS for this reason. https://github.com/LuaJIT/LuaJIT/issues/285 https://github.com/LuaJIT/LuaJIT/issues/285
    - aeadio 2y ago
      This is strictly about how much memory LuaJIT can address on that platform. There’s no “significant performance issues”. If your workload fits inside 2 GiB (IIRC) it will be fast. LuaJIT is a world class JIT. For a long time a new garbage collector has been an open issue for LuaJIT, which would fix this particular issue, and make the language even faster. Last I checked this was actively being worked on.
gustavopezzi 2y ago
Lua's LPeg library would also be a great option here. A couple of years ago I taught an interpreters class together with Roberto Ierusalimschy and we used LPeg for lexing and parsing. The end result was great. Even if you're not using PEGs for your implementation I would recommend spending 30 minutes and looking at LPeg.
jFriedensreich 2y ago
trailing closure syntax looks similar but is probably even a bit nicer. I just wish js would get this feature so we could stop using jsx.
mjmas 2y ago
MicroLua for the RP2040 has a nice dsl for the PIO assembly with some special use of methods: https://github.com/MicroLua/MicroLua/blob/main/lib/pico/hardware.pio.asm.test.lua#L45 https://github.com/MicroLua/MicroLua/blob/main/lib/pico/hard... function pio_timer(_ENV) public(start): set(y, 0) mov(y, ~y) wrap_target() delay: pull() mov(x, osr) loop: jmp(x_dec, loop) mov(isr, ~y) push() jmp(y_dec, delay) end
- flykespice 2y ago
  Was that really achieved with vanilla Lua or MicroLua has some language extensions?
  - SyrupThinker 2y ago
    While I can't speak for MicroLua, this can indeed be accomplished with vanilla Lua. There are two things in play here: First, the _ENV variable here is special. It is implicitly used to lookup any identifier that has no visible binding, locally or in the surrounding scopes. Thus a script just containing print("Hello world") is really _ENV.print("Hello world"). Usually _ENV is just implicitly defined to be the global environment (available as _G), but it can be overridden within a lexical scope to a custom value. Second, the jump labels just make use of the alternate function call syntax of object:method(args), which is equivalent to object.method(object, args). The whitespace is non-significant, which allows it to be written like that. In combination with metatables you can use _ENV to track variable reads, calls etc. within a function, which you can (abuse) to create DSLs. You can get an idea of whats possible by just tracking what we can intercept with _ENV. Adding the following code after the pio_timer function, and running the script with Lua 5.4, already gives us quite a bit for just the first three lines of the function. local loggerMetatable = {} function loggerMetatable:__bnot() print("Called bnot on " .. rawget(self, "name")) end function loggerMetatable:__call(...) local name = rawget(self, "name") print("Invoked " .. name .. " with " .. #table.pack(...) .. " args") -- Return another logger table to visualize variable interactions return setmetatable({ name = name .. "(...)" }, loggerMetatable) end function loggerMetatable:__index(key) local name = rawget(self, "name") print("Accessed key " .. key .. " on " .. name) -- Return another logger table to visualize variable interactions return setmetatable({ name = name .. "." .. key }, loggerMetatable) end local pio_env = setmetatable({ name = "_ENV" }, loggerMetatable) pio_timer(pio_env) -- Output: Accessed key public on _ENV Accessed key start on _ENV Invoked _ENV.public with 1 args Accessed key set on _ENV.public(...) Accessed key y on _ENV Invoked _ENV.public(...).set with 3 args Accessed key mov on _ENV Accessed key y on _ENV Accessed key y on _ENV Called bnot on _ENV.y Invoked _ENV.mov with 1 args [...snip...]
binary132 2y ago
It never fails to strike me as odd how polarizing Lua is. Some people (like me) find it really pleasant and fluid to use. Other people get hives from things like 1-based indexing and global-by-default. The intense complaints are just so silly, especially when the “better alternative” presented is typically Python, which is one of the worst languages I’ve ever used. Lua’s syntax and semantics are so much smaller and cleaner that it’s mindblowing how simple it makes rather complex and custom abstractions. When I use Lua I feel like I’m working on another level of abstraction than in other languages, almost a lisp, but with just the right amount of convenient / quality-of-life features that it makes everything trivial. Multiple inheritance, mixins, type constructors are all trivial classes, if you want them, or parts of them, or to do your own thing entirely, or you can choose none of it at all and merely build a very primitive C-composition scripting API for your library, and yes embedding languages inside it is therefore also trivial. I’ll never understand the mindset that leads people to hating it. “But bro, Python has C libraries!!” and? Lua is built out of C semantics and has a beautiful C API if I want native libraries, I’ll just take a little time to write a trivial wrapper for precisely as much functionality as I want from the library

93 points by flykespice 2y ago

This is a nice example of one way to implement DSLs in a language that doesn't support them well. IIUC, they're using Lua's language's function application syntax, combined with the ability to introduce a different namespace environment for function lookup, and then they're evaluating the tree of function calls at run time. In languages that support doing this at compile time, through compile-time evaluation or compile-time syntax transformation, you can make run time faster. For example of syntax transformation, https://www.neilvandyke.org/racket/html-template/ https://www.neilvandyke.org/racket/html-template/ shows an example of expanded syntax, where it's rendered some of the angle brackets and coalesced that with (escaped) HTML CDATA. If you wanted to extend that, to support chunks of traditional HTML angle-bracket syntax inline with code (without using string literals), like in React, you could define a Racket reader to parse those chunks and transform them to parentheses syntax above.

wahern 2y ago
> This is a nice example of one way to implement DSLs in a language that doesn't support them well. It's not a coincidence that Lua supports such syntax for such use cases.

alberth 2y ago

I've always wanted to recreate a Python like syntax for Lua, this might be a good guide for that.

stdbrouw 2y ago
Around the time when CoffeeScript was popular, there was also https://moonscript.org/ https://moonscript.org/ for Lua. Made by the very same prolific Leaf who authored this DSL tutorial.
- isr 2y ago
  Yup, and it's still in active production use (itch.io, for example). There's even a fork of moonscript, called yuescript, which has a speedier parser, and includes macro's. Effectively, it's moonscript-2.0, and it's bundled directly into the dora-ssr game engine (the same dev is involved with both). Of course, the "big dog" in the compile-to-lua space is fennel. The great thing with all of these options is that they don't have a runtime component- it's all lua semantics, all the way down. So you can mix & match from project to project, as your whim takes you - and all the while you're really just gaining more experience in building lua solutions (while not having to fight against some of lua's syntax)

moonlet 2y ago

While I’m surprised to see they aren’t writing about https://www.inf.puc-rio.br/~roberto/lpeg/ https://www.inf.puc-rio.br/~roberto/lpeg/ it looks like they’ve covered LPEG in a previous post, and their chosen method this time around is neat anyway!

dan353hehe 2y ago

I love Lua for being able to do things like this. I was building a bunch of html pages for an htmx frontend and a golang back end, and got really exhausted from using the builtin `html/template` library in golang. I kept trying to build reusable components, and it just didn’t work as I wanted it to. I ended up doing this exact thing as mentioned in this blog post. https://github.com/DBarney/Glua https://github.com/DBarney/Glua Granted this is just for me, and for prototyping. I put it on GitHub so I wouldn’t loose it. Writing html got so much easier this way: local function Component(data) return div{"my name is", data.name} end return function(params) html{ head{title="this is my page"}, body{ h1{"This is my small page I wrote"}, p{ "some content", "more content" }, div{"page has some dynamic values",params.count}, Component{name="daniel"} } } end Edit: formatting

arccy 2y ago
you should try https://pkg.go.dev/maragu.dev/gomponents https://pkg.go.dev/maragu.dev/gomponents and do it all in go

dang 2y ago

Related: Writing a DSL in Lua (2015) - https://news.ycombinator.com/item?id=22223542 https://news.ycombinator.com/item?id=22223542 - Feb 2020 (41 comments)

roxolotl 2y ago

Lua is a much cooler and more useful language than I think it, publicly, gets credit for. I know it’s widely used in many industries but so many times I’ve read an article like this and thought that it could be even more widely useful. I’m currently writing a just for me static site generator in Fennel and it’s sad to see just how many powerful packages are abandoned. A lot of it is because they are mostly complete and part of why I like it is lack of churn but a last update on 18 months ago is very different from 10 years. I want to host knitting patterns with my site generator and techniques like this make it super easy to build a little pattern dsl. It’s stuff like that which makes me wish it was more widespread.

epcoa 2y ago
> Lua is a much cooler and more useful language than I think it, publicly, gets credit for. I think it’s gets about the right amount of credit. It isn’t obscure by any means, openresty is used quite a bit in odd places, so many games have used it. PyTorch (well what became it) was originally written in it (it was just torch but basically LuaTorch) It doesn’t get more adoption because its warts and limitations really start to show once pushed into more into a “real” language. It doesn’t have the browser monopoly. It’s also quite slow, and LuaJIT doesn’t always magically fix that.
- Yliaho 2y ago
  > LuaJIT doesn’t always magically fix that. Not to mention there are some significant performance issues on aarch64. Love2D, a popular game framework, opted to not use JIT on macOS for this reason. https://github.com/LuaJIT/LuaJIT/issues/285 https://github.com/LuaJIT/LuaJIT/issues/285
  - aeadio 2y ago
    This is strictly about how much memory LuaJIT can address on that platform. There’s no “significant performance issues”. If your workload fits inside 2 GiB (IIRC) it will be fast. LuaJIT is a world class JIT. For a long time a new garbage collector has been an open issue for LuaJIT, which would fix this particular issue, and make the language even faster. Last I checked this was actively being worked on.

gustavopezzi 2y ago

Lua's LPeg library would also be a great option here. A couple of years ago I taught an interpreters class together with Roberto Ierusalimschy and we used LPeg for lexing and parsing. The end result was great. Even if you're not using PEGs for your implementation I would recommend spending 30 minutes and looking at LPeg.

jFriedensreich 2y ago

trailing closure syntax looks similar but is probably even a bit nicer. I just wish js would get this feature so we could stop using jsx.

mjmas 2y ago

MicroLua for the RP2040 has a nice dsl for the PIO assembly with some special use of methods: https://github.com/MicroLua/MicroLua/blob/main/lib/pico/hardware.pio.asm.test.lua#L45 https://github.com/MicroLua/MicroLua/blob/main/lib/pico/hard... function pio_timer(_ENV) public(start): set(y, 0) mov(y, ~y) wrap_target() delay: pull() mov(x, osr) loop: jmp(x_dec, loop) mov(isr, ~y) push() jmp(y_dec, delay) end

flykespice 2y ago
Was that really achieved with vanilla Lua or MicroLua has some language extensions?
- SyrupThinker 2y ago
  While I can't speak for MicroLua, this can indeed be accomplished with vanilla Lua. There are two things in play here: First, the _ENV variable here is special. It is implicitly used to lookup any identifier that has no visible binding, locally or in the surrounding scopes. Thus a script just containing print("Hello world") is really _ENV.print("Hello world"). Usually _ENV is just implicitly defined to be the global environment (available as _G), but it can be overridden within a lexical scope to a custom value. Second, the jump labels just make use of the alternate function call syntax of object:method(args), which is equivalent to object.method(object, args). The whitespace is non-significant, which allows it to be written like that. In combination with metatables you can use _ENV to track variable reads, calls etc. within a function, which you can (abuse) to create DSLs. You can get an idea of whats possible by just tracking what we can intercept with _ENV. Adding the following code after the pio_timer function, and running the script with Lua 5.4, already gives us quite a bit for just the first three lines of the function. local loggerMetatable = {} function loggerMetatable:__bnot() print("Called bnot on " .. rawget(self, "name")) end function loggerMetatable:__call(...) local name = rawget(self, "name") print("Invoked " .. name .. " with " .. #table.pack(...) .. " args") -- Return another logger table to visualize variable interactions return setmetatable({ name = name .. "(...)" }, loggerMetatable) end function loggerMetatable:__index(key) local name = rawget(self, "name") print("Accessed key " .. key .. " on " .. name) -- Return another logger table to visualize variable interactions return setmetatable({ name = name .. "." .. key }, loggerMetatable) end local pio_env = setmetatable({ name = "_ENV" }, loggerMetatable) pio_timer(pio_env) -- Output: Accessed key public on _ENV Accessed key start on _ENV Invoked _ENV.public with 1 args Accessed key set on _ENV.public(...) Accessed key y on _ENV Invoked _ENV.public(...).set with 3 args Accessed key mov on _ENV Accessed key y on _ENV Accessed key y on _ENV Called bnot on _ENV.y Invoked _ENV.mov with 1 args [...snip...]

binary132 2y ago

It never fails to strike me as odd how polarizing Lua is. Some people (like me) find it really pleasant and fluid to use. Other people get hives from things like 1-based indexing and global-by-default. The intense complaints are just so silly, especially when the “better alternative” presented is typically Python, which is one of the worst languages I’ve ever used. Lua’s syntax and semantics are so much smaller and cleaner that it’s mindblowing how simple it makes rather complex and custom abstractions. When I use Lua I feel like I’m working on another level of abstraction than in other languages, almost a lisp, but with just the right amount of convenient / quality-of-life features that it makes everything trivial. Multiple inheritance, mixins, type constructors are all trivial classes, if you want them, or parts of them, or to do your own thing entirely, or you can choose none of it at all and merely build a very primitive C-composition scripting API for your library, and yes embedding languages inside it is therefore also trivial. I’ll never understand the mindset that leads people to hating it. “But bro, Python has C libraries!!” and? Lua is built out of C semantics and has a beautiful C API if I want native libraries, I’ll just take a little time to write a trivial wrapper for precisely as much functionality as I want from the library