8 ms·

A Lua 5.3 VM and compiler written in Go

196 points by erwan 9y ago

jzelinskie 9y ago
There are a couple different Lua implementations in Go. Has anyone researched the difference between them? https://github.com/yuin/gopher-lua https://github.com/yuin/gopher-lua https://github.com/Shopify/go-lua https://github.com/Shopify/go-lua https://github.com/milochristiansen/lua https://github.com/milochristiansen/lua https://github.com/afitz/golua https://github.com/afitz/golua
- wahern 9y ago
  Milo Christiansen's seems to be the only Lua 5.3 implementation. Note that Lua's versioning scheme is not like most other languages. Lua 5.1, 5.2, and 5.3 are not the same language. Though most code will still work across those versions, the Lua developers are more aggressive at adopting new features and killing old ones, even for core functionality. 5.2 dropped function environments in favor of lexical _ENV, which was a huge change for people implementing sandboxing, reflection, module frameworks, etc. 5.3 added an integer type whereas previously all numbers were floating point, and because Lua is dynamically typed you can't move an existing body of Lua code to 5.3 without some careful attention. Similarly, the APIs also have incompatible changes that while mostly simpe to deal with require porting attention. This more aggressive style of stewardship is what has made Lua an amazingly elegant and powerful language, but reduces it's attractiveness for large, monolithic, long-lived frameworks unless they're prepared to periodically upgrade languages and migrate their user base. Since Lua is more typically used for extension and embedding (i.e. not monolithic, pure-Lua projects), this is less of an issue. I was explaining to a colleague the other day that the reason you don't hear so much about Lua is because choosing Lua isn't a decision that dominates a project. A project that heavily uses Perl is a Perl project; one that uses Python is a project. For the most part, you don't build Lua applications; you build C, C++, or w'ever applications that use Lua, even when 90% of the code is in Lua. This is a reflection of the carefully crafted API and language semantics. But IMO it's the biggest reason why Lua hasn't yet had a break-out moment, despite Lua 5.1, 5.2, and 5.3 being (IMNSHO) far superior languages to JavaScript, Perl, Python, Ruby, or other similar dynamic languages. Lua is a better language because it aggressively improves itself. But that evolution is too rapid to support many large, marquee projects.
  - fao_ 9y ago
    In my experience the difference between 5.2 and 5.3 was more towards "negligible" than you're making out. While the functional difference may have been major, most programs in my experience operate exactly as you would expect them. For the majority of programs I doubt that they were even felt, aside from a few minor changes like "unpack" -> "table.unpack"
    - TheCycoONE 9y ago
      The biggest issue I hit when adding Lua 5.3 support to CorsiTH was that floating point numbers now crash when sent to a C function that expects a lua_Integer whereas before they would be silently truncated. Since we still support 5.1 as well I ended up scattering a lot of math.floor to compensate. That said it was trivial compared to the _ENV change for 5.2
    - wahern 9y ago
      That's been my experience, too. And the consensus on the lua-l list seems to be that the migration to mixed number types went relatively smoothly for most people despite the anxiety. But people were still anxious, even gung-ho Lua users. Whether or not justified, people and companies contemplating large projects are especially averse to that sort of environment. My open-source project experiences notwithstanding, before I left my day job a few months ago we still hadn't migrated our 10+ year-old project to 5.2, let alone 5.3, even though LuaJIT wasn't involved, and even though lua_callk and lua_pcallk would have significantly simplified the libevent async I/O integration. In a corporate environment planned obsolescence is a hard to sell, even when it's objectively the better choice.
  - fosk 9y ago
    > Since Lua is more typically used for extension and embedding The LuaJIT (http://luajit.org http://luajit.org) is another reason why Lua is very popular (although it only officially supports Lua 5.1). It’s a very fast implementation of the Lua VM. OpenResty (https://openresty.org/ https://openresty.org/) is a very popular framework that allows Lua scripting on top of NGINX and LuaJIT, and it’s the same stack that Cloudflare uses. Basically over 5% of the world internet traffic goes through Lua, but only a few people know it. This is the same stack that Kong (https://github.com/Mashape/kong https://github.com/Mashape/kong) also uses, probably the most popular OpenResty-based application. Lua on top of LuaJIT can be extremely fast especially if you leverage FFI. Lua is an extremely simple language which can be pretty much embedded anywhere, and the LuaJIT just does a great job at making it extremely embeddable and fast. Disclaimer: I am a core committed at Kong.
gaigepr 9y ago
Is the security policy that is used as a reason to not implement a bunch of stuff documented somewhere?
- loeg 9y ago
  6th paragraph of README.md: > Anything to do with the OS or file IO is not provided. Such things do not belong in the core libraries of an embedded scripting language (do you really want scripts to be able to read and write random files without restriction?).
  - otempomores 9y ago
    Embedded pride world wide. Would you allow some metal-phobe byte-code language to meddle with the binaries ;-)
lasfter 9y ago
What is the use in Lua without pattern matching or coroutines?
- wruza 9y ago
  This project may have a value as interpreter/vm implementation, not as substitution of any sort. Table iterators and weak references are also missing. Name choice seems pretty arbitrary, so author surely could just mention that this vm was inspired by Lua and not name it alike. The statement that he could easily implement coroutines via go seems like oversimplification or lack of problem domain understanding.
  - weberc2 9y ago
    > The statement that he could easily implement coroutines via go seems like oversimplification or lack of problem domain understanding. For someone with a lack of problem domain understanding, could you elaborate on this? :)
    - wruza 9y ago
      vm.go/l.call doesn't seem to be reentrant (though I'm not native go speaker). This means that yielding across, e.g. for-loop [edit:]iterators, would leave vm with broken invariants. There are also pcalls that must be dealt with. But since this language is just similar and differs in too many things, explicit restrictions may help solve that, like it was done for <= 5.1.
throwaway2016a 9y ago
This is really interesting for me personally because I just completed writing an interpreter in Golang and it's great to compare and contrast. One thing I noticed is that the parser is hand written. Which is interesting. I always preferred to write my parsers in Yacc and Go actually has Yacc as a built in tool (although know Lex). Edit: I went through the list someone else posted and only one of the implementations actually has a Yacc grammar file in it.
- villedepommes 9y ago
  > Go actually has Yacc as a built in tool (although know Lex). Interesting. Did they combine both a lexer and a parser into one tool? lex,flex, and re2c are typically standalone tools (lexers) that tokenize input, whereas yacc and bison are parsers that parse those tokens into ASTs (usually).
  - throwaway2016a 9y ago
    > Did they combine both a lexer and a parser into one tool? No. They give you an interface definition for the lexer and you can implement the methods however you see fit as long as it matches the interface. Which I found annoying as there are some tedious parts to writing a lexer as soon as you have a language requiring multi-rune tokens with common prefixes.[1] Apparently the back story is that go needed a parser generator but they didn't need a lexer so they only built just the one out of necessity. I read some places that they would welcome pull requests for a lex/flex equivalent. [1] A `rune` in GoLang is like a `char` in C only it stores multi-byte characters where as in C it is more analogous to a single `byte`
    - hnlmorg 9y ago
      For clarity (I know you know what a rune is but your description felt a little misleading imo): A rune is just an alias for int32. But it's context usually means unicode chars which are indeed multi-byte. You can still use byte arrays (or "slice" if you're writing idiomatic Go, but that's another tangent again) too though. Also "byte" in Go is another alias, but for int8. But that shouldn't come as a surprise to anyone :)
ComputerGuru 9y ago
A _partial_ implementation without IO, pattern matching, or coroutines. Nothing to see here, really. Just remaps basic Lua syntax to Go without actually _implementing_ what makes Lua useful.
- Impossible 9y ago
  In an embedded context lua is definitely still useful without those things, although coroutines are nice to have almost all of the lua work I've done hasn't used any of those features. IO and pattern matching seem important if you're using lua for text processing, which isn't the only useful reason to use Lua.
- pygy_ 9y ago
  ... or weak tables (because it relies on the Go GC which doesn't have weak references).
- nine_k 9y ago
  Using a subset of Lua for scripting beats inventing your own wheel most of the time.
bpizzi 9y ago
Slightly related: I'm in the process of choosing the best way to give our Go binaries some scripting capabilities (it's entreprise stuff, the idea is "let's the customer script this behaviour"). I'm aware of Lua and JS interpreters, some fully native, some made of binding against existing interpreters. Native JS interpreters have my preference right now: easier to build/maintain, I can bear the loss of perfs, and JS seems easier to sell than Lua (remember it's entreprise stuff). Does anyone have some insight or experience in that area and care to give feedback to a fellow HNer? And did I miss other scripting languages?
- Derbasti 9y ago
  Lua is incredibly easy to embed. The whole Lua interpreter including the standard library is about 150kb. That's Kilo, not mega! You can swap out the memory allocator or the number type on one simple header file. Lua was built specifically for embedding! What is more, it is a joyously well designed language that is well suited for building DSLs. And it is pretty fast and memory efficient for a dynamic language. Other options are TKL, but the syntax is a bit unorthodox. Or Python, but it is much bigger, and harder to embed. JS is not a great fit, I would say, mostly because it is comparatively HUGE and not particularly optimized for embedding.
  - hnlmorg 9y ago
    Perl 5 is another option. There's a few embedded Perl libraries out there. But Perl has a bad reputation (undeservingly in my personal opinion) so potentially another hard sell for enterprise.
    - hnlmorg 9y ago
      I guess it's too much to ask for whoever downvoted my comment to explain why they did? Otherwise it's going to look little more than a knee-jerk reaction from some Perl hater.
      - Yaggo 9y ago
        I didn't downvote you, but I'd like to know what makes Perl a good option for embedded language and what implementations are available.
alvil 9y ago
Crippled Lua
- hnlmorg 9y ago
  Honestly, comments like this help noone. The author wrote this because it solved a particular problem s/he had and was excited enough by it to share. S/he has also released it with a permissive licence so that you can contribute to it or even fork and enhance if you so wish. Or if you really don't share the author's enthusiasm for the project then you can simply move on and use someone else's Lua library (or perhaps write your own and risk other internet randoms criticising your work without offering any constructive pointers).
  - vfclists 9y ago
    Pedant note - The pronoun you want for the unspecified gender is 'they', not 's/he', and it was well established before the gender-fluidity brigade adopted it.

196 points by erwan 9y ago

There are a couple different Lua implementations in Go. Has anyone researched the difference between them? https://github.com/yuin/gopher-lua https://github.com/yuin/gopher-lua https://github.com/Shopify/go-lua https://github.com/Shopify/go-lua https://github.com/milochristiansen/lua https://github.com/milochristiansen/lua https://github.com/afitz/golua https://github.com/afitz/golua

wahern 9y ago
Milo Christiansen's seems to be the only Lua 5.3 implementation. Note that Lua's versioning scheme is not like most other languages. Lua 5.1, 5.2, and 5.3 are not the same language. Though most code will still work across those versions, the Lua developers are more aggressive at adopting new features and killing old ones, even for core functionality. 5.2 dropped function environments in favor of lexical _ENV, which was a huge change for people implementing sandboxing, reflection, module frameworks, etc. 5.3 added an integer type whereas previously all numbers were floating point, and because Lua is dynamically typed you can't move an existing body of Lua code to 5.3 without some careful attention. Similarly, the APIs also have incompatible changes that while mostly simpe to deal with require porting attention. This more aggressive style of stewardship is what has made Lua an amazingly elegant and powerful language, but reduces it's attractiveness for large, monolithic, long-lived frameworks unless they're prepared to periodically upgrade languages and migrate their user base. Since Lua is more typically used for extension and embedding (i.e. not monolithic, pure-Lua projects), this is less of an issue. I was explaining to a colleague the other day that the reason you don't hear so much about Lua is because choosing Lua isn't a decision that dominates a project. A project that heavily uses Perl is a Perl project; one that uses Python is a project. For the most part, you don't build Lua applications; you build C, C++, or w'ever applications that use Lua, even when 90% of the code is in Lua. This is a reflection of the carefully crafted API and language semantics. But IMO it's the biggest reason why Lua hasn't yet had a break-out moment, despite Lua 5.1, 5.2, and 5.3 being (IMNSHO) far superior languages to JavaScript, Perl, Python, Ruby, or other similar dynamic languages. Lua is a better language because it aggressively improves itself. But that evolution is too rapid to support many large, marquee projects.
- fao_ 9y ago
  In my experience the difference between 5.2 and 5.3 was more towards "negligible" than you're making out. While the functional difference may have been major, most programs in my experience operate exactly as you would expect them. For the majority of programs I doubt that they were even felt, aside from a few minor changes like "unpack" -> "table.unpack"
  - TheCycoONE 9y ago
    The biggest issue I hit when adding Lua 5.3 support to CorsiTH was that floating point numbers now crash when sent to a C function that expects a lua_Integer whereas before they would be silently truncated. Since we still support 5.1 as well I ended up scattering a lot of math.floor to compensate. That said it was trivial compared to the _ENV change for 5.2
  - wahern 9y ago
    That's been my experience, too. And the consensus on the lua-l list seems to be that the migration to mixed number types went relatively smoothly for most people despite the anxiety. But people were still anxious, even gung-ho Lua users. Whether or not justified, people and companies contemplating large projects are especially averse to that sort of environment. My open-source project experiences notwithstanding, before I left my day job a few months ago we still hadn't migrated our 10+ year-old project to 5.2, let alone 5.3, even though LuaJIT wasn't involved, and even though lua_callk and lua_pcallk would have significantly simplified the libevent async I/O integration. In a corporate environment planned obsolescence is a hard to sell, even when it's objectively the better choice.
- fosk 9y ago
  > Since Lua is more typically used for extension and embedding The LuaJIT (http://luajit.org http://luajit.org) is another reason why Lua is very popular (although it only officially supports Lua 5.1). It’s a very fast implementation of the Lua VM. OpenResty (https://openresty.org/ https://openresty.org/) is a very popular framework that allows Lua scripting on top of NGINX and LuaJIT, and it’s the same stack that Cloudflare uses. Basically over 5% of the world internet traffic goes through Lua, but only a few people know it. This is the same stack that Kong (https://github.com/Mashape/kong https://github.com/Mashape/kong) also uses, probably the most popular OpenResty-based application. Lua on top of LuaJIT can be extremely fast especially if you leverage FFI. Lua is an extremely simple language which can be pretty much embedded anywhere, and the LuaJIT just does a great job at making it extremely embeddable and fast. Disclaimer: I am a core committed at Kong.

gaigepr 9y ago

Is the security policy that is used as a reason to not implement a bunch of stuff documented somewhere?

loeg 9y ago
6th paragraph of README.md: > Anything to do with the OS or file IO is not provided. Such things do not belong in the core libraries of an embedded scripting language (do you really want scripts to be able to read and write random files without restriction?).
- otempomores 9y ago
  Embedded pride world wide. Would you allow some metal-phobe byte-code language to meddle with the binaries ;-)

lasfter 9y ago

What is the use in Lua without pattern matching or coroutines?

wruza 9y ago
This project may have a value as interpreter/vm implementation, not as substitution of any sort. Table iterators and weak references are also missing. Name choice seems pretty arbitrary, so author surely could just mention that this vm was inspired by Lua and not name it alike. The statement that he could easily implement coroutines via go seems like oversimplification or lack of problem domain understanding.
- weberc2 9y ago
  > The statement that he could easily implement coroutines via go seems like oversimplification or lack of problem domain understanding. For someone with a lack of problem domain understanding, could you elaborate on this? :)
  - wruza 9y ago
    vm.go/l.call doesn't seem to be reentrant (though I'm not native go speaker). This means that yielding across, e.g. for-loop [edit:]iterators, would leave vm with broken invariants. There are also pcalls that must be dealt with. But since this language is just similar and differs in too many things, explicit restrictions may help solve that, like it was done for <= 5.1.

throwaway2016a 9y ago

This is really interesting for me personally because I just completed writing an interpreter in Golang and it's great to compare and contrast. One thing I noticed is that the parser is hand written. Which is interesting. I always preferred to write my parsers in Yacc and Go actually has Yacc as a built in tool (although know Lex). Edit: I went through the list someone else posted and only one of the implementations actually has a Yacc grammar file in it.

villedepommes 9y ago
> Go actually has Yacc as a built in tool (although know Lex). Interesting. Did they combine both a lexer and a parser into one tool? lex,flex, and re2c are typically standalone tools (lexers) that tokenize input, whereas yacc and bison are parsers that parse those tokens into ASTs (usually).
- throwaway2016a 9y ago
  > Did they combine both a lexer and a parser into one tool? No. They give you an interface definition for the lexer and you can implement the methods however you see fit as long as it matches the interface. Which I found annoying as there are some tedious parts to writing a lexer as soon as you have a language requiring multi-rune tokens with common prefixes.[1] Apparently the back story is that go needed a parser generator but they didn't need a lexer so they only built just the one out of necessity. I read some places that they would welcome pull requests for a lex/flex equivalent. [1] A `rune` in GoLang is like a `char` in C only it stores multi-byte characters where as in C it is more analogous to a single `byte`
  - hnlmorg 9y ago
    For clarity (I know you know what a rune is but your description felt a little misleading imo): A rune is just an alias for int32. But it's context usually means unicode chars which are indeed multi-byte. You can still use byte arrays (or "slice" if you're writing idiomatic Go, but that's another tangent again) too though. Also "byte" in Go is another alias, but for int8. But that shouldn't come as a surprise to anyone :)

ComputerGuru 9y ago

A _partial_ implementation without IO, pattern matching, or coroutines. Nothing to see here, really. Just remaps basic Lua syntax to Go without actually _implementing_ what makes Lua useful.

Impossible 9y ago
In an embedded context lua is definitely still useful without those things, although coroutines are nice to have almost all of the lua work I've done hasn't used any of those features. IO and pattern matching seem important if you're using lua for text processing, which isn't the only useful reason to use Lua.
pygy_ 9y ago
... or weak tables (because it relies on the Go GC which doesn't have weak references).
nine_k 9y ago
Using a subset of Lua for scripting beats inventing your own wheel most of the time.

bpizzi 9y ago

Slightly related: I'm in the process of choosing the best way to give our Go binaries some scripting capabilities (it's entreprise stuff, the idea is "let's the customer script this behaviour"). I'm aware of Lua and JS interpreters, some fully native, some made of binding against existing interpreters. Native JS interpreters have my preference right now: easier to build/maintain, I can bear the loss of perfs, and JS seems easier to sell than Lua (remember it's entreprise stuff). Does anyone have some insight or experience in that area and care to give feedback to a fellow HNer? And did I miss other scripting languages?

Derbasti 9y ago
Lua is incredibly easy to embed. The whole Lua interpreter including the standard library is about 150kb. That's Kilo, not mega! You can swap out the memory allocator or the number type on one simple header file. Lua was built specifically for embedding! What is more, it is a joyously well designed language that is well suited for building DSLs. And it is pretty fast and memory efficient for a dynamic language. Other options are TKL, but the syntax is a bit unorthodox. Or Python, but it is much bigger, and harder to embed. JS is not a great fit, I would say, mostly because it is comparatively HUGE and not particularly optimized for embedding.
- hnlmorg 9y ago
  Perl 5 is another option. There's a few embedded Perl libraries out there. But Perl has a bad reputation (undeservingly in my personal opinion) so potentially another hard sell for enterprise.
  - hnlmorg 9y ago
    I guess it's too much to ask for whoever downvoted my comment to explain why they did? Otherwise it's going to look little more than a knee-jerk reaction from some Perl hater.
    - Yaggo 9y ago
      I didn't downvote you, but I'd like to know what makes Perl a good option for embedded language and what implementations are available.

alvil 9y ago

Crippled Lua

hnlmorg 9y ago
Honestly, comments like this help noone. The author wrote this because it solved a particular problem s/he had and was excited enough by it to share. S/he has also released it with a permissive licence so that you can contribute to it or even fork and enhance if you so wish. Or if you really don't share the author's enthusiasm for the project then you can simply move on and use someone else's Lua library (or perhaps write your own and risk other internet randoms criticising your work without offering any constructive pointers).
- vfclists 9y ago
  Pedant note - The pronoun you want for the unspecified gender is 'they', not 's/he', and it was well established before the gender-fluidity brigade adopted it.