6 ms·

Luajit a wonderfully clever piece of code that only one person understands and is able to support. Before everyone chimes in that I'm mistaken ask yourself why

by nmpl 10y ago

Luajit a wonderfully clever piece of code that only one person understands and is able to support. Before everyone chimes in that I'm mistaken ask yourself why Luajit has not evolved to be compatible with the latest Lua 5.3 release.

pygy_ 10y ago
It stayed at v5.1 (with some v5.2 features that don't break 5.1 code) because Mike Pall disagreed with some v5.2 features: swapping `setfenv` for `_ENV`, and some details of the bundled bitlib. He also criticized 128 bits interpreter values mandated by the support of 64 bits integer in v5.3. These take twice as much cache space and damage performance.
- daurnimator 10y ago
  Except luajit has boxed 64bit integers: there's no reason you couldn't have them act just like lua 5.3 64 bit ints. In the majority of real use cases you can narrow down to the 32bit ints already supported by luajit.
  - pygy_ 10y ago
    Even from the Lua C API standpoint? I'm not very familiar with it.
    - daurnimator 10y ago
      Currently in luajit you're just "not meant to" work with ffi cdata from the C api. You can... even if it's not pretty. However, I was more saying that there isn't a technical reason that lua 5.3 integer behaviour couldn't reuse the existing (boxed) 64bit integer support in luajit. Which would mean that the slim tagged NaN tagged value approach can be kept.
      - blep 10y ago
        I think the main problem here is that there are several semantic differences between 5.3's and LuaJIT's integers which would make this a bit dodgy; eg. rounding behavior and mixed float-integer arithmetic are different. IMO 5.3 should have done what LuaJIT did and require a suffix for 64-bit integer literals (say `L` to avoid conflict with LuaJIT's `LL`). Given how rarely Lua code needs true integers this large, that probably would have worked better. It also would have avoided the gotchas of porting code to 5.3 where the integral numbers you never add `.0` to are suddenly real integers instead of floats. I'd love for LuaJIT to get 5.3's bitwise ops but the semantic differences between the integer implementations would make this interesting to say the least.
    - rtsisyk 10y ago
      Tarantool[1], a general-purpose database and app server based on LuaJIT, has Lua/C API to work with 64-bit numbers and even custom cdata objects [2]. I can extract this code into a separate library for you. [1]: http://tarantool.org/ http://tarantool.org/ [2]: https://github.com/tarantool/tarantool/blob/1.7/src/lua/utils.h#L382-L430 https://github.com/tarantool/tarantool/blob/1.7/src/lua/util...
- zde 10y ago
  Not sure what you talk about.. lua5.3 stores 64bit integers in the same tagged union it stored doubles so the value size didn't grow at all. 64bit ints only need a new tag value (which was available before).
  - pygy_ 10y ago
    IIRC, Lua 5.2 uses NaN tagging as LuaJIT does, and has 64bits values.
    - zde 10y ago
      All Lua5.x versions use 8-byte union plus 1-byte tag, that's 12-byte TValue (16-byte on 64bit targets). (somewhat off topic) Mike authored a cunning patch to un-box short strings, sadly it got never mainlined. http://lua-users.org/wiki/FastStringPatch http://lua-users.org/wiki/FastStringPatch
      - pygy_ 10y ago
        Edit, yes, v5.2 supports the NaN trick (see the comments above http://www.lua.org/source/5.2/lobject.h.html#NNMARK http://www.lua.org/source/5.2/lobject.h.html#NNMARK), but only on x86. I suppose the Lua authors wanted to avoid the memory limitations of LuaJIT on 64bits machines. ---- Original below: I thought v5.2 used NaN tagging but I may be wrong. At least the beta versions did. I misremembered that post to refer to the final version: http://lua-users.org/lists/lua-l/2011-07/msg00180.html http://lua-users.org/lists/lua-l/2011-07/msg00180.html Quoted in full: ---- Lua 5.2 beta implements the "NaN trick": it packs all Lua values into a single double value by representing non-numeric values as signaled NaNs, which are (usually) not produced by the system. This trick should work on most 32-bit machines that follow IEEE 754-2008 for double representation. However, currently it is being enabled only by predefined macro __i386__ (and similars). We would like to know what other platforms could benefit from this trick (and what macros should we check in luaconf.h). You can force the trick by compiling Lua with -DLUA_NANTRICKLE (for little endian systems) or -DLUA_NANTRICKBE (for big endian). (Of course, we would also like to know whether there are problems with this implementation.) -- Roberto ---- I also think I remember people (maybe Mike Pall) reporting perf regressions between 5.2 and 5.3 because of the cache bloat introduced by mandatory 128bits values (on 64bits systems).
TheAceOfHearts 10y ago
From what I've read, Lua has apparently done a poor job at maintaining backwards compatibility and Mike Pall is unwilling to support those kinds of changes. I don't have enough context to know if breaking backwards compatibility is justified or not. I generally think it's a poor decision that's hostile to developers to break backwards compatibility in a programming language without providing a clear migration path. Issues between 5.1, 5.2, and 5.3 definitely seem to have splintered the community. http://www.freelists.org/post/luajit/Port-bitop-to-53,1 http://www.freelists.org/post/luajit/Port-bitop-to-53,1
- wwwigham 10y ago
  I feel that the language itself has remained remarkably compatible through releases. The difficult change, I believe, has been the changes to the API for native extensions (if I'm remembering my history right).
- catwell 10y ago
  The backward compatibility issue Mike is ranting about in this post is not in the language, it is in the source code of Lua itself (actually a define in its configuration header file). There was a very good reason for this change, which is that Lua introduced an integer type instead of using floating point numbers for everything. This means that LUA_NUMBER_DOUBLE and LUA_NUMBER_FLOAT do not really make sense anymore (when you check those you may want to check the type of the integer numbers or the type of the floating-point numbers). In any case, porting luabitop doesn't really matter that much: if LuaJIT supported Lua 5.3 people would use the native 64 bit bitwise operators it introduced instead.
wruza 10y ago
From the business point of view the question is why 5.3 has not evolved to be 5.1 + some mt/env/yield fixes.
copx 10y ago
Lua was pretty much perfect at version 5.1 (the version LuaJIT supports). Lua is meant to be a simple language, not one of those languages which accumulate more and more features and thus complexity over time (I am looking at you C#, Java, etc.). So unlike those languages Lua can really be considered "done". Version 5.2 was a lot like C99, maybe people considered one or two features "nice to have" but most Lua programmers saw no need to upgrade. Others like Mike Pall even considered 5.2 inferior to 5.1. Now we have 5.3 which is really all about one special-purpose feature: 64 bit integers. That is a feature most Lua users do not need and it comes with a massive complexity and performance price tag. Unless you really need 64 bit integer math in your scripts 5.3 is the inferior language. I really enjoy that in classic Lua a number is a number, that is such a relief compared to the hell that is C's countless numeric types and all the potential bugs resulting from (silent) conversions between them. Lua 5.3 brought that problem to Lua e.g. a = 9223372036854775000 b = 0.5 c = a + b c = c - b print(c) // 9.2233720368548e+18 The third line silently converts a 64-bit integer variable to a 64-bit floating point variable, corrupting its value. And of course Lua 5.3 is slower and needs more memory because of the additional complexity which must be handled. Thanks but no thanks. I mean Lua 5.3 still makes sense if you really need 64-bit integers in your Lua scripts. But otherwise it is a downgrade.
- fit2rule 10y ago
  As a long-term Lua fan, I completely agree with you, even though its not a popular opinion. Too many times I've been debugging some Lua code, tearing my hair out, only to discover "oh yeah, we're using Lua5.3 for this - is that a problem?" .. but at least we're catching up with Python in this regard. /ducks
- pritambaral 10y ago
  > print(c) // 9.2233720368548e+18 I have little experience in lua myself, and I know only that lua is meant to be a simple language. That said, I can't see from your example what's wrong with lua5.3. I ran the same code with lua5.1, lua5.2, and lua5.3 – and the output was identical across all: $ diff <(lua5.1 numcheck.lua) <(lua5.3 numcheck.lua) $ diff <(lua5.2 numcheck.lua) <(lua5.3 numcheck.lua) $
anonymoushn 10y ago
It looks like the main change in 5.3 is an overhaul of how numbers work to add integers and integer division. This seems like a bad call and I'd rather not deal with it.
rtsisyk 10y ago
Hi, Tarantool[1] team maintains production-ready fork of LuaJIT [2]. Commercial support contract for Tarantool covers problems with LuaJIT as the integral part of the product. [1]: http://tarantool.org/ http://tarantool.org/ [2]: https://github.com/tarantool/luajit https://github.com/tarantool/luajit P.S. there are shortcoming plans to establish a non-profit foundation to continue development of LuaJIT. // Disclaimer, I'm a contributor of [1]

by nmpl 10y ago

pygy_ 10y ago

It stayed at v5.1 (with some v5.2 features that don't break 5.1 code) because Mike Pall disagreed with some v5.2 features: swapping `setfenv` for `_ENV`, and some details of the bundled bitlib. He also criticized 128 bits interpreter values mandated by the support of 64 bits integer in v5.3. These take twice as much cache space and damage performance.

daurnimator 10y ago
Except luajit has boxed 64bit integers: there's no reason you couldn't have them act just like lua 5.3 64 bit ints. In the majority of real use cases you can narrow down to the 32bit ints already supported by luajit.
- pygy_ 10y ago
  Even from the Lua C API standpoint? I'm not very familiar with it.
  - daurnimator 10y ago
    Currently in luajit you're just "not meant to" work with ffi cdata from the C api. You can... even if it's not pretty. However, I was more saying that there isn't a technical reason that lua 5.3 integer behaviour couldn't reuse the existing (boxed) 64bit integer support in luajit. Which would mean that the slim tagged NaN tagged value approach can be kept.
    - blep 10y ago
      I think the main problem here is that there are several semantic differences between 5.3's and LuaJIT's integers which would make this a bit dodgy; eg. rounding behavior and mixed float-integer arithmetic are different. IMO 5.3 should have done what LuaJIT did and require a suffix for 64-bit integer literals (say `L` to avoid conflict with LuaJIT's `LL`). Given how rarely Lua code needs true integers this large, that probably would have worked better. It also would have avoided the gotchas of porting code to 5.3 where the integral numbers you never add `.0` to are suddenly real integers instead of floats. I'd love for LuaJIT to get 5.3's bitwise ops but the semantic differences between the integer implementations would make this interesting to say the least.
  - rtsisyk 10y ago
    Tarantool[1], a general-purpose database and app server based on LuaJIT, has Lua/C API to work with 64-bit numbers and even custom cdata objects [2]. I can extract this code into a separate library for you. [1]: http://tarantool.org/ http://tarantool.org/ [2]: https://github.com/tarantool/tarantool/blob/1.7/src/lua/utils.h#L382-L430 https://github.com/tarantool/tarantool/blob/1.7/src/lua/util...
zde 10y ago
Not sure what you talk about.. lua5.3 stores 64bit integers in the same tagged union it stored doubles so the value size didn't grow at all. 64bit ints only need a new tag value (which was available before).
- pygy_ 10y ago
  IIRC, Lua 5.2 uses NaN tagging as LuaJIT does, and has 64bits values.
  - zde 10y ago
    All Lua5.x versions use 8-byte union plus 1-byte tag, that's 12-byte TValue (16-byte on 64bit targets). (somewhat off topic) Mike authored a cunning patch to un-box short strings, sadly it got never mainlined. http://lua-users.org/wiki/FastStringPatch http://lua-users.org/wiki/FastStringPatch
    - pygy_ 10y ago
      Edit, yes, v5.2 supports the NaN trick (see the comments above http://www.lua.org/source/5.2/lobject.h.html#NNMARK http://www.lua.org/source/5.2/lobject.h.html#NNMARK), but only on x86. I suppose the Lua authors wanted to avoid the memory limitations of LuaJIT on 64bits machines. ---- Original below: I thought v5.2 used NaN tagging but I may be wrong. At least the beta versions did. I misremembered that post to refer to the final version: http://lua-users.org/lists/lua-l/2011-07/msg00180.html http://lua-users.org/lists/lua-l/2011-07/msg00180.html Quoted in full: ---- Lua 5.2 beta implements the "NaN trick": it packs all Lua values into a single double value by representing non-numeric values as signaled NaNs, which are (usually) not produced by the system. This trick should work on most 32-bit machines that follow IEEE 754-2008 for double representation. However, currently it is being enabled only by predefined macro __i386__ (and similars). We would like to know what other platforms could benefit from this trick (and what macros should we check in luaconf.h). You can force the trick by compiling Lua with -DLUA_NANTRICKLE (for little endian systems) or -DLUA_NANTRICKBE (for big endian). (Of course, we would also like to know whether there are problems with this implementation.) -- Roberto ---- I also think I remember people (maybe Mike Pall) reporting perf regressions between 5.2 and 5.3 because of the cache bloat introduced by mandatory 128bits values (on 64bits systems).

TheAceOfHearts 10y ago

From what I've read, Lua has apparently done a poor job at maintaining backwards compatibility and Mike Pall is unwilling to support those kinds of changes. I don't have enough context to know if breaking backwards compatibility is justified or not. I generally think it's a poor decision that's hostile to developers to break backwards compatibility in a programming language without providing a clear migration path. Issues between 5.1, 5.2, and 5.3 definitely seem to have splintered the community. http://www.freelists.org/post/luajit/Port-bitop-to-53,1 http://www.freelists.org/post/luajit/Port-bitop-to-53,1

wwwigham 10y ago
I feel that the language itself has remained remarkably compatible through releases. The difficult change, I believe, has been the changes to the API for native extensions (if I'm remembering my history right).
catwell 10y ago
The backward compatibility issue Mike is ranting about in this post is not in the language, it is in the source code of Lua itself (actually a define in its configuration header file). There was a very good reason for this change, which is that Lua introduced an integer type instead of using floating point numbers for everything. This means that LUA_NUMBER_DOUBLE and LUA_NUMBER_FLOAT do not really make sense anymore (when you check those you may want to check the type of the integer numbers or the type of the floating-point numbers). In any case, porting luabitop doesn't really matter that much: if LuaJIT supported Lua 5.3 people would use the native 64 bit bitwise operators it introduced instead.

wruza 10y ago

From the business point of view the question is why 5.3 has not evolved to be 5.1 + some mt/env/yield fixes.

copx 10y ago

Lua was pretty much perfect at version 5.1 (the version LuaJIT supports). Lua is meant to be a simple language, not one of those languages which accumulate more and more features and thus complexity over time (I am looking at you C#, Java, etc.). So unlike those languages Lua can really be considered "done". Version 5.2 was a lot like C99, maybe people considered one or two features "nice to have" but most Lua programmers saw no need to upgrade. Others like Mike Pall even considered 5.2 inferior to 5.1. Now we have 5.3 which is really all about one special-purpose feature: 64 bit integers. That is a feature most Lua users do not need and it comes with a massive complexity and performance price tag. Unless you really need 64 bit integer math in your scripts 5.3 is the inferior language. I really enjoy that in classic Lua a number is a number, that is such a relief compared to the hell that is C's countless numeric types and all the potential bugs resulting from (silent) conversions between them. Lua 5.3 brought that problem to Lua e.g. a = 9223372036854775000 b = 0.5 c = a + b c = c - b print(c) // 9.2233720368548e+18 The third line silently converts a 64-bit integer variable to a 64-bit floating point variable, corrupting its value. And of course Lua 5.3 is slower and needs more memory because of the additional complexity which must be handled. Thanks but no thanks. I mean Lua 5.3 still makes sense if you really need 64-bit integers in your Lua scripts. But otherwise it is a downgrade.

fit2rule 10y ago
As a long-term Lua fan, I completely agree with you, even though its not a popular opinion. Too many times I've been debugging some Lua code, tearing my hair out, only to discover "oh yeah, we're using Lua5.3 for this - is that a problem?" .. but at least we're catching up with Python in this regard. /ducks
pritambaral 10y ago
> print(c) // 9.2233720368548e+18 I have little experience in lua myself, and I know only that lua is meant to be a simple language. That said, I can't see from your example what's wrong with lua5.3. I ran the same code with lua5.1, lua5.2, and lua5.3 – and the output was identical across all: $ diff <(lua5.1 numcheck.lua) <(lua5.3 numcheck.lua) $ diff <(lua5.2 numcheck.lua) <(lua5.3 numcheck.lua) $

anonymoushn 10y ago

It looks like the main change in 5.3 is an overhaul of how numbers work to add integers and integer division. This seems like a bad call and I'd rather not deal with it.

rtsisyk 10y ago

Hi, Tarantool[1] team maintains production-ready fork of LuaJIT [2]. Commercial support contract for Tarantool covers problems with LuaJIT as the integral part of the product. [1]: http://tarantool.org/ http://tarantool.org/ [2]: https://github.com/tarantool/luajit https://github.com/tarantool/luajit P.S. there are shortcoming plans to establish a non-profit foundation to continue development of LuaJIT. // Disclaimer, I'm a contributor of [1]