4 ms·

One interesting change in this release is the move to a stackful VM. This creates a pretty small (edit: actually huge, 1978 stack frames in a lua 5.4 repl insta

One interesting change in this release is the move to a stackful VM. This creates a pretty small (edit: actually huge, 1978 stack frames in a lua 5.4 repl installed with luaver just now) limit on the depth of the Lua call stack. I wonder what motivated this decision.

moonchild 6y ago
Was it not a stack machine already? The embedding API explicitly exposes a stack, so I assumed that it was implemented that way.
- anonymoushn 6y ago
  it had a stack, but that stack was on the heap. In lua 5.4 under normal circumstances, 20 frames of Lua call stack corresponds to 20ish frames of C call stack. In lua 5.3 the number of C stack frames was some smaller number unrelated to the depth of the Lua call stack.
  - moonchild 6y ago
    Ah - so it uses the C native stack.
wahern 6y ago
Citation? I don't think anything significant changed in that regard. Coroutines are still "stackless" in terms of the C stack, but "stackful" in terms of the language semantics. The virtual data stack is still clearly on the heap. Lua 5.4 adds a new function, lua_setcstacklimit, but this merely exposes what was a compile-time constant, LUAI_MAXCCALLS, in earlier versions. Lua can't avoid making some use of the C stack as it supports intermingling of lua_call's from C code with in-language VM calls, which necessarily will use some amount of C stack. Lua has lua_callk for yield/resuming across C code invocations, but not everybody makes use of this and in any event it's only for coroutine semantics, not for recursion.
- shakna 6y ago
  > Citation? I don't think anything significant changed in that regard. The limits for recursion have undertaken a significant change. (Though it might not be that concerning, if you see my sibling answer). > As already discussed, Lua 5.4 is using a "non stackless" implementation. That means that it uses the C stack for recursion in Lua functions. Because of that, the maximum depth of recursion is much lower in 5.4 than it was in 5.3. - Roberto Ierusalimschy [0] [0] http://lua-users.org/lists/lua-l/2019-06/msg00083.html http://lua-users.org/lists/lua-l/2019-06/msg00083.html
- anonymoushn 6y ago
  See this prior discussion https://news.ycombinator.com/item?id=23297500 https://news.ycombinator.com/item?id=23297500 and this commit https://github.com/lua/lua/commit/196c87c9cecfacf978f37de4ec69eba0a5971256 https://github.com/lua/lua/commit/196c87c9cecfacf978f37de4ec...
- TheCycoONE 6y ago
  In CorsixTH we're hitting C Stack Overflow issues on 5.4. I'm having particular trouble with our serializer for saving games which includes recursive calls to lua_gettable.
shakna 6y ago
I believe, but am not certain, that the change was made to help speed up Lua. As an embedded language, and a language that runs on embedded devices, performance is one of the key criteria of design that goes into the language. The compile-to-bytecode step may have seen some improvement with the change. So instead of speculating on why the change happened, I've found a few bits of information around the limits that make me feel like it often simply won't be a problem. There's some interesting discussion around the limits across various platforms here [0]. Whilst I'm not entirely clear on the motivation, even Lua running inside a browser under emcripten has a stack limit of over 4000, which is somewhat decent even for recursive functions. Whereas on Linux you seem to be able to tweak a config when building and safely have it in the region of 20,000, and quite possibly more. Even under a tight restriction, Lua seems to cope fairly well with it when recursing (the example crashed at 1758 recursions for a ulimit of 400). Whilst Python has a terrible limit of 1000 for the default recursion limit which is somewhat comparable to this aggressive test, Python also has no tail call elimination - but Lua does. I have a somewhat large Lua project (6000 lines of C, 10,000 lines of Lua) that is a recursive parser for a format that I passionately hate (think self-modifying LaTeX). With the default 5.4 limits it never hits a point where it crashes, though I had expected it to. [0] http://lua-users.org/lists/lua-l/2019-06/msg00083.html http://lua-users.org/lists/lua-l/2019-06/msg00083.html
- schwartzworld 6y ago
  Doesn't Lua have tail call recursion built in to the language? Stack overflows shouldn't be an issue on properly written recursive functions.
  - shakna 6y ago
    Yes, it has tail call elimination (which I think I mentioned). But you can still right badly behaved recursive functions that avoid tail call elimination, and that's when you'll hit your stack overflow. I have a few of those badly behaved recursive functions in the parser I mentioned at the end of my comment, and expected them to overflow, but they didn't hit the default limit.
- anonymoushn 6y ago

moonchild 6y ago

Was it not a stack machine already? The embedding API explicitly exposes a stack, so I assumed that it was implemented that way.

anonymoushn 6y ago
it had a stack, but that stack was on the heap. In lua 5.4 under normal circumstances, 20 frames of Lua call stack corresponds to 20ish frames of C call stack. In lua 5.3 the number of C stack frames was some smaller number unrelated to the depth of the Lua call stack.
- moonchild 6y ago
  Ah - so it uses the C native stack.

wahern 6y ago

Citation? I don't think anything significant changed in that regard. Coroutines are still "stackless" in terms of the C stack, but "stackful" in terms of the language semantics. The virtual data stack is still clearly on the heap. Lua 5.4 adds a new function, lua_setcstacklimit, but this merely exposes what was a compile-time constant, LUAI_MAXCCALLS, in earlier versions. Lua can't avoid making some use of the C stack as it supports intermingling of lua_call's from C code with in-language VM calls, which necessarily will use some amount of C stack. Lua has lua_callk for yield/resuming across C code invocations, but not everybody makes use of this and in any event it's only for coroutine semantics, not for recursion.

shakna 6y ago
> Citation? I don't think anything significant changed in that regard. The limits for recursion have undertaken a significant change. (Though it might not be that concerning, if you see my sibling answer). > As already discussed, Lua 5.4 is using a "non stackless" implementation. That means that it uses the C stack for recursion in Lua functions. Because of that, the maximum depth of recursion is much lower in 5.4 than it was in 5.3. - Roberto Ierusalimschy [0] [0] http://lua-users.org/lists/lua-l/2019-06/msg00083.html http://lua-users.org/lists/lua-l/2019-06/msg00083.html
anonymoushn 6y ago
See this prior discussion https://news.ycombinator.com/item?id=23297500 https://news.ycombinator.com/item?id=23297500 and this commit https://github.com/lua/lua/commit/196c87c9cecfacf978f37de4ec69eba0a5971256 https://github.com/lua/lua/commit/196c87c9cecfacf978f37de4ec...
TheCycoONE 6y ago
In CorsixTH we're hitting C Stack Overflow issues on 5.4. I'm having particular trouble with our serializer for saving games which includes recursive calls to lua_gettable.

shakna 6y ago

I believe, but am not certain, that the change was made to help speed up Lua. As an embedded language, and a language that runs on embedded devices, performance is one of the key criteria of design that goes into the language. The compile-to-bytecode step may have seen some improvement with the change. So instead of speculating on why the change happened, I've found a few bits of information around the limits that make me feel like it often simply won't be a problem. There's some interesting discussion around the limits across various platforms here [0]. Whilst I'm not entirely clear on the motivation, even Lua running inside a browser under emcripten has a stack limit of over 4000, which is somewhat decent even for recursive functions. Whereas on Linux you seem to be able to tweak a config when building and safely have it in the region of 20,000, and quite possibly more. Even under a tight restriction, Lua seems to cope fairly well with it when recursing (the example crashed at 1758 recursions for a ulimit of 400). Whilst Python has a terrible limit of 1000 for the default recursion limit which is somewhat comparable to this aggressive test, Python also has no tail call elimination - but Lua does. I have a somewhat large Lua project (6000 lines of C, 10,000 lines of Lua) that is a recursive parser for a format that I passionately hate (think self-modifying LaTeX). With the default 5.4 limits it never hits a point where it crashes, though I had expected it to. [0] http://lua-users.org/lists/lua-l/2019-06/msg00083.html http://lua-users.org/lists/lua-l/2019-06/msg00083.html

schwartzworld 6y ago
Doesn't Lua have tail call recursion built in to the language? Stack overflows shouldn't be an issue on properly written recursive functions.
- shakna 6y ago
  Yes, it has tail call elimination (which I think I mentioned). But you can still right badly behaved recursive functions that avoid tail call elimination, and that's when you'll hit your stack overflow. I have a few of those badly behaved recursive functions in the parser I mentioned at the end of my comment, and expected them to overflow, but they didn't hit the default limit.
anonymoushn 6y ago