5 ms·
Pointers do not ever 'move'. You also should never store variables (to userdata, memory chunks, strings, etc.) by pointers or anything like that according to do
by FRex 9y ago
Pointers do not ever 'move'. You also should never store variables (to userdata, memory chunks, strings, etc.) by pointers or anything like that according to documentation because they might be collected at any time. To stuff like tables or values you never ever get a pointer and having a pointer to a "Lua value" would make no sense because it's a small struct with just the type number (in case of nil that's all that matters actually) and then either an inline value (light userdata, bool, number, etc.) or a pointer to the actual GC object (thread, string, table, function, internal data structures, etc.). Lack of exposing internal data structures (there is Buffer in luaL but it's the only one I can think of and it's so simple and just to collect few results together it doesn't matter) is probably the reasons why LuaJIT is ABI and API compatible with Lua 5.1 despite implementing stuff very differently internally (like values in NaN tagged doubles to keep them in a 64 bit boundary instead of just placing fields normally in structs and letting compiler figure it out like stock Lua does).
There is a feature in luaL part of the C API called 'refs' but that's basically storing stuff in a table array and addressing it by ints and it's done all using the C API itself, it's not a VM or language feature.
There is also a weak reference system (where table can be marked to have key, value or both be 'weak' and allow collection if there are no 'strong' references anywhere to these objects) and that is part of the VM itself (GC specifically, the table processing code) and has some caveats with stuff like strings (they are interned in Lua) and it doesn't remove intelligently form tables serving as arrays.
To call a function from C you push it and its arguments onto the stack (of the main thread or the coroutine) and execute it with one of API functions (depending on what you're trying to do)[1], to call it in Lua you 'just' call it or pass it to pcall or a when making a coroutine.
A C function exposed to Lua like that must implement its 'protocol'[2] which is getting and returning arguments by interacting with the lua state pointer so it's an extra effort to wrap them or you can use a third party library (there are some, especially C++ ones which use templates to automatically wrap stuff, but I don't use any myself).
C and Lua functions are indistinguishable in most cases, they are called the same way, etc. but sometimes they differ, i.e. Lua functions can be dumped with string.dump.
There is no FFI (as there is in C#, JNI, etc.) as such. LuaJIT has an FFI that parses C declarations and it's quite nice IMO. There are also third party FFI libs but I know nothing about them. Using that also loses you all the nice to haves that Lua has like multiple return values (it's customary to return false or nil and an error message string on errors from many functions and it's impossible to write an iterator without multiple return values), interacting with lua state in the function, arg type check calls with automatic errors, etc.
The GC is a mark and sweep one, in 5.0 it was a stop the world one - two color, from 5.1 onwards it's incremental tri color with forward and back write barriers(see [0] for an explanation of what that means and how it works). In 5.2 there was also (off by default) generational one. I have no idea how that one worked and it was removed in 5.3. LuaJIT also uses mark and sweep from 5.1 but with improvements and for 3.0 there was supposed to be a quad color one. You of course have full control over it (stop, start, adjust speed, force a full cycle or a single step, etc.) to complete right now) from Lua and C via the right function. It does nothing fancy like compaction or copying/moving.
You are also responsible for the memory allocation (in 64 bit LuaJIT it's taken over and attempting to do that is an error) and the default alloc is just a malloc wrapper[3]. From 5.2 onwards you also get told during initial (i.e. not the resize) allocation what kind of Lua object is being allocated (it doesn't tell you about internal structures though, only the user visible Lua types) so that might help you make some decisions in the allocator. Lua VM never allocs memory or touches the system on its own in any way either.
If you want you can remove all the standard libs like os, io, etc. and not use any of luaL functions (they all are implemented using Lua C API only so they are 'userspace' and optional, not part of VM, they do not use internals of the VM itself). Even the require mechanism for importing libs is all userspace implemented (unlike like Python where it's part of the language, I think). It's also possible to recompile Lua and change what type the number is (double so 64 bit by default, plus 64 bit int in 5.3, there is convenient macro to force both to 32 bit in 5.3, for LuaJIT I'm not sure, it uses NaN tagging in a double and a handwritten asm to get its speed and compactness and I'm not familiar with its codebase like I am with the vanilla Lua), add a bit of extra space in front of the VM for direct instant access from C (space for 1 pointer by default in 5.3, no space before that) and a bunch of other configurable things. The codebase of PUC-Rio Lua (i.e. not some third party implementation like LuaJIT or Havok Script[4] - which I think they don't even sell but that Google still indexes and they have a testimony on the front page for...) is also easy to jump into if you know C and very small (c.a. 10k lines if you ignore the lexer+parser+compiler part, std libs and standalone lua and luac program sources).
Feel free to ask about anything if I was unclear. I am knee deep in Lua by now and I've even written about Lua internals for my engineer's thesis (in Polish) so I'd say I'm quite up there with my knowledge of its working and history and such compared to average Lua knowing person. I thought about recycling my notes to write an article about that but I don't really see a point because I'd largely be rahashing between source code itself (very readable!) and various resources like no frills intro to vm instructions[5], articles from the three Brazilian devs themselves, mailing lists, details on LuaJIT page, etc. If someone is interested about such an article then you can drop me an email( my email address is visible once you're logged in at https://github.com/FRex/ https://github.com/FRex/ ) and if I get a few I might do it.
[0] - http://wiki.luajit.org/New-Garbage-Collector#introduction http://wiki.luajit.org/New-Garbage-Collector#introduction
[1] - http://www.lua.org/manual/5.3/manual.html#lua_call http://www.lua.org/manual/5.3/manual.html#lua_call
[2] - http://www.lua.org/manual/5.3/manual.html#lua_CFunction http://www.lua.org/manual/5.3/manual.html#lua_CFunction
[3] - http://www.lua.org/manual/5.3/manual.html#lua_Allo http://www.lua.org/manual/5.3/manual.html#lua_Allo
[4] - https://www.havok.com/wp-content/uploads/Havok_Brochures/Havok_Script_2015.pdf https://www.havok.com/wp-content/uploads/Havok_Brochures/Hav...
[5] - http://luaforge.net/docman/83/98/ANoFrillsIntroToLua51VMInstructions.pdf http://luaforge.net/docman/83/98/ANoFrillsIntroToLua51VMInst...
- daurnimator 9y agoFYI Lua 5.4 is bringing back a generational collector in a slightly different style. Also, what is your thesis?
- FRex 9y agoI don't really follow the development of Lua in any way. Lua 5.3 was very underwhelming to me, I didn't care for the ints and bit operators at all (and before 5.3 the one number type was a selling point) and it's annoying how there is such a clear strong split between 5.3 and LuaJIT because it effectively finalized the fork that 5.2 has sort of started. Some things (like Love2D, OpenResty and other performance conscious stuff) stuck with LuaJIT so basically a 5.1 on steroids with few 5.2 features. 5.1 also had the killer feature of incremental GC and came at a nice time and was the main Lua for almost 6 years so it stuck into gamedev a lot (which is where I picked Lua from, my hobbist gamedeving). I also stick to 5.1 by default in my hobby gamedev to be able to bring in LuaJIT at any time and ignore 5.2 and 5.3 pretty much. My thesis was about history of Lua a bit (rehashing what's on the main website plus few Roberto's articles pretty much) and then a lot about how stock Lua 5.1 is implemented (most of it is 1:1 transferable to other stock Luas) like how bytecode is executed, how does GC work, what does it mean Lua is register VM and not stack one, how are internal structures laid out, how functions and closures and threads work, etc. It's (official) English title was "Analysis of code of original implementation of Lua 5.1" but it is written in Polish ("Analiza kodu oryginalnej implementacji jezyka Lua 5.1"). I don't think it's available anywhere. That was my engineering (inżynier) thesis which I defended in February and I'm now doing a master thesis but it's purely mathematical CS stuff and not related to Lua at all.
- marktangotango 9y agoAs an aside, there’s stuff on the lua wiki about sandboxing lua, how would you go about sandboxing lua to limit heap, cpu cycles, and os level api calls? For lua 5.1? For luajit? An example use case would be letting users upload scripts to run in game?
- 9y ago