The classic Doug Lea's Malloc with #define ONLY_MSPACES is great for this. That way you can constrain not only Lua's memory usage, but also it's fragmentation.

The classic Doug Lea's Malloc with #define ONLY_MSPACES is great for this. That way you can constrain not only Lua's memory usage, but also it's fragmentation. We'd also manually run a fixed number of GC steps per frame in our games. The higher average CPU use was worth the much lower variance.