5 ms·

It's not just a tiny C compiler - it's a compiler you can use as a static library that can compile your C code directly to memory. And then you can call into it

It's not just a tiny C compiler - it's a compiler you can use as a static library that can compile your C code directly to memory. And then you can call into it, if you dare. I used it as a scripting backend for a long time, but eventually you will realize that you are not the only one that could write scripts, but you also really need to have a trust boundary (or just a different address space), so that errors in the script don't drag the whole thing down. That's where Lua and LuaJIT shines: It's simple, and it's sandboxed. However, there is still one thing missing. These days, several programming languages have several targets that they can compile to, and so imagine if you could emulate some platform at high performance with low overhead, in a sandbox. You would then be able to script in whatever language you so desire, provided it has that target. Unfortunately, those languages tend to be system languages and not the absolute best for scripting. With one big exception: Compile-time programming support.

BubRoss 6y ago
Have you actually gotten the static library and compile to memory mode to work?
- saagarjha 6y ago
  I have tried the latter and it seems to work fine.
- fwsgonzo 6y ago
  Yes, I used it in a game engine for many years, on Windows and Linux. No problems on that part. I don't really recommend doing it, unless you want to do it as a learning experiment. It was a fun thing to do.
- aidenn0 6y ago
  I got it to work as a DLL for calling into C from common-lisp when I needed something that was header-file dependent. It was really just a proof of concept though because if you distribute source, you can require that they have CC installed just as easily as libtcc, and if you distribute a binary, you can grovel ahead of time and don't need libtcc...
rubber_duck 6y ago
I feel like embedding a web assembly runtime and then exposing scripting API to it would give you similar benefits (you would have to add a compile C to WASM step before loading) while giving you a really secure sandbox. It is more complicated than just embedding a scripting language (or a scripting compiler :)) but more general and secure. Especially if WASM evolves to a point where you can compile C# and other high level languages to it (you can to a point already but it's not on par with native runtimes) - it would be the most general embedding runtime.
- makapuf 6y ago
  Is a wasm runtime complex to write? Not with jit or anything fancy, is it at the level of making a GB emulator or a llvm backend ?
haberman 6y ago
LuaJIT is not sandboxed. Mike Pall told me this in no uncertain terms: http://lua-users.org/lists/lua-l/2011-02/msg01582.html http://lua-users.org/lists/lua-l/2011-02/msg01582.html > The only reasonably safe way to run untrusted/malicious Lua scripts is to sandbox it at the process level. Everything else has far too many loopholes.
- fwsgonzo 6y ago
  I see, that's unfortunate. I don't have those issues in my emulated environment though. I don't use Lua anymore - I'm running emulated RISC-V in my own emulator. I have been benchmarking against LuaJIT for the longest time because I thought it was an equal in that respect. Guess it should have been against regular Lua.
  - haberman 6y ago
    For what it's worth, I think his statement applies to regular Lua too, see: http://lua-users.org/lists/lua-l/2011-02/msg01595.html http://lua-users.org/lists/lua-l/2011-02/msg01595.html
    - pansa2 6y ago
      I think there’s a difference between trying to prevent untrusted code from escaping a sandbox, and trying to prevent it from locking up the CPU. Lua is commonly used for scripting games. If I load an untrusted script into the game I’m playing and it can access all of my files, it’s potentially catastrophic. OTOH if the worst an untrusted script can do is use 100% CPU (and cause the game I’m playing to lock up so I need to restart), it’s a minor annoyance. AFAIK in the absence of bugs (and regular Lua is pretty close to that [0]), Lua’s sandboxing features are sufficient to protect against the former but not the latter. Am I right to think that’s often good enough? It seems a huge improvement over embedding, say, Python - I wonder how it compares to a embedding a JavaScript engine like V8? [0] https://www.lua.org/bugs.html https://www.lua.org/bugs.html
- wahern 6y ago
  The issue isn't that Lua can't nominally provide a sandboxed environment--it can, better than almost any other language. The central issue is whether LuaJIT, PUC Lua, or any other particular piece of software can be made sufficiently free of bugs that you can trust such a sandbox to run potentially malicious code. The answer in the case of LuaJIT is definitely no, because the JIT engine is sufficiently complex that exploits are inevitable. Note that this is also the case with JavaScript. Many browser exploits start with some codegen bug in V8 or SpiderMonkey. And there are many more eyeballs looking to fix bugs in V8 and SpiderMonkey than LuaJIT, so in the case of LuaJIT the prudent answer is that you should never trust it to run potentially malicious code. The case for PUC Lua is more nuanced. Lua used to have a bytecode verifier, but it was removed for 5.2 because too many bugs were found in the verifier, and because the VM relied heavily on the verifier to filter bad opcode patterns, that led to sandbox breakouts. This made the PUC developers believe that the verifier made Lua worse off as compared to a more wholistic emphasis on correctness and robustness, so they dropped the verifier. They also dropped any pretense that you could safely sandbox bytecode (i.e. precompiled scripts); if you want a sandbox in Lua you should only load untrusted code as plain Lua scripts into the sandboxed environment. To that end Lua 5.2 added a parameter to all APIs for loading code that specified whether to accept text scripts or binary bytecode. In other words, the bytecode verifier was considered a third wheel, so they removed it and redirected attention to the compiler and the rest of the VM. So for PUC Lua the issue really comes down to how prudent it is to draw a trust boundary around a pure Lua sandbox; or rather, what your adversarial model is, precisely. PUC Lua is committed to Lua's sandboxing features, but many developers are fairly of the opinion that the only way to run untrusted code, if you're to run untrusted code at all, is using either a hardware VM or a very strict seccomp jail. If you're of the latter opinion, the language is irrelevant--you shouldn't trust Lua, JavaScript, Java, or any other language environment, period. In practice, however, even people of the latter opinion generally apply the principle of defense in depth. That's why browser JavaScript APIs and capabilities are still relatively limited, even though browsers execute JavaScript in OS-based sandboxes. In most practical contexts Lua's sandboxing features still provide great value; it just needs to be understood that they're a complement to rather than substitute for process sandboxing. The same analysis applies to WebAssembly, FWIW, especially JIT'ing WASM environments. Anybody who thinks WASM is a magic cureall for running untrusted code is mistaken.
monocasa 6y ago
You can see this technique being used in TCCBOOT, using tcc as a boot loader to compile A Linux kernel into memory and then run it in a handful of seconds. https://bellard.org/tcc/tccboot.html https://bellard.org/tcc/tccboot.html
- SomeoneFromCA 6y ago
  Yeah well, modern Linux cannot be compiled in 15 seconds. The link is from 2004. OTOH, I wonder how quickly modern Ryzen would be able to boot 2004 kernel from the source.
  - monocasa 6y ago
    Kernel compile time has apparently stayed pretty static over the years (or at least tracked with perf improvements). That being said, I think that takes into account SMP which probably wouldn't be easily accessible to tccboot.
  - tyingq 6y ago
    The Threadripper 3970x can compile the 5.x kernel in ~24 seconds. That's not too shabby. https://www.phoronix.com/scan.php?page=article&item=amd-linux-3960x-3970x&num=9 https://www.phoronix.com/scan.php?page=article&item=amd-linu...
karteum 6y ago
> and so imagine if you could emulate some platform at high performance with low overhead, in a sandbox. You would then be able to script in whatever language you so desire, provided it has that target. My understanding is that this is what Google's PNaCl was about (https://developer.chrome.com/native-client/nacl-and-pnacl https://developer.chrome.com/native-client/nacl-and-pnacl) : leveraging LLVM bitcode for both portability (source and target) and native performance, being able to sandbox properly the resulting executable.

BubRoss 6y ago

Have you actually gotten the static library and compile to memory mode to work?

saagarjha 6y ago
I have tried the latter and it seems to work fine.
fwsgonzo 6y ago
Yes, I used it in a game engine for many years, on Windows and Linux. No problems on that part. I don't really recommend doing it, unless you want to do it as a learning experiment. It was a fun thing to do.
aidenn0 6y ago
I got it to work as a DLL for calling into C from common-lisp when I needed something that was header-file dependent. It was really just a proof of concept though because if you distribute source, you can require that they have CC installed just as easily as libtcc, and if you distribute a binary, you can grovel ahead of time and don't need libtcc...

rubber_duck 6y ago

I feel like embedding a web assembly runtime and then exposing scripting API to it would give you similar benefits (you would have to add a compile C to WASM step before loading) while giving you a really secure sandbox. It is more complicated than just embedding a scripting language (or a scripting compiler :)) but more general and secure. Especially if WASM evolves to a point where you can compile C# and other high level languages to it (you can to a point already but it's not on par with native runtimes) - it would be the most general embedding runtime.

makapuf 6y ago
Is a wasm runtime complex to write? Not with jit or anything fancy, is it at the level of making a GB emulator or a llvm backend ?

haberman 6y ago

LuaJIT is not sandboxed. Mike Pall told me this in no uncertain terms: http://lua-users.org/lists/lua-l/2011-02/msg01582.html http://lua-users.org/lists/lua-l/2011-02/msg01582.html > The only reasonably safe way to run untrusted/malicious Lua scripts is to sandbox it at the process level. Everything else has far too many loopholes.

fwsgonzo 6y ago
I see, that's unfortunate. I don't have those issues in my emulated environment though. I don't use Lua anymore - I'm running emulated RISC-V in my own emulator. I have been benchmarking against LuaJIT for the longest time because I thought it was an equal in that respect. Guess it should have been against regular Lua.
- haberman 6y ago
  For what it's worth, I think his statement applies to regular Lua too, see: http://lua-users.org/lists/lua-l/2011-02/msg01595.html http://lua-users.org/lists/lua-l/2011-02/msg01595.html
  - pansa2 6y ago
    I think there’s a difference between trying to prevent untrusted code from escaping a sandbox, and trying to prevent it from locking up the CPU. Lua is commonly used for scripting games. If I load an untrusted script into the game I’m playing and it can access all of my files, it’s potentially catastrophic. OTOH if the worst an untrusted script can do is use 100% CPU (and cause the game I’m playing to lock up so I need to restart), it’s a minor annoyance. AFAIK in the absence of bugs (and regular Lua is pretty close to that [0]), Lua’s sandboxing features are sufficient to protect against the former but not the latter. Am I right to think that’s often good enough? It seems a huge improvement over embedding, say, Python - I wonder how it compares to a embedding a JavaScript engine like V8? [0] https://www.lua.org/bugs.html https://www.lua.org/bugs.html
wahern 6y ago
The issue isn't that Lua can't nominally provide a sandboxed environment--it can, better than almost any other language. The central issue is whether LuaJIT, PUC Lua, or any other particular piece of software can be made sufficiently free of bugs that you can trust such a sandbox to run potentially malicious code. The answer in the case of LuaJIT is definitely no, because the JIT engine is sufficiently complex that exploits are inevitable. Note that this is also the case with JavaScript. Many browser exploits start with some codegen bug in V8 or SpiderMonkey. And there are many more eyeballs looking to fix bugs in V8 and SpiderMonkey than LuaJIT, so in the case of LuaJIT the prudent answer is that you should never trust it to run potentially malicious code. The case for PUC Lua is more nuanced. Lua used to have a bytecode verifier, but it was removed for 5.2 because too many bugs were found in the verifier, and because the VM relied heavily on the verifier to filter bad opcode patterns, that led to sandbox breakouts. This made the PUC developers believe that the verifier made Lua worse off as compared to a more wholistic emphasis on correctness and robustness, so they dropped the verifier. They also dropped any pretense that you could safely sandbox bytecode (i.e. precompiled scripts); if you want a sandbox in Lua you should only load untrusted code as plain Lua scripts into the sandboxed environment. To that end Lua 5.2 added a parameter to all APIs for loading code that specified whether to accept text scripts or binary bytecode. In other words, the bytecode verifier was considered a third wheel, so they removed it and redirected attention to the compiler and the rest of the VM. So for PUC Lua the issue really comes down to how prudent it is to draw a trust boundary around a pure Lua sandbox; or rather, what your adversarial model is, precisely. PUC Lua is committed to Lua's sandboxing features, but many developers are fairly of the opinion that the only way to run untrusted code, if you're to run untrusted code at all, is using either a hardware VM or a very strict seccomp jail. If you're of the latter opinion, the language is irrelevant--you shouldn't trust Lua, JavaScript, Java, or any other language environment, period. In practice, however, even people of the latter opinion generally apply the principle of defense in depth. That's why browser JavaScript APIs and capabilities are still relatively limited, even though browsers execute JavaScript in OS-based sandboxes. In most practical contexts Lua's sandboxing features still provide great value; it just needs to be understood that they're a complement to rather than substitute for process sandboxing. The same analysis applies to WebAssembly, FWIW, especially JIT'ing WASM environments. Anybody who thinks WASM is a magic cureall for running untrusted code is mistaken.

monocasa 6y ago

You can see this technique being used in TCCBOOT, using tcc as a boot loader to compile A Linux kernel into memory and then run it in a handful of seconds. https://bellard.org/tcc/tccboot.html https://bellard.org/tcc/tccboot.html

SomeoneFromCA 6y ago
Yeah well, modern Linux cannot be compiled in 15 seconds. The link is from 2004. OTOH, I wonder how quickly modern Ryzen would be able to boot 2004 kernel from the source.
- monocasa 6y ago
  Kernel compile time has apparently stayed pretty static over the years (or at least tracked with perf improvements). That being said, I think that takes into account SMP which probably wouldn't be easily accessible to tccboot.
- tyingq 6y ago
  The Threadripper 3970x can compile the 5.x kernel in ~24 seconds. That's not too shabby. https://www.phoronix.com/scan.php?page=article&item=amd-linux-3960x-3970x&num=9 https://www.phoronix.com/scan.php?page=article&item=amd-linu...

karteum 6y ago

> and so imagine if you could emulate some platform at high performance with low overhead, in a sandbox. You would then be able to script in whatever language you so desire, provided it has that target. My understanding is that this is what Google's PNaCl was about (https://developer.chrome.com/native-client/nacl-and-pnacl https://developer.chrome.com/native-client/nacl-and-pnacl) : leveraging LLVM bitcode for both portability (source and target) and native performance, being able to sandbox properly the resulting executable.