2 ms·
About 20 years ago, I used Tcl as object serialization format in a game engine, and then made the next mistake to allow injecting manually written 'scriptlets'
by flohofwoe 6y ago
About 20 years ago, I used Tcl as object serialization format in a game engine, and then made the next mistake to allow injecting manually written 'scriptlets' into this serialization format (after all, serialized objects were just Tcl scripts).
These two things combined were probably the biggest software design mistake I ever made. We still have to drag a stripped down Tcl interpreter along in our asset pipeline 20 years later because it's nearly impossible to switch away (because of the manually written 'scriptlets' lurking everywhere in asset source files - of course it would be possible, but it was never on the right side of the cost-benefit equation).
Never mix code and data, folks :)
- enriquto 6y agoa stripped down tcl interpreter is one of the tiniest and less harmful dependendencies you can have nowadays. I do not see a problem with that. And code is data, and has always been.
- kstenerud 6y agoUntil someone sends serialized code-data like: for i = 1, 1000000 do for j = 1, 1000000 do for k = 1, 1000000 do for l = 1, 1000000 do ... Once you move your publicly injectable data format from declarative to something else, you're in for a world of hurt.
- nmadden 6y agoTcl has had sandboxed interpreters with resource limits for a long time [1]. You can even remove built-in commands (for/while etc) if you want to. You can limit the language to a purely declarative subset and selectively expose commands and resources as you wish. [1]: https://www.tcl.tk/man/tcl8.5/TclCmd/interp.htm#M46 https://www.tcl.tk/man/tcl8.5/TclCmd/interp.htm#M46
- dividuum 6y agoIf you use Lua only for your configuration data, patching the Lua interpreter to limit the CPU time is rather easy: Increment a counter within your lua_State struct here: https://github.com/lua/lua/blob/c33b1728aeb7dfeec4013562660e07d32697aa6b/lvm.c#L1150 https://github.com/lua/lua/blob/c33b1728aeb7dfeec4013562660e..., then abort once you reach a limit.
- CyberDildonics 6y agoIf you follow the mantra of 'code is data' then you have arbitrary and difficult to debug execution anywhere you have data. Mixing the two intentionally is a blunder of creating software that seems to be repeated over and over, maybe because people keep encouraging it without realizing the cost in non trivial projects.
- albertzeyer 6y agoDespite having an additional dependency (Tcl; but is this really such a big dependency?), can you maybe share other experiences of why this was a bad idea? The article actually suggest pro doing this, i.e. using a script engine (Lua here) to store your configs/settings/data, and I agree, it can make certain things much simpler, and more clean.
- flohofwoe 6y agoThe main problem is allowing "free scripting", because those script snippets are hard to automatically convert to something else. But if you don't allow or need that, why have the runtime overhead (in performance and size) of a scripting engine for loading data at all. But even when the scripting engine has already been integrated for other things it's still a bad idea to use it for data files, unless you forbid any sort of human scripting (so basically like JSON vs Javascript). If you have pure data file formats, like JSON, XML or even your own custom binary formats, it's trivial to upgrade or change them (such as switching from XML to JSON to your own binary format and back). But once you have human-scripted parts such an automatic conversion suddenly gets much harder or impossible, since now you need to transpile from one scripting language to another. That's pretty much the worst case, but the problems already start when APIs or data layouts change. If a data item goes away, it's trivial to automatically inspect a pure data representation like JSON and remove or ignore such obsolete data items. But with free scripting, you suddenly get an obscure scripting runtime error. And finally, if you want to process such mixed data / code files in another tool, you always need to integrate that specific scripting engine, which is a much bigger dependency than (for instance) a simple JSON or XML reader/writer, or even just your own binary file format readers/writers. For instance the Emscripten SDK's .emscripten config file suffers from this: at a glance this looks like a simple text file with key/value string pairs, but on closer inspection it must actually be evaluated through a python interpreter to be any useful. So what could be a simple text parsing tool to extract filesystem paths from this config file suddenly requires a full blown python environment. So basically, by allowing "turing complete" config or data file formats, you make future changes much harder then they need to be. Of course all of this isn't all that important for small throwaway projects. But sometimes those small projects (or quick'n'dirty solutions in general) become much more important and longer-lived than expected.