4 ms·

Despite having an additional dependency (Tcl; but is this really such a big dependency?), can you maybe share other experiences of why this was a bad idea? The

• flohofwoe 6y ago
  The main problem is allowing "free scripting", because those script snippets are hard to automatically convert to something else. But if you don't allow or need that, why have the runtime overhead (in performance and size) of a scripting engine for loading data at all. But even when the scripting engine has already been integrated for other things it's still a bad idea to use it for data files, unless you forbid any sort of human scripting (so basically like JSON vs Javascript). If you have pure data file formats, like JSON, XML or even your own custom binary formats, it's trivial to upgrade or change them (such as switching from XML to JSON to your own binary format and back). But once you have human-scripted parts such an automatic conversion suddenly gets much harder or impossible, since now you need to transpile from one scripting language to another. That's pretty much the worst case, but the problems already start when APIs or data layouts change. If a data item goes away, it's trivial to automatically inspect a pure data representation like JSON and remove or ignore such obsolete data items. But with free scripting, you suddenly get an obscure scripting runtime error. And finally, if you want to process such mixed data / code files in another tool, you always need to integrate that specific scripting engine, which is a much bigger dependency than (for instance) a simple JSON or XML reader/writer, or even just your own binary file format readers/writers. For instance the Emscripten SDK's .emscripten config file suffers from this: at a glance this looks like a simple text file with key/value string pairs, but on closer inspection it must actually be evaluated through a python interpreter to be any useful. So what could be a simple text parsing tool to extract filesystem paths from this config file suddenly requires a full blown python environment. So basically, by allowing "turing complete" config or data file formats, you make future changes much harder then they need to be. Of course all of this isn't all that important for small throwaway projects. But sometimes those small projects (or quick'n'dirty solutions in general) become much more important and longer-lived than expected.
  • lioeters 6y ago
    As someone who's had a love-hate thing with domain-specific languages - reading your story was a good learning experience. There's great value in sticking with "inert" data formats like XML or JSON, which have standard parsers in most languages, developed over years. Once data crosses over into code, there's a whole host of potential issues to consider, including the requirement of an interpreter/compiler and a sandboxed environment, with all that entails. The Emscripten config needing Python is a perfect example. Another example I thought of, is Markdown and MDX. The latter allows the use of JavaScript and React components mixed with markup, for dynamic content. It's brilliant in its own way, but I'm wary of using it as a document format precisely because it's no longer just data. Documents in MDX can't be consumed by other languages, and will always require the full JS/React environment. Tcl actually feels like it could reasonable to use for data, due to its ubiquity/longevity. Lua, or better yet some kind of Lisp, also seem attractive as a serialization format. The interpreter could be small and stable, and the latter at least might even be simpler to parse than XML or JSON. > We still have to drag a stripped down Tcl interpreter along in our asset pipeline 20 years later because it's nearly impossible to switch away This makes me see how the decision created a long-term dependency to a language, where the data was no longer in a portable format. Good to keep this in mind next time I'm tempted to allow data to become code.