7 ms·

No -- you're confusing generation and parsing. Your program is still (often, but necessarily) going to consume a static data structure. Properly handled, using

No -- you're confusing generation and parsing. Your program is still (often, but necessarily) going to consume a static data structure. Properly handled, using code to generate a config at runtime can reduce the number of `if` statements and weird logic you have inside of your main program. Purely as an example, let's say I'm writing a program that wants to consume a list of files in a JSON config. { "files": ["/path/a", "/path/b"] } Now, let's say I want to pass in a wildcard pattern, or I want to exclude a specific directory, or I want to dynamically load the file paths based on a REST request, or some kind of other complicated crap. If I want to have a static JSON file as the config, I have a couple of options: First, I can put logic inside of my program to handle wildcards, and to distinguish between filepaths and URLs, and to have exclude paths. This is not ideal because it makes my program much more complicated, and it's not even complication for a particular useful reason. 99% of the time I'm not going to need those options, so I'm writing all of this code to try and anticipate weird inputs that most people don't need. It also forces me to make opinionated decisions about stupid stuff like, "what should the glob format be? What Regex variant are we using?" None of these are questions I care about or want to answer. The second option, which would be a lot better, is I could have a 3rd-party program generate a config on the fly, write it to disk, and then launch my actual program when it's done. This means I don't need to have a bunch of extra logic in my main program, and everything is a lot less opinionated. BUT, it also means that the logic to launch my program is split up across multiple files, and now I'm adding some weird dependencies to my runtime. It's a hacky solution that's layering a bunch of complexity on top of what should be a simple, "launch the program." ---- So, what's a third option? I write my config file with code, which my program will execute, and which will return the original static config that's nice to consume. module.exports = { files: [ //some big list of files, maybe populated with a glob. //exclude any filepath that contains the letter 'c' ].filter((file) => !file.includes('c')); }; I still need the code to spit out an array of files at the end, but I can do that after making some network requests, or running custom filter functions on that array, or doing really whatever the heck I want. In many cases this does not introduce any new dependencies to my runtime. It's a lot less error prone -- I can launch multiple instances of my app at the same time without worrying about files overwriting each other. It's also a lot cleaner and easier to refactor, because all of my filter functions and Regexes and generation logic lives in one file. And most importantly, for users who aren't taking advantage of the dynamic nature of the config, I'm still ultimately shipping a smaller, simpler program with a less complicated API. I don't need to ship a bunch of error-prone logic dedicated to reconciling 20 different flags or options from random pull requests. ---- Now, taking this a step further, you don't necessarily need to return a static data object. If I'm building a static site generator, maybe instead of embedding a bunch of logic in my system to load/parse Pug templates, instead I just consume a function from the config that takes a string and returns a string. And the reason that might be attractive (beyond what I list above) is that the programmatic APIs for a library like Pug are already just as short and easy to use as any set of YAML options you can come up with. So instead of forcing your users to learn your new APIs to slot in a custom template engine, they just use the APIs they already know and enjoy, and your documentation is shorter and easier to read, and everyone wins. And again, importantly, getting rid of that extra logic has reduced the amount of code in your main project and made stuff like validation and loading easier to reason about and easier to debug. The way you want to think about a config that returns a function is the same way you already think about functions-as-parameters in functional languages like Lisp or Javascript. The trick is realizing that there is no real difference between a config file on a filesystem and an API for a library you're importing at runtime.

pdonis 7y ago
> you're confusing generation and parsing I understand the distinction you are making, but I don't think it's what the article is illustrating in its examples. > I write my config file with code, which my program will execute If you, the developer, are writing this code (for example, to provide some pre-packaged standard configs for different use cases, and the user just selects which one they want to use), that's one thing. If your users are writing this code, you have a whole new set of problems that you don't have if users can only provide text config files in a safe format. I can see providing a separate config editor program users can use to generate configs for your main program; you can write the code for that program to ensure that the generated config files are safe. But such a program would be generating configs separate from the running of your main program. If your configs are really complicated enough that they need to be generated at the time your main program runs, to me that means your code needs refactoring. Either that, or what you are calling a "config" is really a plugin/extension/scripting system, where your users can write their own code and your main program is more of an interpreter for it. That's a whole separate use case that I don't think is well described by the term "config". Certainly I would say that's the case if your "config" can return a function to your main program.
- danShumway 7y ago
  > If your configs are really complicated enough that they need to be generated at the time your main program runs, to me that means your code needs refactoring. I'm not sure that's true. In fact, I would go so far as to say for some programs, it's the opposite. In the example I describe above (forgetting about lambda functions or anything like that) I'm shipping a program that consumes a small set of very rigid input options. I consume a single array of strings that need to resolve to file paths. That's incredibly easy to validate: that's the type of limited input where I can be pretty certain that my program will not have any bugs or vulnerabilities around the input. It's also an incredibly simple API to teach to users, it's literally one option. And importantly, my program is still pretty powerful -- you can do whatever you need to do to get that file list without me needing to extend what input I accept. I can't think of a good way to convert that to a pure JSON/YAML config without either making my software more error prone or making my software much less useful. If returning lambda functions from a config file scares you, forcing the config to be a static object literal but allowing that object literal to be generated at runtime will still (often) result in a runtime that is less error prone, easier to validate, and easier to debug when something goes wrong. This is because at the end of the user's dynamic config, they'll still be returning a small, easy-to-understand object literal -- whatever logic was used to build it. If instead you put that logic in your main program, and you accept globs for files, and you start doing complicated things like allowing arrays of multiple globs and file excludes -- well, suddenly you have a big chunk of parsing logic in your main program that end-users can't debug when something goes wrong. It becomes harder to validate whether or not you have some bug around the order that patterns are passed to you, or whether there's an error if you try to use a globbed filepath as an excluded filepath, or whatever. In other words, it is usually better to use procedural logic to generate declarative logic than it is to use declarative logic to generate procedural logic. Of course, your program might be simple enough that you can get away with a config file that's turned into a bunch of `if` statements when its parsed. But it probably would be simpler if you got rid of as much parsing code as possible and focused on more rigidly validating a smaller, stricter set of options. > I can see providing a separate config editor program users can use to generate configs for your main program If you ignore the stuff about lambda functions, that's exactly what we're doing. We're just piggybacking on a real language to do it instead of wasting time writing our own graphical/command-line utility. ---- Where returning functions is concerned: > Either that, or what you are calling a "config" is really a plugin/extension/scripting system I think this is a kind of blurry line. I'm not going to say there is no difference between a plugin system and a config, but I have run into programs that have both configs and plugin systems, and where the plugin system exists mostly to deal with the fact that the config API is not very useful. As a user, given the choice between learning one API and two, I usually prefer to learn one API. A lot of static site generators come to mind; I don't want to learn two different APIs for a templating engine just so I can change how links render. I don't want to learn both your plugin API and your config API. Just give me one, single API for templates that can be fully described in a single paragraph of text. I dunno. I've been doing functional programming long enough that the idea of lambdas just doesn't scare me. Functional paradigms are wildly useful when building library APIs that get imported; I don't see why we should throw them away just because our API lives in a dotfile.

pdonis 7y ago

> you're confusing generation and parsing I understand the distinction you are making, but I don't think it's what the article is illustrating in its examples. > I write my config file with code, which my program will execute If you, the developer, are writing this code (for example, to provide some pre-packaged standard configs for different use cases, and the user just selects which one they want to use), that's one thing. If your users are writing this code, you have a whole new set of problems that you don't have if users can only provide text config files in a safe format. I can see providing a separate config editor program users can use to generate configs for your main program; you can write the code for that program to ensure that the generated config files are safe. But such a program would be generating configs separate from the running of your main program. If your configs are really complicated enough that they need to be generated at the time your main program runs, to me that means your code needs refactoring. Either that, or what you are calling a "config" is really a plugin/extension/scripting system, where your users can write their own code and your main program is more of an interpreter for it. That's a whole separate use case that I don't think is well described by the term "config". Certainly I would say that's the case if your "config" can return a function to your main program.

danShumway 7y ago
> If your configs are really complicated enough that they need to be generated at the time your main program runs, to me that means your code needs refactoring. I'm not sure that's true. In fact, I would go so far as to say for some programs, it's the opposite. In the example I describe above (forgetting about lambda functions or anything like that) I'm shipping a program that consumes a small set of very rigid input options. I consume a single array of strings that need to resolve to file paths. That's incredibly easy to validate: that's the type of limited input where I can be pretty certain that my program will not have any bugs or vulnerabilities around the input. It's also an incredibly simple API to teach to users, it's literally one option. And importantly, my program is still pretty powerful -- you can do whatever you need to do to get that file list without me needing to extend what input I accept. I can't think of a good way to convert that to a pure JSON/YAML config without either making my software more error prone or making my software much less useful. If returning lambda functions from a config file scares you, forcing the config to be a static object literal but allowing that object literal to be generated at runtime will still (often) result in a runtime that is less error prone, easier to validate, and easier to debug when something goes wrong. This is because at the end of the user's dynamic config, they'll still be returning a small, easy-to-understand object literal -- whatever logic was used to build it. If instead you put that logic in your main program, and you accept globs for files, and you start doing complicated things like allowing arrays of multiple globs and file excludes -- well, suddenly you have a big chunk of parsing logic in your main program that end-users can't debug when something goes wrong. It becomes harder to validate whether or not you have some bug around the order that patterns are passed to you, or whether there's an error if you try to use a globbed filepath as an excluded filepath, or whatever. In other words, it is usually better to use procedural logic to generate declarative logic than it is to use declarative logic to generate procedural logic. Of course, your program might be simple enough that you can get away with a config file that's turned into a bunch of `if` statements when its parsed. But it probably would be simpler if you got rid of as much parsing code as possible and focused on more rigidly validating a smaller, stricter set of options. > I can see providing a separate config editor program users can use to generate configs for your main program If you ignore the stuff about lambda functions, that's exactly what we're doing. We're just piggybacking on a real language to do it instead of wasting time writing our own graphical/command-line utility. ---- Where returning functions is concerned: > Either that, or what you are calling a "config" is really a plugin/extension/scripting system I think this is a kind of blurry line. I'm not going to say there is no difference between a plugin system and a config, but I have run into programs that have both configs and plugin systems, and where the plugin system exists mostly to deal with the fact that the config API is not very useful. As a user, given the choice between learning one API and two, I usually prefer to learn one API. A lot of static site generators come to mind; I don't want to learn two different APIs for a templating engine just so I can change how links render. I don't want to learn both your plugin API and your config API. Just give me one, single API for templates that can be fully described in a single paragraph of text. I dunno. I've been doing functional programming long enough that the idea of lambdas just doesn't scare me. Functional paradigms are wildly useful when building library APIs that get imported; I don't see why we should throw them away just because our API lives in a dotfile.