7 ms·

move over templated yaml files, get ready for templated python files. I heard you like config files, so I set up a config file for your config file so you can c

move over templated yaml files, get ready for templated python files. I heard you like config files, so I set up a config file for your config file so you can config while you config.

danShumway 7y ago
Real talk though, at the point where people are building templating engines for YAML files, doesn't it become kind of obvious that YAML isn't powerful enough or expressive enough for at least some of the projects that are using it? We've got a project in Python, but Python is too hard to read, so instead we have YAML. But YAML is too hard to write, so now we have YTT. We've now taken a program that was written in one language, and magically transformed it into 3 languages instead :)
- pdonis 7y ago
  > We've got a project in Python, but Python is too hard to read You said the kind of scripting system you were describing was for cases where your users can write code. If your users can write code, I don't see why Python would be too hard to read. If your users can't write code, then, as you agreed, you shouldn't expect them to, which means the kind of system you were describing won't work. And that means you should rewrite your code so it doesn't need such a complicated configuration that it seems like writing code is necessary to express it.
  - danShumway 7y ago
    Yeah, I was trying to joke -- if your users are OK with Python, then just let them use Python. What I was getting at was that this is a scenario where concerns over Python users needing to actually use Python has morphed into those same users now needing to use 3 languages.
- ZenPsycho 7y ago
  If you're templating YAML, it just means that you don't know that you can generate YAML. Dev ops is a very different world. Yes they can script, but that doesn't mean they will modify any part of your program, even a python config file directly. They will template it. If you find yourself thinking that your config language isn't powerful enough, the solution is to either generate it, or to fix the program to not require a complex config file. There is a long term, 10 year cost to making config turing complete that I don't believe is worth paying. It'll save you time up front, maybe, but I certainly am not going to be impressed with having to modify your program just to "configure" it. Finally, given that's the environment we're dealing with, from now on the rational thing to do would be to design config file formats expecting them to eventually get templated. Because that's how they get automated.
  - danShumway 7y ago
    > Finally, given that's the environment we're dealing with, from now on the rational thing to do would be to design config file formats expecting them to eventually get templated. But... that's kind of what a dynamic config file is. If you're writing a JS program that imports and runs a JS file to generate an object literal for its options, you have just made a config file format that you expect to be generated, or templated, or whatever you'd like to call it. The only difference is that you're using using 1 language instead of 3, and your users don't need to have a separate step during build to generate the file. People keep on circling around to, "but what if my user doesn't know the language or isn't comfortable with it?" And sure, if your user isn't comfortable programming, don't do this. But if your user does know how to program, don't force them to learn a new templating format, just let them write the code that they already know. Or honestly, just play the joke out straight and build a template engine for your source language. I get that you're making a joke above, but a templating engine that spits out valid Python/JS code from valid JSON/YAML is trivial to write. var template_source = fs.readFileSync(process.argv[1]); var output = JSON.stringify(JSON.parse(template_source), null, 2); /* check malformatted input and prettify */ fs.writeFileSync('config.js', `module.exports=${output};`); So if you go this route and a few of your users end up wanting to template JSON/YAML anyway, then fine, they still can.
    - ZenPsycho 7y ago
      great now integrate that in all the automation tools that devops use, and train them how to use the integration instead of just templating the config file. or put another way, stop trying to reinvent the wheel and expect that OTHER people will script the generation of the config file, in their own way using their own tools and languages, and just make that easier for them. when devops are involved, the question isn't whether they're comfortable with whatever language, the question is how easy it will be for them to template the file, because they simply will not learn your special configuration language, whether they possibly could or not. The point isn't about whether we should make the automation people learn a new templating language. the point is they simply will not. they will find where the relevant parts of the config file are, and they'll stick a ${my_variable} in there, process it using their own tool, whatever it is, regardless of how cleverly (or not) you designed the scripting language around the config. And now whether your config file is python yaml or json is just irrelevant. They will not use python to loop through a file list, even if they could, or know how. They will template that. all you've done by switching your config to python is create a breeding ground for script injection attacks. So ultimately it's the wrong question. What I'm proposing is, design our config file expecting the ${subsititution_variable} will get edited in at some point, and you guard against script injection assuming that is comimng.
spc476 7y ago
You might think you are joking, but that's exactly what happened with sendmail. It's configuration file is so complex (and I think Turing complete) that people came up with a completely different configuration file to generate the configuration file. Thank god I no longer have to deal with sendmail.
- ZenPsycho 7y ago
  I was making a joke that you can either laugh at or cry at.

move over templated yaml files, get ready for templated python files. I heard you like config files, so I set up a config file for your config file so you can config while you config.

danShumway 7y ago

Real talk though, at the point where people are building templating engines for YAML files, doesn't it become kind of obvious that YAML isn't powerful enough or expressive enough for at least some of the projects that are using it? We've got a project in Python, but Python is too hard to read, so instead we have YAML. But YAML is too hard to write, so now we have YTT. We've now taken a program that was written in one language, and magically transformed it into 3 languages instead :)

pdonis 7y ago
> We've got a project in Python, but Python is too hard to read You said the kind of scripting system you were describing was for cases where your users can write code. If your users can write code, I don't see why Python would be too hard to read. If your users can't write code, then, as you agreed, you shouldn't expect them to, which means the kind of system you were describing won't work. And that means you should rewrite your code so it doesn't need such a complicated configuration that it seems like writing code is necessary to express it.
- danShumway 7y ago
  Yeah, I was trying to joke -- if your users are OK with Python, then just let them use Python. What I was getting at was that this is a scenario where concerns over Python users needing to actually use Python has morphed into those same users now needing to use 3 languages.
ZenPsycho 7y ago
If you're templating YAML, it just means that you don't know that you can generate YAML. Dev ops is a very different world. Yes they can script, but that doesn't mean they will modify any part of your program, even a python config file directly. They will template it. If you find yourself thinking that your config language isn't powerful enough, the solution is to either generate it, or to fix the program to not require a complex config file. There is a long term, 10 year cost to making config turing complete that I don't believe is worth paying. It'll save you time up front, maybe, but I certainly am not going to be impressed with having to modify your program just to "configure" it. Finally, given that's the environment we're dealing with, from now on the rational thing to do would be to design config file formats expecting them to eventually get templated. Because that's how they get automated.
- danShumway 7y ago
  > Finally, given that's the environment we're dealing with, from now on the rational thing to do would be to design config file formats expecting them to eventually get templated. But... that's kind of what a dynamic config file is. If you're writing a JS program that imports and runs a JS file to generate an object literal for its options, you have just made a config file format that you expect to be generated, or templated, or whatever you'd like to call it. The only difference is that you're using using 1 language instead of 3, and your users don't need to have a separate step during build to generate the file. People keep on circling around to, "but what if my user doesn't know the language or isn't comfortable with it?" And sure, if your user isn't comfortable programming, don't do this. But if your user does know how to program, don't force them to learn a new templating format, just let them write the code that they already know. Or honestly, just play the joke out straight and build a template engine for your source language. I get that you're making a joke above, but a templating engine that spits out valid Python/JS code from valid JSON/YAML is trivial to write. var template_source = fs.readFileSync(process.argv[1]); var output = JSON.stringify(JSON.parse(template_source), null, 2); /* check malformatted input and prettify */ fs.writeFileSync('config.js', `module.exports=${output};`); So if you go this route and a few of your users end up wanting to template JSON/YAML anyway, then fine, they still can.
  - ZenPsycho 7y ago
    great now integrate that in all the automation tools that devops use, and train them how to use the integration instead of just templating the config file. or put another way, stop trying to reinvent the wheel and expect that OTHER people will script the generation of the config file, in their own way using their own tools and languages, and just make that easier for them. when devops are involved, the question isn't whether they're comfortable with whatever language, the question is how easy it will be for them to template the file, because they simply will not learn your special configuration language, whether they possibly could or not. The point isn't about whether we should make the automation people learn a new templating language. the point is they simply will not. they will find where the relevant parts of the config file are, and they'll stick a ${my_variable} in there, process it using their own tool, whatever it is, regardless of how cleverly (or not) you designed the scripting language around the config. And now whether your config file is python yaml or json is just irrelevant. They will not use python to loop through a file list, even if they could, or know how. They will template that. all you've done by switching your config to python is create a breeding ground for script injection attacks. So ultimately it's the wrong question. What I'm proposing is, design our config file expecting the ${subsititution_variable} will get edited in at some point, and you guard against script injection assuming that is comimng.

spc476 7y ago

You might think you are joking, but that's exactly what happened with sendmail. It's configuration file is so complex (and I think Turing complete) that people came up with a completely different configuration file to generate the configuration file. Thank god I no longer have to deal with sendmail.

ZenPsycho 7y ago
I was making a joke that you can either laugh at or cry at.