5 ms·

Do you have an example of when logic would help a configuration file? Their value and use are entirely dependent on the context. Also I can't make a head or ta

Do you have an example of when logic would help a configuration file? Their value and use are entirely dependent on the context. Also I can't make a head or tail of your statement, no offense intended—I can make out parts, but the whole just has no meaning to me.

DonHopkins 7y ago
Given any non-trivial data-only config file, it will always grow to the point that you'll end up needing to generate it automatically with logic. And that goes double for God's config file.
- SomeOldThrow 7y ago
  Do you have an example? I don't think I've ever needed to generate a config automatically (except in niche cases like generating configs for services in chef), and if I'm understanding correctly, nothing proposed in this thread would help with that scenario. I suspect you're front-loading a lot of logic from the app bootup into the config file, and I'm not sure what you stand to gain conflating those two things. Like—what's the execution order of a config file? Can you refer to a value later in the file? If so, how does it determine which value is executed before the next? If not, how do you setup circular dependencies—can you redefine config values halfway through a file? If not, you're gonna have to fall back on at-boot pre-processing anyway with sufficient complexity, so just treat a config like dumb data to begin with and do all your logic in the bootup and put all the values/whatever in the config file. And god knows, I would be strongly tempted to murder an engineer that introduced a config file capable of rewriting itself—that engineer has clearly never needed to debug another person's shitty code before.
  - DonHopkins 7y ago
    I've developed programs with tens of thousands of lines of expanded JSON in their config files. No fucking way I'm maintaining all that by hand as pure data. See: https://news.ycombinator.com/item?id=20735231 https://news.ycombinator.com/item?id=20735231 Also: https://en.wikipedia.org/wiki/Don%27t_repeat_yourself https://en.wikipedia.org/wiki/Don%27t_repeat_yourself and https://en.wikipedia.org/wiki/Single_source_of_truth https://en.wikipedia.org/wiki/Single_source_of_truth The opposite of DRY (Don't Repeat Yourself) is WET (Write Everything Twice or We Enjoy Typing or Waste Everyone's Time) -- but twice could be ten times or more, there's no limit. Writing it all out again and again by hand as pure literal data, and hoping I didn't make any typos or omissions in each of the ten repetitions, without some sort of logical algorithmic compression and abstraction, would be idiotic.
    - SomeOldThrow 7y ago
      > I've developed programs with tens of thousands of lines of expanded JSON in their config files. To what end? I don't get what you could possibly be putting in these files that could consume so much space without refactoring.
      - DonHopkins 7y ago
        I posted some examples above. (Or maybe it's below. The long message with lots of links.)
  - DrJosiah 7y ago
    This thread hurts so bad. I never used helm / kubernetes before 3 months ago. Not 2 weeks ago I needed to loop in a helm config file in order to basically say "all this same config, libraries, etc., just run this other command instead" ... because someone who makes those decisions had ~100 lines of environment-injected configuration + boilerplate in the yaml that I couldn't get rid of, needed, and would have otherwise needed to copy / paste. Since then, those environment variables have been pulled out into a different file (refactoring!), and now we replaced a loop over 100 lines of config, with 2x sets of 15-20 lines of config boilerplate. Better, but still a lot of bull. I don't know what the right answer is, because we've got less helm templating bullshit in there, but we still need boilerplate. Because it's not like I can tear down an entire kubernetes + helm infrastructure because I don't like how the config files are written. Configs / config generation is hard, and generally awful. If you don't see it that way, congratulations; you're either a genius in your field, you've got not enough experience, and/or you're wrong. If you believe it's easy, and we're all missing something - please, by all means, write a book on how / why configurations aren't as hard as the rest of us say they are. Best of luck to you.
    - SomeOldThrow 7y ago
      > Configs / config generation is hard, and generally awful. If you don't see it that way, congratulations; you're either a genius in your field, you've got not enough experience, and/or you're wrong. The point I'm trying to make is that you're describing broken frameworks, data flows, and work flows, and blaming it on config generation. If you have a counter example, I'd love to see it. Discussing these things in the abstract is pretty pointless and based in emotional language/semantic quibbling rather than meaningful things people can reason about and discuss, like code comparison or time tradeoffs. Hell, because no specific GOOD examples of configuration-as-code have been brought up, literally everyone in this thread could be considering a different pet example of theirs. It's OBVIOUSLY a waste of everyone's time without examples. Why bother comment at all—to go out of your way to punch down without contributing to the discourse?
      - DonHopkins 7y ago
        Lots of examples in my other post, including links to some open source code (UnityJS).
  - DonHopkins 7y ago
    FYI, here are some concrete examples, and some demos of a multi player cross platform networked AR system I developed that's based on shitloads of JSON config files describing objects, behaviors, catalog entries, user interface, a multi player networking protocol, etc. Pantomime extensively (and extensibly) uses the simple JSON templating system I described in this other link, which I wrote in C#. Everything you see is a plug-in object, and they're all described and configured in JSON, and implemented in C#: https://news.ycombinator.com/item?id=20735366 https://news.ycombinator.com/item?id=20735366 Pantomime Playground - Immersive Virtual Worlds for the Rest of Us https://vimeo.com/137924152 https://vimeo.com/137924152 Pantomime Creatures - Real-Time Augmented Reality https://vimeo.com/183132061 https://vimeo.com/183132061 Bug Farm – First Minute, First Person 3D https://vimeo.com/148805981 https://vimeo.com/148805981 Consumer Augmented Reality Arrives – with Pantomime https://vimeo.com/149319403 https://vimeo.com/149319403 If I were to rewrite it from scratch, I'd simply use JavaScript instead of rolling my own JSON templating system, because it would have been much more flexible and powerful. Oh wait -- I DID rewrite at least some of that stuff from scratch! To illustrate that superior JavaScript-centric approach, here's an example of some other JSON based systems I developed with Unity3D and JavaScript, one for scripting ARKit on iOS, and the other for scripting financial visualization on WebGL, both using UnityJS (an extension I developed for scripting and configuring and debugging Unity3D in JavaScript). One nice thing about it is that you can debug and live code your Unity3D apps running on the mobile device or in the web browser while it's running, using the standard JavaScript debugging tools! UnityJS is a plugin for Unity 5 that integrates JavaScript and web browser components into Unity, including a JSON messaging system and a C# bridge, using JSON.net. https://github.com/SimHacker/UnityJS https://github.com/SimHacker/UnityJS WovenAR Tools with ARKit and Pantomime (built with an early version of UnityJS): https://vimeo.com/240612327 https://vimeo.com/240612327 A description of UnityJS, and how I use JavaScript and JSON to define, configure and control Unity3D objects: https://news.ycombinator.com/item?id=19804242 https://news.ycombinator.com/item?id=19804242 https://news.ycombinator.com/item?id=19748582 https://news.ycombinator.com/item?id=19748582 https://news.ycombinator.com/item?id=19745034 https://news.ycombinator.com/item?id=19745034 Here's a demo of another more recent application using UnityJS that reads shitloads of JSON data from spreadsheets, including both financial data, configuration, parameters, object templates, etc. https://reasonstreet.co/2019/01/31/how-does-amazon-make-money-fy2018/ https://reasonstreet.co/2019/01/31/how-does-amazon-make-mone... https://reasonstreet.co/2019/08/09/tech-giants-acquisitions/ https://reasonstreet.co/2019/08/09/tech-giants-acquisitions/ Here is an article about how the JSON spreadsheet system works, and discusses some ideas about JSON definition, editing and templating with spreadsheets, which is about a year old, but that I've developed it a lot further since writing that article. https://medium.com/@donhopkins/representing-and-editing-json-with-spreadsheets-8f8832db5239 https://medium.com/@donhopkins/representing-and-editing-json... >Representing and Editing JSON with Spreadsheets >I’ve been developing a convenient way of representing and editing JSON in spreadsheets, that I’m very happy with, and would love to share! >I‘ve been successfully synergizing JSON with spreadsheets, and have developed a general purpose approach and sample implementation that I’d like to share. So I’ll briefly describe how it works (and share the code and examples), in the hopes of receiving some feedback and criticism. Here is the question I’m trying to answer: >How can you conveniently and compactly represent, view and edit JSON in spreadsheets, using the grid instead of so much punctuation? >My goal is to be able to easily edit JSON data in any spreadsheet, conveniently copy and paste grids of JSON around as TSV files (the format that Google Sheets puts on your clipboard), and efficiently export and import those spreadsheets as JSON. >So I’ve come up with a simple format and convenient conventions for representing and editing JSON in spreadsheets, without any sigils, tabs, quoting, escaping or trailing comma problems, but with comments, rich formatting, formulas, and leveraging the full power of the spreadsheet. >It’s especially powerful with Google Sheets, since it can run JavaScript code to export, import and validate JSON, provide colorized syntax highlighting, error feedback, interactive wizard dialogs, and integrations with other services. Then other apps and services can easily retrieve those live spreadsheets as TSV files, which are super-easy to parse into 2D arrays of strings to convert to JSON. [...] >Philosophy: The goal is to leverage the spreadsheet grid format to reduce syntax and ambiguity, and eliminate problems with brackets, braces, quotes, colons, commas, missing commas, tabs versus spaces, etc. >Instead, you enjoy important benefits missing from JSON like like comments, rich formatting, formulas, and the ability to leverage the spreadsheet’s power, flexibility, programmability, ubiquity and familiarity. More info: https://news.ycombinator.com/item?id=17384078 https://news.ycombinator.com/item?id=17384078 https://news.ycombinator.com/item?id=18860467 https://news.ycombinator.com/item?id=18860467 https://news.ycombinator.com/item?id=18171571 https://news.ycombinator.com/item?id=18171571
  - kbr2000 7y ago
    https://panoptic.com/wiki/aolserver/Conf/Examples https://panoptic.com/wiki/aolserver/Conf/Examples
DonHopkins 7y ago
It sounds like you're assuming that configurations are only created before running a program. But you can also create them while programs are running, to configure dynamically created objects or structures, too. And you can send those configurations as messages, to implement, for example, a distributed network object system for a multi player game. So you may be programmatically creating hundreds of dynamic parameterized "configuration files" per second.

DonHopkins 7y ago

Given any non-trivial data-only config file, it will always grow to the point that you'll end up needing to generate it automatically with logic. And that goes double for God's config file.

SomeOldThrow 7y ago
Do you have an example? I don't think I've ever needed to generate a config automatically (except in niche cases like generating configs for services in chef), and if I'm understanding correctly, nothing proposed in this thread would help with that scenario. I suspect you're front-loading a lot of logic from the app bootup into the config file, and I'm not sure what you stand to gain conflating those two things. Like—what's the execution order of a config file? Can you refer to a value later in the file? If so, how does it determine which value is executed before the next? If not, how do you setup circular dependencies—can you redefine config values halfway through a file? If not, you're gonna have to fall back on at-boot pre-processing anyway with sufficient complexity, so just treat a config like dumb data to begin with and do all your logic in the bootup and put all the values/whatever in the config file. And god knows, I would be strongly tempted to murder an engineer that introduced a config file capable of rewriting itself—that engineer has clearly never needed to debug another person's shitty code before.
- DonHopkins 7y ago
  I've developed programs with tens of thousands of lines of expanded JSON in their config files. No fucking way I'm maintaining all that by hand as pure data. See: https://news.ycombinator.com/item?id=20735231 https://news.ycombinator.com/item?id=20735231 Also: https://en.wikipedia.org/wiki/Don%27t_repeat_yourself https://en.wikipedia.org/wiki/Don%27t_repeat_yourself and https://en.wikipedia.org/wiki/Single_source_of_truth https://en.wikipedia.org/wiki/Single_source_of_truth The opposite of DRY (Don't Repeat Yourself) is WET (Write Everything Twice or We Enjoy Typing or Waste Everyone's Time) -- but twice could be ten times or more, there's no limit. Writing it all out again and again by hand as pure literal data, and hoping I didn't make any typos or omissions in each of the ten repetitions, without some sort of logical algorithmic compression and abstraction, would be idiotic.
  - SomeOldThrow 7y ago
    > I've developed programs with tens of thousands of lines of expanded JSON in their config files. To what end? I don't get what you could possibly be putting in these files that could consume so much space without refactoring.
    - DonHopkins 7y ago
      I posted some examples above. (Or maybe it's below. The long message with lots of links.)
- DrJosiah 7y ago
  This thread hurts so bad. I never used helm / kubernetes before 3 months ago. Not 2 weeks ago I needed to loop in a helm config file in order to basically say "all this same config, libraries, etc., just run this other command instead" ... because someone who makes those decisions had ~100 lines of environment-injected configuration + boilerplate in the yaml that I couldn't get rid of, needed, and would have otherwise needed to copy / paste. Since then, those environment variables have been pulled out into a different file (refactoring!), and now we replaced a loop over 100 lines of config, with 2x sets of 15-20 lines of config boilerplate. Better, but still a lot of bull. I don't know what the right answer is, because we've got less helm templating bullshit in there, but we still need boilerplate. Because it's not like I can tear down an entire kubernetes + helm infrastructure because I don't like how the config files are written. Configs / config generation is hard, and generally awful. If you don't see it that way, congratulations; you're either a genius in your field, you've got not enough experience, and/or you're wrong. If you believe it's easy, and we're all missing something - please, by all means, write a book on how / why configurations aren't as hard as the rest of us say they are. Best of luck to you.
  - SomeOldThrow 7y ago
    > Configs / config generation is hard, and generally awful. If you don't see it that way, congratulations; you're either a genius in your field, you've got not enough experience, and/or you're wrong. The point I'm trying to make is that you're describing broken frameworks, data flows, and work flows, and blaming it on config generation. If you have a counter example, I'd love to see it. Discussing these things in the abstract is pretty pointless and based in emotional language/semantic quibbling rather than meaningful things people can reason about and discuss, like code comparison or time tradeoffs. Hell, because no specific GOOD examples of configuration-as-code have been brought up, literally everyone in this thread could be considering a different pet example of theirs. It's OBVIOUSLY a waste of everyone's time without examples. Why bother comment at all—to go out of your way to punch down without contributing to the discourse?
    - DonHopkins 7y ago
      Lots of examples in my other post, including links to some open source code (UnityJS).
- DonHopkins 7y ago
  FYI, here are some concrete examples, and some demos of a multi player cross platform networked AR system I developed that's based on shitloads of JSON config files describing objects, behaviors, catalog entries, user interface, a multi player networking protocol, etc. Pantomime extensively (and extensibly) uses the simple JSON templating system I described in this other link, which I wrote in C#. Everything you see is a plug-in object, and they're all described and configured in JSON, and implemented in C#: https://news.ycombinator.com/item?id=20735366 https://news.ycombinator.com/item?id=20735366 Pantomime Playground - Immersive Virtual Worlds for the Rest of Us https://vimeo.com/137924152 https://vimeo.com/137924152 Pantomime Creatures - Real-Time Augmented Reality https://vimeo.com/183132061 https://vimeo.com/183132061 Bug Farm – First Minute, First Person 3D https://vimeo.com/148805981 https://vimeo.com/148805981 Consumer Augmented Reality Arrives – with Pantomime https://vimeo.com/149319403 https://vimeo.com/149319403 If I were to rewrite it from scratch, I'd simply use JavaScript instead of rolling my own JSON templating system, because it would have been much more flexible and powerful. Oh wait -- I DID rewrite at least some of that stuff from scratch! To illustrate that superior JavaScript-centric approach, here's an example of some other JSON based systems I developed with Unity3D and JavaScript, one for scripting ARKit on iOS, and the other for scripting financial visualization on WebGL, both using UnityJS (an extension I developed for scripting and configuring and debugging Unity3D in JavaScript). One nice thing about it is that you can debug and live code your Unity3D apps running on the mobile device or in the web browser while it's running, using the standard JavaScript debugging tools! UnityJS is a plugin for Unity 5 that integrates JavaScript and web browser components into Unity, including a JSON messaging system and a C# bridge, using JSON.net. https://github.com/SimHacker/UnityJS https://github.com/SimHacker/UnityJS WovenAR Tools with ARKit and Pantomime (built with an early version of UnityJS): https://vimeo.com/240612327 https://vimeo.com/240612327 A description of UnityJS, and how I use JavaScript and JSON to define, configure and control Unity3D objects: https://news.ycombinator.com/item?id=19804242 https://news.ycombinator.com/item?id=19804242 https://news.ycombinator.com/item?id=19748582 https://news.ycombinator.com/item?id=19748582 https://news.ycombinator.com/item?id=19745034 https://news.ycombinator.com/item?id=19745034 Here's a demo of another more recent application using UnityJS that reads shitloads of JSON data from spreadsheets, including both financial data, configuration, parameters, object templates, etc. https://reasonstreet.co/2019/01/31/how-does-amazon-make-money-fy2018/ https://reasonstreet.co/2019/01/31/how-does-amazon-make-mone... https://reasonstreet.co/2019/08/09/tech-giants-acquisitions/ https://reasonstreet.co/2019/08/09/tech-giants-acquisitions/ Here is an article about how the JSON spreadsheet system works, and discusses some ideas about JSON definition, editing and templating with spreadsheets, which is about a year old, but that I've developed it a lot further since writing that article. https://medium.com/@donhopkins/representing-and-editing-json-with-spreadsheets-8f8832db5239 https://medium.com/@donhopkins/representing-and-editing-json... >Representing and Editing JSON with Spreadsheets >I’ve been developing a convenient way of representing and editing JSON in spreadsheets, that I’m very happy with, and would love to share! >I‘ve been successfully synergizing JSON with spreadsheets, and have developed a general purpose approach and sample implementation that I’d like to share. So I’ll briefly describe how it works (and share the code and examples), in the hopes of receiving some feedback and criticism. Here is the question I’m trying to answer: >How can you conveniently and compactly represent, view and edit JSON in spreadsheets, using the grid instead of so much punctuation? >My goal is to be able to easily edit JSON data in any spreadsheet, conveniently copy and paste grids of JSON around as TSV files (the format that Google Sheets puts on your clipboard), and efficiently export and import those spreadsheets as JSON. >So I’ve come up with a simple format and convenient conventions for representing and editing JSON in spreadsheets, without any sigils, tabs, quoting, escaping or trailing comma problems, but with comments, rich formatting, formulas, and leveraging the full power of the spreadsheet. >It’s especially powerful with Google Sheets, since it can run JavaScript code to export, import and validate JSON, provide colorized syntax highlighting, error feedback, interactive wizard dialogs, and integrations with other services. Then other apps and services can easily retrieve those live spreadsheets as TSV files, which are super-easy to parse into 2D arrays of strings to convert to JSON. [...] >Philosophy: The goal is to leverage the spreadsheet grid format to reduce syntax and ambiguity, and eliminate problems with brackets, braces, quotes, colons, commas, missing commas, tabs versus spaces, etc. >Instead, you enjoy important benefits missing from JSON like like comments, rich formatting, formulas, and the ability to leverage the spreadsheet’s power, flexibility, programmability, ubiquity and familiarity. More info: https://news.ycombinator.com/item?id=17384078 https://news.ycombinator.com/item?id=17384078 https://news.ycombinator.com/item?id=18860467 https://news.ycombinator.com/item?id=18860467 https://news.ycombinator.com/item?id=18171571 https://news.ycombinator.com/item?id=18171571
- kbr2000 7y ago
  https://panoptic.com/wiki/aolserver/Conf/Examples https://panoptic.com/wiki/aolserver/Conf/Examples

DonHopkins 7y ago

It sounds like you're assuming that configurations are only created before running a program. But you can also create them while programs are running, to configure dynamically created objects or structures, too. And you can send those configurations as messages, to implement, for example, a distributed network object system for a multi player game. So you may be programmatically creating hundreds of dynamic parameterized "configuration files" per second.