6 ms·

Not directly related to copilot but in context of ever increasing yml/json configurations for cloud apps, am I the only one who feels overwhelmed with the confi

Not directly related to copilot but in context of ever increasing yml/json configurations for cloud apps, am I the only one who feels overwhelmed with the configuration options? I see tons of articles with just drop this json here and that yml there and run this cli command - where is the reference for these json/ymls? Does everyone fully know/understands the structure/options? How do people keep track of the changes to that between upgrades? I feel increasingly dumb the more I see these configs.

Mandatum 6y ago
There's definitely room for config-viewer plugins in VSCode, Sublime, etc. to help write and maintain the 100's of config files we have to be knowledgeable about in DevOps these days. Does anyone have good suggestions in this space? Or do I just have to google docs and reference documentation when I need to figure out what the hell all these parameters mean?
f6v 6y ago
I think clouds have been trying to fool developers into thinking that cloud is easy. Don't feel bad, building a reliable infrastructure is incredibly hard. There're ton of services, some of them are redundant, and it takes a lot of time to discover all the nitty-gritty.
- jmchuster 6y ago
  Maybe it's messaging intended to be for grizzled old veterans going "look how we've taken the old complicated thing and given you powerful tools to allow you to manage it more easily with less steps" but then gets interpreted by neophytes as "look how easy it is for a complete beginner to come in and set up a complicated system".
  - gk1 6y ago
    Great observation. A lot of messaging for cloud products is comparing against legacy or dominant solutions. A beginner doesn’t have that reference point.
    - ethbro 6y ago
      Messaging is written by marketing and sales departments. Employees there usually don't have depth of technical expertise or experience to draw from, and are incentivized to make the product sound as easy to use as possible. This combination results in {really complicated thing} -> {simple abstraction}. Ultimately, everyone internalizes this and starts forgetting that {simple abstraction} is actually {really complicated thing}. Happens with any popular technology. Networking, DB, cloud, ML. Unfortunately, this flies because the primary buyers are also non-technical.
leokennis 6y ago
Oof yes. The times I read stuff like "to make sure your WidgyPidy will automatically balance your BoolyPooly cluster, just set the autoEnableBalanceBoolyClusterGenerated in your WiPoBaClConfig.yml file to the desired floating point and drop a reference in the ServerClusteringPoolDirective.json in your configured ShimpyDimpy config folder" like it's obvious and simple...
- ssrs 6y ago
  hahaha.. lol.
- luisfmh 6y ago
  This starts to sound more and more like how they manufacture plumbuses
  - leetrout 6y ago
    I always wondered how plumbuses got made.
    - auggierose 6y ago
      Me too! Now I know: https://www.youtube.com/watch?v=eMJk4y9NGvE https://www.youtube.com/watch?v=eMJk4y9NGvE
  - stjo 6y ago
    High IQ joke
- joshxyz 6y ago
  This is just.. accurate.
- ethbro 6y ago
  Finally, DevOps has caught up to the rate of progress in js framework land. Rejoice!
iamacyborg 6y ago
Urgh, yes. I'm not a dev by any stretch of the imagination but deploying PHP apps on servers was significantly easier than this mess we have now.
- Aeolun 6y ago
  Rsync/ln is still the fastest deployment process I’ve had so far.
  - iamacyborg 6y ago
    Still too complex for me, let me just login to an sFTP and drop some files in
thesimon 6y ago
>I feel increasingly dumb the more I see these configs. Especially since there is no IDE support. I've come back to enjoy XML with nice schemas and IDE validation and autocompletion instead of YAML and liquibase changes not going through properly due to an indention problem.
- ithkuil 6y ago
  You can use schemas with json/yaml (e.g. jsonschema or protobufs); I wish people used them more though
realharo 6y ago
I feel like trying to use a "dumb" declarative config language for everything is part of the problem. The domain is not simple enough for that, so you end up with all kinds of weird scripts around it all, or other templating/codegen solutions, plus all kinds of ad-hoc conventions that are extremely easy to break and very annoying to navigate. Having a full blown statically typed language like TypeScript - where things can reference each other strongly (not just by names and other strings), naming conventions can be encoded in functions (or even enforced by types), and structures on every level (even your project-specific ones) have a well-defined and checked schema - would help a lot in my opinion. Maybe deno could find a use in this niche.
- scrollaway 6y ago
  You're (accidentally, I think?) describing Pulumi in striking detail. https://www.pulumi.com/ https://www.pulumi.com/
  - realharo 6y ago
    Kind of. From a brief look, it looks very framework-y and all-or-nothing (though I could be wrong, they do have a page about gradual adoption at least https://www.pulumi.com/docs/guides/adopting/ https://www.pulumi.com/docs/guides/adopting/). I was picturing something more low level and less opinionated, though I can only guess how such a thing would perform in practice. Not even necessarily a specific product, just the adoption of workflows that use already existing tools (e.g. the various API client libraries - for AWS, Kubernetes, git, docker registry, etc.) to achieve this. I realize that that could lead to reinventing the wheel, and eventually an in-house framework anyway, but it feels a lot more comfortable to start there, as you can design things closer to your exact needs. Pulumi definitely looks interesting though.
- codebeaker 6y ago
  It sounds like you're describing HashiCorp's HCL2 and Terraform Configs which are at least a graph. I find using AWS without Terraform to be nearly impossible.
Hikikomori 6y ago
For aws, I usually read the cloudformation/api/boto3 docs. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui... I don't know all the options, and I don't need to either. Can look at examples to get started and customise as I need. What changes? Idk if other ways are better, just looking at large text config files for apache or other linux software boggles my mind just as much if not more. Once you have lots of options it's just going to be complex to understand.
rorykoehler 6y ago
Track changes in version control (git) using infra as code tooling
ghaff 6y ago
And get the second order effect of one of those configuration options wrong and wake up to a huge AWS bill. And, yes, with a bit of groveling they'll probably refund you after you've sweated a bit--the first time at least.
- gk1 6y ago
  Or worse, you fudge a YAML line and bring the application down. There are tools to catch and prevent that sort of thing: Datree, AWS CloudFormation templates, and Sentinel, just to name a few. They apply and enforce different policies to prevent YAML/JSON (or “infrastructure as code”) misconfigurations. (Disclaimer: I work with Datree, but not the others.)
robbyt 6y ago
No, I'm not overwhelmed by documented/structured config files. 20 years ago I was a Windows admin, discovered Linux, and it blew my mind how much easier it was to configure, experiment, validate, and roll-back config changes. GUIs are nice for discovery and exploration, but there's a very good reason they're not used for server and infrastructure configuration.
crispyambulance 6y ago
> yml/json configurations for [...] We learned this lesson the hard way with XML back in the early naughts when behind everything was a nauseating morass of xml with overcomplicated designed-by-committee schemas. People moved out of development and tried hard to forget it, but neglected to instruct the next generation about the pitfalls of shitty configuration files that get out of control. Now we got json and yaml. No more </>'s, yay, but somehow worse, less maintainable and less understandable.
bdcravens 6y ago
The new J2EE.

Mandatum 6y ago

There's definitely room for config-viewer plugins in VSCode, Sublime, etc. to help write and maintain the 100's of config files we have to be knowledgeable about in DevOps these days. Does anyone have good suggestions in this space? Or do I just have to google docs and reference documentation when I need to figure out what the hell all these parameters mean?

f6v 6y ago

I think clouds have been trying to fool developers into thinking that cloud is easy. Don't feel bad, building a reliable infrastructure is incredibly hard. There're ton of services, some of them are redundant, and it takes a lot of time to discover all the nitty-gritty.

jmchuster 6y ago
Maybe it's messaging intended to be for grizzled old veterans going "look how we've taken the old complicated thing and given you powerful tools to allow you to manage it more easily with less steps" but then gets interpreted by neophytes as "look how easy it is for a complete beginner to come in and set up a complicated system".
- gk1 6y ago
  Great observation. A lot of messaging for cloud products is comparing against legacy or dominant solutions. A beginner doesn’t have that reference point.
  - ethbro 6y ago
    Messaging is written by marketing and sales departments. Employees there usually don't have depth of technical expertise or experience to draw from, and are incentivized to make the product sound as easy to use as possible. This combination results in {really complicated thing} -> {simple abstraction}. Ultimately, everyone internalizes this and starts forgetting that {simple abstraction} is actually {really complicated thing}. Happens with any popular technology. Networking, DB, cloud, ML. Unfortunately, this flies because the primary buyers are also non-technical.

leokennis 6y ago

Oof yes. The times I read stuff like "to make sure your WidgyPidy will automatically balance your BoolyPooly cluster, just set the autoEnableBalanceBoolyClusterGenerated in your WiPoBaClConfig.yml file to the desired floating point and drop a reference in the ServerClusteringPoolDirective.json in your configured ShimpyDimpy config folder" like it's obvious and simple...

ssrs 6y ago
hahaha.. lol.
luisfmh 6y ago
This starts to sound more and more like how they manufacture plumbuses
- leetrout 6y ago
  I always wondered how plumbuses got made.
  - auggierose 6y ago
    Me too! Now I know: https://www.youtube.com/watch?v=eMJk4y9NGvE https://www.youtube.com/watch?v=eMJk4y9NGvE
- stjo 6y ago
  High IQ joke
joshxyz 6y ago
This is just.. accurate.
ethbro 6y ago
Finally, DevOps has caught up to the rate of progress in js framework land. Rejoice!

iamacyborg 6y ago

Urgh, yes. I'm not a dev by any stretch of the imagination but deploying PHP apps on servers was significantly easier than this mess we have now.

Aeolun 6y ago
Rsync/ln is still the fastest deployment process I’ve had so far.
- iamacyborg 6y ago
  Still too complex for me, let me just login to an sFTP and drop some files in

thesimon 6y ago

>I feel increasingly dumb the more I see these configs. Especially since there is no IDE support. I've come back to enjoy XML with nice schemas and IDE validation and autocompletion instead of YAML and liquibase changes not going through properly due to an indention problem.

ithkuil 6y ago
You can use schemas with json/yaml (e.g. jsonschema or protobufs); I wish people used them more though

realharo 6y ago

I feel like trying to use a "dumb" declarative config language for everything is part of the problem. The domain is not simple enough for that, so you end up with all kinds of weird scripts around it all, or other templating/codegen solutions, plus all kinds of ad-hoc conventions that are extremely easy to break and very annoying to navigate. Having a full blown statically typed language like TypeScript - where things can reference each other strongly (not just by names and other strings), naming conventions can be encoded in functions (or even enforced by types), and structures on every level (even your project-specific ones) have a well-defined and checked schema - would help a lot in my opinion. Maybe deno could find a use in this niche.

scrollaway 6y ago
You're (accidentally, I think?) describing Pulumi in striking detail. https://www.pulumi.com/ https://www.pulumi.com/
- realharo 6y ago
  Kind of. From a brief look, it looks very framework-y and all-or-nothing (though I could be wrong, they do have a page about gradual adoption at least https://www.pulumi.com/docs/guides/adopting/ https://www.pulumi.com/docs/guides/adopting/). I was picturing something more low level and less opinionated, though I can only guess how such a thing would perform in practice. Not even necessarily a specific product, just the adoption of workflows that use already existing tools (e.g. the various API client libraries - for AWS, Kubernetes, git, docker registry, etc.) to achieve this. I realize that that could lead to reinventing the wheel, and eventually an in-house framework anyway, but it feels a lot more comfortable to start there, as you can design things closer to your exact needs. Pulumi definitely looks interesting though.
codebeaker 6y ago
It sounds like you're describing HashiCorp's HCL2 and Terraform Configs which are at least a graph. I find using AWS without Terraform to be nearly impossible.

Hikikomori 6y ago

For aws, I usually read the cloudformation/api/boto3 docs. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui... I don't know all the options, and I don't need to either. Can look at examples to get started and customise as I need. What changes? Idk if other ways are better, just looking at large text config files for apache or other linux software boggles my mind just as much if not more. Once you have lots of options it's just going to be complex to understand.

rorykoehler 6y ago

Track changes in version control (git) using infra as code tooling

ghaff 6y ago

And get the second order effect of one of those configuration options wrong and wake up to a huge AWS bill. And, yes, with a bit of groveling they'll probably refund you after you've sweated a bit--the first time at least.

gk1 6y ago
Or worse, you fudge a YAML line and bring the application down. There are tools to catch and prevent that sort of thing: Datree, AWS CloudFormation templates, and Sentinel, just to name a few. They apply and enforce different policies to prevent YAML/JSON (or “infrastructure as code”) misconfigurations. (Disclaimer: I work with Datree, but not the others.)

robbyt 6y ago

No, I'm not overwhelmed by documented/structured config files. 20 years ago I was a Windows admin, discovered Linux, and it blew my mind how much easier it was to configure, experiment, validate, and roll-back config changes. GUIs are nice for discovery and exploration, but there's a very good reason they're not used for server and infrastructure configuration.

crispyambulance 6y ago

> yml/json configurations for [...] We learned this lesson the hard way with XML back in the early naughts when behind everything was a nauseating morass of xml with overcomplicated designed-by-committee schemas. People moved out of development and tried hard to forget it, but neglected to instruct the next generation about the pitfalls of shitty configuration files that get out of control. Now we got json and yaml. No more </>'s, yay, but somehow worse, less maintainable and less understandable.

bdcravens 6y ago

The new J2EE.