6 ms·

This delusion is especially visible in the DevOps space. For some reason we have decided as an industry that instead of writing some code in whatever 'real' lan

• UserIsUnused 7y ago
  Thank god trends like Pulumi and the new AWS sdk is emerging. General purpose programing languages are getting more expressive by the day, why do we use data serialization languages instead for configs? it doesn't make any sense. Configuration is code not data.
  • q3k 7y ago
    I still have my reservations against Pulumi in particular, though. I've only really skimmed their documentation, but the idea that calling 'new' against a VM class instantiates a VM in production seems to be to magical for my taste, and might be an indication that it's yet another product that targets the happy path more than what we tend to experience in production: failures.
  • reallydontask 7y ago
    The main problem is that, in my experience, the majority of DevOps teams are Ops teams that have been renamed and refocused towards automation. These are people that by and large don't want to code, not saying that they can't or won't. To be fair this has been in Windows shops, where scripting has only recently (last 5-10 years) taken off, so you've got a lot of windows admins that the closest they've been to code is Batch scripting with a bit of Powershell. This is a big change for them As it happens i read about pulumi recently and I've put it on mt list of todo things, but I can't see that I'll be able to sell it to our team and our team is blessed (cursed?) with three former developers
    • toyg 7y ago
      > These are people that by and large don't want to code I disagree. That's the stereotype that tool builders have of such people. Good ops people have always loved coding, or we wouldn't be living on the mountains of Bash scripts also known as "Linux distributions". (Besides, people who don't like to code won't like writing tons of declarative markup either. So there is little point in the current approach either way.)
      • reallydontask 7y ago
        It might be a stereotype but it's also my experience, which granted is limited and Windows based, which as I pointed out in the previous comment hasn't been really onboard with the scripting experience until relatively recently As to the declarative markup, for instance Azure Devops still doesn't have feature parity between YAML pipelines and classic pipelines, so while you could well be right about the same resistance to yaml, it's not, necessarily an issue yet.
        • toyg 7y ago
          Most devops tools are built on Linux for Linux, then ported to Windows later as an afterthought, so I don’t think platforms are much of a factor. It’s a defect in the production pipeline somewhere, something like feedback from potential users not reaching developers until it’s too late. It doesn’t help that, in some cases, vendors just impose what is going to happen, and the community is simply forced to put up with it.
    • dragonwriter 7y ago
      > The main problem is that, in my experience, the majority of DevOps teams are Ops teams that have been renamed and refocused towards automation. I'm on a relatively new team kind of like that (it's not a former Ops team but most of the team was pulled from Ops/DBA/analysis teams, and fits the description, and it's not a really a DevOps but more of a Dev+Ops team) and the general reaction of the team to being introduceed to the AWS CDK has been “we need to move to that as soon as we can”, even from people who very vocally never wanted to be programmers. And that's with, in many cases, a couple months experience with both programming and YAML IAC in the form of CloudFormation.
  • solatic 7y ago
    > General purpose programing languages are getting more expressive by the day You know, once upon a time, we understood that declarative approaches to software engineering were superior to imperative approaches, when declarative approaches are feasible. Declarative approaches are much safer and easier to test, at a cost of only being able to express what the tool accepting the declarative approach can understand. Imperative approaches are strictly worse for any problem set where a declarative approach solves the problem within performance requirements. The additional expressiveness of languages like Pulumi is the last thing I want. YAML is a horrible language for declarative system configuration because a) any sufficiently complex system will require you to generate your declarative codebase in the name of maintainability, b) generating code for any language where whitespace is significant will lead you to an early death, and c) stringly-typed languages are fundamentally unmaintainable at sufficient scale. But this is not an indictment of a declarative approach! It is an indictment of YAML. > Configuration is code not data. Data > code. Data does not need to be debugged. The best code you can have is deleted code - deleted code does not need to be maintained, updated, or patched. Code is a necessary evil we write in order to build operable systems, not a virtue in and of itself.
    • UserIsUnused 7y ago
      There are declarative general purpose programing languages. That data you are talking about does need to be debugged, like Helm charts and pipeline definitions. Sure data is better, but config is code, not data.
      • solatic 7y ago
        Generators need to be debugged, not data. It's very easy to test a generator - a few unit tests checking whether, for a given input, the generator produced the expected output, and you're set. Data sometimes needs to be cleaned, but there's no such thing as a bug in data. Whether the generated declarative output produces the expected behavior on the part of the tool interpreting the declarative output is part of the tool's contract, not the generator or the declarative output. If you need to check the tool's behavior then either a) you wrote the tool or b) you're writing an acceptance test for the tool, which is an entirely different endeavor.
  • falcolas 7y ago
    > Thank god trends like Pulumi and the new AWS sdk is emerging. The more things change, the more they stay the same. The pendulum is swinging back towards scripting languages. But give it a few years, and we'll be railing against scripting languages for not being idempotent enough, and moving back to the latest version of YAML and XML (perhaps we'll go with TOML this time) with custom interpreters. We have seen this half a dozen times already, between turing complete DSLs in ruby, to plain bash, perl, and python scripts. The other side are the DSLs written using YAML, JSON, and XML (and every other config language written in history).
  • sgslo 7y ago
    > new AWS SDK Could you share some details on that? A quick Google search didn't reveal anything about major changes to the SDK.
    • dragonwriter 7y ago
      Probably referring to the CDK (Cloud Development Kit), a newish SDK for developing Cloudformation infrastructure-as-code in general-purpose programming languages. https://aws.amazon.com/cdk/ https://aws.amazon.com/cdk/
    • JonathanHedley 7y ago
      I believe the OP is referring to the AWS CDK (Cloud Development Kit) - https://docs.aws.amazon.com/cdk/latest/guide/home.html https://docs.aws.amazon.com/cdk/latest/guide/home.html
  • jimbokun 7y ago
    > Configuration is code not data. As any Lisper knows, code is data is code.
• collyw 7y ago
  To play devils advocate, isn't the idea with Ansible at least that it's idempotent. The YAML should describe the final state and should be concerned with branching or lower level features like that. (It's the same with SQL, you describe what you want back rather than how its achieved. A declarative approach works fairly well there.)
  • UserIsUnused 7y ago
    There are plenty of declarative programming language families. Lisps, MLs, SQLs. Configuration is code. Use a programming language.
    • collyw 7y ago
      So how do you differentiate a declarative language like SQL with a declarative YAML file for Ansible? What would be the benefit of adding a declaritive language into the mix over YAML?
      • maweki 7y ago
        If you want to build a DSL, build a DSL. DSLs are easily embeddable in the mentioned programming languages. YAML is not typed and you can basically do whatever. The worst of all worlds.
      • UserIsUnused 7y ago
        Tooling. Data transformation. Libraries. Emacs is configured using Lisp. Because of that it is amazingly configurable. XMonad uses Haskell, which gives types to avoid lots of error cases. The thing is providing an actually programming language doesn't remove anything specially if the same config can be written as cleanly. And you would avoid "YAML templates for creating YAML", when you actually need to process some data (even if it's just for pre/suffix creation in names). Secrets retrieval is another thing that you also need to do and template. Also, YAML is a terrible to parse language, with can give out weird error cases. Other languages compilers/interpreters are more mature. And ofc, if you provide SDKs for your IaC tool, you should be able to use the language that your developers are more familiar with. Taking advantage of the good practices their are used to. Don't limit with "declarative languages". Use a programing language that make more sense, and leave data languages for data.
• dangerface 7y ago
  You can implement LUA in less lines of code than YAML, lua has been around longer, easier syntax, loads of support, proper community around it. Yet people still use YAML on hype alone, it's awful.
  • falcolas 7y ago
    So, those two things are quite different. One is a programming language, the other a markup specification. And LUA has its own hype, just not in configuration scenarios.
• ckuehne 7y ago
  This sounds a lot like the "We do everything with XML now"-phase that the Java world has gone through.
  • q3k 7y ago
    Agreed, this is a very strong parallel.
• jordanbeiber 7y ago
  Well, I would have to disagree (but agree somewhat, as well :)). These insanity levels you describe, and I agree with you here, are actually pushed through projects by coders for coders. Here in lie much of the ”devops” problem, imo, as coders seem to want to allow easy configuration management through “simple“ declarations, inventing almost a new language in the process. Having worked for 20 years with systems/configuration management, mainly as a developer I can tell many coders have not. Not at scale, and not supporting 100s of different services simultaneously. Hence square wheels gets re-invented. To be rid of all yaml templating and loosely coupled technical integrations you will have to treat config/service delivery/ci/cd/what-have-you as a business domain of it’s own, and develop it the same way. I guess we agree, completely perhaps?! :) I’ve been fortunate to have been able to work everything from small upstart to big corporate enterprises and to me this is more true now than ever. As a developer I build my own version of a CMDB, I do not use crazy templating - I write an API, that leverages the CMDB and produces sane output. It’s still configuration as code, just something that is easier to scale and adapt. Guess I’m only disagreeing on whom to blame for this mess.
• dragonwriter 7y ago
  The ironic thing is that—specifically in the cloud space—we call this no-code abomination “infrastructure as code”. But at least there is the AWS CDK and Pulumi now to enable “infrastructure as code” as code.
• solatic 7y ago
  > Especially with tools like Helm I believe we reached peak insanity levels. Instead of using a general purpose or configuration-specific (like Jsonnet/Cue/Dhall) programming language to build and then emit in-memory objects to YAML manifests, the industry is using YAML to define templated YAML and then parametrize this templating with even more YAML. As a big proponent of Dhall, I gotta say, probably the primary reason for that at this point is due to a lack of language bindings. Most tools in the Kubernetes ecosystem are written in Go. Currently, there is no official Go language binding for Dhall - one implementation[1] is pretty close, but the marshalling features need some work, and it's not the author's day job. The only way to get Dhall configuration into a Go program today is to either shell out and call the Haskell-based reference implementation directly or to script your Go binary such that the Haskell-based reference implementation translates the Dhall configuration to JSON and then feeds that into the Go program. Even that's not ideal - you end up maintaining your schema in Go code and in Dhall simultaneously. I very much believe that Dhall will solve the stringly-typed insanity which you refer to, but the language is not quite there yet. While I'd love for some of the Kubernetes core developers to step up, I mean, I can understand people's unwillingness to adopt something that isn't 100% perfectly well-supported and handed to them on a silver platter. [1]: https://github.com/philandstuff/dhall-golang https://github.com/philandstuff/dhall-golang
  • marcosdumay 7y ago
    There is a levels of indirection problem with that explanation. Using Dhall to create the YML you will use isn't great, but it's quite possible today and is much better than using YAML to script a YAML generator that will read some more YAML to create the YAML you will use. I guess some people simply like YAML.
    • solatic 7y ago
      I agree, which is why I use Dhall at work. But the fact of the matter is that `kubectl apply -f config.dhall` isn't realistic yet, and there's a big segment of the market that's just not willing to accept anything but that.
• runlevel1 7y ago
  YAML is okay for representing a simple declarative configuration. It starts to break down when you try to encode too much operational logic into it. If we step back, there are a few different strategies for dealing with operational logic. From least scalable to most (roughly): 1. Human processes (ex: checklist) 2. In the config itself (ex: Kubernetes' YAML) 3. A config generator (ex: Helm, Kustomize) 4. Software (ex: Kubernetes Operator) I see a lot of people get stuck doing one for too long. When doing one becomes unwieldy, it's time to consider the next. That or find a way to reduce the need for all that operational logic.
  • marcosdumay 7y ago
    > simple declarative configuration The point of all this is that software configuration is never a simple declaration. It is always a mess of behaviors and derived values. I am not sure I totally agree, but it at least correct enough to always break something. And adding abstraction layers just because you choose to use an underpowered notation at the bottom isn't a good practice.
• nnq 7y ago
  Maybe because we've chosen such rigid, inexpressive and DSL-hostile programming languages for devops... I mean, Python, Go... ofc you'd rather write your DSL in YAML instead and forego any checking the language might provide since everything else would be too awkward. Something like a "statically typed Ruby" would probably shine here.
  • toyg 7y ago
    TL;DR: "your language sucks, my language would shine" Language doesn't matter. What matters is the attitude of tool builders that everything should "simply" be described as data. When that turns out to be insufficient, as it inevitably does, hacks are introduced and sooner or later you end up with crippled and idiosyncratic pseudo-languages.
• jillesvangurp 7y ago
  Exactly, the fix is also obvious but not quite happening just yet. Several languages now support internal domain specific languages. Several of those are strongly typed. And several of those can compile to native code and integrate with other stuff around them and also come typically with nice tool support in the form of syntax highlighting, auto complete, etc. Basically all the stuff that yaml sucks at can be fixed by having a richer language model than free form strings in free form objects and lists with no semantics whatsoever; no auto complete; and nothing but run time errors to tell you you copy pasted it wrong (because that's what happens when there is no tool support). IMHO a big part of the problem is that the companies involved are actively incentivized to justify their existence by making things more complicated instead of less complicated. If things are too simple, nobody needs their consulting, training, support, etc. Worse most of these companies are competing for attention and typically only provide part of the solution instead of the whole solution. So there's a lot of money in cross integrating multiple complicated solutions from different companies. Indeed many of the companies involved are serial offenders when it comes to this. As soon as you get companies like Red Hat, Amazon, Oracle, etc. get interested in stuff, brace yourself for some convoluted vendor lockin. That's basically what happened to Kubernetes. When it comes to languages, I'd love something that has a compiler and auto complete. I'd also like something that has a notion of internal domain specific languages. IMHO Typescript, Kotlin or Rust could work for this. The point of this would be leveraging a type system for static checks and code completion. That kind of rules out dynamic languages like javascript, python, or ruby (all of which have been used in devops in the past). They are nice but not nice enough. However, typescript has enough static typing that you get some meaningful autocomplete. Also, it's already popular in the node.js world (i.e. lots of stuff available to integrate). Kotlin would also be able to fit in that world since there's a transpiler to javascript, it can utilize typescript type definitions and it integrates with node.js and npm. Rust would be a bit of an outlier here and probably a bit too much in terms of language complexity for the average yaml loving devops type with a mere few years of full stack experience. With Kotlin, the transition of gradle from groovy to Kotlin has been interesting. Groovy basically is a dynamically typed language and they've slowly been replacing it with the more strictly type Kotlin. That sort of stabilized with v5 and with v6 it got usable for a lot of projects and recent updates to Intellij have improved how it is able to support developers with e.g. code completion and other features. Something like that aimed at devops could work.
• majkinetor 7y ago
  Powershell shines here. DSC or pure powershell has bunch of useful tools and a real programming language to use in between. Yaml like configuration can be achieved with nested HashTables that have almost equivalent readability without loosing any of the devlopment capabilities.
• downerending 7y ago
  I'm reminded a bit of systemd. Whatever their problems, when things went south, it was usually fairly easy for someone who could code to walk through the init scripts involved and figure it out. The config files of systemd at first glance look quite simple and declarative, but if things don't work, one almost immediately slams into a wall of complexity. Sometimes one doesn't "need" a coder because the system is such that it wouldn't do you any good anyway.