5 ms·

I had to basically give up my full time role as a Data Engineer who is supposed to solve "big data" problems (Storage, Streaming, Machine Learning pipelines) et

I had to basically give up my full time role as a Data Engineer who is supposed to solve "big data" problems (Storage, Streaming, Machine Learning pipelines) etc, and had to solely focus on DevOps because there was no other expert in the team. We (a colleague and I) basically learned Terraform, then Kubernetes, and then Helm in an extremely stressful environment. I wouldn't recommend that to most people. (Also, I can hear "you don't need k8s", but tbh, it makes the difficult things possible, and that's what we needed). I've done Ops in bare-metal environments before, and at least there, even if there were new skills to learn, the environment just wasn't so hostile for beginners, and the learning curve was simpler. The YAML (or heaven forbid, jsonnet) does *not* help. TBH, I'm convinced if the IaC bits were done in a regular Java/Go/Rust SDK style instead of mixing and matching YAML glued together via labels and tags, I would have had a much easier time. The positive side of that is that now the IaC bits are mature enough that even the Data Scientists in our team write their own Terraform code, and I've upskilled myself quite a lot. I'm slowly going back to solving Storage/Data/ML issues, which I enjoy a lot more. Maybe I'm just becoming a cranky Software Dev as I enter my 30s. I do _not_ enjoy reading YAML.

lukevp 5y ago
If you haven’t heard about cdktf, check it out! It lets you do infrastructure as code but in full programming languages like Python, C# and TypeScript. It’s by hashicorp, and under the covers it runs Terraform, so you have the whole set of terraform providers at your disposal.
- manish_gill 5y ago
  Ah, CDKTF seems to be the Pulumi equivalent for Terraform. I think I remember seeing it. We're far too invested into Terraform at this point to learn yet another tool. Oh well. Hopefully the next time I have to implement something like this, K8s ecosystem would also have similar tools (or maybe I just missed something)
  - chucky_z 5y ago
    There’s a straight up conversion tool iirc, and you can do things like include hcl as well I believe. Cdktf is just terraform at the bottom.
- cloudcalvin 5y ago
  If you havn't heard, check out Pulumi..
- topochico3 5y ago
  I don’t understand the appeal of CDK, it’s the most difficult to understand overly complex part of IaaC I have ever come across. I want simple yaml/hcl that is easy to understand and write. I don’t need to feel smart writing TF, I need it to be easy to understand to reduce MTTR when things go wrong.
  - kazen44 5y ago
    the issue with yaml is that is breaks down under real life complexities in larger/complexer environments.
  - ryan_lane 5y ago
    Try to TF manage a large github organization with just terraform and get back to me. I'm having to wrap it with python code already, and my next step is the CDK. The biggest issue is that parts of the TF for github requires references, and it fetches each of those references individually (which which quickly make you hit your ratelimit, while also taking 10+ minutes for just a plan). Wrapping this in python, and doing more efficient fetches ahead of time, and feeding the references in yourself can turn a 10+ minute plan into a 20 second plan for 1000+ repos.
StreamBright 5y ago
You are on the right track. I have exactly the same story, data engineer, working on automation a lot. YAML, JSON and HCL should be purged from IaC asap. Luckily Dhall can be integrated everywhere we need YAML otherwise we would go nuts. The bigges failure of k8s is to understand that YAML does not scale to teams and beyond 100 lines, especially not good to nest it (hello ArgoCD). A type safe not accidentally Turing complete configuration language or for that matter any langue (hello Pulumi) is much better than YAML ever will be. Anyways, if I can give you an advise, drop k8s like its hot. I is much more trouble than worth. We have phased it out everywhere we could. The companies still using it are in 1 year long implementation cycles with it with 10+ consultants. It is going very well. For consultants, obviously. https://dhall-lang.org/ https://dhall-lang.org/
- manish_gill 5y ago
  Dhall looks interesting. Thanks! I went into the rabbithole of terraform/HCL and came up with Terragrunt to do dynamic code generation. Combining it together with generous use of `make` keeps me sane, but it's not a long term solution. Even now, there are things within the system where the boundaries should be separate, but they aren't. We literally got exhausted and burned out writing our automation, and decided to leave it as it is and then slowly build it up over the next few months. :/ PS: You're 100% right re K8s/YAML. It feels like everything in the Kubernetes world is optimized for writing the code just once. Which, of course, is easy if you already know how all the various bits fit together (true in any system). But Reading/Debugging kubernetes world is a dreadful experience. Edit: typo
  - StreamBright 5y ago
    > But Reading/Debugging kubernetes world is a dreadful experience. Exactly. It is abstraction, over abstraction, over abstraction for no obvious benefits.
- jjtheblunt 5y ago
  "turning complete" : typo?
ludamad 5y ago
I think the YAML issues center around a simple fact, few are taking e.g. 1000+ line YAML (or YAML for X) dev seriously. Complex YAML becomes code, and requires as much e.g. refactoring, idioms, reuse, static checking. At least it's not terribly hard to make a script output it as a format
laumars 5y ago
DevOps specialist here and I agree with your complaints about IaC meaning JSON and YAML documents. sigh
benlivengood 5y ago
I've been in two production environments where IaC was done with Turing-complete languages (Google and Facebook). It's really not better (it just lets the truly impossible/unmanageable become possible) because they were both dynamically typed. If anything, Terraform comes the closest to being a good IaC language because providers form an API to production resources as opposed to unstructured code/config like Helm or Jenkins/Puppet/Chef. Production in reality is well-typed, and unstructured configuration makes static analysis very difficult and makes diffs only a guess at what will actually be changing when code changes are applied. What I'd like to see is Haskell (or similar) typeclasses as production API; make it impossible to construct malformed configurations in the first place but allow expressive functions to replace the limited templating functionality that current languages have (why can't helm templatize inside values.yaml? Why can't Terraform templatize backends and why aren't providers first-class objects?). It's very difficult to keep current IaC DRY, and if it's not DRY then it has bugs and there are misunderstandings about how environments are actually configured.
- girvo 5y ago
  In terms of “Haskell type classes as production API”, I assume you’ve come across Dhall? https://dhall-lang.org/ https://dhall-lang.org/
- manish_gill 5y ago
  One can write good and bad code in any language. It just so happens that our daily driver - procedural or functional languages, happen to be ones where a Software Engineer's mental model + Existing tooling is evolved enough to make understanding it _easier_ (as opposed to just easy). The problem with DSLs and Config languages is that someone will inevitably try to use them in a way they are not designed to be used, and then circumvent the way you're "supposed to" do things. Not only do you lose all the advances in tooling, but people will end up reinventing things that already exist in the other languages. PS: Of course, I'll pick Static Typing every time.
- lima 5y ago
  Cuelang is a nice compromise: https://cuelang.org https://cuelang.org Composition over encapsulation, expressive enough for complex configs, not expressive enough to be turing complete.
sofixa 5y ago
> The YAML (or heaven forbid, jsonnet) does not help. TBH, I'm convinced if the IaC bits were done in a regular Java/Go/Rust SDK style instead of mixing and matching YAML glued together via labels and tags, I would have had a much easier time. I have mixed feelings about this. I've seen some absolutely atrocious Python and JavaScript code, and i wouldn't want to use either of them, and the flexibility ( to shoot yourself in the foot) they provide. Go is better. On the other hand, YAML is just terrible. I think DSLs such as HCL are better than either option, as long as they're flexible enough while staying limit in order not to get illegible/unmaintainable.
flas9sd 5y ago
could you give me a hint why you didn't like jsonnet? I dreaded k8s yaml when needing to (write) and use helm packages and think of indenting levels, but jsonnet via grafanas tanka gave me the option to be concise, writing functions and creating reusable bits. Of course then you're not far away from using a real language "sdk style". As you so aptly put, k8s learning curve is beginner hostile and needs system knowledge before you go at it, so I'm not sure if this was also with being more familiar with the abstractions - after having looked at k8s for 2-3 years, jsonnet made me finally like interfacing with the k8s apiserver.
- manish_gill 5y ago
  Yeah, I think the difference is that I was a) Learning K8s and Helm Concepts (both different things) b) Reading/Debugging K8s and Helm YAML (both different things) c) Ending up at Helm Chart Configs (another config) d) Figuring out how to set all this to helm me deploy the solution I needed and customize it acc to my needs ...all within the span of a few weeks. So by the time I got to some GitHub repos which were using Jsonnet, I was basically done. I'm sure it has its uses and given enough time and experience, I can see how to utilize it more effectively.
  - flas9sd 5y ago
    yes, both different things. I guess helm is still the most common entry point for people to handle the contents of a cluster. I'm checking out other suggestions in this thread (dhall), but from my personal experience I can already tell, grafanas k8s jsonnet-libs was a second chance to kubernetes I'm glad I got to know (thanks coworkers).
commandlinefan 5y ago
> regular Java/Go/Rust SDK style I thought that until I started working with Gradle (which is a build script tool who's syntax is nearly-fully-functioning Groovy). I miss Makefiles - they were strictly declarative and, while I've seen some hairy Makefiles, you'll never need a debugger to figure out what they're actually doing wrong.
notimrelakatos 5y ago
> it makes the difficult things possible, and that's what we needed Indeed, the problem of just saying "you don't need k8s" is you may accidentally invent an even worse in-house version of k8s.
- boublepop 5y ago
  Though for 80-90% og cases out there you seriously don’t need k8s, and when the small internal IT dept that used to “manage” a dozen virtual machines starts talking about “need to keep up with the times and use new tech” what they are really saying is “let’s gamble infrastructure stability on my personal self education in k8s for my next job” and it’s perfectly reasonable to give them a hard no.
totetsu 5y ago
any suggestions on learning materials?
- manish_gill 5y ago
  You'll find plethora of learning documentation, but by far, the one talk/presentation that made it click for me was this one: https://kube-decon.carson-anderson.com/ https://kube-decon.carson-anderson.com/

lukevp 5y ago

If you haven’t heard about cdktf, check it out! It lets you do infrastructure as code but in full programming languages like Python, C# and TypeScript. It’s by hashicorp, and under the covers it runs Terraform, so you have the whole set of terraform providers at your disposal.

manish_gill 5y ago
Ah, CDKTF seems to be the Pulumi equivalent for Terraform. I think I remember seeing it. We're far too invested into Terraform at this point to learn yet another tool. Oh well. Hopefully the next time I have to implement something like this, K8s ecosystem would also have similar tools (or maybe I just missed something)
- chucky_z 5y ago
  There’s a straight up conversion tool iirc, and you can do things like include hcl as well I believe. Cdktf is just terraform at the bottom.
cloudcalvin 5y ago
If you havn't heard, check out Pulumi..
topochico3 5y ago
I don’t understand the appeal of CDK, it’s the most difficult to understand overly complex part of IaaC I have ever come across. I want simple yaml/hcl that is easy to understand and write. I don’t need to feel smart writing TF, I need it to be easy to understand to reduce MTTR when things go wrong.
- kazen44 5y ago
  the issue with yaml is that is breaks down under real life complexities in larger/complexer environments.
- ryan_lane 5y ago
  Try to TF manage a large github organization with just terraform and get back to me. I'm having to wrap it with python code already, and my next step is the CDK. The biggest issue is that parts of the TF for github requires references, and it fetches each of those references individually (which which quickly make you hit your ratelimit, while also taking 10+ minutes for just a plan). Wrapping this in python, and doing more efficient fetches ahead of time, and feeding the references in yourself can turn a 10+ minute plan into a 20 second plan for 1000+ repos.

StreamBright 5y ago

You are on the right track. I have exactly the same story, data engineer, working on automation a lot. YAML, JSON and HCL should be purged from IaC asap. Luckily Dhall can be integrated everywhere we need YAML otherwise we would go nuts. The bigges failure of k8s is to understand that YAML does not scale to teams and beyond 100 lines, especially not good to nest it (hello ArgoCD). A type safe not accidentally Turing complete configuration language or for that matter any langue (hello Pulumi) is much better than YAML ever will be. Anyways, if I can give you an advise, drop k8s like its hot. I is much more trouble than worth. We have phased it out everywhere we could. The companies still using it are in 1 year long implementation cycles with it with 10+ consultants. It is going very well. For consultants, obviously. https://dhall-lang.org/ https://dhall-lang.org/

manish_gill 5y ago
Dhall looks interesting. Thanks! I went into the rabbithole of terraform/HCL and came up with Terragrunt to do dynamic code generation. Combining it together with generous use of `make` keeps me sane, but it's not a long term solution. Even now, there are things within the system where the boundaries should be separate, but they aren't. We literally got exhausted and burned out writing our automation, and decided to leave it as it is and then slowly build it up over the next few months. :/ PS: You're 100% right re K8s/YAML. It feels like everything in the Kubernetes world is optimized for writing the code just once. Which, of course, is easy if you already know how all the various bits fit together (true in any system). But Reading/Debugging kubernetes world is a dreadful experience. Edit: typo
- StreamBright 5y ago
  > But Reading/Debugging kubernetes world is a dreadful experience. Exactly. It is abstraction, over abstraction, over abstraction for no obvious benefits.
jjtheblunt 5y ago
"turning complete" : typo?

ludamad 5y ago

I think the YAML issues center around a simple fact, few are taking e.g. 1000+ line YAML (or YAML for X) dev seriously. Complex YAML becomes code, and requires as much e.g. refactoring, idioms, reuse, static checking. At least it's not terribly hard to make a script output it as a format

laumars 5y ago

DevOps specialist here and I agree with your complaints about IaC meaning JSON and YAML documents. sigh

benlivengood 5y ago

I've been in two production environments where IaC was done with Turing-complete languages (Google and Facebook). It's really not better (it just lets the truly impossible/unmanageable become possible) because they were both dynamically typed. If anything, Terraform comes the closest to being a good IaC language because providers form an API to production resources as opposed to unstructured code/config like Helm or Jenkins/Puppet/Chef. Production in reality is well-typed, and unstructured configuration makes static analysis very difficult and makes diffs only a guess at what will actually be changing when code changes are applied. What I'd like to see is Haskell (or similar) typeclasses as production API; make it impossible to construct malformed configurations in the first place but allow expressive functions to replace the limited templating functionality that current languages have (why can't helm templatize inside values.yaml? Why can't Terraform templatize backends and why aren't providers first-class objects?). It's very difficult to keep current IaC DRY, and if it's not DRY then it has bugs and there are misunderstandings about how environments are actually configured.

girvo 5y ago
In terms of “Haskell type classes as production API”, I assume you’ve come across Dhall? https://dhall-lang.org/ https://dhall-lang.org/
manish_gill 5y ago
One can write good and bad code in any language. It just so happens that our daily driver - procedural or functional languages, happen to be ones where a Software Engineer's mental model + Existing tooling is evolved enough to make understanding it _easier_ (as opposed to just easy). The problem with DSLs and Config languages is that someone will inevitably try to use them in a way they are not designed to be used, and then circumvent the way you're "supposed to" do things. Not only do you lose all the advances in tooling, but people will end up reinventing things that already exist in the other languages. PS: Of course, I'll pick Static Typing every time.
lima 5y ago
Cuelang is a nice compromise: https://cuelang.org https://cuelang.org Composition over encapsulation, expressive enough for complex configs, not expressive enough to be turing complete.

sofixa 5y ago

> The YAML (or heaven forbid, jsonnet) does not help. TBH, I'm convinced if the IaC bits were done in a regular Java/Go/Rust SDK style instead of mixing and matching YAML glued together via labels and tags, I would have had a much easier time. I have mixed feelings about this. I've seen some absolutely atrocious Python and JavaScript code, and i wouldn't want to use either of them, and the flexibility ( to shoot yourself in the foot) they provide. Go is better. On the other hand, YAML is just terrible. I think DSLs such as HCL are better than either option, as long as they're flexible enough while staying limit in order not to get illegible/unmaintainable.

flas9sd 5y ago

could you give me a hint why you didn't like jsonnet? I dreaded k8s yaml when needing to (write) and use helm packages and think of indenting levels, but jsonnet via grafanas tanka gave me the option to be concise, writing functions and creating reusable bits. Of course then you're not far away from using a real language "sdk style". As you so aptly put, k8s learning curve is beginner hostile and needs system knowledge before you go at it, so I'm not sure if this was also with being more familiar with the abstractions - after having looked at k8s for 2-3 years, jsonnet made me finally like interfacing with the k8s apiserver.

manish_gill 5y ago
Yeah, I think the difference is that I was a) Learning K8s and Helm Concepts (both different things) b) Reading/Debugging K8s and Helm YAML (both different things) c) Ending up at Helm Chart Configs (another config) d) Figuring out how to set all this to helm me deploy the solution I needed and customize it acc to my needs ...all within the span of a few weeks. So by the time I got to some GitHub repos which were using Jsonnet, I was basically done. I'm sure it has its uses and given enough time and experience, I can see how to utilize it more effectively.
- flas9sd 5y ago
  yes, both different things. I guess helm is still the most common entry point for people to handle the contents of a cluster. I'm checking out other suggestions in this thread (dhall), but from my personal experience I can already tell, grafanas k8s jsonnet-libs was a second chance to kubernetes I'm glad I got to know (thanks coworkers).

commandlinefan 5y ago

> regular Java/Go/Rust SDK style I thought that until I started working with Gradle (which is a build script tool who's syntax is nearly-fully-functioning Groovy). I miss Makefiles - they were strictly declarative and, while I've seen some hairy Makefiles, you'll never need a debugger to figure out what they're actually doing wrong.

notimrelakatos 5y ago

> it makes the difficult things possible, and that's what we needed Indeed, the problem of just saying "you don't need k8s" is you may accidentally invent an even worse in-house version of k8s.

boublepop 5y ago
Though for 80-90% og cases out there you seriously don’t need k8s, and when the small internal IT dept that used to “manage” a dozen virtual machines starts talking about “need to keep up with the times and use new tech” what they are really saying is “let’s gamble infrastructure stability on my personal self education in k8s for my next job” and it’s perfectly reasonable to give them a hard no.

totetsu 5y ago

any suggestions on learning materials?

manish_gill 5y ago
You'll find plethora of learning documentation, but by far, the one talk/presentation that made it click for me was this one: https://kube-decon.carson-anderson.com/ https://kube-decon.carson-anderson.com/