5 ms·

Increasingly it seems like people think there is a need for tools that bridge the gap between simple and complicated IaC. I think this is a better idea than a b

by flurie 4y ago

Increasingly it seems like people think there is a need for tools that bridge the gap between simple and complicated IaC. I think this is a better idea than a bespoke cli tool that does something similar[1], but I don't think it escapes the core problems that tool has, which are the same problems that people and organizations encounter with other complicated infrastructure/deployment tooling: getting good at these things is a time sink, and shortcuts that magic away some of the pain also magic away the ability to learn. [1] https://github.com/nathants/libaws https://github.com/nathants/libaws

igorzij 4y ago
This is an amazing project you have there! We have a very similar philosophy - but taking it one step further. We believe that developers do not and should not care about things like VPCs, security groups, etc. We believe that the current state of infrastructure-as-code is akin to that of assembly language. Low-level and not truly reusable - because using Terraform modules still requires understanding of low-level concepts. What we are trying to build is essentially a compiler of "human" concepts into Terraform. But because even simply running Terraform is a daunting task to un-initiated developers, we also need to make this part easy. Hence we run it on our servers. This way the user is completely shielded from complexity without losing the ability to go low-level and customise whatever they like.
- rajamaka 4y ago
  As a Devops engineer who spends all day provisioning infra with IaC, I always wonder why developers intentionally avoid concepts like "security groups and VPCs". They concepts are actually really simple and sure as shit are simpler than learning nearly any dev fundamentals. Sometimes I get concerned that one day developers will figure out how simple our job is compared to actual programming, but I enjoy the perpetuation of the mystique of basic infra concepts in the meantime for the sake of my employment.
  - francis-io 4y ago
    People have a finite amount of bandwidth to focus on certain things. A DevOps Engineer should be a programmer focused on infrastructure, but we share toolkits.
    - skyzyx 4y ago
      This.
  - devonkim 4y ago
    As an infrastructure engineer it is my job IMO to provide tooling, information, and services to reduce cognitive overhead for my peers and customers. For my job wouldn’t exist unless we had software to deliver in the first place. Utilizing existing cognates developers have is important for communicating the trade-offs we make in infrastructure. Developers are already asked a lot of them these days and only increasing (part of why I’m not a straightforward feature developer whatsoever anymore). With some level of empathy for our stakeholders we can be more effective as a team and customer-vendor and so forth. I am not really concerned by developers figuring out that this is that easy partly because it frankly isn’t yet to help make systems scale. And the day infrastructure is legitimately brain dead simple I’ll breathe a huge sigh of relief that we managed to tame this mess of short sighted decisions and bits and bytes plaguing humanity. So sad to see how something so important to our society is this stupidly unnecessarily complex, but once we look into many other disciplines we can see how they have their issues as well and that we as engineers and laymen tend to just be blind to the inherent complexities of human systems.
- web007 4y ago
  > We believe that developers do not and should not care about things like VPCs, security groups, etc. I very strongly and fundamentally disagree with this sentiment. I know they don't, but they should! Developers should understand how and where and why/why not their code is deployed. They should understand what their security model is AND HOW IT WORKS, because if they don't they will 100% without fail open their SG to 0.0.0.0/0 and embed hard coded passwords (usually "password", or "p@ssw0rd" at best). They should know what a VPC is, but maybe they don't need to know anything beyond "private network" and probably "NAT is involved". Hiding these concepts is a double-edged sword. You get the simplicity of "make me a database" but don't understand why or how it works. You don't have to fiddle with security groups, until you do and then because you don't understand them you go with 0/0. You don't have to think about VPCs until you rack up a $10k bill because you never considered how you could route between instances vs round-tripping out of the VPC and back in through a LB - twice.
  - igorzij 4y ago
    I see your point clearly, and it's a good one. However, consider the following counterargument: the exact same thing could be said (and was said) extensively about low-level programming concepts such as registers, later pointers, memory allocation, etc. It was essential for every software engineer to master... until it wasn't. Same with lower-level OS fundamentals. Same thing earlier with pre-PC hardware. We've come full circle with the present-day cloud providers like AWS. 15 years ago they started as simplification, but today they are boxes of 200+ specialised building blocks that are anything but trivial to put together. A human specialist has to be involved. But why? This complexity is completely artificial, its sole purpose is to keep their customers locked in. This must and will be solved with better tooling. That's what we are trying to make.
    - skyzyx 4y ago
      I disagree that it’s complex for the sake of vendor lock-in. Having spent four years working at AWS I can tell you that it’s complex for two reasons: 1. Some of the stuff is genuinely complex. 2. The software engineers and development managers regularly ignore the user experience teams and their feedback.
- flurie 4y ago
  I apologize if I made it appear as though that project were mine, because it is not, but I do consider yours a better version of the same concept. At a basic level, I agree with you on the current state of IaC, but I feel like things such as CDK and Pulumi are attempting to address that issue more directly, so I'm curious why you decided not to use those instead. I believe it is much more difficult to get out of "cloud assembly" than these tools make it seem, and I think you have an interesting approach, but I agree with the sibling commenter who said that attempting to escape that completely is a bad idea. One of the major problems solutions like this one tend to have is that they have sharp edges about what they can and cannot handle, and they also have places where there's a steep drop back into cloud assembly, for which the user attempting to avoid it in the first place is now woefully unprepared. This is true of AWS's myriad of simplification solutions, and this is also true of tools like Serverless. If I give a naïve user access to a service catalog that they have the ability to override at a low level, I might as well teach them how to handle the low level in the first place. It's not clear to me how you plan to address that. Abstracting away the need to run Terraform directly is useful, but there are plenty of products that have made that their focus, including HashiCorp's, so I'm not sure how you differentiate yourself from those as well.

by flurie 4y ago

igorzij 4y ago

This is an amazing project you have there! We have a very similar philosophy - but taking it one step further. We believe that developers do not and should not care about things like VPCs, security groups, etc. We believe that the current state of infrastructure-as-code is akin to that of assembly language. Low-level and not truly reusable - because using Terraform modules still requires understanding of low-level concepts. What we are trying to build is essentially a compiler of "human" concepts into Terraform. But because even simply running Terraform is a daunting task to un-initiated developers, we also need to make this part easy. Hence we run it on our servers. This way the user is completely shielded from complexity without losing the ability to go low-level and customise whatever they like.

rajamaka 4y ago
As a Devops engineer who spends all day provisioning infra with IaC, I always wonder why developers intentionally avoid concepts like "security groups and VPCs". They concepts are actually really simple and sure as shit are simpler than learning nearly any dev fundamentals. Sometimes I get concerned that one day developers will figure out how simple our job is compared to actual programming, but I enjoy the perpetuation of the mystique of basic infra concepts in the meantime for the sake of my employment.
- francis-io 4y ago
  People have a finite amount of bandwidth to focus on certain things. A DevOps Engineer should be a programmer focused on infrastructure, but we share toolkits.
  - skyzyx 4y ago
    This.
- devonkim 4y ago
  As an infrastructure engineer it is my job IMO to provide tooling, information, and services to reduce cognitive overhead for my peers and customers. For my job wouldn’t exist unless we had software to deliver in the first place. Utilizing existing cognates developers have is important for communicating the trade-offs we make in infrastructure. Developers are already asked a lot of them these days and only increasing (part of why I’m not a straightforward feature developer whatsoever anymore). With some level of empathy for our stakeholders we can be more effective as a team and customer-vendor and so forth. I am not really concerned by developers figuring out that this is that easy partly because it frankly isn’t yet to help make systems scale. And the day infrastructure is legitimately brain dead simple I’ll breathe a huge sigh of relief that we managed to tame this mess of short sighted decisions and bits and bytes plaguing humanity. So sad to see how something so important to our society is this stupidly unnecessarily complex, but once we look into many other disciplines we can see how they have their issues as well and that we as engineers and laymen tend to just be blind to the inherent complexities of human systems.
web007 4y ago
> We believe that developers do not and should not care about things like VPCs, security groups, etc. I very strongly and fundamentally disagree with this sentiment. I know they don't, but they should! Developers should understand how and where and why/why not their code is deployed. They should understand what their security model is AND HOW IT WORKS, because if they don't they will 100% without fail open their SG to 0.0.0.0/0 and embed hard coded passwords (usually "password", or "p@ssw0rd" at best). They should know what a VPC is, but maybe they don't need to know anything beyond "private network" and probably "NAT is involved". Hiding these concepts is a double-edged sword. You get the simplicity of "make me a database" but don't understand why or how it works. You don't have to fiddle with security groups, until you do and then because you don't understand them you go with 0/0. You don't have to think about VPCs until you rack up a $10k bill because you never considered how you could route between instances vs round-tripping out of the VPC and back in through a LB - twice.
- igorzij 4y ago
  I see your point clearly, and it's a good one. However, consider the following counterargument: the exact same thing could be said (and was said) extensively about low-level programming concepts such as registers, later pointers, memory allocation, etc. It was essential for every software engineer to master... until it wasn't. Same with lower-level OS fundamentals. Same thing earlier with pre-PC hardware. We've come full circle with the present-day cloud providers like AWS. 15 years ago they started as simplification, but today they are boxes of 200+ specialised building blocks that are anything but trivial to put together. A human specialist has to be involved. But why? This complexity is completely artificial, its sole purpose is to keep their customers locked in. This must and will be solved with better tooling. That's what we are trying to make.
  - skyzyx 4y ago
    I disagree that it’s complex for the sake of vendor lock-in. Having spent four years working at AWS I can tell you that it’s complex for two reasons: 1. Some of the stuff is genuinely complex. 2. The software engineers and development managers regularly ignore the user experience teams and their feedback.
flurie 4y ago
I apologize if I made it appear as though that project were mine, because it is not, but I do consider yours a better version of the same concept. At a basic level, I agree with you on the current state of IaC, but I feel like things such as CDK and Pulumi are attempting to address that issue more directly, so I'm curious why you decided not to use those instead. I believe it is much more difficult to get out of "cloud assembly" than these tools make it seem, and I think you have an interesting approach, but I agree with the sibling commenter who said that attempting to escape that completely is a bad idea. One of the major problems solutions like this one tend to have is that they have sharp edges about what they can and cannot handle, and they also have places where there's a steep drop back into cloud assembly, for which the user attempting to avoid it in the first place is now woefully unprepared. This is true of AWS's myriad of simplification solutions, and this is also true of tools like Serverless. If I give a naïve user access to a service catalog that they have the ability to override at a low level, I might as well teach them how to handle the low level in the first place. It's not clear to me how you plan to address that. Abstracting away the need to run Terraform directly is useful, but there are plenty of products that have made that their focus, including HashiCorp's, so I'm not sure how you differentiate yourself from those as well.