4 ms·

This is true to some extent. I'm building out the infrastructure for my company, and there is a lot of effort on my part required to automate stuff. Something

by lykr0n 7y ago

This is true to some extent. I'm building out the infrastructure for my company, and there is a lot of effort on my part required to automate stuff. Something like kafka requires figuring out the configuration you want, putting that into Salt, adding the correct configuration options, deploying it and making sure zookeeper works, and then generating certificates and what not. It's not a simple process. Setting up monitoring, and other things like floating IPs is a pain. Custom wrappers for Terraform scripts and other components required to deploy the systems you need to run an app. It's a lot

sosuke 7y ago
You know that your job has abstracted into insanity when use the names of products but the sentence doesn't read like it. Kafka requirements adding salt to keep your zookeeper happy. JavaScript went into this stage not too many years back. "I test my react code with mocha chai cucumber in gulp with puppeteer."
- Legogris 7y ago
  In this case, I'm not sure if it's abstraction-insanity as much as having common solutions to common problems. You know what would be an insanity of abstraction? Rolling all that yourself. Given that there are better and worse abstractions, most are not suitable for most use-cases, a lot of times simpler and more centralized is actually more stable. But when you start needing high-availability-coordination you're going to be damn happy that there are things such as Consul, etcd and Zookeeper. As for Salt; by all means, start off by having bash scripts in source control, which at some points becomes a pain-point and you'll realize you need some kind of configuration management to not go insane. Several of these tools are just putting a framework and common language for tasks we have to do either way.
anonytrary 7y ago
Ugh this comment triggers my hate for the current state of infrastructure. It's just configuration hell at this point. We thought all of these wrappers and services would make infrastructure easy, and they did to a certain degree, but it also created brand new problems that are arguably just as annoying as the old ones. Edit: As a developer, I would rather sketch out the infrastructure that I need, then hand it off to someone whose entire job is to set up infrastructure. I work at a small startup where the devs are still doing a lot of infrastructure, and all it does is create technical debt. Without a solid team that owns infrastructural concerns, developers just end up digging themselves a grave.
- weberc2 7y ago
  I think the problems are mostly around configuration that is so complex and redundant that we need programmatic abstractions to deal with it—we need something like a general purpose programming language to DRY up and simplify that configuration, but the powers that be are clinging desperately to YAML (despite encoding a shitty, half-baked AST on top of it a la CloudFormation or “generating” YAML with text templates a la Helm) as the human/configuration interface presumably because the “it’s as easy as YAML!” is such a good marketing schtick. A certain amount of configuration complexity will always be there, but there’s still a lot of incidental complexity that could be cut away if we just generated these config a with a general purpose language.
  - imtringued 7y ago
    There are worse things than YAML. Dealing with hundreds of snow flake configuration formats that do not follow common sense rules cost me more time than working around mistakes in YAML. There are obviously some really, really stupid ideas like generating YAML with text templates but the problem isn't with YAML. It's the tool that generated the YAML file.
  - chucky_z 7y ago
    HCL2 feels like the farthest ahead right now, if you don't want to deal with straight up python or ruby.
  - greyskull 7y ago
    In case you haven't seen AWS' answer: https://github.com/aws/aws-cdk https://github.com/aws/aws-cdk
- hinkley 7y ago
  It doesn't feel like we've come very far at all from the first time I realized I could use nginx as a forward proxy and have the code just make all service calls to localhost. That seriously reduced the complexity for the development team. Some days I just want a load balancer that can route traffic to whatever servers are up now and then I'd just have Docker, a service registry, a load balancer and I'd be done for a while. The easy bits should be easy, and they can't even manage that.
lwansbrough 7y ago
Isn’t this stuff supposed to be solved by products like DC/OS?
- halbritt 7y ago
  DC/OS is dead, Jim. Well, kinda. Mesosphere changed their name to D2IQ and is now supporting kubernetes. You can, however install the banzaicloud kafka operator with only a few commands and very little config and yield a production-grade kafka cluster pretty quickly. https://github.com/banzaicloud/kafka-operator https://github.com/banzaicloud/kafka-operator

by lykr0n 7y ago

sosuke 7y ago

You know that your job has abstracted into insanity when use the names of products but the sentence doesn't read like it. Kafka requirements adding salt to keep your zookeeper happy. JavaScript went into this stage not too many years back. "I test my react code with mocha chai cucumber in gulp with puppeteer."

Legogris 7y ago
In this case, I'm not sure if it's abstraction-insanity as much as having common solutions to common problems. You know what would be an insanity of abstraction? Rolling all that yourself. Given that there are better and worse abstractions, most are not suitable for most use-cases, a lot of times simpler and more centralized is actually more stable. But when you start needing high-availability-coordination you're going to be damn happy that there are things such as Consul, etcd and Zookeeper. As for Salt; by all means, start off by having bash scripts in source control, which at some points becomes a pain-point and you'll realize you need some kind of configuration management to not go insane. Several of these tools are just putting a framework and common language for tasks we have to do either way.

anonytrary 7y ago

Ugh this comment triggers my hate for the current state of infrastructure. It's just configuration hell at this point. We thought all of these wrappers and services would make infrastructure easy, and they did to a certain degree, but it also created brand new problems that are arguably just as annoying as the old ones. Edit: As a developer, I would rather sketch out the infrastructure that I need, then hand it off to someone whose entire job is to set up infrastructure. I work at a small startup where the devs are still doing a lot of infrastructure, and all it does is create technical debt. Without a solid team that owns infrastructural concerns, developers just end up digging themselves a grave.

weberc2 7y ago
I think the problems are mostly around configuration that is so complex and redundant that we need programmatic abstractions to deal with it—we need something like a general purpose programming language to DRY up and simplify that configuration, but the powers that be are clinging desperately to YAML (despite encoding a shitty, half-baked AST on top of it a la CloudFormation or “generating” YAML with text templates a la Helm) as the human/configuration interface presumably because the “it’s as easy as YAML!” is such a good marketing schtick. A certain amount of configuration complexity will always be there, but there’s still a lot of incidental complexity that could be cut away if we just generated these config a with a general purpose language.
- imtringued 7y ago
  There are worse things than YAML. Dealing with hundreds of snow flake configuration formats that do not follow common sense rules cost me more time than working around mistakes in YAML. There are obviously some really, really stupid ideas like generating YAML with text templates but the problem isn't with YAML. It's the tool that generated the YAML file.
- chucky_z 7y ago
  HCL2 feels like the farthest ahead right now, if you don't want to deal with straight up python or ruby.
- greyskull 7y ago
  In case you haven't seen AWS' answer: https://github.com/aws/aws-cdk https://github.com/aws/aws-cdk
hinkley 7y ago
It doesn't feel like we've come very far at all from the first time I realized I could use nginx as a forward proxy and have the code just make all service calls to localhost. That seriously reduced the complexity for the development team. Some days I just want a load balancer that can route traffic to whatever servers are up now and then I'd just have Docker, a service registry, a load balancer and I'd be done for a while. The easy bits should be easy, and they can't even manage that.

lwansbrough 7y ago

Isn’t this stuff supposed to be solved by products like DC/OS?

halbritt 7y ago
DC/OS is dead, Jim. Well, kinda. Mesosphere changed their name to D2IQ and is now supporting kubernetes. You can, however install the banzaicloud kafka operator with only a few commands and very little config and yield a production-grade kafka cluster pretty quickly. https://github.com/banzaicloud/kafka-operator https://github.com/banzaicloud/kafka-operator