6 ms·

None of this is simple. We're drowning in YAML files. It's difficult to maintain. Jinja + ansible + helm... it's a joke. There must be a better way of doing al

None of this is simple. We're drowning in YAML files. It's difficult to maintain. Jinja + ansible + helm... it's a joke. There must be a better way of doing all this.

SingAlong 6y ago
I'm working on a command-line deployment tool that deploys to DigitalOcean and AWS LightSail (to start with). This is based on my experience deploying apps. I expect to finish the remaining work in the next few weeks. Can I contact you to try it out? (My email is on my profile)
- techntoke 6y ago
  You mean like Terraform?
  - SingAlong 6y ago
    I'm building a bunch of opinionated Ansible playbooks + tooling, with a Heroku-like experience.
    - leetrout 6y ago
      I’m curious- why ansible instead of terraform for dealing with the cloud provider APIs for your base layer of infrastructure?
      - SingAlong 6y ago
        TBH because Ansible is what I’ve been using for a long time. And based on what FunnyLookinHat mentions in another comment, Terraform seems to offer a lot less. I have no first-hand experience with Terraform to confirm that.
        leetrout 6y ago
        Yea it’s focused on the infrastructure layer. It has a separate state mechanism to keep things in sync that Ansible didn’t have the last time I tried only using Ansible. They’re a match made in heaven when put together IMO.
        scaryclam 6y ago
        Terraform goes quite a lot further than FunnyLookinHat mentions. I actually don't advocate for using it further than infrastructure myself (I like using specific tooling for specific jobs), but that doesn't mean it can't go a lot further (https://www.terraform.io/docs/providers/index.html https://www.terraform.io/docs/providers/index.html for a list of the providers, and it's possible to use the null provider to write some more custom things and hack arounds). I've had a lot of success coupling Terraform with provisioners like Ansible or Saltstack. Of course, if Ansible is what you're used to and it works for you, there's no real benefit to using something else right now :) I'm a big fan of Terraform, so I hope you also have a play around with it to see if it can help with what you're doing in the future.
  - FunnyLookinHat 6y ago
    Terraform is so heavy handed - I wouldn't call it a deployment tool. It's more of a means to build out infrastructure in AWS. Once you get your AWS account setup there's still virtually no tooling to actually manage deploys of new code into that infrastructure. We're going to hand roll some tooling on top of aws-cli most likely.
    - nolite 6y ago
      What's wrong with https://aws.amazon.com/codedeploy/ https://aws.amazon.com/codedeploy/ ?
      - FunnyLookinHat 6y ago
        CodeDeploy is totally fine, but honestly I'm tired of being nickel-and-dimed in AWS. I'm already paying AWS a massive amount for compute, storage, and bandwidth - I don't want to pay more just be able to use it efficiently. Maybe it's the FOSS nerd in me - but I can't stand the walled-gardens we're essentially creating in these IaaS providers.
        nolite 6y ago
        I understand.. alot of hidden costs that add up.. (just fyi though, codedeploy is free for EC2 / lambda usages: https://aws.amazon.com/codedeploy/pricing/ https://aws.amazon.com/codedeploy/pricing/ )
    - poxrud 6y ago
      It took me half an hour and a 30-line (with spaces) terraform file to get a Lightsail instance up with static ip, running wordpress.
    - techntoke 6y ago
      I agree Terraform isn't great for installing software. Cloud-init works but it can be cumbersome. If that is your major pain point though then I'd look into EKS and using Kubernetes to manage and install software. There is a CLI tool called eksctl that makes setting up an EKS instance a breeze. I don't know all the intricacies with AWS, but with Google Cloud you can setup a single node "cluster". If you're already planning on standing up at least 3 compute instances though, might as well run EKS in a cluster.
techntoke 6y ago
Ansible user Jinja templating. Might as well say Jinja + Jinja + Helm. Helm uses Go templates which are awesome. It is what Hugo is built off. The main issue is that you still have to manage indentation with Helm. YAML is easier to read than JSON and TOML. If you don't like it, then what do you suggest is better?
bamboozled 6y ago
Just moving the problem around ...
- WrtCdEvrydy 6y ago
  Abstract it up a layer - The Industry's Motto.
bbarnett 6y ago
The real problem is, people are deploying dozens of different coding languages, any technology that whimsically passes by, and replacing simple, streamlined monolith technology with 100 micro services. All of this is endlessly pushed by AWS, Google, Docker, and anyone else with a foot in the "Snag as much cash from DEVs" crowd. Other old timers will explain how they ran thousands of hits/second on 6 or 7 bare metal servers, 15 years ago, without CDNs, using LAMP, ajax/js frontends, and more. The key was code optimization, SQL queries without inane and poorly written MVC SQL logic, and the list goes on. I am relentlessly gobsmacked at how people are spending quite literally 100 to 1000 times the cash to host on AWS, using microservices. And often amused how new devs just can't get it through their head, that yes, this is 100% factually true. Docker replaced something that was already done ; identical PROD and DEV environments. All DEVs I had working with me, were issued VMs with 100% replicated PROD. VMs auto-build using debootstrap + SVN checkout. Rollbacks in prod? Handled by SVN rollbacks. When I look at the insane complexity being displayed by containers on top of VMs on top of baremetal, the MASSIVE loss in performance (yup, it's there.. especially for IO)... I just don't get it. Many DEVs have all been sold a pack of complete and total lies. I get called in again, and again by clients to reduce cost, optimize resource usage. I literally bring 1000x performance boosts to the SQL layer with minimal improvements. Anyhow. Yes, this was a rant. Sorry it's a reply to you specifically.
- secondcoming 6y ago
  It's a totally valid rant, and one that I enthusiastically agree with. I work in adtech (apologies). We have maybe 10 - 15 instances (16 vCPUs, 31GB RAM) that each handle 10k+ HTTP requests per second. There's a push to dockerise all this. I don't see the point. I've often wondered about the potential performance loss of Dockerising all this, do you have public numbers available? We recently hired a ex-Googler to a management position who claims that on GCP, running docker may actually perform better than VMs. If true, that's really interesting, but I can't find anything to back it up. Code optimisation (or even profiling) seems to be a dying art since people think that it's easier to just throw CPUs at the problem.
  - rubber_duck 6y ago
qppo 6y ago
Someone has probably already done this, but you could write a dashboard generator for all the services that can be configured with just YAML files and another generator to glue them all together into one page. Then you'd have your own NoCode startup generator

None of this is simple. We're drowning in YAML files. It's difficult to maintain. Jinja + ansible + helm... it's a joke. There must be a better way of doing all this.

SingAlong 6y ago

I'm working on a command-line deployment tool that deploys to DigitalOcean and AWS LightSail (to start with). This is based on my experience deploying apps. I expect to finish the remaining work in the next few weeks. Can I contact you to try it out? (My email is on my profile)

techntoke 6y ago
You mean like Terraform?
- SingAlong 6y ago
  I'm building a bunch of opinionated Ansible playbooks + tooling, with a Heroku-like experience.
  - leetrout 6y ago
    I’m curious- why ansible instead of terraform for dealing with the cloud provider APIs for your base layer of infrastructure?
    - SingAlong 6y ago
      TBH because Ansible is what I’ve been using for a long time. And based on what FunnyLookinHat mentions in another comment, Terraform seems to offer a lot less. I have no first-hand experience with Terraform to confirm that.
      - leetrout 6y ago
        Yea it’s focused on the infrastructure layer. It has a separate state mechanism to keep things in sync that Ansible didn’t have the last time I tried only using Ansible. They’re a match made in heaven when put together IMO.
      - scaryclam 6y ago
        Terraform goes quite a lot further than FunnyLookinHat mentions. I actually don't advocate for using it further than infrastructure myself (I like using specific tooling for specific jobs), but that doesn't mean it can't go a lot further (https://www.terraform.io/docs/providers/index.html https://www.terraform.io/docs/providers/index.html for a list of the providers, and it's possible to use the null provider to write some more custom things and hack arounds). I've had a lot of success coupling Terraform with provisioners like Ansible or Saltstack. Of course, if Ansible is what you're used to and it works for you, there's no real benefit to using something else right now :) I'm a big fan of Terraform, so I hope you also have a play around with it to see if it can help with what you're doing in the future.
- FunnyLookinHat 6y ago
  Terraform is so heavy handed - I wouldn't call it a deployment tool. It's more of a means to build out infrastructure in AWS. Once you get your AWS account setup there's still virtually no tooling to actually manage deploys of new code into that infrastructure. We're going to hand roll some tooling on top of aws-cli most likely.
  - nolite 6y ago
    What's wrong with https://aws.amazon.com/codedeploy/ https://aws.amazon.com/codedeploy/ ?
    - FunnyLookinHat 6y ago
      CodeDeploy is totally fine, but honestly I'm tired of being nickel-and-dimed in AWS. I'm already paying AWS a massive amount for compute, storage, and bandwidth - I don't want to pay more just be able to use it efficiently. Maybe it's the FOSS nerd in me - but I can't stand the walled-gardens we're essentially creating in these IaaS providers.
      - nolite 6y ago
        I understand.. alot of hidden costs that add up.. (just fyi though, codedeploy is free for EC2 / lambda usages: https://aws.amazon.com/codedeploy/pricing/ https://aws.amazon.com/codedeploy/pricing/ )
  - poxrud 6y ago
    It took me half an hour and a 30-line (with spaces) terraform file to get a Lightsail instance up with static ip, running wordpress.
  - techntoke 6y ago
    I agree Terraform isn't great for installing software. Cloud-init works but it can be cumbersome. If that is your major pain point though then I'd look into EKS and using Kubernetes to manage and install software. There is a CLI tool called eksctl that makes setting up an EKS instance a breeze. I don't know all the intricacies with AWS, but with Google Cloud you can setup a single node "cluster". If you're already planning on standing up at least 3 compute instances though, might as well run EKS in a cluster.

techntoke 6y ago

Ansible user Jinja templating. Might as well say Jinja + Jinja + Helm. Helm uses Go templates which are awesome. It is what Hugo is built off. The main issue is that you still have to manage indentation with Helm. YAML is easier to read than JSON and TOML. If you don't like it, then what do you suggest is better?

bamboozled 6y ago

Just moving the problem around ...

WrtCdEvrydy 6y ago
Abstract it up a layer - The Industry's Motto.

bbarnett 6y ago

The real problem is, people are deploying dozens of different coding languages, any technology that whimsically passes by, and replacing simple, streamlined monolith technology with 100 micro services. All of this is endlessly pushed by AWS, Google, Docker, and anyone else with a foot in the "Snag as much cash from DEVs" crowd. Other old timers will explain how they ran thousands of hits/second on 6 or 7 bare metal servers, 15 years ago, without CDNs, using LAMP, ajax/js frontends, and more. The key was code optimization, SQL queries without inane and poorly written MVC SQL logic, and the list goes on. I am relentlessly gobsmacked at how people are spending quite literally 100 to 1000 times the cash to host on AWS, using microservices. And often amused how new devs just can't get it through their head, that yes, this is 100% factually true. Docker replaced something that was already done ; identical PROD and DEV environments. All DEVs I had working with me, were issued VMs with 100% replicated PROD. VMs auto-build using debootstrap + SVN checkout. Rollbacks in prod? Handled by SVN rollbacks. When I look at the insane complexity being displayed by containers on top of VMs on top of baremetal, the MASSIVE loss in performance (yup, it's there.. especially for IO)... I just don't get it. Many DEVs have all been sold a pack of complete and total lies. I get called in again, and again by clients to reduce cost, optimize resource usage. I literally bring 1000x performance boosts to the SQL layer with minimal improvements. Anyhow. Yes, this was a rant. Sorry it's a reply to you specifically.

secondcoming 6y ago
It's a totally valid rant, and one that I enthusiastically agree with. I work in adtech (apologies). We have maybe 10 - 15 instances (16 vCPUs, 31GB RAM) that each handle 10k+ HTTP requests per second. There's a push to dockerise all this. I don't see the point. I've often wondered about the potential performance loss of Dockerising all this, do you have public numbers available? We recently hired a ex-Googler to a management position who claims that on GCP, running docker may actually perform better than VMs. If true, that's really interesting, but I can't find anything to back it up. Code optimisation (or even profiling) seems to be a dying art since people think that it's easier to just throw CPUs at the problem.
- rubber_duck 6y ago

qppo 6y ago

Someone has probably already done this, but you could write a dashboard generator for all the services that can be configured with just YAML files and another generator to glue them all together into one page. Then you'd have your own NoCode startup generator