6 ms·

So I'm heading a shop with some serious legacy deployment woes. I've got plans for everything but scaling. Now, we're likely to hire out Ops eventually, but for

So I'm heading a shop with some serious legacy deployment woes. I've got plans for everything but scaling. Now, we're likely to hire out Ops eventually, but for now I'd like to learn devops (backend eng currently) and cut my teeth by getting the ball rolling on a slightly more robust setup. With that said, what do you recommend? We're a small shop, and our auto scaling needs a very very minimal. We're non-public facing, so our user base is pretty predictable. Yet, I'd like to migrate us to a more modern and mature system. Naturally I was looking towards K8s because of the overwhelming support I see for K8s on HN/etc, but your comment makes me question that. Honestly, it's the vast options that really screw with my head. There are just so many tools. And where they lack is not immediately evident from the outside. It's difficult to pick a direction. At the moment I'm reviewing Nomad, as it sounds interesting to work with our existing by-hand infrastructure.

politelemon 8y ago
I'd suggest going in baby steps. Start with the absolute basics - standalone docker containers + docker-compose. Work in those containers, deploy those same containers. Build your deployment pipeline. That'll get you familiar with the workings of containers in the docker context, and you'll be forced to iron out issues you may find with this way of working. Then slowly start to spread further - you could start with docker swarm, which is a smaller next-step for devs. Again it's to help you figure out what kind of deployment pipelines you would create and what constraints you need to introduce. The reason I call it a smaller next-step is because the docker compose file used for development is the same one you can use for deployment to standalone or swarm. By the time you start hitting the limitations of swarm, you should be ready to move to k8s and look at the various options available to you.
- djsumdog 8y ago
  I like this. This is good advice. Containers are still fairly portable and I've worked at a shop where we moved from CoreOS+Consul to Marathon/DCOS and there were almost no app changes. That is one of the big benefits of Docker. We had some stuff just running on nodes with Docker/docker-compose and even that stuff slowly got migrated over to the big platform.
  - asdkhadsj 8y ago
    Agreed, this method sounds nice. The only thing I'd like to find on top of it is some software to manage apps across container resources and physical resources. Ie, I've got some non-cloud old windows boxes that I want to deploy to, so ideally I need a mechanism further abstracts that. I'm currently looking at Nomad, though i'm very early in research.
- JonasJSchreiber 8y ago
  Currently on this path almost exactly. Problem description and steps all the way up to limitations of swarm. Our swarm currently handles 1M/requests per day we haven't really hit any limitations yet. Good advice though, really
toomuchtodo 8y ago
Have you considered hiring an ops/devops consultant for a few hours to help provide direction? Even at low four figures, it would be money well spent based on where you're currently at.
brightball 8y ago
What you are going through is why “Just use Heroku” is still great advice from wise people. There are so many options out there that until you actually hit the point where Heroku is too expensive or no longer makes sense...Heroku is your best option (if AWS is an option).
- asdkhadsj 8y ago
  I'm hoping to keep it cheap though, and cloud deployment services scare me a bit on price. We're a ~15y old company (of which I'm recent) and in a domain where margins are fairly thin. I'd rather spend the sweat learning this (or hiring for it) than paying Heroku, fwiw. Plus lock-in scares me.. a lot.
  - brightball 8y ago
    Oh, Heroku doesn’t lock you in. It’s just running your code on containers. There’s even an open source library called Dokku that uses their open source build packs to give run the same thing in Docker containers. The minute you price out the time and training investment to factor into your deployment solution, Heroku starts winning in a landslide. Existing scale certainly changes things, but the fact that it forces you to go ahead and build for 12 factor makes moving elsewhere pretty simple.
    - asdkhadsj 8y ago
      Good to know, thank you! On that note, we have some odd ball legacy machines that are dictated/required by our customers. Things like WinXP. So I'm surveying Nomad now as a potential to support non-traditional OSs (though I wouldn't be surprised in Nomad doesn't support WinXP). Do you recommend anything for odd deployment needs like this? I imagine we'll write custom plugins to handle the odd ones, since who the hell releases builds for WinXP or Win95 these days lol. Nevertheless, I figured I'd ask. Especially since broader solutions like Heroku are not going to help our WindowsX deployment woes. (note, they're physical machines currently, though virtualization might be possible)
archgrove 8y ago
You almost certainly don't need containers. Just use a PAAS (platform as a service). Heroku is the "go to" default, and will be worth much more than they charge. If you're worried about scaling, I've run off the shelve PAAS providers past 5000+ TPS without any issues. If for some reason you can't use Heroku, you could try a hosted Cloudfoundry (https://run.pivotal.io https://run.pivotal.io). If you want something that hybridises PAAS-like behaviour with the big IAAS services, Amazon's Elastic Beanstalk isn't entirely awful. All these setups largely honour the PAAS haiku of "Here is my source code / Run it on the cloud for me / I do not care how" (https://twitter.com/sramji/status/599255828772716544?lang=en https://twitter.com/sramji/status/599255828772716544?lang=en). Don't waste your time on getting container orchestration going, or even containerising your app yet. For internal, low scaling, app-focused scenarios, you'd be mad to go down this road _unless_ your code has been broken into many discrete deployable artefacts _and_ you need to run them all on the same machine (either locally or in production). You have an app. You want to run it. PAASes do _exactly_ this. Edit: I see you're worried about costs. PAAS are pricier than the equivalent AWS instance when you consider raw compute, sure. However, factor in your time in creating the same level of redundancy, deployability and monitoring, I'd be shocked if Heroku wasn't an order of magnitude cheaper. Note that PWS Cloudfoundry will host you any 12 factor app for $21.60/GbRAM/month with no bandwidth limit you'll ever hit.
- asdkhadsj 8y ago
  I should have made it more clear, but one need of mine (as mentioned in another comment) is a deployment system that supports the ability to manage very old, archaic, wonky systems. WinXP, Win95, etc. Do you recommend any tooling to support this? PAASs aren't going to help me much in managing our resources like physical legacy machines. Thoughts?
  - archgrove 8y ago
    Ah, a bit out of my experience alas. Sorry! However, I'm still not sure how container orchestration would help with this. I can still endorse PAAS for any "cloud" applications you may want to deploy.
xorcist 8y ago
To be honest it sounds like you probably shouldn't. There is a risk to researching "the best" technology, it can be a full time job and you still wouldn't get any closer to your goal. You are likely to have a lot of low hanging fruit in a legacy systems shop. Start with those and move yourself forward from there. Without any specific knowledge of your situation it's hard to be anything but generic advice. But first you need eyes and ears. Under no circumstances should customers know of problems before you do. Are you monitoring the right things? If there is a product, take a long hard look at the build process. When those mostly work without that special someone that always does them because no one else can, start mapping out all the infrastructure, document it and place it under version control. Implement a configuration management system. No exceptions. With repeatable builds and repeatable infrastructure you are in a much better place to see what your real requirements are.

politelemon 8y ago

I'd suggest going in baby steps. Start with the absolute basics - standalone docker containers + docker-compose. Work in those containers, deploy those same containers. Build your deployment pipeline. That'll get you familiar with the workings of containers in the docker context, and you'll be forced to iron out issues you may find with this way of working. Then slowly start to spread further - you could start with docker swarm, which is a smaller next-step for devs. Again it's to help you figure out what kind of deployment pipelines you would create and what constraints you need to introduce. The reason I call it a smaller next-step is because the docker compose file used for development is the same one you can use for deployment to standalone or swarm. By the time you start hitting the limitations of swarm, you should be ready to move to k8s and look at the various options available to you.

djsumdog 8y ago
I like this. This is good advice. Containers are still fairly portable and I've worked at a shop where we moved from CoreOS+Consul to Marathon/DCOS and there were almost no app changes. That is one of the big benefits of Docker. We had some stuff just running on nodes with Docker/docker-compose and even that stuff slowly got migrated over to the big platform.
- asdkhadsj 8y ago
  Agreed, this method sounds nice. The only thing I'd like to find on top of it is some software to manage apps across container resources and physical resources. Ie, I've got some non-cloud old windows boxes that I want to deploy to, so ideally I need a mechanism further abstracts that. I'm currently looking at Nomad, though i'm very early in research.
JonasJSchreiber 8y ago
Currently on this path almost exactly. Problem description and steps all the way up to limitations of swarm. Our swarm currently handles 1M/requests per day we haven't really hit any limitations yet. Good advice though, really

toomuchtodo 8y ago

Have you considered hiring an ops/devops consultant for a few hours to help provide direction? Even at low four figures, it would be money well spent based on where you're currently at.

brightball 8y ago

What you are going through is why “Just use Heroku” is still great advice from wise people. There are so many options out there that until you actually hit the point where Heroku is too expensive or no longer makes sense...Heroku is your best option (if AWS is an option).

asdkhadsj 8y ago
I'm hoping to keep it cheap though, and cloud deployment services scare me a bit on price. We're a ~15y old company (of which I'm recent) and in a domain where margins are fairly thin. I'd rather spend the sweat learning this (or hiring for it) than paying Heroku, fwiw. Plus lock-in scares me.. a lot.
- brightball 8y ago
  Oh, Heroku doesn’t lock you in. It’s just running your code on containers. There’s even an open source library called Dokku that uses their open source build packs to give run the same thing in Docker containers. The minute you price out the time and training investment to factor into your deployment solution, Heroku starts winning in a landslide. Existing scale certainly changes things, but the fact that it forces you to go ahead and build for 12 factor makes moving elsewhere pretty simple.
  - asdkhadsj 8y ago
    Good to know, thank you! On that note, we have some odd ball legacy machines that are dictated/required by our customers. Things like WinXP. So I'm surveying Nomad now as a potential to support non-traditional OSs (though I wouldn't be surprised in Nomad doesn't support WinXP). Do you recommend anything for odd deployment needs like this? I imagine we'll write custom plugins to handle the odd ones, since who the hell releases builds for WinXP or Win95 these days lol. Nevertheless, I figured I'd ask. Especially since broader solutions like Heroku are not going to help our WindowsX deployment woes. (note, they're physical machines currently, though virtualization might be possible)

archgrove 8y ago

You almost certainly don't need containers. Just use a PAAS (platform as a service). Heroku is the "go to" default, and will be worth much more than they charge. If you're worried about scaling, I've run off the shelve PAAS providers past 5000+ TPS without any issues. If for some reason you can't use Heroku, you could try a hosted Cloudfoundry (https://run.pivotal.io https://run.pivotal.io). If you want something that hybridises PAAS-like behaviour with the big IAAS services, Amazon's Elastic Beanstalk isn't entirely awful. All these setups largely honour the PAAS haiku of "Here is my source code / Run it on the cloud for me / I do not care how" (https://twitter.com/sramji/status/599255828772716544?lang=en https://twitter.com/sramji/status/599255828772716544?lang=en). Don't waste your time on getting container orchestration going, or even containerising your app yet. For internal, low scaling, app-focused scenarios, you'd be mad to go down this road _unless_ your code has been broken into many discrete deployable artefacts _and_ you need to run them all on the same machine (either locally or in production). You have an app. You want to run it. PAASes do _exactly_ this. Edit: I see you're worried about costs. PAAS are pricier than the equivalent AWS instance when you consider raw compute, sure. However, factor in your time in creating the same level of redundancy, deployability and monitoring, I'd be shocked if Heroku wasn't an order of magnitude cheaper. Note that PWS Cloudfoundry will host you any 12 factor app for $21.60/GbRAM/month with no bandwidth limit you'll ever hit.

asdkhadsj 8y ago
I should have made it more clear, but one need of mine (as mentioned in another comment) is a deployment system that supports the ability to manage very old, archaic, wonky systems. WinXP, Win95, etc. Do you recommend any tooling to support this? PAASs aren't going to help me much in managing our resources like physical legacy machines. Thoughts?
- archgrove 8y ago
  Ah, a bit out of my experience alas. Sorry! However, I'm still not sure how container orchestration would help with this. I can still endorse PAAS for any "cloud" applications you may want to deploy.

xorcist 8y ago

To be honest it sounds like you probably shouldn't. There is a risk to researching "the best" technology, it can be a full time job and you still wouldn't get any closer to your goal. You are likely to have a lot of low hanging fruit in a legacy systems shop. Start with those and move yourself forward from there. Without any specific knowledge of your situation it's hard to be anything but generic advice. But first you need eyes and ears. Under no circumstances should customers know of problems before you do. Are you monitoring the right things? If there is a product, take a long hard look at the build process. When those mostly work without that special someone that always does them because no one else can, start mapping out all the infrastructure, document it and place it under version control. Implement a configuration management system. No exceptions. With repeatable builds and repeatable infrastructure you are in a much better place to see what your real requirements are.