11 ms·

I'm wondering if Kubernetes would be the right choice in our use case. Or if something like Ansible would be better suited and easier to setup and use? I'll so

I'm wondering if Kubernetes would be the right choice in our use case. Or if something like Ansible would be better suited and easier to setup and use? I'll soon have to manage 50 remote bare-metal servers, all of them in different cities. Each one of them will run the same dockerized application. It's not 1 application distributed in 50 servers; it's 50 instances of the same application running in 50 servers, independently. A frequent use case is to deploy a new version of this application on every server (nowadays I do it manually, it's OK since I manage only like 10 servers). A nice-to-have would be to avoid downtimes also, when I update the application (nowadays I have a downtime of 2-5 minutes (when it goes well), which matters for us).

aprdm 7y ago
I would personally go with ansible and consul if your application is something like a rest api, should be trivial to setup.
clhodapp 7y ago
At a high level, your use-case maps well to a Kubernetes DaemonSet, which is a resource that represents the desire to run a copy of something containerized on every node within your cluster. Kubernetes would give you some things that might be useful: * A unified control plane (monitoring, statusing) * An automated rolling-upgrade system * "self-healing" in the form of automatically restarting your containers if they go awry However, for your use-case, it is likely more effort than it's worth: * Settin up and maintaining Kubernetes on bare metal servers is a huge amount of work * A lot of Kubernetes features are not needed for your use-case * The Kubernetes networking model essentially requires you to set up private network spanning all of your sites in order to work but provides almost no benefit for your use-case
- kissgyorgy 7y ago
  Technically you could use a DaemonSet for that, but the primary use-case is not that for DaemonSets. It should be used for "daemon" containers, like logging daemons (e.g. fluentd), network plugin daemons like calico, flannel or monitoring like prometheus. If you want to run multiple instances of your application, you should define a replicaset or deployment with setting Pod affinity instead.
  - clhodapp 7y ago
    That's super fiddly though: you will be leaning on (anti) affinity and counts to retain the 1:1 relationship between hosts and pods. I would argue that daemonsets aren't for any specific workload, they're more about how you want scheduling to happen (1 per node, on some subset of your nodes).
    - manigandham 7y ago
      This is where K8S becomes messy because of its rapid development history. Applications should just be a `Deployment` manifest with the type of distribution being a child setting (X replicas with affinity or 1-per-host). There's no reason to have competing ReplicaSets and DaemonSets just for this detail.
      - kissgyorgy 7y ago
        I disagree on this one, DaemonSets are for really a different use-case. There is no replicas field you can set, and strictly 1 Pod will run on 1 node. A Deployment is more flexible, because you can run even more Pods in different cities if necessary. What if there are more/less nodes are in different cities or different specs, which could and should run more Pods? You can't fine-tune DaemonSets for those cases, it really is for daemons.
        manigandham 7y ago
        It's not about use-case, it's that "X number of replicas with affinity" OR "1-per-node" should just be a setting within a Deployment. No need to use a completely separate DaemonSet definition just to run 1-per-node.
dragonsh 7y ago
No kubernetes will be a bad choice for your use case. But if you learn all the intricacies and manage a behemoth besides your own application it will help in your resume to brag that you are a modern devops and a fashionable one using kubernetes. If you need a practical suggestion use simple solution like LXD [1]. But as you are already using docker stay with it and ansible. As you are using ansible you are already doing infrastructure as a code, you are way ahead of curve already. Probably when you grow a bit bigger use packer [2] and terraform [3], but I think ansible will do just fine. Kubernetes was designed for Google kind of problem the burden to maintain is quite high with lots of moving parts, unless you use Google GKE or Amazon or managed kubernetes service. So don't fall for it unless you need it for your resume. [1] https://linuxcontainers.org/ https://linuxcontainers.org/ [2] https://www.packer.io/intro/ https://www.packer.io/intro/ [3] https://www.terraform.io/ https://www.terraform.io/
- samvher 7y ago
  GKE and EKS are very expensive which in my opinion makes using them unattractive for smaller scale projects. But Digital Ocean now offers managed Kubernetes for free, I've played around with it and it actually seems really easy to set up and manage. Would be great if the competition would follow suit.
  - jorams 7y ago
    > GKE and EKS are very expensive > Digital Ocean now offers managed Kubernetes for free GKE is just as "free" as DO, in that you only pay for the nodes you actually use (those nodes are probably a bit more expensive, admittedly). EKS is the outlier in this trio in that you also have to pay (a lot) for the master. Could you clarify what GKE could do to "follow suit"?
    - samvher 7y ago
      It seems, nothing! I was just wrong about GKE being expensive, not sure why I thought that (though it seems that until late 2017 they did have costs per cluster).
deleted 7y ago
[deleted]
kissgyorgy 7y ago
Kubernetes is supposed to installed nodes close to each other, typically in the same server room, or datacenter, otherwise the latency would kill the etcd cluster. Ansible is perfectly fine and simple for your use case, why would you bother even thinking about overcomplicating it? Simplicity is king!
rochacon 7y ago
No, as @dragonsh said, Kubernetes will be a bad choice for you. Kubernetes is complex, specially its network stack, you'll need to setup an VPN from each site to your control plane site to keep some basic functionality working (kubectl logs, for example). If you don't care about a centralized API to probe status and manage each instance, Ansible should be enough to orchestrate these installations and with little effort (that also depends on the application at hand) getting zero-downtime rollouts with Docker can be easily done with it. However, if you want a single control place to probe status and want to avoid writing your own rollout scripts, Hashicorp's Nomad [1] might be a good solution for this. It is a lot simpler than Kubernetes while still giving you nice primitives to describe jobs/services, health checks, rollouts strategies and etc. Treat every site as a datacenter of its own, setup a job of type "system" (akin to Kubernetes DaemonSets) and all you need on these sites is internet access to your HTTPS endpoint of the Nomad control plane. If you want to talk more about this, hit me up on Twitter or Telegram, I use @rochacon as my handle virtually everywhere. [1] https://www.nomadproject.io https://www.nomadproject.io Edit: grammar and typos
- SomaticPirate 7y ago
  I would also investigate Terraform if the Ansible path appeals to you.
- MuffinFlavored 7y ago
  How can zero downtime deploys be done with Docker? I had to write my own custom blue/green deploy script to hot reload traffic to proxy_pass definitions in nginx upstream configs since I don’t use Docker.
  - jeremija 7y ago
    IIRC, if you use docker swarm it wil handle the routing for you. So if an app is running on port 8080 on two swarm nodes, if you access port 8080 on each node, you might actually be accessing another node. Since docker swarm knows if an instance is down, it will know to not use that instance. See https://docs.docker.com/engine/swarm/ingress/ https://docs.docker.com/engine/swarm/ingress/
yashap 7y ago
Very curious about what kind of service has these requirements! Servers in 50 different cities suggest you likely care A LOT about latency. Going bare metal likely means you’re dealing with huge amounts of computation. But tolerating 2-5 mins of downtime on every deploy (until now) means availability isn’t a big deal. I’ve never seen this combo of requirements on any service before, what is it? Or are my assumptions just wrong about what these things suggest about your business?
- ggregoire 7y ago
  We detect and monitor commercials airing on terrestrial TV networks. :)
  - cfors 7y ago
    Pretty cool stuff. I'm assuming you have to set up Antenna's where you have your servers colocated and monitoring the transport stream's going through? Obviously you don't have to spill the company sauce but I'm trying to imagine the benefits of having all of this data.
  - mherdeg 7y ago
    Cool, so, looks like the product generates daily reports for advertisers about performance of ad spots over a trailing 1 day, lining up info about (a) what ad ran in what markets at what times plus (b) what increase in mobile/web traffic/conversion events happened from those markets coincidental with advertising, if I understand https://www.decidata.tv/advertisers.html https://www.decidata.tv/advertisers.html right. Looks like a complementary product (maybe running as a second process on the same nodes, or maybe running as part of the logic of the thing that parses the video) also produces quality reports about aired content ( https://www.decidata.tv/operators.html https://www.decidata.tv/operators.html ). I don't really know video very well, probably some very clever stuff you can do here to parallelize the processing. So hmmmm, a few things to think about for the tooling: () How quickly can you notice a single-node service outage and react / restore service? () How much downtime is ok? Is it ever ok to fail to process data for an entire day for one of the ~50 markets (can you just say "this report is incomplete" and reprocess it later)? How much wiggle room do you have (how long does it take 1 instance of the application to do its local processing job on 24 hours of data, and how long does the downline data pipeline have?) () Are you responsible for the part that watches TV + saves it to disk, and ALSO the part that batch-processes the saved video and produces the ad metadata / quality metadata? Are those different services deployed separately? () Yikes, is there, like, a person on call in each of 50 places who can unplug the thing that's watching TV and plug it back in, if a wire burns out or if the machine that's watching TV fails? That sounds like it could be an operational nightmare but maybe this is a solved problem / maybe you can buy watch-TV-and-save-it-to-disk as a service? () How do you make sure that every node is running the same version of the software with the same version of all the configuration? () How do you release new versions of the software? Do you (a) have a non-production environment which runs side-by-side at every node, processing the same data, and measure that the output is equally correct or better, the performance is the same or better, and there are no new errors or failures -- then promote the non-production code to production automatically after a while? Or do you (b) deploy a "canary" to one of the 50 nodes and watch how it performs for a while, then deploy to the rest if they all behave okay? Or do you (c) just ship the latest code to all 50 nodes on Friday nights when no one's looking, and check some health metrics over the weekend, and roll it back on Monday if a metric looks bad? () How do you keep track of what software versions you released when? If someone asks "What changed on 'X date'" does your tooling let you tell them pretty quickly and pretty accurately? () If you discover a bad bug (the parser is corrupting data; the new parsing job has a slow memory leak and fails catastrophically after 21 days of uptime) and you need to make a change in a hurry, do your runbooks let you make that kind of change quickly? Doesn't seem like you really need k8s or ansible or chef or whatever for this, as much as you need to write down your operational requirements and decide what technology will meet them.
wmf 7y ago
This sounds like Chick-fil-A's use case where they decided to put a k8s cluster in each restaurant. https://medium.com/@cfatechblog/edge-computing-at-chick-fil-a-7d67242675e2 https://medium.com/@cfatechblog/edge-computing-at-chick-fil-... https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at-chick-fil-a-scale-7b0607bd3541 https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at... If you have a single node not a cluster in each location then k8s buys you a lot less. Some people still advocate single-node k8s "clusters" for benefits like rolling deployment.
- 19ylram49 7y ago
  I’m still stuck on the Chick-fil-A part. Haha.
  - jbigelow76 7y ago
    It's handy to have all those k8s instances scale to zero on Sundays.
    - GhettoMaestro 7y ago
      I am fairly certain the second the old dude dies they are going to be like "Motion to open on Sundays..." in the board room.
      - mr_cyborg 7y ago
        S. Truett Cathy (who I assume is the old dude you mentioned) passed away in September of 2014.
        GhettoMaestro 7y ago
        Ah I did not know that. I figured he was still hanging out and getting a kick out of refusing to open on Sundays, given how much of meme it has become.
- merb 7y ago
yongjik 7y ago
Echoing others, you really don't want Kubernetes here. I only have limited experience, but Kubernetes works very hard to abstract away the actual machines, and it works best that way: you just say "Deploy 20 instances of job X", and k8s will somehow find the machines to deploy them. You don't care where they are running - k8s handles that. Once you start to care about actual machines and which jobs are running where, k8s starts to make less sense. You're paying for huge complexity (required to abstract away the machines), but you're getting none of the benefits - it just becomes a glorified wrapper around Docker daemon.
- thawkins 7y ago
  Docker in production is a potential disaster, it runs everything as root from the daemon, we are heavy k8s users, and we are transitioning to rootless cri-o containers specificaly because of this. Investigate podman which is a docker cli clone that builds cri-o containers from docker files and allows you to run them under a designated user id, the only restriction is that in rootless mode you cant map ports that are less than 1024, for that you need to run them under sudo which defeats tbe purpose. Howver we use an ingress controller to map inbound requests to containers so it can do the port mapping. We used to use traefik, but now we use nginx as ibgress, becuase we can run the excellent lua-waf module. We run on multiple clouds, so vendor apis and k8s implementations are not usefull to us.
- deleted 7y ago
  [deleted]
- MuffinFlavored 7y ago
  I struggle to understand when Docker is supposed to be introduced to a stack. If you have a 100 user SaaS platform, do you run Linux containers on top of Linux just for isolation?
  - inapis 7y ago
    If you need strong guarantees of isolation and if every user has their own dedicated installation. If you are a general service where isolation is enforced at the application level then just deploy straight away. Docker is useful when you don't want to care about the underlying operating system or you might be deploying on different operating systems etc. Also if you want to quickly switch providers (like going from Ubuntu 18.04 on digital ocean to Windows Server on Azure) since it's a self contained image. This self contained image is useful during development too. I can cleanly separate dependencies between applications and dev/test/prod. For example I develop a node app based on node 12 + pg 11 but I tinker with ghost too which requires node 10 + mysql. Having this in Docker keeps cruft and complexity off my macOS installation at the cost of space. Having multiple versions of a runtime/database etc can quickly become a nightmare to manage on local operating systems. Operationally if you can control deployment and you don't need many servers, deploying to the base operating system makes sense. Docker/Kubernetes shine when you have scale + need to provision many servers across different platform providers.
halbritt 7y ago
Check out BalenaCloud. It's a container orchestration service for edge devices.
gchamonlive 7y ago
You could also go full docker and experiment with docker swarm. I have little experience with swarm itself, but AWS ECS has deployment strategy. You could say you want 50 container instances on a fleet of 50 machines with deployment strategy of spreading across all machines and you would guarantee that every single machine is populated with a docker container. Maybe try to look for a similar functionality thinking about you bare metal machines as a single fleet of instances. Do you manage your containers configs with environment variables? I think docker also let's you share the hosts environment with the docker container, so you can pre provision the server with the necessary information and once the swarm is deployed, the container figures out what to do with the hosts env. Applying that to the spread across machines deployment strategy and you solve your problem.
dgoog 7y ago
Ansible all the way. I'm really upset with the way Google let their marketing team run roughshod all over the place with that software. Kubernetes is almost never the tool to use. It's entirely insecure, overly complicated and almost never fits the intended supposed benefit. Worse it feels that the entire CNCF ecosystem is ran entirely by marketing people with "developer evangelists" that have never coded a single line of code in their day - it's a real shame and quite honestly an insult to professional engineers.
- MuffinFlavored 7y ago
  I don’t understand when to use Ansible, Puppet, Chef, or Terraform...
  - wmf 7y ago
    Terraform: First you use this to create VMs. Ansible/Puppet/Chef/Salt: Then you use this to install your stuff in the VMs. Just pick one of these and stick with it.
    - jordanbeiber 7y ago
      Install k8s with a/p/c/s. :) K8s is not about configuration management, it’s about dynamic application management. Some parts infringe on areas where CM tools work as well, but k8s is all about managing containerized applications. Trying to set up flows for ”works on my laptop”-dev, ci/cd, loadtesting, a/b tests, canary releases, autoscaling and rollbacks for multiple teams of devs? K8s really simplifies these things. The idea is that you have one api spec to rule the whole stack (from a dev perspective). If you go down a more light-weight stack, a lot can still be achieved. More duct tape required though. That being said- I love duct tape!
    - gerbilly 7y ago
      Ansible can be used to create VMs too. I used to use it to provision AWS instances and to configure them after. https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html https://docs.ansible.com/ansible/latest/modules/list_of_clou...
deleted 7y ago
[deleted]
ninetax 7y ago
I've used SaltStack in a similar setup with some success, it's easy to keep all the servers in sync and prevent configuration drift.
jordanbeiber 7y ago
You could rather easily manage this flow with ansible. K8s abstracts a lot of the pieces in your ansible plays away, but comes back at you in form of a rather complex stack. Say you want to host instances of this app in a DC/cloud, k8s would be more appropriate, and perhaps then it would make sense to go single node “micro-k8s” style (for single server locations), to give you a common api for managing deployments, network configuration, security etc. Win with k8s is that once you’re tuned, a lot of “config crap” is abstracted away, basically. But - there's a reason ansible is supported specifically through the “operators” SDK, something we use to manage seamless upgrades of a k8s deployed app: https://github.com/operator-framework/operator-sdk/blob/master/doc/proposals/ansible-operator.md https://github.com/operator-framework/operator-sdk/blob/mast... Edit - forgot that if meteics based autoscaling of the app is a requirement my experience is that k8s makes this a breeze. Probably not a req if using single node hosts though...
mattbillenstein 7y ago
I haven't used ansible in a few years, but I sorta loathed the transport after a time with even dozens of boxes -- constantly creating ssh connections was slowing things down significantly. I switched to saltstack some time back and it's much better while having a similar sorta interface. The docs are worse than ansible imo tho. ymmv
- pferde 7y ago
  You shouldn't be creating ssh connections all the time, but instead using persistent connection to each server that ansible opens once at the beginning, and reuses for all subsequent tasks. See ControlPersist/ControlPath OpenSSH client configuration.
  - mattbillenstein 7y ago
    Never found this incredibly reliable - and if you change networks, sleep, or power off your machine, those connections die. Salt is just better at all of this imho.
    - mattbillenstein 7y ago
      I mean, I guess if you use a dedicated host, it's more reliable - but I've always just pushed from the client using Ansible whereas with Salt you have the master running the cloud running somewhere most usually.
    - pferde 7y ago
      Do you often do these things in the middle of running a playbook? I would hope not. :) It's not meant to have the connections persist forever, just long enough for repeated short ssh connections to have almost no overhead, and therefore be much faster.
      - mattbillenstein 7y ago
        Yeah, good point.
speedplane 7y ago
Kubernetes is complicated, it's a whole new world of terminology and abstractions that you need to learn, and it changes quickly. That said, maintaining a large cluster is complex too. Often, there is much more involved than making sure each image's config files are up to date. Kubernetes shines when you have a number of different docker images, where each needs its own unique network, scaling, cpu, and disk settings, i.e., hardware requirements that can't really be controlled with just config files. In your case, the fact that all of your images are identical probably cuts against using Kubernetes. However, the fact that these images are running on different networks where each may require their own tweaking leans towards it. If you expect to start adding many different docker images to your cluster, with different hardware requirements, it's probably worth the learning curve to get on to Kubernetes. Disclaimer: I've never run Kubernetes on bare metal, just hosted environments. Bare metal may make Kubernetes much less attractive.
OJFord 7y ago
Summary: 50 production instances of a single application. No, you don't need Kubernetes (though fwiw if you used it what you'd have would be 50 nodes with a single daemonset, 1 pod of the app per node, if that helps understand) - but I'd suggest not using Ansible. Instead, use Packer (or similar) to create a machine image/snapshot/whatever your server provider calls it, and then deploy that same image on all 50. You might have some small amount of host-specific things left to configure, and by all means use ansible for that if you want, but there's no need for error-prone running of the same large playbook 50 times. Personally, I find that final setup is little enough to do it with Terraform (which is provisioning the servers anyway).
hackermailman 7y ago
I do this on ~30 servers with GuixSD, so there is a single file to configure all of them. To update the application I wrote a Guix pkg for it that pulls from our repository (simple to set up with Guix) everything is controlled in emacs by me to change/start/stop all the servers. https://news.ycombinator.com/item?id=17084561 https://news.ycombinator.com/item?id=17084561 is similar to what I'm doing. Caveat: you have to go on google scholar to read guix and nix papers to understand the system architecture, then their documentation also downtimes aren't an issue for us since these are internal servers, and this is done during non critical time but it's usually just restarting the new app so a few seconds. Other caveats, security fixes we just edit in changes not run guix pull but I also have that automated now after mailing list help. You may have totally different req otherwise look into nix/guix, Guix the pkg manager runs on any distro.

aprdm 7y ago

I would personally go with ansible and consul if your application is something like a rest api, should be trivial to setup.

clhodapp 7y ago

At a high level, your use-case maps well to a Kubernetes DaemonSet, which is a resource that represents the desire to run a copy of something containerized on every node within your cluster. Kubernetes would give you some things that might be useful: * A unified control plane (monitoring, statusing) * An automated rolling-upgrade system * "self-healing" in the form of automatically restarting your containers if they go awry However, for your use-case, it is likely more effort than it's worth: * Settin up and maintaining Kubernetes on bare metal servers is a huge amount of work * A lot of Kubernetes features are not needed for your use-case * The Kubernetes networking model essentially requires you to set up private network spanning all of your sites in order to work but provides almost no benefit for your use-case

kissgyorgy 7y ago
Technically you could use a DaemonSet for that, but the primary use-case is not that for DaemonSets. It should be used for "daemon" containers, like logging daemons (e.g. fluentd), network plugin daemons like calico, flannel or monitoring like prometheus. If you want to run multiple instances of your application, you should define a replicaset or deployment with setting Pod affinity instead.
- clhodapp 7y ago
  That's super fiddly though: you will be leaning on (anti) affinity and counts to retain the 1:1 relationship between hosts and pods. I would argue that daemonsets aren't for any specific workload, they're more about how you want scheduling to happen (1 per node, on some subset of your nodes).
  - manigandham 7y ago
    This is where K8S becomes messy because of its rapid development history. Applications should just be a `Deployment` manifest with the type of distribution being a child setting (X replicas with affinity or 1-per-host). There's no reason to have competing ReplicaSets and DaemonSets just for this detail.
    - kissgyorgy 7y ago
      I disagree on this one, DaemonSets are for really a different use-case. There is no replicas field you can set, and strictly 1 Pod will run on 1 node. A Deployment is more flexible, because you can run even more Pods in different cities if necessary. What if there are more/less nodes are in different cities or different specs, which could and should run more Pods? You can't fine-tune DaemonSets for those cases, it really is for daemons.
      - manigandham 7y ago
        It's not about use-case, it's that "X number of replicas with affinity" OR "1-per-node" should just be a setting within a Deployment. No need to use a completely separate DaemonSet definition just to run 1-per-node.

dragonsh 7y ago

No kubernetes will be a bad choice for your use case. But if you learn all the intricacies and manage a behemoth besides your own application it will help in your resume to brag that you are a modern devops and a fashionable one using kubernetes. If you need a practical suggestion use simple solution like LXD [1]. But as you are already using docker stay with it and ansible. As you are using ansible you are already doing infrastructure as a code, you are way ahead of curve already. Probably when you grow a bit bigger use packer [2] and terraform [3], but I think ansible will do just fine. Kubernetes was designed for Google kind of problem the burden to maintain is quite high with lots of moving parts, unless you use Google GKE or Amazon or managed kubernetes service. So don't fall for it unless you need it for your resume. [1] https://linuxcontainers.org/ https://linuxcontainers.org/ [2] https://www.packer.io/intro/ https://www.packer.io/intro/ [3] https://www.terraform.io/ https://www.terraform.io/

samvher 7y ago
GKE and EKS are very expensive which in my opinion makes using them unattractive for smaller scale projects. But Digital Ocean now offers managed Kubernetes for free, I've played around with it and it actually seems really easy to set up and manage. Would be great if the competition would follow suit.
- jorams 7y ago
  > GKE and EKS are very expensive > Digital Ocean now offers managed Kubernetes for free GKE is just as "free" as DO, in that you only pay for the nodes you actually use (those nodes are probably a bit more expensive, admittedly). EKS is the outlier in this trio in that you also have to pay (a lot) for the master. Could you clarify what GKE could do to "follow suit"?
  - samvher 7y ago
    It seems, nothing! I was just wrong about GKE being expensive, not sure why I thought that (though it seems that until late 2017 they did have costs per cluster).

deleted 7y ago

[deleted]

kissgyorgy 7y ago

Kubernetes is supposed to installed nodes close to each other, typically in the same server room, or datacenter, otherwise the latency would kill the etcd cluster. Ansible is perfectly fine and simple for your use case, why would you bother even thinking about overcomplicating it? Simplicity is king!

rochacon 7y ago

No, as @dragonsh said, Kubernetes will be a bad choice for you. Kubernetes is complex, specially its network stack, you'll need to setup an VPN from each site to your control plane site to keep some basic functionality working (kubectl logs, for example). If you don't care about a centralized API to probe status and manage each instance, Ansible should be enough to orchestrate these installations and with little effort (that also depends on the application at hand) getting zero-downtime rollouts with Docker can be easily done with it. However, if you want a single control place to probe status and want to avoid writing your own rollout scripts, Hashicorp's Nomad [1] might be a good solution for this. It is a lot simpler than Kubernetes while still giving you nice primitives to describe jobs/services, health checks, rollouts strategies and etc. Treat every site as a datacenter of its own, setup a job of type "system" (akin to Kubernetes DaemonSets) and all you need on these sites is internet access to your HTTPS endpoint of the Nomad control plane. If you want to talk more about this, hit me up on Twitter or Telegram, I use @rochacon as my handle virtually everywhere. [1] https://www.nomadproject.io https://www.nomadproject.io Edit: grammar and typos

SomaticPirate 7y ago
I would also investigate Terraform if the Ansible path appeals to you.
MuffinFlavored 7y ago
How can zero downtime deploys be done with Docker? I had to write my own custom blue/green deploy script to hot reload traffic to proxy_pass definitions in nginx upstream configs since I don’t use Docker.
- jeremija 7y ago
  IIRC, if you use docker swarm it wil handle the routing for you. So if an app is running on port 8080 on two swarm nodes, if you access port 8080 on each node, you might actually be accessing another node. Since docker swarm knows if an instance is down, it will know to not use that instance. See https://docs.docker.com/engine/swarm/ingress/ https://docs.docker.com/engine/swarm/ingress/

yashap 7y ago

Very curious about what kind of service has these requirements! Servers in 50 different cities suggest you likely care A LOT about latency. Going bare metal likely means you’re dealing with huge amounts of computation. But tolerating 2-5 mins of downtime on every deploy (until now) means availability isn’t a big deal. I’ve never seen this combo of requirements on any service before, what is it? Or are my assumptions just wrong about what these things suggest about your business?

ggregoire 7y ago
We detect and monitor commercials airing on terrestrial TV networks. :)
- cfors 7y ago
  Pretty cool stuff. I'm assuming you have to set up Antenna's where you have your servers colocated and monitoring the transport stream's going through? Obviously you don't have to spill the company sauce but I'm trying to imagine the benefits of having all of this data.
- mherdeg 7y ago
  Cool, so, looks like the product generates daily reports for advertisers about performance of ad spots over a trailing 1 day, lining up info about (a) what ad ran in what markets at what times plus (b) what increase in mobile/web traffic/conversion events happened from those markets coincidental with advertising, if I understand https://www.decidata.tv/advertisers.html https://www.decidata.tv/advertisers.html right. Looks like a complementary product (maybe running as a second process on the same nodes, or maybe running as part of the logic of the thing that parses the video) also produces quality reports about aired content ( https://www.decidata.tv/operators.html https://www.decidata.tv/operators.html ). I don't really know video very well, probably some very clever stuff you can do here to parallelize the processing. So hmmmm, a few things to think about for the tooling: () How quickly can you notice a single-node service outage and react / restore service? () How much downtime is ok? Is it ever ok to fail to process data for an entire day for one of the ~50 markets (can you just say "this report is incomplete" and reprocess it later)? How much wiggle room do you have (how long does it take 1 instance of the application to do its local processing job on 24 hours of data, and how long does the downline data pipeline have?) () Are you responsible for the part that watches TV + saves it to disk, and ALSO the part that batch-processes the saved video and produces the ad metadata / quality metadata? Are those different services deployed separately? () Yikes, is there, like, a person on call in each of 50 places who can unplug the thing that's watching TV and plug it back in, if a wire burns out or if the machine that's watching TV fails? That sounds like it could be an operational nightmare but maybe this is a solved problem / maybe you can buy watch-TV-and-save-it-to-disk as a service? () How do you make sure that every node is running the same version of the software with the same version of all the configuration? () How do you release new versions of the software? Do you (a) have a non-production environment which runs side-by-side at every node, processing the same data, and measure that the output is equally correct or better, the performance is the same or better, and there are no new errors or failures -- then promote the non-production code to production automatically after a while? Or do you (b) deploy a "canary" to one of the 50 nodes and watch how it performs for a while, then deploy to the rest if they all behave okay? Or do you (c) just ship the latest code to all 50 nodes on Friday nights when no one's looking, and check some health metrics over the weekend, and roll it back on Monday if a metric looks bad? () How do you keep track of what software versions you released when? If someone asks "What changed on 'X date'" does your tooling let you tell them pretty quickly and pretty accurately? () If you discover a bad bug (the parser is corrupting data; the new parsing job has a slow memory leak and fails catastrophically after 21 days of uptime) and you need to make a change in a hurry, do your runbooks let you make that kind of change quickly? Doesn't seem like you really need k8s or ansible or chef or whatever for this, as much as you need to write down your operational requirements and decide what technology will meet them.

wmf 7y ago

This sounds like Chick-fil-A's use case where they decided to put a k8s cluster in each restaurant. https://medium.com/@cfatechblog/edge-computing-at-chick-fil-a-7d67242675e2 https://medium.com/@cfatechblog/edge-computing-at-chick-fil-... https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at-chick-fil-a-scale-7b0607bd3541 https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at... If you have a single node not a cluster in each location then k8s buys you a lot less. Some people still advocate single-node k8s "clusters" for benefits like rolling deployment.

19ylram49 7y ago
I’m still stuck on the Chick-fil-A part. Haha.
- jbigelow76 7y ago
  It's handy to have all those k8s instances scale to zero on Sundays.
  - GhettoMaestro 7y ago
    I am fairly certain the second the old dude dies they are going to be like "Motion to open on Sundays..." in the board room.
    - mr_cyborg 7y ago
      S. Truett Cathy (who I assume is the old dude you mentioned) passed away in September of 2014.
      - GhettoMaestro 7y ago
        Ah I did not know that. I figured he was still hanging out and getting a kick out of refusing to open on Sundays, given how much of meme it has become.
merb 7y ago

yongjik 7y ago

Echoing others, you really don't want Kubernetes here. I only have limited experience, but Kubernetes works very hard to abstract away the actual machines, and it works best that way: you just say "Deploy 20 instances of job X", and k8s will somehow find the machines to deploy them. You don't care where they are running - k8s handles that. Once you start to care about actual machines and which jobs are running where, k8s starts to make less sense. You're paying for huge complexity (required to abstract away the machines), but you're getting none of the benefits - it just becomes a glorified wrapper around Docker daemon.

thawkins 7y ago
Docker in production is a potential disaster, it runs everything as root from the daemon, we are heavy k8s users, and we are transitioning to rootless cri-o containers specificaly because of this. Investigate podman which is a docker cli clone that builds cri-o containers from docker files and allows you to run them under a designated user id, the only restriction is that in rootless mode you cant map ports that are less than 1024, for that you need to run them under sudo which defeats tbe purpose. Howver we use an ingress controller to map inbound requests to containers so it can do the port mapping. We used to use traefik, but now we use nginx as ibgress, becuase we can run the excellent lua-waf module. We run on multiple clouds, so vendor apis and k8s implementations are not usefull to us.
deleted 7y ago
[deleted]
MuffinFlavored 7y ago
I struggle to understand when Docker is supposed to be introduced to a stack. If you have a 100 user SaaS platform, do you run Linux containers on top of Linux just for isolation?
- inapis 7y ago
  If you need strong guarantees of isolation and if every user has their own dedicated installation. If you are a general service where isolation is enforced at the application level then just deploy straight away. Docker is useful when you don't want to care about the underlying operating system or you might be deploying on different operating systems etc. Also if you want to quickly switch providers (like going from Ubuntu 18.04 on digital ocean to Windows Server on Azure) since it's a self contained image. This self contained image is useful during development too. I can cleanly separate dependencies between applications and dev/test/prod. For example I develop a node app based on node 12 + pg 11 but I tinker with ghost too which requires node 10 + mysql. Having this in Docker keeps cruft and complexity off my macOS installation at the cost of space. Having multiple versions of a runtime/database etc can quickly become a nightmare to manage on local operating systems. Operationally if you can control deployment and you don't need many servers, deploying to the base operating system makes sense. Docker/Kubernetes shine when you have scale + need to provision many servers across different platform providers.

halbritt 7y ago

Check out BalenaCloud. It's a container orchestration service for edge devices.

gchamonlive 7y ago

You could also go full docker and experiment with docker swarm. I have little experience with swarm itself, but AWS ECS has deployment strategy. You could say you want 50 container instances on a fleet of 50 machines with deployment strategy of spreading across all machines and you would guarantee that every single machine is populated with a docker container. Maybe try to look for a similar functionality thinking about you bare metal machines as a single fleet of instances. Do you manage your containers configs with environment variables? I think docker also let's you share the hosts environment with the docker container, so you can pre provision the server with the necessary information and once the swarm is deployed, the container figures out what to do with the hosts env. Applying that to the spread across machines deployment strategy and you solve your problem.

dgoog 7y ago

Ansible all the way. I'm really upset with the way Google let their marketing team run roughshod all over the place with that software. Kubernetes is almost never the tool to use. It's entirely insecure, overly complicated and almost never fits the intended supposed benefit. Worse it feels that the entire CNCF ecosystem is ran entirely by marketing people with "developer evangelists" that have never coded a single line of code in their day - it's a real shame and quite honestly an insult to professional engineers.

MuffinFlavored 7y ago
I don’t understand when to use Ansible, Puppet, Chef, or Terraform...
- wmf 7y ago
  Terraform: First you use this to create VMs. Ansible/Puppet/Chef/Salt: Then you use this to install your stuff in the VMs. Just pick one of these and stick with it.
  - jordanbeiber 7y ago
    Install k8s with a/p/c/s. :) K8s is not about configuration management, it’s about dynamic application management. Some parts infringe on areas where CM tools work as well, but k8s is all about managing containerized applications. Trying to set up flows for ”works on my laptop”-dev, ci/cd, loadtesting, a/b tests, canary releases, autoscaling and rollbacks for multiple teams of devs? K8s really simplifies these things. The idea is that you have one api spec to rule the whole stack (from a dev perspective). If you go down a more light-weight stack, a lot can still be achieved. More duct tape required though. That being said- I love duct tape!
  - gerbilly 7y ago
    Ansible can be used to create VMs too. I used to use it to provision AWS instances and to configure them after. https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html https://docs.ansible.com/ansible/latest/modules/list_of_clou...

deleted 7y ago

[deleted]

ninetax 7y ago

I've used SaltStack in a similar setup with some success, it's easy to keep all the servers in sync and prevent configuration drift.

jordanbeiber 7y ago

You could rather easily manage this flow with ansible. K8s abstracts a lot of the pieces in your ansible plays away, but comes back at you in form of a rather complex stack. Say you want to host instances of this app in a DC/cloud, k8s would be more appropriate, and perhaps then it would make sense to go single node “micro-k8s” style (for single server locations), to give you a common api for managing deployments, network configuration, security etc. Win with k8s is that once you’re tuned, a lot of “config crap” is abstracted away, basically. But - there's a reason ansible is supported specifically through the “operators” SDK, something we use to manage seamless upgrades of a k8s deployed app: https://github.com/operator-framework/operator-sdk/blob/master/doc/proposals/ansible-operator.md https://github.com/operator-framework/operator-sdk/blob/mast... Edit - forgot that if meteics based autoscaling of the app is a requirement my experience is that k8s makes this a breeze. Probably not a req if using single node hosts though...

mattbillenstein 7y ago

I haven't used ansible in a few years, but I sorta loathed the transport after a time with even dozens of boxes -- constantly creating ssh connections was slowing things down significantly. I switched to saltstack some time back and it's much better while having a similar sorta interface. The docs are worse than ansible imo tho. ymmv

pferde 7y ago
You shouldn't be creating ssh connections all the time, but instead using persistent connection to each server that ansible opens once at the beginning, and reuses for all subsequent tasks. See ControlPersist/ControlPath OpenSSH client configuration.
- mattbillenstein 7y ago
  Never found this incredibly reliable - and if you change networks, sleep, or power off your machine, those connections die. Salt is just better at all of this imho.
  - mattbillenstein 7y ago
    I mean, I guess if you use a dedicated host, it's more reliable - but I've always just pushed from the client using Ansible whereas with Salt you have the master running the cloud running somewhere most usually.
  - pferde 7y ago
    Do you often do these things in the middle of running a playbook? I would hope not. :) It's not meant to have the connections persist forever, just long enough for repeated short ssh connections to have almost no overhead, and therefore be much faster.
    - mattbillenstein 7y ago
      Yeah, good point.

speedplane 7y ago

Kubernetes is complicated, it's a whole new world of terminology and abstractions that you need to learn, and it changes quickly. That said, maintaining a large cluster is complex too. Often, there is much more involved than making sure each image's config files are up to date. Kubernetes shines when you have a number of different docker images, where each needs its own unique network, scaling, cpu, and disk settings, i.e., hardware requirements that can't really be controlled with just config files. In your case, the fact that all of your images are identical probably cuts against using Kubernetes. However, the fact that these images are running on different networks where each may require their own tweaking leans towards it. If you expect to start adding many different docker images to your cluster, with different hardware requirements, it's probably worth the learning curve to get on to Kubernetes. Disclaimer: I've never run Kubernetes on bare metal, just hosted environments. Bare metal may make Kubernetes much less attractive.

OJFord 7y ago

Summary: 50 production instances of a single application. No, you don't need Kubernetes (though fwiw if you used it what you'd have would be 50 nodes with a single daemonset, 1 pod of the app per node, if that helps understand) - but I'd suggest not using Ansible. Instead, use Packer (or similar) to create a machine image/snapshot/whatever your server provider calls it, and then deploy that same image on all 50. You might have some small amount of host-specific things left to configure, and by all means use ansible for that if you want, but there's no need for error-prone running of the same large playbook 50 times. Personally, I find that final setup is little enough to do it with Terraform (which is provisioning the servers anyway).

hackermailman 7y ago

I do this on ~30 servers with GuixSD, so there is a single file to configure all of them. To update the application I wrote a Guix pkg for it that pulls from our repository (simple to set up with Guix) everything is controlled in emacs by me to change/start/stop all the servers. https://news.ycombinator.com/item?id=17084561 https://news.ycombinator.com/item?id=17084561 is similar to what I'm doing. Caveat: you have to go on google scholar to read guix and nix papers to understand the system architecture, then their documentation also downtimes aren't an issue for us since these are internal servers, and this is done during non critical time but it's usually just restarting the new app so a few seconds. Other caveats, security fixes we just edit in changes not run guix pull but I also have that automated now after mailing list help. You may have totally different req otherwise look into nix/guix, Guix the pkg manager runs on any distro.