9 ms·

So... systemd + Ansible? I really disliked SystemD before I got my hands dirty with it. Assuming you're developing with any modern language, the language itsel

by erulabs 10y ago

So... systemd + Ansible? I really disliked SystemD before I got my hands dirty with it. Assuming you're developing with any modern language, the language itself probably wraps OS differences anyways (Node / Golang / Rust / Ruby / Python / anything-libUV-based to name a few), not like you can convince developers to change their habits anyways. People act like Docker makes builds reproducible by magic, then go on to not pin any Golang deps or curl internet resources or not generate lockfiles for NPM, ad infinitum. There is and never will be a shield operations people can put around their developers. You just need good developers. If you want SoA scaling or automatic cloud configuration, why not just use the most established tools available? Linux's init system and kernel _already do_ resource management, even outside of LXC. People act like a "cloud scheduler" is the only thing that lets multiple processes work together. This whole idea of isolation is also really inane. Each app will have different disk, io, network, cpu requirements. I also dream of the day Kubernetes can do this "sorting" better than an experienced operations team can, but that day is _far off_ (and I will die before I put redis/mysql/postgres/cassandra/any-other-mature-datastore behind docker and iptables)

jmspring 10y ago
You've done nothing to justify your "systemd + sensible" in this comment. You've just ranted against other technologies. I don't see the point.
- erulabs 10y ago
  I suppose systemd could be swapped for "any modern linux init system" and Ansible could be swapped for "any modern cloud configuration tool". There are probably 3 or 4 items that come to mind for both categories. The idea is that Linux already solves that vast majority of these problems, as do cloud providers. I then ranted on a couple of the "problems" that containers and container orchestration "solve". You're right tho - a bit of a ramble. Been a long day / on-call week :P I will think about how I could have been more eloquent for next time.
  - dozzie 10y ago
    > [...] Ansible could be swapped for "any modern cloud configuration tool". Deployment tool, not configuration tool. Ansible is awful for keeping environment configured, it was from ground up designed as somebody's deployment script.
    - geggam 10y ago
      Amen. Ansible is great for fire and forget machines you don't need or care about. You need real config management with an audit trail when you go enterprise.
    - aeorgnoieang 10y ago
      What's an example of an configurable aspect of some environment that Ansible handles awfully but that another configuration tool handles well? Generally, all these tools are doing the same thing, i.e. running a bunch of commands on a bunch of servers. I haven't discovered what specific details tho indicate Ansible being less suitable (or unsuitable) for configuration management. I could easily imagine that the 'modules' in Ansible aren't as robust as those in, e.g. Chef, at least for some platforms. But my experience so far having used both Chef and Ansible with Windows VMs (on VMWare) is that Ansible is much better.
      - dozzie 10y ago
        > What's an example of an configurable aspect of some environment that Ansible handles awfully but that another configuration tool handles well? Hosts that were temporarily down when Ansible tried to ship configuration. CFEngine or Puppet just apply the changes some time later without operator's intervention. > Generally, all these tools are doing the same thing, i.e. running a bunch of commands on a bunch of servers. So does Ant or Gradle. Something is different about the operation, though, because nobody uses Gradle to manage server configuration. > I haven't discovered what specific details tho indicate Ansible being less suitable (or unsuitable) for configuration management. The architecture and the protocol. With Ansible you get synchronous sequence of operations that breaks on first problem (e.g. host is down for maintenance). This is OK for deployment, but not for ordering a configuration change, for which you don't care when it will be applied, as long as it's soon enough. Note that deployment and managing configuration are similar, but different things, and it's not uncommon to need to do both (although not at the same time). Then the protocol. Ansible uses brain-dead idea of using the debugging/interactive channel (SSH) to configure things, including possibly SSH itself. One small mistake can cut off your all remote access at once. It's negligible risk if you only configure SSH daemon when deploying the server, but it increases quickly in the log-term. Not to mention that managing SSH access properly (e.g. not ignoring warnings about hosts' public keys, but not just that) is much more tedious than keeping a CA, so SSH is a bad choice for unattended operations.
awinter-py 10y ago
Yeah. Would be really interesting to see a container technology that people were eager to use for DB management. Persistent storage is the thing that hasn't clicked yet. My guess is this technology will be some kind of cluster management. If there were a 'platform' for horizontally scaling data that managed sharding, rebalancing & backups plus gave me user-space hooks to override policies, databases might slowly & cautiously build on top of it. Most DBs have their own of these (pgpool, mongos, redis sentinel, all of cassandra & hadoop) but would be cool to centralize the common reqs and collaborate.
- erulabs 10y ago
  > would be cool to centralize the common reqs and collaborate Oh baby the desire to pitch my WIP SaaS product. This is something I'm extremely keen on and I couldn't agree more!
- regularfry 10y ago
  The problem is that the constraints of wanting your data and CPU on the same node, needing data synced to more than one node, and avoiding being too much of a performance bottleneck, all combine to make it very hard to do better than something which has privileged knowledge of how the database works under the hood. Performant consistency is hard.
- william20111 10y ago
  smartOS gives you persistent storage btw. And its lovely ZFS storage..
  - justinsaccount 10y ago
    saying smartos gives you persistent storage is like saying docker gives you persistent storage because you can use docker run .... -v /storage:/storage myapp That's not what people are talking about. If you run a container on smartos and the physical machine it is running on blows up, what happens to your persistent storage?
    - 10y ago
shitloadofbooks 10y ago
As an Ops guy, I preach Ansible + systemd all day everyday, but so many of our Devs (and Ops) have drunk the containerization Kool-aid. Unfortunately, the same culture which fucked modern web design (a new JS framework every 6 months) is starting to creep into Ops; some REALLY smart people make a tool which promises to be X but with X's main problem solved. They do the easy 80% and then abandon the project when the hard 20% needs to be fixed (and forget documentation).
- lobster_johnson 10y ago
  The JavaScript situation is grossly exaggerated. It's doing very well. It's just that the people who aren't productive are also the ones shouting the loudest on HN. Meanwhile, a lot of people are being very quiet getting tons of stuff done really efficiently. (Source: Me, very happy React developer who is so glad we're finally out of the terrible "wilderness years" of server templates and jQuery.)
  - stephenr 10y ago
    > terrible "wilderness years" of server templates wat.
    - deleted 10y ago
      [deleted]
    - lumpypua 10y ago
      That's fine, it's: > terrible "wilderness years" of server templates and jQuery where things go off the rails.
      - yellowapple 10y ago
        I reckon > terrible "wilderness years" of ... jQuery is the part where things go off the rails (and literally off the Rails).
        10y ago
lobster_johnson 10y ago
The idea with Kubernetes isn't necessarily that it can do resource allocation more carefully and more intelligently than a human ops person, but it can do so at a level of automation and speed that a human can't. Pods die, Kubernetes spawns new ones. Server goes full, Kubernetes spawns new node. Server runs out of disk space, Kubernetes evicts nodes until everything is good again. Server goes down, Kubernetes shuffles all the pods over to a different node. And so on. It does this at a scale of hundreds or thousands of machines. The utility it brings also applies to small clusters. You don't remove ops, you just flatten it into Kubernetes' operational model. Equally importantly, Kubernetes also brings a declarative deployment model. You have the same with Systemd, but at the node level. Kubernetes is a bit like Systemd at the cluster level; its "units" are pods declared in YAML/JSON files, but unlike Systemd they run on any node the cluster, and are subject to a resource allocation model that includes things like persistent storage (e.g. AWS EBS volumes). You don't get anything close to that with Systemd, which is much more analogous to Docker. I haven't heard anyone ever claim that Docker brings reproducible builds -- why would anyone assume that? What Docker give you is reproducible deploys; build once, deploy many times from the exact same image. Maybe that's what you're thinking of?
- erulabs 10y ago
  I don't dislike Kubernetes, in fact im a (very minor) contributor and regular user and am a member of the federated kubernetes workgroup. A small group of microservices very happily lives on Kube in production. But, for the sake of the argument: > Pods die, Kubernetes spawns new ones. AWS ASGs also solve for this, as does every other cloud provider at the infrastructure level. For applications, systemd and any init system can restart an application when it crashes... > Server goes full, Kubernetes spawns new node. I assuming you mean utilization - again, ASGs/CloudWatch solve for this as do every other cloud provider (if they didn't a few minutes worth of Sensu plugins and good-to-go). > Server runs out of disk space, Kubernetes evicts nodes until everything is good again Right but this can and has lead to disaster. I got paged for bad response time and found out the issue was that containers were being evicted and pods were going critical because _all_ hosts were filling up on diskspace and there were no healthy nodes on which to put pods. This due to one bad service writing out tons and tons of log data). Had that been dedicated, I would have just gotten paged for one full disk and fixed the problem when it showed up instead of waiting for a bigger issue. This is super fixable in Kube (and was - that service now has its own volume and still lives in kube) - but my point is that it's a problem for everyone. > Server goes down, Kubernetes shuffles all the pods over to a different node. This effects response time of user facing applications as new 1GiB+ containers are shuffled around (I know, thats an unfair example). ASGs and Cloudwatch solve for this in a saner / safer way. As for the reproducible builds, it was in response to the article which says: > The ‘repeatable builds’ argument is the most interesting to me, in part because this is the same promise that java made in the 90s (write once run anywhere). You may be correct that Docker doesn't promise this explicitly, but it is a big misconception. Additionally, `tar` solves for "reproducible deploys"... I'm not an AWS employee, just figured I'd use a common cloud as an example.
mianos 10y ago
Systemd + ansible is very close to what we have. We are python shop so we use: - salstack to deploy and update virtualenvs and applications. - salstack to install and configure upstart jobs - logging goes to /var/log/upstart but this gets shipped automagically by salstack configured per app awslogs config. - All config is read from zookeeper. - The only thing that is initialised on the machine is a ZK host environment variable (by salt-cloud). Everything is ummutable. (Using Postgres RDS for the moment). A system dies we can just make another one with salt. We can run multiple services on one machine or one per machine or install them in lxc containers trivially.
- amorphid 10y ago
  Minor nitpick... I believe you mean "saltstack". Yes?
  - mianos 10y ago
    Sorry, yes saltstack. Typing on my phone while my kids jump on my head. saltstack is like a toolkit for all this sort of stuff.
    - amorphid 10y ago
      Off topic, I just looked up the definition of nitpick. Apparently it refers to removing nits, the eggs of lice :O https://en.wikipedia.org/wiki/Nitpicking https://en.wikipedia.org/wiki/Nitpicking I did not mean to imply your comment has lice :)
      - mianos 10y ago
        I have kids in primary school, there are probably some lice in there somewhere.
lima 10y ago
I really like the CoreOS approach of using systemd for container supervision.
aprdm 10y ago
We use ansible and systemd with around 100 services in 3 continents and works well.
- dyu- 10y ago
  Cool. If any of it is opensource, please post the links in the reply, I'm curious.
dyu- 10y ago
+1 for systemd + ansible. That is exactly my setup on ubuntu 16.04 deployments.
zedpm 10y ago
I see a lot of "me too" responses to this, and for the life of me, I can't figure out what systemd brings to the table. I understand the Ansible part, but what does systemd do that's notable? My understanding is that it's an alternative to init, and I can't imagine calling out init as part of my deployment/config strategy. What am I missing?
- collinmanderson 10y ago
  I'm wondering the same thing, as someone who has only started using systemd. From what I can tell it handles: - starting the various pieces of the app in the correct order - permissions (uid, cgroup, maybe even virtual file system) - environment variables, working directory - passing around sockets - restarting things if needed - monitoring resource usage - centralized logging

by erulabs 10y ago

jmspring 10y ago

You've done nothing to justify your "systemd + sensible" in this comment. You've just ranted against other technologies. I don't see the point.

erulabs 10y ago
I suppose systemd could be swapped for "any modern linux init system" and Ansible could be swapped for "any modern cloud configuration tool". There are probably 3 or 4 items that come to mind for both categories. The idea is that Linux already solves that vast majority of these problems, as do cloud providers. I then ranted on a couple of the "problems" that containers and container orchestration "solve". You're right tho - a bit of a ramble. Been a long day / on-call week :P I will think about how I could have been more eloquent for next time.
- dozzie 10y ago
  > [...] Ansible could be swapped for "any modern cloud configuration tool". Deployment tool, not configuration tool. Ansible is awful for keeping environment configured, it was from ground up designed as somebody's deployment script.
  - geggam 10y ago
    Amen. Ansible is great for fire and forget machines you don't need or care about. You need real config management with an audit trail when you go enterprise.
  - aeorgnoieang 10y ago
    What's an example of an configurable aspect of some environment that Ansible handles awfully but that another configuration tool handles well? Generally, all these tools are doing the same thing, i.e. running a bunch of commands on a bunch of servers. I haven't discovered what specific details tho indicate Ansible being less suitable (or unsuitable) for configuration management. I could easily imagine that the 'modules' in Ansible aren't as robust as those in, e.g. Chef, at least for some platforms. But my experience so far having used both Chef and Ansible with Windows VMs (on VMWare) is that Ansible is much better.
    - dozzie 10y ago
      > What's an example of an configurable aspect of some environment that Ansible handles awfully but that another configuration tool handles well? Hosts that were temporarily down when Ansible tried to ship configuration. CFEngine or Puppet just apply the changes some time later without operator's intervention. > Generally, all these tools are doing the same thing, i.e. running a bunch of commands on a bunch of servers. So does Ant or Gradle. Something is different about the operation, though, because nobody uses Gradle to manage server configuration. > I haven't discovered what specific details tho indicate Ansible being less suitable (or unsuitable) for configuration management. The architecture and the protocol. With Ansible you get synchronous sequence of operations that breaks on first problem (e.g. host is down for maintenance). This is OK for deployment, but not for ordering a configuration change, for which you don't care when it will be applied, as long as it's soon enough. Note that deployment and managing configuration are similar, but different things, and it's not uncommon to need to do both (although not at the same time). Then the protocol. Ansible uses brain-dead idea of using the debugging/interactive channel (SSH) to configure things, including possibly SSH itself. One small mistake can cut off your all remote access at once. It's negligible risk if you only configure SSH daemon when deploying the server, but it increases quickly in the log-term. Not to mention that managing SSH access properly (e.g. not ignoring warnings about hosts' public keys, but not just that) is much more tedious than keeping a CA, so SSH is a bad choice for unattended operations.

awinter-py 10y ago

Yeah. Would be really interesting to see a container technology that people were eager to use for DB management. Persistent storage is the thing that hasn't clicked yet. My guess is this technology will be some kind of cluster management. If there were a 'platform' for horizontally scaling data that managed sharding, rebalancing & backups plus gave me user-space hooks to override policies, databases might slowly & cautiously build on top of it. Most DBs have their own of these (pgpool, mongos, redis sentinel, all of cassandra & hadoop) but would be cool to centralize the common reqs and collaborate.

erulabs 10y ago
> would be cool to centralize the common reqs and collaborate Oh baby the desire to pitch my WIP SaaS product. This is something I'm extremely keen on and I couldn't agree more!
regularfry 10y ago
The problem is that the constraints of wanting your data and CPU on the same node, needing data synced to more than one node, and avoiding being too much of a performance bottleneck, all combine to make it very hard to do better than something which has privileged knowledge of how the database works under the hood. Performant consistency is hard.
william20111 10y ago
smartOS gives you persistent storage btw. And its lovely ZFS storage..
- justinsaccount 10y ago
  saying smartos gives you persistent storage is like saying docker gives you persistent storage because you can use docker run .... -v /storage:/storage myapp That's not what people are talking about. If you run a container on smartos and the physical machine it is running on blows up, what happens to your persistent storage?
  - 10y ago

shitloadofbooks 10y ago

As an Ops guy, I preach Ansible + systemd all day everyday, but so many of our Devs (and Ops) have drunk the containerization Kool-aid. Unfortunately, the same culture which fucked modern web design (a new JS framework every 6 months) is starting to creep into Ops; some REALLY smart people make a tool which promises to be X but with X's main problem solved. They do the easy 80% and then abandon the project when the hard 20% needs to be fixed (and forget documentation).

lobster_johnson 10y ago
The JavaScript situation is grossly exaggerated. It's doing very well. It's just that the people who aren't productive are also the ones shouting the loudest on HN. Meanwhile, a lot of people are being very quiet getting tons of stuff done really efficiently. (Source: Me, very happy React developer who is so glad we're finally out of the terrible "wilderness years" of server templates and jQuery.)
- stephenr 10y ago
  > terrible "wilderness years" of server templates wat.
  - deleted 10y ago
    [deleted]
  - lumpypua 10y ago
    That's fine, it's: > terrible "wilderness years" of server templates and jQuery where things go off the rails.
    - yellowapple 10y ago
      I reckon > terrible "wilderness years" of ... jQuery is the part where things go off the rails (and literally off the Rails).
      - 10y ago

lobster_johnson 10y ago

The idea with Kubernetes isn't necessarily that it can do resource allocation more carefully and more intelligently than a human ops person, but it can do so at a level of automation and speed that a human can't. Pods die, Kubernetes spawns new ones. Server goes full, Kubernetes spawns new node. Server runs out of disk space, Kubernetes evicts nodes until everything is good again. Server goes down, Kubernetes shuffles all the pods over to a different node. And so on. It does this at a scale of hundreds or thousands of machines. The utility it brings also applies to small clusters. You don't remove ops, you just flatten it into Kubernetes' operational model. Equally importantly, Kubernetes also brings a declarative deployment model. You have the same with Systemd, but at the node level. Kubernetes is a bit like Systemd at the cluster level; its "units" are pods declared in YAML/JSON files, but unlike Systemd they run on any node the cluster, and are subject to a resource allocation model that includes things like persistent storage (e.g. AWS EBS volumes). You don't get anything close to that with Systemd, which is much more analogous to Docker. I haven't heard anyone ever claim that Docker brings reproducible builds -- why would anyone assume that? What Docker give you is reproducible deploys; build once, deploy many times from the exact same image. Maybe that's what you're thinking of?

erulabs 10y ago
I don't dislike Kubernetes, in fact im a (very minor) contributor and regular user and am a member of the federated kubernetes workgroup. A small group of microservices very happily lives on Kube in production. But, for the sake of the argument: > Pods die, Kubernetes spawns new ones. AWS ASGs also solve for this, as does every other cloud provider at the infrastructure level. For applications, systemd and any init system can restart an application when it crashes... > Server goes full, Kubernetes spawns new node. I assuming you mean utilization - again, ASGs/CloudWatch solve for this as do every other cloud provider (if they didn't a few minutes worth of Sensu plugins and good-to-go). > Server runs out of disk space, Kubernetes evicts nodes until everything is good again Right but this can and has lead to disaster. I got paged for bad response time and found out the issue was that containers were being evicted and pods were going critical because _all_ hosts were filling up on diskspace and there were no healthy nodes on which to put pods. This due to one bad service writing out tons and tons of log data). Had that been dedicated, I would have just gotten paged for one full disk and fixed the problem when it showed up instead of waiting for a bigger issue. This is super fixable in Kube (and was - that service now has its own volume and still lives in kube) - but my point is that it's a problem for everyone. > Server goes down, Kubernetes shuffles all the pods over to a different node. This effects response time of user facing applications as new 1GiB+ containers are shuffled around (I know, thats an unfair example). ASGs and Cloudwatch solve for this in a saner / safer way. As for the reproducible builds, it was in response to the article which says: > The ‘repeatable builds’ argument is the most interesting to me, in part because this is the same promise that java made in the 90s (write once run anywhere). You may be correct that Docker doesn't promise this explicitly, but it is a big misconception. Additionally, `tar` solves for "reproducible deploys"... I'm not an AWS employee, just figured I'd use a common cloud as an example.

mianos 10y ago

Systemd + ansible is very close to what we have. We are python shop so we use: - salstack to deploy and update virtualenvs and applications. - salstack to install and configure upstart jobs - logging goes to /var/log/upstart but this gets shipped automagically by salstack configured per app awslogs config. - All config is read from zookeeper. - The only thing that is initialised on the machine is a ZK host environment variable (by salt-cloud). Everything is ummutable. (Using Postgres RDS for the moment). A system dies we can just make another one with salt. We can run multiple services on one machine or one per machine or install them in lxc containers trivially.

amorphid 10y ago
Minor nitpick... I believe you mean "saltstack". Yes?
- mianos 10y ago
  Sorry, yes saltstack. Typing on my phone while my kids jump on my head. saltstack is like a toolkit for all this sort of stuff.
  - amorphid 10y ago
    Off topic, I just looked up the definition of nitpick. Apparently it refers to removing nits, the eggs of lice :O https://en.wikipedia.org/wiki/Nitpicking https://en.wikipedia.org/wiki/Nitpicking I did not mean to imply your comment has lice :)
    - mianos 10y ago
      I have kids in primary school, there are probably some lice in there somewhere.

lima 10y ago

I really like the CoreOS approach of using systemd for container supervision.

aprdm 10y ago

We use ansible and systemd with around 100 services in 3 continents and works well.

dyu- 10y ago
Cool. If any of it is opensource, please post the links in the reply, I'm curious.

dyu- 10y ago

+1 for systemd + ansible. That is exactly my setup on ubuntu 16.04 deployments.

zedpm 10y ago

I see a lot of "me too" responses to this, and for the life of me, I can't figure out what systemd brings to the table. I understand the Ansible part, but what does systemd do that's notable? My understanding is that it's an alternative to init, and I can't imagine calling out init as part of my deployment/config strategy. What am I missing?

collinmanderson 10y ago
I'm wondering the same thing, as someone who has only started using systemd. From what I can tell it handles: - starting the various pieces of the app in the correct order - permissions (uid, cgroup, maybe even virtual file system) - environment variables, working directory - passing around sockets - restarting things if needed - monitoring resource usage - centralized logging