5 ms·

I published this article a couple of weeks ago: https://sourcehut.org/blog/2020-04-20-prioritizing-simplitity/ https://sourcehut.org/blog/2020-04-20-prioritizi

I published this article a couple of weeks ago: https://sourcehut.org/blog/2020-04-20-prioritizing-simplitity/ https://sourcehut.org/blog/2020-04-20-prioritizing-simplitit... Dozens of people wrote to me, aghast at the idea that I provision servers manually, wondering how I could refuse to use the rube goldberg machines that define the modern "cloud-scale" computing trends. It looks like it didn't take long for my approach to be validated. Salt is complicated as hell. k8s, Docker, Ansible, they're all the same. Complicated systems break more often and more severely than simple systems. It's a simple fact, and I hope that our industry gets over its collective devops fetish sooner rather than later.

tptacek 7y ago
No, Salt is distinctive: it uses resident agents, unlike other tools; infamously, they managed to implement e=1 RSA in their transport protocol. You cannot reasonably extrapolate from Salt to Terraform or Ansible.
- ddevault 7y ago
  I can, and I will. It's not just security that's affected, it's availability, reliability, and maintainability as well, and it fosters a culture which doesn't value any of those things nearly enough, leading to a bunch of hulking monoliths lumbering over to the pile of VC money for dinner. I have no confidence in any of these systems.
  - geerlingguy 7y ago
    A system like Ansible, in particular, doesn't install anything resident on the servers it manages, nor manages state which could be exploited on a management machine. And in one sense, it's like writing a shell script and running it on the server via SSH. Manual provisioning scales well enough for a single server but is much less secure when you scale past one server, or need to be able to rebuild that server in a hurry, like when your cloud provider inevitably deletes it.
    - ddevault 7y ago
      Ansible will faithfully deploy any security vulnerabilities you tell it to across your whole fleet, in just seconds! Don't make the mistake of thinking it's immune to errors. https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=ansible https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=ansible Your servers ought to be simple, too. Once you're simple enough, the problems shrink and shrink until they're barely there. Or, you can go as complex as possible and just hide them in mountains of garbage. "There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies."
      - ghshephard 7y ago
        Okay - I'm intrigued. How do you recommend managing infrastructure which consists of thousands of servers? With k8s - you just delete the pod and it gets recreated with a clean image within a few seconds. What's your proposal?
        ddevault 7y ago
        I recommend not having thousands of servers. Who has thousands of servers? Maybe 10 companies, worldwide, have an actual demonstrable need for that. The rest of the companies are just engaging in devops masturbation.
        akerl_ 7y ago
        What about the plethora of companies that provide B2B IT services? If your company’s business is “deploy $insert_products on-premise for each customer”, and you’re doing good business, you’re going to quickly approach thousands of servers.
        AmericanChopper 7y ago
        I was doing some work for a bank last month. It’s a reasonably small bank, and you likely haven’t heard of it. They have thousands of servers. They had over 100 logstash instances alone. All their desktops were virtualized, and that was a couple thousand servers at least.
- tialaramex 7y ago
  I feel like the resident agent thing is the wrong focus. Goofing RSA is a more troubling sign, because it means there probably isn't anybody who understands how this stuff works in the room where plans to use it are made, and that means there will be security bugs forever. Presumably other people don't agree with me or else there wouldn't be important stuff running on Salt?
  - tptacek 7y ago
    I don't even know if they still use the transport protocol they came up with. I remember more clearly the Red Hat security guy claiming, on the same Github issue thread, that e=3 RSA was dangerous because of GPUs, which is almost an equally fucky thing to say. The agent thing, though, would squick me out almost no matter which team implemented it.
- psanford 7y ago
  Isn't it fairly common for config management tools to use agents? Chef and Puppet both do. Once your configuration management is tied into your CI/CD pipeline, I don't really see much of a difference between an agent running on each machine vs a master node that has a root ssh key into your whole fleet.
  - gnufx 7y ago
    And sshd is a fairly common thing to fall over in my experience (though particularly on badly-managed clusters). Presumably more so on otherwise-inaccessible systems, by the laws of life. The agent saved systems isolated in places like x-ray hutches a number of times back when I thought using cfengine was a good idea.
adrr 7y ago
Or you just don’t expose these services externally. Even with Ansible, why open up sshd to publicly facing interfaces. Use a jumpbox or VPN. Why increase your threat surface?
- ddevault 7y ago
  This attitude is what turns a relatively minor breach into 100,000 credit cards being sold to the highest bidder. Ever heard of "lateral movement"?
wolfgang42 7y ago
I agree that keeping a system as simple as possible is a good idea. But, you may want to take a step back and consider why (as of this writing) every single comment from you—and nobody else’s—is grey in this thread. (It’s not often on HN that you see two luminaries such as you and 'tptacek get into such a vigorous and apparently one-sided debate.) I was going to attempt to summarize both sides of the argument here, but instead I think it’s better if I just suggest that you take some time to attempt doing that yourself and try to understand the points that everyone else is trying to make. I don’t think it’s your point about excess complexity that’s garnering the downvotes (at least, I hope it isn’t), but rather (a) that you’ve tarred a very broad swathe of software with exactly the same brush, refusing to acknowledge the significant differences in design, implementation, and use between such diverse softwares as Kubernetes and Ansible; and (b) the dogmatism about there being no conceivable need for such any such tools under any circumstances whatsoever. I’d be very interested in a blog post (if one hasn’t been published already) about how system management is done at Sourcehut. A lot of people (myself included) turn to the tools you’re disparaging to help them manage things like change tracking and testing server configurations before applying to production, and it would be interesting to read about how you manage these sorts of problems without introducing a tool to do it.

tptacek 7y ago

No, Salt is distinctive: it uses resident agents, unlike other tools; infamously, they managed to implement e=1 RSA in their transport protocol. You cannot reasonably extrapolate from Salt to Terraform or Ansible.

ddevault 7y ago
I can, and I will. It's not just security that's affected, it's availability, reliability, and maintainability as well, and it fosters a culture which doesn't value any of those things nearly enough, leading to a bunch of hulking monoliths lumbering over to the pile of VC money for dinner. I have no confidence in any of these systems.
- geerlingguy 7y ago
  A system like Ansible, in particular, doesn't install anything resident on the servers it manages, nor manages state which could be exploited on a management machine. And in one sense, it's like writing a shell script and running it on the server via SSH. Manual provisioning scales well enough for a single server but is much less secure when you scale past one server, or need to be able to rebuild that server in a hurry, like when your cloud provider inevitably deletes it.
  - ddevault 7y ago
    Ansible will faithfully deploy any security vulnerabilities you tell it to across your whole fleet, in just seconds! Don't make the mistake of thinking it's immune to errors. https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=ansible https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=ansible Your servers ought to be simple, too. Once you're simple enough, the problems shrink and shrink until they're barely there. Or, you can go as complex as possible and just hide them in mountains of garbage. "There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies."
    - ghshephard 7y ago
      Okay - I'm intrigued. How do you recommend managing infrastructure which consists of thousands of servers? With k8s - you just delete the pod and it gets recreated with a clean image within a few seconds. What's your proposal?
      - ddevault 7y ago
        I recommend not having thousands of servers. Who has thousands of servers? Maybe 10 companies, worldwide, have an actual demonstrable need for that. The rest of the companies are just engaging in devops masturbation.
        akerl_ 7y ago
        What about the plethora of companies that provide B2B IT services? If your company’s business is “deploy $insert_products on-premise for each customer”, and you’re doing good business, you’re going to quickly approach thousands of servers.
        AmericanChopper 7y ago
        I was doing some work for a bank last month. It’s a reasonably small bank, and you likely haven’t heard of it. They have thousands of servers. They had over 100 logstash instances alone. All their desktops were virtualized, and that was a couple thousand servers at least.
tialaramex 7y ago
I feel like the resident agent thing is the wrong focus. Goofing RSA is a more troubling sign, because it means there probably isn't anybody who understands how this stuff works in the room where plans to use it are made, and that means there will be security bugs forever. Presumably other people don't agree with me or else there wouldn't be important stuff running on Salt?
- tptacek 7y ago
  I don't even know if they still use the transport protocol they came up with. I remember more clearly the Red Hat security guy claiming, on the same Github issue thread, that e=3 RSA was dangerous because of GPUs, which is almost an equally fucky thing to say. The agent thing, though, would squick me out almost no matter which team implemented it.
psanford 7y ago
Isn't it fairly common for config management tools to use agents? Chef and Puppet both do. Once your configuration management is tied into your CI/CD pipeline, I don't really see much of a difference between an agent running on each machine vs a master node that has a root ssh key into your whole fleet.
- gnufx 7y ago
  And sshd is a fairly common thing to fall over in my experience (though particularly on badly-managed clusters). Presumably more so on otherwise-inaccessible systems, by the laws of life. The agent saved systems isolated in places like x-ray hutches a number of times back when I thought using cfengine was a good idea.

adrr 7y ago

Or you just don’t expose these services externally. Even with Ansible, why open up sshd to publicly facing interfaces. Use a jumpbox or VPN. Why increase your threat surface?

ddevault 7y ago
This attitude is what turns a relatively minor breach into 100,000 credit cards being sold to the highest bidder. Ever heard of "lateral movement"?

wolfgang42 7y ago

I agree that keeping a system as simple as possible is a good idea. But, you may want to take a step back and consider why (as of this writing) every single comment from you—and nobody else’s—is grey in this thread. (It’s not often on HN that you see two luminaries such as you and 'tptacek get into such a vigorous and apparently one-sided debate.) I was going to attempt to summarize both sides of the argument here, but instead I think it’s better if I just suggest that you take some time to attempt doing that yourself and try to understand the points that everyone else is trying to make. I don’t think it’s your point about excess complexity that’s garnering the downvotes (at least, I hope it isn’t), but rather (a) that you’ve tarred a very broad swathe of software with exactly the same brush, refusing to acknowledge the significant differences in design, implementation, and use between such diverse softwares as Kubernetes and Ansible; and (b) the dogmatism about there being no conceivable need for such any such tools under any circumstances whatsoever. I’d be very interested in a blog post (if one hasn’t been published already) about how system management is done at Sourcehut. A lot of people (myself included) turn to the tools you’re disparaging to help them manage things like change tracking and testing server configurations before applying to production, and it would be interesting to read about how you manage these sorts of problems without introducing a tool to do it.