6 ms·
The more complex the process you use to automate tasks, the more difficult it is to troubleshoot and maintain, and the more impossible it is to inevitably repla
by peterwwillis 13y ago
The more complex the process you use to automate tasks, the more difficult it is to troubleshoot and maintain, and the more impossible it is to inevitably replace parts of it with a new system. https://xkcd.com/1319/ https://xkcd.com/1319/ is not just a comic, it's a truism.
I am basically a Perl developer by trade, and have been building and maintaining customized Linux distributions for large clusters of enterprise machines for years. I would still rather use shell scripts to maintain it all than Perl, or Python, or Ruby, or anything else, and would rather use a system of 'stupid' shell scripts than invest more time in another complicated configuration management scheme.
Why use shell? It forces you to think simpler, and it greatly encourages you to extend existing tools rather than create your own. Even when you do create your own tools with it, they can be incredibly simple and yet work together to manage any aspect of a system at all. And of course, anyone can maintain it [especially non-developers].
As an example of how incredibly dumb it can be to reinvent the wheel, i've worked for a company that wanted a tool that could automate any task, and that anyone could use. They ended up writing a large, clunky program with a custom configuration format and lots of specific functions for specific tasks. It came to the point where if I needed to get something done I would avoid it and just write expect scripts, because expect was simpler. Could the proprietary program have been made as simple as expect? Sure! But what the hell would be the point of creating and maintaining something that is already done better in an existing ages-old tool?
That said, there are certain tasks i'd rather leave to a robust configuration management system (of which there are very few in the open source world [if any] that contain all the functionality you need in a large org). But it would be quite begrudgingly. The amount of times i've ripped out my hair trying to get the thing to do what I wanted it to do while in a time and resource crunch is not something i'd like to revisit.
- rjzzleep 13y agoi both agree and disagree. you can create rather huge complex incomprehensible systems in bash too. also, it's not always better to use existing systems. sometimes rolling something new is indeed better. but i had to work in a place where everyone was a ruby fetishist. don't get me wrong ruby paid my rent for the last couple of years, but i too thought that using it for basic system administration tasks made absolutely no sense. at least what they used it for. it's used in chef/puppet for one reason only. because it's easy to make a dsl in ruby, and it's good for that purpose actually. although i find chef terribly overengineered
- cwp 13y agoJust to be sure I've understood correctly, what you're saying is "Bah, humbug!" Is that right? Edit: OK, sarcasm aside, I can't see how this comment amounts to more than "I don't need anything fancier than shell scripts." Fine, you don't. But there are lots of people out there that do. For several years, I ran a cluster that consisted of 10 or so EC2 instances that had to rebuilt and replaced every day. Shell scripts didn't cut it for me. Chef and Puppet were painful. I ended up using Fabric, boto and a pile of custom Python code. And 10 machines is nothing. How do you maintain a 1000-machine cluster with frequent application deployments, configuration changes and variable load? If you've got a bunch of shell scripts that make that easy, that would be fantastic news. I kind of doubt it, though.
- rhizome 13y ago10 or so EC2 instances that had to rebuilt and replaced every day Just to be understood correctly, you're saying the exception proves the rule? Sidestepping the necessity of the task above, which I'm sure was rendered easier by Fabric or whatever, but it's not like that's a typical use case. You would have to ignore nuance to extrapolate that to generic VM instances from known-state images, etc. that might respond just as well to a bunch (not even a bunch) of scripts.
- toomuchtodo 13y ago> And 10 machines is nothing. How do you maintain a 1000-machine cluster with frequent application deployments, configuration changes and variable load? If you've got a bunch of shell scripts that make that easy, that would be fantastic news. I kind of doubt it, though. Four years ago, I was going this with shell scripts, a little bit of Python, and a cluster management tool called Rocks (built in Python and shell scripts). This was for a 5500+ Linux server cluster processing data from the CMS detector at the LHC. I had the ability to do a rolling reinstall of Scientific Linux/CentOS while the cluster was actively processing data with no interruption to the end user jobs. Everyone keeps reinventing the wheel, but in most cases, shell scripts can do a lot of heavy lifting before you need to start writing one-off python scripts.
- jessaustin 13y ago
- haberman 13y ago> i've worked for a company that wanted a tool that could automate any task, and that anyone could use. They ended up writing a large, clunky program with a custom configuration format Lots of people design terrible, overly complicated systems. This is an argument for good design, not for ad hoc shell scripts for everything. Would you write a build system in shell? Why not? Because "make" is far better suited to the job. (Even "make" has a lot of room for improvement, but that's a story for another day). As another example, even Debian (known for its conservatism) decided that systemd was a net benefit over sysvinit, where the latter could quite accurately be described as a pile of shell scripts. Abstractions that suit the problem domain are far superior to ad hoc imperative bags of commands.
- justizin 13y agosystemd, not launchd.
- haberman 13y agoThanks for the correction.
- peterwwillis 13y agoAnd my argument was that a simple design is good design, and in my opinion, better than something that isn't simple. You could definitely write a build system in shell. In my opinion, "the job" is subjective and you shouldn't have to pick one tool for it. But I would prefer a simple one where I can see pretty much everything it's doing up front and not have to download source code or search through docs. Debian picked up systemd because it was a popular decision, not because it was better. As far as i'm aware there was nothing broken in sysvinit that systemd fixed; there was just a vocal minority that wanted some new features and insisted it be shipped as default. And in defense of shell scripts, nearly every single operating system you can think of other than Windows uses scripts to initialize (hell, even Windows has some startup scripts), and that is an abstraction that's suited to the problem domain. It's not like for the past 50 years people have been too stupid to come up with something better than shell scripts. The whole point of init scripts is to let anyone be able to modify their operating system's start-up procedures quickly and easily. If we didn't need to modify them to suit our individual cases they'd be statically compiled C programs or assembly (which, by the way, I have done in the past; it's not a good idea)
- njharman 13y ago> Why use shell? It forces you to think simpler I have the shell scripts to prove it makes you think more convultedly in order to get things done that are cleaner in other languages.
- cinquemb 13y agoFunny enough, I'm using a combination of small shell scrips that are executed within an internal application depending on the status of the the machine. So I guess I'm in the "why not use both" position. Currently, I have automated system set up where basically a cluster of vm's in nodes of two communicate with each other to pass over where the other server in a node left off in its operation (or dies from timeouts) running nginx/gunicorn/django. And those servers on each node are monitored by another server (running apache/php/mysql) that checks the progress/status of the operations and may send requests (reinitialized a node that stopped running) to the nodes where those bash scripts (concatenating files, finding a specific place in one of the files to help reinitialize a process in a node) are executed and piped through back to the monitoring server. It is way more complex now, but surprisingly, I don't have troubleshoot not even close to as much as before since I automated that.
- sparkie 13y agoThe troubleshooting issue is one of the biggest benefits of Nix IMO - because the complex automation process which it significantly reduces the need to troubleshoot is the GNU[1] build system. ([1] replace GNU with other build system of your choice.) The GNU build system (+pkg_config) is really awful to work with when it doesn't magically work the first time. You'll often get errors such as m4 macros not being defined (but which don't hint at what dependency is missing), and more often than not you'll need to build several dependencies for the software you want, which are often poorly specified. It's not uncommon for one to waste hours or whole days trying to build a piece of software (and if you're lucky, he will blog about his experience so that you can spend less time when you inevitably google the problem, because that's always easier than trying to understand and debug the build scripts yourself.) Nix reduces this by requiring exact dependencies and build instructions - there's no "hidden knowledge" that the software developer/package maintainer had which the user should not also have available, so at most, only one person needs to experience the pain of building the software and others can reproduce based on his precise instructions, rather than his blog post. I can't honestly say that there will be no problems when using Nix, but a build error in nix is a bug in the distributed nix packages, not just some unknown problem on the users system which he needs to fix. I agree with you that reinventing the wheel is usually bad, and you get a long way by reusing existing programs with shell scripts, but sometimes you need to throw away your assumptions about how things work because a new model is significant enough to rethink the old. (We didn't build aeroplanes by attaching wings to cars).
- mercurial 13y ago> As an example of how incredibly dumb it can be to reinvent the wheel That's a misunderstanding. It's like saying "Java is a reinvention of Haskell". Neither is a reinvention of the other, they are two very different beasts. You don't have any OS whose entire state (package and configuration) is described at a central location and can be recreated by a single command (outside of research projects, possibly). You may disagree with the approach they take, but such a blunt dismissal dismisses at the same time any kind of original approach to a problem. Speaking of which, you speak in favour of shell as a solution to automation. What you are saying is that you push the complexity on the side of the shell script developer. In my experience, declarative systems tend to have more complex innards, but it also means more maintainability, and a flow much easier to understand. This is the kind of tradeoff you make when forsaking sysvinit in favour of systemd.