12 ms·
Fucking Shell Scripts
- qwerty_asdf 13y ago"Wanna just use fucking shell scripts to configure a server? Read on!" Step 0: Install the gem # gem install fucking_shell_scripts # gem: command not found wat.
- raverbashing 13y agoExactly If I want to use FSS to configure the server, I already do. I don't need the "latest fad in configuration tool" to do something FSSs already do! Like fabric. If you're not using their advanced features, you're writing a shell script in python. Thanks, I'll write a shell script
- dogas 13y agoworks for me! http://rubygems.org/gems/fucking_shell_scripts http://rubygems.org/gems/fucking_shell_scripts
- qwerty_asdf 13y agoYeah, but like, that's not a shell script, man.
- atomic-penguin 13y agoIts just fucking shell scripts, already! * * Depends on Ruby.
- phazmatis 13y agoMost people coming from chef would have done the drudgery of msking surr ruby works.
- justizin 13y agochef installs as omnibus, so you're suggesting that people should install the massive chef stack, and then the FSS gem? _brilliant_ !
- phazmatis 13y agoWell this is clearly aimed at people who are frustrated with Chef, above and beyond the frustrations of figuring out how to install ruby (just install from source, rvm sucks). If someone doesn't already have Chef installed, why would they even have clicked on this link?
- deleted 13y ago[deleted]
- cmbaus 13y agoI agree. This would be far more awesome if you could install it with a fucking shell script. It is called fucking shell scripts. Not fucking ruby scripts.
- cmbaus 13y agoJoking aside, there is something to be said about doing CM with the shell. If somebody created a list of requirements for the perfect language for doing CM it would probably be similar to: - Always available on every machine - Proven reliable and stable - Built to interface with the OS and other utilities Hmm, that sounds a lot like the shell. If your application doesn't already depend on Ruby, bringing Ruby in as a dependency, is a lot of extra overhead. There is something to be said about this concept, but it should be shell all the way down.
- icebraining 13y agoOn the other hand, Python comes installed by default on Red Hat, CentOS, Scientific Linux, Gentoo, Debian, Ubuntu, MacOS X, Solaris and OpenIndiana, at least, so a "full" CM system like Ansible is potentially more portable than FSS :)
- suprjami 13y agoHahaha, I love that the install process is more complicated than the example shell script.
- deleted 13y ago[deleted]
- Fasebook 13y agoWait, a gem, a yaml file? What exactly is this doing with my scripts? Can I really expect this to make my shell scripts work across clouds?
- deleted 13y ago[deleted]
- deleted 13y ago[deleted]
- akerl_ 13y agoSo... this is basically just one module from Ansible, plus some obscenities?
- drewvolpe 13y agoWhich is another way of saying it's Ansible without all of the stuff you don't need (plus obscenities).
- akerl_ 13y agoExcept all the modules you don't use aren't getting in your way. I'd be totally for this as a simpler solution for simpler use cases, if it was actually easier than Ansible to get installed and set up. Since it's not, you're paying the same upfront cost that you'd pay for the more powerful tool, except you'll hit a wall if you try to expand it later.
- mpdehaan2 13y agoWe can add obscenities if you want :) We do have cowsay integration though. (There are some questionable cowsay modes!) That all being said, the thing missing here between any of these tools is most obviously the resource model -- and the templating system and where you put variables and things to manage variance between systems. Thus, it will blast out some commands for you, but that is the easiest part of the equation -- not too much different than say, doing something from Capistrano or Fabric. Even in Ansible, that's the part we built first. Achieving idempotence in shell scripts is the reason most people move away from shell scripts, and also ... well, the desire to program less :) Then you'll want the rolling update features, or provisioning, or dry run, or a way to pull inventory from cloud sources, or... and you'll streamroller a bit. The balancing act for us is achieving the right level of features vs language complexity and keeping in that sweet spot. I still think you should have an easy time getting started, see also things like http://galaxy.ansible.com http://galaxy.ansible.com for community roles to download to go faster -- most people should be able to do basic things in a few minutes. The script module in particular is a great way of pushing a f'n shell script :) http://docs.ansible.com/script_module.html http://docs.ansible.com/script_module.html (full disclosure: I created Ansible, but I did not shoot the deputy)
- EGreg 13y agoThere are two ways to debug something: 1) Step through the program, try to think of logic errors etc. 2) Do a binary search through your latest commits and see where something broke. Those are your two clues.
- RyanZAG 13y agoHonestly this is not a bad idea. Since bringing up a new instance VM will always be in exactly the same state, the shell script is completely deterministic. If any line fails to run, you could simply log an error and automatically shut down the VM. The bash script itself is also extremely straight forward and is easily testable with a built in REPL (you know, bash). Anybody who vaguely knows unix is also going to be able to understand and maintain the shell script. Adding in new dependencies is simple. It's easily portable and you don't need to do any extra work to add in new features. I can't even think of a drawback.
- nawitus 13y agoTypically shell scripts are fine in the beginning, but over time the complexity rises and it becomes an unmaintanable mess, and you'll end up reimplementing it in a proper programming language. Shell scripts are not easily portable either, unless by "portable" you mean "works in Linux". Node.js scripts, for example, really (mostly) work on Linux and other platforms like Windows. Until you hit some nasty "path is longer than 254 characters" bugs. Oh well..
- michaelmior 13y agoWhy do you really need it to be portable outside of Linux? I don't imagine many services will evolve to run on other platforms in their lifetime without significant configuration changes anyway. I can't really argue the point about complexity though.
- tachion 13y agoBecause there are other operating systems out there too, with their own features making them better choice (or just a preference) for certain tasks over Linux, including FreeBSD, OpenBSD, DragonFlyBSD, SmartOS, Illumos... World doesnt end with Linux, neither starts with it ;)
- nbouscal 13y agoI think the point was that it's not common for someone to switch from running a server on Linux to running a server on something else, and it's even less common for someone to do that without changing the configuration pretty extensively.
- hedwall 13y agoWho in their right mind wgets and builds software on a production machine...?
- sgt 13y agoI actually do this with one piece of software, and that is PostgreSQL. I know it in and out, it's easy to upgrade via source, and I prefer to put the database in a bundled location e.g. /data/postgres or /home/postgres... instead of /var/lib/postgresql/9.1/main, /etc/postgresql/9.1 et cetera.
- fletchowns 13y agoHopefully: over https, not from the central repo, and only after verification of the sha256sum
- icebraining 13y agoWhy not just build your own packages? fpm[1] is very simple to use. [1] https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm
- justizin 13y agoThe authors of Chef.
- area51org 13y agoUh ... not really.
- dogas 13y agoCoauthor here. Well, this is a surprise! This project was mainly borne out of our frustrations with Chef and then Ansible for configuring our servers. A few thoughts: 1) This project is incomplete and is more of a concept than anything. We wrote it literally in an afternoon, and were giggling like schoolgirls the whole time. 2) There is value in what Ansible provides. It does a lot of things for you. FSS did not replace ansible where we work. All that being said however, I feel like this (or a project like this) does have merit, if executed properly. I feel that once you get used to how FSS works, it might be a good solution for you. We intentionally tried to keep it as simple as possible.
- sdegutis 13y agoWe're using Chef right now, and evaluating other options. They seem to either be between way too over-engineered, or not flexible enough, but nowhere in between. I'd love to see a tool like yours be brought to full maturity, then I think I'd love to use it.
- frist45 13y agoOther co-author here: I couldn't agree with you enough. We settled on Ansible, but have had our fair share of speed bumps along the way.
- mikegirouard 13y agoOut of pure curiosity, what makes Ansible the best choice for you? For my team, I decided on Ansible because it was the simplest option (no agents to install, pure ssh).
- michaelmior 13y agoIt would be nice if there was a way to use this without having to deal with obscenities. Not everyone finds this funny. Useful idea though :) At least for early stage projects, I can imagine it getting complicated quickly.
- adamors 13y ago> gem install fucking_shell_scripts This fucking sucks.
- fredsanford 13y agoI gotta know... Do the scripts reproduce with all that fscking going on?
- cordite 13y agoGot a good laugh out of me, seems a little hypocritical to depend on ruby (which chef et al. use) So far I only have experience with puppet, and it seems really annoying to tie phases like "install postgres" -> "Reinitialize the database in UTF-8, but known to work on ubuntu" -> "Update the configs so non-localhost connections can actually connect" -> "Okay, now it can start." The above problem is something I do not want to deal with when it comes to a provisioning system.
- frist45 13y agoAuthor here: fair point. Ruby is only required for the client that's building machines. Ruby is our primary language so it's on all of our servers and laptops. Less about thought and more about lowest barrier to entry ;)
- cordite 13y agoSeems like bash (or insert your favorite shell here) would be THE lowest barrier of entry on any Linux, BSD, whatever host--perfect for this project! :-)
- agwa 13y agoThe nice thing about an approach like this is that you don't actually need to use shell for your scripts - you could use Perl, Python, etc. The point is you can just write in whatever scripting language you like/gets the job done and not need to worry about learning a new domain-specific configuration language. Does anyone else here see value in a configuration management system that centers around installing files and running scripts, much like FSS? I'm contributing to, and using in production, a configuration management system that takes this approach. It's a lot more mature than FSS, but unfortunately the primary author isn't ready to open source it yet. If there's some demand for it, maybe I can convince him to hurry up.
- sbt 13y agoThis does not manage a server, it provisions a server. The whole point of configuration management software is to bring your software into a desired state from any actual state the server may be in. This rests on the notion that configuration drifts, i.e. for long running services a server gets misconfigured eventually for a variety of reasons. Bottom line, if you are just running a small startup with a few servers and a few people, then by all means use FSS, but eventually you will need real CM.
- agwa 13y agoThe CM system I'm using "journals" every change it makes to the system, so if you later remove or change a file or script in your CM repository, it undoes the old change. This eliminates virtually all configuration drift, provided you aren't making changes by hand outside of the CM system (which you should never do anyways). This system has been managing about 100 servers and several hundred desktops at a single site since 2009, so the approach does scale. It has some problems that declarative solutions like Puppet solve, but at the same time it solves problems that Puppet/Chef/others have.
- oftenwrong 13y agoWhich CM system?
- 13y ago
- badman_ting 13y agoEvery time I go to write a shell script I'm like "do I need to put quotes around this variable to interpolate it? How do I do looping again?" then I give up and use Python or Perl or something.
- phazmatis 13y agoI guess you could load your favorite language runtime. That sounds like a one line change. In your shell script. Which is the point. I'm betting at some point you look back and realize that much of your config doesn't need a secondary language.
- icebraining 13y agoHow would you take a template file, fill in some values, copy it to a certain path on the server and give it the right permissions? With, say, Python, I'd have to read the file, using mako or jinja to fill in the values, scp the file to some temporary file, then somehow run Python code on the server using SSH to copy the file and give it the right permissions. With Ansible, this is a single line.
- phazmatis 13y agoLet's see... I would take the template file, fill in some values, scp it up to it's final location on the server, and run chmod on the server. I guess I could learn the "one-line" ansible way to do it, but I'd also have to learn to set up ansible. And I'm guessing it's not as flexible as, say, shell scripts.
- icebraining 13y agoThe cool thing about Ansible is that there's no set up. You install it, write the config and run "ansible-playbook <config>". It connects to the nodes using SSH and runs without having to be installed in them. As for not being as flexible as shell scripts, I'd say that's technically impossible, since it has a command for running shell scripts :) Personally, I never had to use it, but it's there.
- 616c 13y agoIf you really want some fucking shell scripts: http://www.nico.schottelius.org/software/cdist/ http://www.nico.schottelius.org/software/cdist/
- razfar 13y agoThe name is bad.
- WoodenChair 13y ago1. You use unprofessional obscenities in your project title 2. I don't take your project seriously (perhaps unfortunately)
- blisterpeanuts 13y agoAgreed. It's like some 17-year-old trying to sound cool.
- npizzolato 13y agoNot all of us get offended by the word fucking. We don't have to be super serious all the time.
- lamby 13y agoDon't get me wrong, I'm the biggest fan of shell scripts and chapeau for writing opinionated software. However, not having idempotent configuration files seems to be not only missing some of the power of shell scripting but also missing a huge principle of "good" server configuration.
- liveoneggs 13y agoanother project that needs to discover rdist
- deleted 13y ago[deleted]
- isaacb 13y agoI really like the idea, but the name is somewhat offputting, especially if I were in a position where I had to sell it to management (luckily I'm not). Will definitely give it a whirl, but I strongly suggest considering a name change.
- obsurveyor 13y agoIf you're going to use ruby, you may as well commit and use sprinkle. You still get to use all the shell scripts you want and you get the deployment power of capistrano. I've really enjoyed using it for my small provisioning requirements.
- nsxwolf 13y agoIt's not a good name. Outside of the Valley, where youth and irreverence do not dominate, the name makes this a non-starter.
- twic 13y agoActually, i think the name goes over rather well in London, where we like swearing (and shell scripts) quite a lot.
- LukeShu 13y agoI too think the name is bad, but for a different reason than everybody else. Without context, it sounds like the name is complaining about shell scripts; where the project itself endorses them. Before I clicked the link, I was expecting to come here and write a defense of shell scripts.
- benched 13y agoGuiys, the fucking swearing in the name suks alot. Not all the managers like it. That's why we have perfesionalism, cause prfesionalism is good. Guiys, we haf to get along. We cant do it with swearing. Please, think about pleasing everyone and offending none. Otherwise ur milenials, ruining our designated corprate productivity spaces.
- eli 13y agoShrug If you want people to use your software -- and you do or you wouldn't be promoting it -- you should consider how you present it. Bad words don't offend me, but they do suggest immaturity and inexperience... qualities I tend to avoid in systems orchestration.
- andrewflnr 13y agoThis actually looks like as much if not more work to set up than Ansible. As soon as you start talking about required directory hierarchies, you've lost the "just f###### do it" feel. Then I still have to write a special YAML file, and then my scripts. The Ansible setup for my personal server started as two files: an inventory file consisting of an ip address, and site.yml. My ignorance (and some installation issues) notwithstanding, it has scaled pretty smoothly from there to copying up config files, templated nginx config, and so on. I don't see much room for anything between that and a literal "just f###### shell script".
- efuquen 13y agoI originally thought this was a lament about how shell scripts suck, like along the lines of "man, those fucking shell scripts ..." My surprise when I saw this is the exact opposite of that ...
- area51org 13y agoThis stuff is all fine and good for a few VMs. I wouldn't try to run a serious endeavor using them, unless I really wanted to spend a bunch of time diagnosing, debugging, and sweating. The reason automation, CM, and devops have taken hold is because people are so goddamned tired of serving technology like it's some kind of god to which we owe worship. Screw that. I want the VMs to go make ME a sammich, not the other way around. Trying to build serious infrastructure using nothing but shell scripts is a quick trip to the temple of server-worship.
- gerhardlazu 13y agoAfter 4 years of heavy Cheffing, I completely agree with most opinions expressed here. I'm pretty sure Puppet is just about the same, I have 0 experience with it though, I won't claim otherwise. I got frustrated enough with the Capistrano, Fabric, Puppi lot that I wrote a pure bash deployment tool with a fitting name htpps://github.com/gerhard/deliver. Ansible on the other hand is something else though. There is some learning curve, agreed, but it's not as bad as awk or sed. And seriously, if you know your bash, you will know both awk and sed. I consider my shell scripting to be above average, and I've attempted a Docker orchestration pure bash tool https://github.com/cambridge-healthcare/dockerize https://github.com/cambridge-healthcare/dockerize, but Ansible just makes the same job easier. It's not everyone's cup of tea, but before you dismiss it, give it a real chance. I should know because I have initially dismissed it thinking that it's too complex, yet another Chef circus, yada yada, but trust me - it's worth it ; )
- noonespecial 13y agoI've been doing almost exactly this for years, always ashamed that I didn't take the time to learn a "real" tool like puppet. Solidarity.
- gerhardlazu 13y agoAnd one more thing: the point of a system management tool is to have no dependencies. If you need to install or use anything other than the provisioner to setup a vanilla distro, you've been hustled. Bash is guaranteed to work everywhere and that's what most love about it - myself including. But Ansible works on my RaspberryPies, on my FreeBSD storage servers and on my production Debians, Ubuntus and ArchLinuxes with no crutches or aids. I had bash scripts before, was really close to open sourcing them, but then I realised there is a much better way ; )
- mattjaynes 13y agoHave to say I'm a bit puzzled by the claim of Ansible being "blow your brains out" difficult. In many cases, Ansible is even easier than shell scripts. I wrote a post about this a few months ago: https://devopsu.com/blog/ansible-vs-shell-scripts/ https://devopsu.com/blog/ansible-vs-shell-scripts/ I completely understand where the sentiment is coming from though. I wrote a comparison book on Puppet, Chef, Salt, and Ansible a few months ago and am currently finishing the 2nd edition ( https://devopsu.com/books/taste-test-puppet-chef-salt-stack-ansible.html https://devopsu.com/books/taste-test-puppet-chef-salt-stack-... ). Even for an experienced sysadmin, using Puppet and Chef to do even a trivial project (replacing a ~10 line shell script) took a painful couple of days. Why? They're overly complex and have confusing broken documentation (that mostly haven't been corrected even 6 months after I gave them a full breakdown on the issues). Salt was pretty smooth, but Ansible was downright easy. Using a shell script to set up a server generally indicates that it will then be managed manually afterwards (sadness and despair!). A huge advantage of using a configuration management (CM) tool is that they're "idempotent". Idempotency basically means that you can run the directives over and over again safely. An idempotent command will verify that the system is how you defined it and will only make changes to bring the system back into alignment with what you defined. That means you can define your system in the language of the CM tool and use it not only for initial system setup, but also for monitoring, updating, and correcting a server's configuration over the life of the server. A CM tool can ultimately act like a self-healing test suite for your systems - neat! Your systems are the "app" that your app runs on. They're the foundation. Not using a CM tool is like not having any tests for your app. Sure, it might seem faster at first, but you'll pay for it in chaos, slowness, bugs, and misery later. Shell scripts are a great first step, but if you're serious about your systems, you need to be using a CM tool. Modern ones like Ansible are simple, easy, and have great docs so there's little excuse left for not using one now.
- atweiden 13y agoI use a Bash script to set up Arch with Btrfs on LUKS and optionally enable SSH [1]. Assuming you have a base machine running, config management tools are great and quickly start to make sense. If you're starting from scratch with a blank physical machine, I think the answer is still shell scripts and then add on CM afterwards. 1: https://github.com/atweiden/pacstrapit https://github.com/atweiden/pacstrapit
- stormbrew 13y agoHaving begun using Chef over the last year I've found that my problem has not been so much with chef (a major issue with how it deals with when/whether to restart services aside) itself but with the community cookbooks. This is where the overcomplexity tends to come in in my experience. I'm not sure you can genuinely have a good provisioning script that treats all of these things as variables in a matrix: - OS (sometimes including windows as well as linuxes and bsds) - OS version - Using system packages or building from source - Package/source version - Every single config variable the program/service can have Every time we use a community cookbook, which includes several hundred plus lines of code to deal with platforms we will never ever deploy to, it ends in tears eventually. Having a 20 line recipe that works on the Ubuntu LTS we've standardized on is much closer to what FSS can achieve, but even then it does often make simple tasks considerably more difficult.
- eldavido 13y agoI spent the last year and a half cross-training from dev to ops. One of the most important lessons I learned: use operating system packages for production; don't compile from source. Compiling from source: - Wait minutes for each instance to come up, as each time requires a fresh compile - Have to download from ruby-lang.org -> you have a dependency on this site being up, bad idea when you hit a load spike, need to scale, and ruby-lang.org is having a bad day - Loss of dependency management. Now you can't tell whether your environment has the right packages installed, nor can other things express a dependency on your code being installed - Very difficult to remove packages for upgrades/maintenance/security fixes Packages: - Host your own repo, removing the need for external dependencies (can do this with source as well, but it's much better when packages know to discover themselves from a repository) - Much cleaner rollback -- almost impossible to trash a system with apt-get/yum, they'll always leave you in a good state in case the package fails to install, or other mayhem ensues
- jamiesonbecker 13y agoAgreed. Tracking all those versions for security vulns is a major pain as well. Unless you absolutely positively must have X new feature that's only in upstream, DON'T compile it yourself unless you're prepared to pay that price.
- blueblob 13y agowhy do I have to use ruby to install shell scripts? Why not wget or curl or GET?
- greatsuccess 13y agoWow that fucking sucks. Installing rails to run a shell script, no thanks I think I would rather blow my brains out.
- shiven 13y agoI ♥ Ansible ... 'nuff said.
- artellectual 13y agoThis is simply brilliant! I've dreamt about such tools. And now you have reached into my dream and turned it into a reality. We might as well be soul mates you and I.
- sp1982 13y agoHaving managed 10's of thousands of servers and multiple hundreds of configurations at a big web company for many years, I completely see where OP is coming from. I think a huge problem with puppet/chef is that they try to do it all, ie deployment and OS state management and make it work with vendor specific package management systems. Unfortunately rpm/debian based package mgmt systems are not well suited for complicated deployment strategies. Most companies I worked at come to a solution similar to: * Install everything you need in /package_versionstr (except say glibc) * Point the current version with a symlink This enables you to do a simple atomic rollback/rollforward and is much easier to reason compared to complicated pkg state. For simple OS config management (say usermgmt, sysctls), we used a system similar to the ideas expressed by OP. 1. Keep it simple. puppet/chef have horrendous DSL's that make it really complicated to reason about. I shouldn't have to debug a backtrace 15 levels deep to understand why an useradd didn't work. 2. Server side logic. Don't try to do "intelligent" stuff based on client side state, it will be almost impossible to get it right. All data needed for state needs to derived by group membership. This helps in * validating all changes upfront * diffs for state changes across a group of nodes. 3. No orchestration. Except any changes to be applied at any time. This acts as an enforcing function to make your scripts idempotent.
- billpg 13y agoPlease do not wget files over HTTP (no S) and just run them, as the example script does. Have the script run the downloaded file through sha256sum and check the output matches what you expect. I know that was only an example script, but we don't want to be encouraging bad practices here.
- upbeatlinux 13y agoA wrapper for shell scripts with a dependency on Ruby but lacking determinism, templating, etc? In other words a remote command dispatch and execution tool.
- fideloper 13y agoAs someone in the PHP community, I've made something similar (Vaprobash - "Vagrant Provision Bash Scripts") with the goal of helping people learn more about what goes on when setting up servers for use (installing web servers, databases, configuration, etc). https://github.com/fideloper/Vaprobash https://github.com/fideloper/Vaprobash
- fndrplayer13 13y agoIf you're going to go this direction wouldn't it make more sense to just upstart another script that runs your scripts? How much time and code does this actually save?
- deleted 13y ago[deleted]
- greatdox 13y agoI'm all for anything that makes managing servers easier, but why use swear words in the name? It is unprofessional.