8 ms·
Show HN: Boulette - Protect you from yourself (even as root).
- p4cmanus3r 2y agoSeems like just aliasing `shutdown` to something with a confirmation on servers and not your local would suffice.
- Rygian 2y agoWhy should my user have the right to shut down a production box? Having a tool to warn me that I'm about to do something I shouldn't be allowed to do is okay. Having proper access control would be better. (Fun: 'boulette' is French for 'dumb mistake')
- roblabla 2y agoI mean, you're probably going to need to give the reboot rights to someone. For instance, the people in charge of applying kernel updates. And those people will, at this point, be prone to making a boulette.
- f1shy 2y agoWe used to have special user for such tasks, with long names and password that should make clear what you want to do. For example shutdown/youwillshutdownproductionXXXXXXX (with Xs being a “real” password) because the random part of the password and sometimes the “fix” part was changed, there was no way that muscle memory would take over.
- ramon156 2y agoWhen you work at a bank and have multiple production boxes, this is pretty typical
- deathanatos 2y ago> Having proper access control would be better. Some of us live on the "granted" side of that access control. I'm an SRE; I have access to a fair number of things. I worry about making dumb mistakes like that in the article due to sleep deprivation. The warning here might very well be a warning that you're about to do something that you are allowed to do. In the specific example … sometimes rebooting a production machine is what needs doing. Ideally, there's a second VM, ideally, the services on the machine would gracefully drain, ideally, it wouldn't matter. Ideally. But the reality is that often they don't, whether because of developer laziness, incompetence, some ugly confluence of bullshit bugs that conspire to prevent sanity, etc., more often than not I want to be careful even when the process/theory would like to say "this is a fully automated & completely safe operation" — and half-asleep is not careful, and accidental allegedly safe commands while half asleep is where Murphy likes to rear his head. I've definitely used techniques like this. Red prompts that scream "You're SSH'd into PROD dummy" also come to mind. I keep my kubectl context fixed to "local laptop" so that I am forced to always type out --context the_production_cluster when I want that. Temet nosce.
- GuB-42 2y agoThe idea is to protect you from yourself. You already have root access to the production server, because you are the admin and you need it to do your job. Fun followup: the image is from the French movie "Le dîner de cons", where the character depicted says "Oh la boulette", after making a gaffe.
- tln 2y agoRunning "shutdown" on remote hosts isn't something I have ever needed to do routinely. Is this for ephemeral dev boxes? Does shutdown suspend billing on AWS/cloud type hosts?
- frizlab 2y agoThis for the case where you think you type it locally but are actually typing it remotely. But I’d personally never type this locally so this use case is not convincing at all for me.
- voidUpdate 2y agoI have a laptop that just runs a bare ubuntu terminal, no DE, and I've accidentally shut down a home server before when I tried to shut down the laptop. Quite annoying, since I was away from home at the time
- hk__2 2y agoWhy do you shut your laptop down instead of just closing the lid?
- yonatan8070 2y agoMany laptops have terrible power usage during sleep, so if you just close the laptop and throw it in your bag, you'll open it tomorow with 30% less battery
- voidUpdate 2y agoBecause the battery is absolutely knackered
- jean_dupont 2y agoThis! Another use case: I use it on nixos-rebuild to update the system, when I don't want to rebuild from tiny hosts but rather from a powerfull machine and then send binaries,because it would take to much time to compile otherwise.
- bandie91 2y agoi don't need anything reminding me that i'm in a remote shell session: i know because it lags. i also know which one from the delay's length.
- hk__2 2y ago> i don't need anything reminding me that i'm in a remote shell session: i know because it lags. It doesn’t if you use a tool like Mosh [1] (I’m not affiliated) [1]: https://mosh.org/ https://mosh.org/
- yonatan8070 2y agoReplacing SSH with a custom protocol sounds quite scary Is it as secure as regular SSH when configured properly?
- techjamie 2y agohttps://news.ycombinator.com/item?id=10220080 https://news.ycombinator.com/item?id=10220080 Here's a conversation on HN about it. Mosh uses an initial SSH connection to establish a session, and I located elsewhere that the communication thereafter is handled via AES-128 encrypted UDP traffic. The server process itself seems to only live the life of the session, and doesn't require escalated permissions. I can't imagine enterprise or government adding it to their stacks, but for connecting to personal stuff doesn't really seem like a big risk.
- matthews2 2y agoYou still authenticate and kick off the Mosh session via SSH, so it shouldn't be any worse than plain SSH. > Mosh doesn't listen on network ports or authenticate users. The mosh client logs in to the server via SSH, and users present the same credentials (e.g., password, public key) as before. Then Mosh runs the mosh-server remotely and connects to it over UDP.
- mdaniel 2y ago> Mosh doesn't listen on network ports ... > ... Then Mosh runs the mosh-server remotely and connects to it over UDP. Are UDP sockets immune from port scanning? Regardless of the answer to that, the sentence should be rewritten because it sounds like nonsense as is
- itsanaccount 2y agoI would like to register a prediction of futility here, just a little more detailed than the trope of "when you try to idiot proof something the universe always makes a better idiot." The number of software systems I've seen designed to allow the clueless to bumble their way through operations is much higher than the number of companies I've seen that attempts to train better operators. Maybe its the experience of working on tractors, where the PTO has fun side effects like "degloving" where, sure theres a guard but more importantly theres attention not to put your hand, hair or clothing near the spinning torque monster of doom. Theres no option to make that purely safe economically. Same goes for machine tools and other heavy equipment. The people who work around such things have been to my experience more capable at accomplishing difficult, diverse tasks. Where was the last company that had anything equivalent to the lowly "forklift certified" for prod? Its a very rare shop I've seen invest in any sort of across the board training for command line skills, outages, pre-requirements for delicate operations. We don't invest in people being better, being more capable. I think because we have internalized an owner/management point of view that workers are fungible and training is a waste, while software system guards are investments. As a worker, I don't agree with that. I don't agree with building systems to be powered by the lowest educated, lowest paid meat popsicles yet I think thats the strategy behind this. Next time you go to build a system like this, consider who's logged into production and can they be trained to be more capable, more attentive operators. In the long run, I think it will end up with a better industry overall.
- frizlab 2y agoIt’s the “Do you want to share XXX” popup all over again. Once you get used to see it, it will be a part of the shutdown command (to take the example in the readme).
- jean_dupont 2y agoHahaha, yes I thought of it while writting it! This is a crazy annoying terminal popup! But it can maybe be useful if used correctly and with parsimony
- InsideOutSanta 2y ago
- ale42 2y agoI'm using molly-guard (htps://salsa.debian.org/debian/molly-guard, mentioned on Boulette's github btw as inspiration) since years for this after I've remotely shut down our file server thinking I was on another tmux pane. It saved me once or twice since. Btw, molly-guard doesn't require setting up aliases.
- remram 2y agoFor anyone wondering how this works, molly-guard uses package diversions to actually rename the shutdown/reboot/... binaries and replace them with molly-guard: https://git.launchpad.net/ubuntu/+source/molly-guard/tree/debian/molly-guard.preinst https://git.launchpad.net/ubuntu/+source/molly-guard/tree/de... This is a cool ability of dpkg that is not well known. You can use it without making a package as well.
- ale42 2y agoDidn't know what specific technique it used to do that, thanks for mentioning this! Btw for those who wonder where the name comes from: https://news.ycombinator.com/item?id=26633320 https://news.ycombinator.com/item?id=26633320
- deleted 2y ago[deleted]
- jean_dupont 2y agoYes! Molly guard is genius! Unfortunately I couldn't get it to work on nixos which is not FHS-complient!
- debarshri 2y agoCombination of bash profiles and alias can be used to achieve this without installing any external thing. You can prevent commands like rm, chmod, cat etc even for root user. You can also prevent root users from accessing directories where the vault, database etc. Data is written. Ps. At Adaptive (http://adaptive.live http://adaptive.live), we have kind of productized something like this.
- samsk 2y agoI've a shell alias for this, requiring valid hostname for reboot, halt etc...
- ncoco 2y agoLoving the name "boulette" here, which means "blinder" or "slip-up".
- rafram 2y agoWhy does it take the command as a single string argument? Seems like it would make more sense to take it as an argument list, like other "wrapper" commands like sudo, nohup, etc., do.
- hk__2 2y agoProbably to avoid mistakes such as `boulette some command || some other command`, which would not happen with `boulette "some command || some other command"`.
- jean_dupont 2y agoIt just didn't cross my mind! You've got a point here! I'll rethink the argument parsing.
- jiscariot 2y agoFor me, what is more common is the likelihood of doing something in the wrong environment (e.g. lab, dev, stage, prod). To help make things a bit more clear, our images now override the PS1 with a `(environment)` at the beginning, which is a different color, lab=green, dev=purple, prod=red. If it saves me once, it was worth it.
- oslem 2y agoI’ve tried using iTerm2’s automatic profile switching feature to adjust the theme depending on the connection, but I’ve never been able to get it to work reliably.
- jean_dupont 2y agoI think such a thing can be achieved easily with the starship prompt!
- binwiederhier 2y agoI've used this successfully in the past for Terminator: https://github.com/GratefulTony/TerminatorHostWatch https://github.com/GratefulTony/TerminatorHostWatch
- reddit_clone 2y agoI too use iTerm's profiles to change background colors, fonts etc. to indicate where I am. Reduces number of fat fingering disasters.
- franktankbank 2y agoDouble deep environment protection. True prod servers can't be deployed to except from special terminal which only appears visible to the ProdGlasses(TM). If you are wearing ProdGlass(TM) you receive mild brain wave stimulation which is collected for "product control purposes"(TM) future product development(patent pending).
- drcongo 2y agoI have iTerm set up to change the background colour of a shell based on whether it's local, docker, staging or production, and on top of that prompt colours change depending on how privileged the user is. Without all this, I'm sure I'd be making terrible mistakes on a regular basis.
- arschfick 2y ago[dead]
- bityard 2y agoThis says it's inspired by molly-guard (which I love and install everywhere and has saved my bacon countless times) but I don't see what's different about it? Molly-guard is a single `apt install` away with no further config needed. Also, the problem with a Y/N question is that when you are bored and/or in a hurry, you only skim the question and muscle memory takes over, and you hit Y and then you realise a few seconds later that you rebooted the wrong machine. This is why molly-guard makes you enter the hostname of the host you want to shut down.
- deleted 2y ago[deleted]
- jean_dupont 2y agoYou can set the challenge type to hostname as well and some others. The only pb with Molly-guard is that I can't have it working properly on my system.
- deleted 2y ago[deleted]
- dmd 2y agoWhat I really want is a way to get a read only root shell. I do a lot of work on a "traditional" multi-user unix host, where hundreds of scientists share a powerful computer. I often need to become root to look at files. I want to be able to do that without the ability to screw anything up.
- 6r17 2y agoMicroOS has something in that spirit I think ? Maybe I'm wrong tough, don't even know if MicroOS is usable in host
- makemyworkforme 2y agoCan you create a group and set a sticky bit for ACLs for your root for the filesystem with read permissions for that group? That way all files created under the filesystem root will have read permissions for that user, but you cannot modify anything.
- dmd 2y agoNot really, there are tons of different filesystems mounted from all over the place. It's a mess.
- spease 2y agoCan you create a second read-only mount of root? Maybe this would help: https://serverfault.com/questions/136515/read-only-bind-mount https://serverfault.com/questions/136515/read-only-bind-moun...
- roydivision 2y agoI've worked as a sysadmin for most of my professional career, around 25 years. Sure I've screwed up, but I've learned each time. Covering everything in bubble wrap to shield you from consequences of mistakes is counter-productive. All that happens is that you learn that there are safety nets and not to think twice about what you're doing. Also these tools are non-standard, so you'll expect them everywhere. Don't do this.
- ape4 2y agoJust remove `/usr/sbin/shutdown` and `/usr/sbin/poweroff`
- tptacek 2y agoI like the idea, but my immediate comment here, in a spirit of thinking about what would get a tool like this wider adoption, is that "boulette" is a clunky thing to type on a terminal. You can be more assertive about the importance of your tool, and give it a shorter, punchier (command-line) name; it might see more use if you do.
- wiether 2y agoYou're not really supposed to type it everytime. Just declare an alias prefixed with boulette replacing the actual risky commands. If you're in a mental state where you can shutdown a prod server while thinking you're on a dev one, you certainly won't think to type "wtf command", no matter "wtf" actually is.
- benhurmarcel 2y agoFor what it’s worth, the name is funny and easy to remember to French-speaking people.
- hn92726819 2y agoIsn't this a little over engineered? You can accomplish the same thing with a 5-line bash script. put protect.sh somewhere in your path: #!/bin/bash if [[ $SSH_TTY ]]; then read -p 'You are in SSH. Are you sure (enter hostname for yes)? ' [[ $REPLY == $(hostname) ]] || exit 2 fi exec "$@" Then in your bashrc or zshrc: alias shutdown='protect.sh shutdown' alias reboot='protect.sh' alias sudo='sudo ' # Don't allow sudo to bypass the protection. Can do the same with doas
- jean_dupont 2y agoUnfortunately we have to detect nested sessions too! When doing sudo -i, the $SSH env var can't be retrieve and we have to browse through the process tree and look for sshd. And we need fancy printing with colors too!
- theamk 2y agoIt's mentioned in the bottom of the page, but I'd like to highlight that "molly-guard" [0] provides a very similar functionality for at least 10 years. Since it's so old, it is present in all the systems - "sudo apt install molly-guard" on your server, and your shutdown, reboot, etc.. are all protected, no need for 3rd party tools. [0] https://www.unix.com/man-page/debian/8/MOLLY-GUARD/ https://www.unix.com/man-page/debian/8/MOLLY-GUARD/
- qwertox 2y agoI use that in combination with base16-shell [0] to set different color profiles as a protection. [0] https://github.com/chriskempson/base16-shell https://github.com/chriskempson/base16-shell
- Fnoord 2y agoZFS snapshots with bootloader support (other CoW filesystem should also work). For the specific purpose of shutdown, there has been a solution for decades: molly-guard.
- wiether 2y agoI guess the name and Pignon's picture only resonate with a French audience, thus why so many serious comments here. Moi ça m'a fait sourire !
- hyperman1 2y agoI've basically stopped these problems by coloring the servers. My production shells have a red PS1 background. Datagrip gives a red color to my prod databases. HTML admin consoles got some red css if configurable.
- eternityforest 2y agoWhat if you alias ssh on the local machine to open a tmux with the top pane being a red warning, and then the whole thing could automatically close when ssh ends so the experience is just like normal ssh? I've never actually used tmux, but maybe I will try it out!