19 ms·
I usually run 'w' first when troubleshooting unknown machines
- subway 9y agow; df; dmesg; top usually there's a loud sob somewhere in between.
- agumonkey 9y agoreading w df in my head made me laugh
- briandear 9y ago> If you want to impress me, set up a system at your company that will reimage a box within 48 hours of someone logging in as root and/or doing something privileged with sudo (or its local equivalent). If you can do that and make it stick, it will keep randos from leaving experiments on boxes ... It’s called Chef.
- sneak 9y agoChef doesn’t reimage anything.
- pdpi 9y agoRachel’s starting point here is a fair few steps ahead of ”it’s called chef”. For one, chef only fixes the things that you explicitly tell it to. When you have enough machines and enough people poking at them, you’re more or less guaranteed that someone somewhere will have gotten a machine or three in a state chef can’t recover from — hence her suggestion of reimaging.
- isostatic 9y agoAlways pxeboot, then "re-imaging" is a matter of "turn it off and on again". If you have a hardware failure how would you rebuild? Of course this all takes time to implement and manage -- it's a tradeoff.
- pdpi 9y agoHow do you manage the network boot server? How will pxeboot interact with hosting your rack switch images?
- isostatic 9y agoHigh availability IP for your dhcp for "next-server", so you can boot one server from another one. Mikrotik routers and switches can boot from dhcp (or bootp?), but yes a typical switch can't. Not a network person, but I assume you can give fine grained enough control that you can't do a "copy running-config startup-config" on a cisco switch, so have a startup config that boots to a known basic state then tftps it's config from your HA dhcp server.
- Fnoord 9y ago> If you want to impress me, set up a system at your company that will reimage a box within 48 hours of someone logging in as root and/or doing something privileged with sudo (or its local equivalent). Why revert it in the first place? Why not deny the changes? Doesn't this indicate that the wrong people have root access?
- softawre 9y agosometimes the best way to test a fix to something, before you make it permanent in the deployment-as-code, is to test it on the box. This just makes you sure actually go back and change the deploy code, else it breaks in ~2 days.
- Fnoord 9y agoYou could just have one or more specific development/test box(ex) to try changes on, and then implement those on production.
- banned1 9y agoIf you want to impress me, go serverless.
- edf13 9y agoAnd next, I usually run "top"... quick snapshop of whats going on.
- tluyben2 9y agoYep w for me as well. History after.
- checkyoursudo 9y agoI run my own web server and email server for my law firm. I'm the only one with credentials to log in (ash), and probably the only one who would even know how to do it, and possibly the only one who knows we have servers. And I still run w first thing almost every time out of habit. What am I looking for? A session I accidentally left open somewhere else? Unauthorized access? A friend? Dunno...
- RobAley 9y agoYou're leaving your firm incredible vulnerable to the "checkyoursudo gets hit by a bus" scenario aren't you? At least let the rest of your firm know you have servers and give some credentials (for another account with sudo permissions) to e.g. a managing partner or similar (with instructions to never use them unless you die).
- HenryBemis 9y agoMy though exactly! For the company of checkyoursudo, he's what is called a Key Person Risk. But in my examples, I never use the "hit-by-a-bus". Sysadmins tend to frown upon that comment. I use the "win-the-lottery-go-to-Fiji-and-never-look-back". It always makes them smile :)
- Symbiote 9y ago> up 23 days > Right there, I can see that okay, the box hasn't been rebooted recently Is the implication of "recently" that it should have been rebooted every couple of weeks or something? My desktop is at 54 days (since I moved desk, I think) and picking a random Hadoop node: > 10:26:47 up 303 days, 11:56, 1 user, load average: 44.12, 50.56, 47.48 It's private, so kernel updates aren't a security issue. (Keeeping this on-topic, "history" tells me the most frequent thing I do on this node is "sudo iftop"; we've been doubting the accuracy of our monitoring system's network utilization graphs.)
- RobAley 9y agoI think her thinking was "the problem started recently (~days), did the box get rebooted recently (~days) which might indicate when the problem started", rather than it should have been rebooted recently.
- rurban 9y ago> If you want to impress me, set up a system at your company that will reimage a box within 48 hours of someone logging in as root and/or doing something privileged with sudo (or its local equivalent). If you can do that and make it stick, it will keep randos from leaving experiments on boxes which persist for months (or years...) and make things unnecessarily interesting for others down the road. Ferrari does this. They do a lot of experimentation on their most expensive internal test equipment, and every now and then the whole box is re-imaged automatically, even if it's completely locked down from outside. It's their internal staff who is corrupting/improving the system. Only if it's a really good and well-tested improvement they will make it stick.
- softawre 9y agoAnybody running Chaos Monkey (from Netflix) does this, at lesat for their stateless services.
- olefoo 9y agoI do that too. I probably wouldn't if I hadn't learned on a shell server where that was how you found out who else was up.
- SubiculumCode 9y agoi did not know w was a command. I've used the 'last' command https://www.cyberciti.biz/faq/linux-unix-last-command-examples/ https://www.cyberciti.biz/faq/linux-unix-last-command-exampl...
- EngineerBetter 9y agoGenuine question: how many HN readers log on to boxes with user accounts that belong to humans, where some state may have been mutated? My experience of the last five years is so heavily weighted to (effectively) immutable infrastructure that checking to see who had been on a box hadn't event crossed my mind.
- sneak 9y agoYou are at the right hand side of the curve. Professionals do it that way, yes. Many smaller or less-well-managed organizations have a lot more warts.
- subway 9y agoeven on immutable hosts it isn't surprising to see somebody has ssh'd in for debugging purposes. they might be attaching a debugger, taking a core dump, or doing a number of other things that could cause a drastic perf hit while not actually mutating the host's state otherwise.
- jameshart 9y agoWhy take the risk of continuing to run a tainted host, though, if you can just tear it down and spin up a new, clean, untouched one? I think there’s another level we need beyond treating servers as pets or cattle, which is treating servers as wild animals; after you’ve captured one and interacted with it, you’ve doomed it because now it has the scent of humans on it.
- subway 9y agoSure, but that does nothing to prevent the person who invoke the debugger from inadvertently triggering prod alerts, possibly encouraging more folks to ssh in to see what's going on.
- gowld 9y agoGood hiring and training practices prevent people from invoking the debugger on live prod machines
- dvh 9y agoI run df first, very often something stopped working because the disk space ran out
- avip 9y agoIf you have a tiny fleet of instances, datadog free plan would solve that for you (not affiliated, it's a good product).
- Symbiote 9y agoThat's one thing I don't need to do. If the disk gets to 80%, I get an email from the monitoring system. 90% and I get a text.
- letientai299 9y agoCould you please share how did you setup to be able to get such notification?
- vk23 9y agoHave a look at nagios. It's a popular open source server/application monitoring tool. You can define custom alerts and monitor basicly everything.
- FuckOffNeemo 9y agoNot OP, but we use Nagios XI for such alerting, it provides SMS and email monitoring for infrastructure (disk space as an example) and service monitoring. [1] Their open source NCPA (Nagios Cross Platform agent) plugin for Windows and Linux hosts is pretty great, though we were stung when we found out it didn't natively support SPARC as of yet unless you compiled your own client from the source (I did). [2] Plenty of other monitoring services equivalent to Nagios offer equivalent services. Nagios was just the flavour I suggested to my current employer because of my familiarity with the product over my last few employments. If you don't want to fork out the licensing for the supported version (XI), their open source version is free and relatively easy to deploy using Ansible if you don't mind writing in PHP. [3] Edit: words, grammar. [1] https://www.nagios.com/products/nagios-xi/ https://www.nagios.com/products/nagios-xi/ [2] https://github.com/NagiosEnterprises/ncpa https://github.com/NagiosEnterprises/ncpa [3] https://hobo.house/2016/06/24/automate-nagios-deployment-with-ansible/ https://hobo.house/2016/06/24/automate-nagios-deployment-wit...
- chris_wot 9y agoI never even knew about that command...
- Freak_NL 9y agoMe neither. I just asked two colleagues (one on MacOS, one on Ubuntu) and got the same “huh, I never knew?” response I had. Sure there are tons of unknown commands on any OS, but a one-letter command you've never heard of somehow amplifies the amazement.
- jlgaddis 9y agoIn a (normal/typical) shell, hit "<Tab><Tab>y" and be amazed.
- sgaduuw 9y agoMy standard set of commands in context of troubleshooting, in no particular order is * w * last * dmesg
- xstartup 9y agoI run history first, followed by df then finally I run htop.
- known 9y agosudo dmesg -w -H --level=err
- deleted 9y ago[deleted]
- ktpsns 9y agoInteresting that nobody mentioned htop so far (https://en.wikipedia.org/wiki/Htop https://en.wikipedia.org/wiki/Htop). It is my favourite command to get a quick glance on the computational facilities of a computer (memory, cores) and what it is doing (load, fancy ps/top). htop is not installed everywhere, but it is easy to make a static build and scp it to the questioned host. Another very handy command is sudo netstat -atpn which shows you the processes and owners of open TCP/UDP ports. The argument combination is as weird as "ps aux" that I just memorized it by heart.
- BuildTheRobots 9y agoI use both htop and atop, usually htop first as it's more graphical, but atop gives nice IO, wait and NET stats too which is extremely useful.
- poooogles 9y agonetstat is deprecated and has been for a while, you should be using ss from iproute2. Most of the commands should map directly so it's pretty drop in.
- mmillin 9y agoOnly on certain Linux systems is it deprecated. On the BSDs netstat is still the preferred tool.
- 9y ago
- thedonkeycometh 9y agoI like the 48 hour wipe after root access
- sente 9y agoatop creates world readable log files. Does anyone else think this is a security vulnerability?
- cesarb 9y agoThey have nearly the same information a normal user could get by running atop themselves, which reads the world-readable virtual files found in /proc. If you are using something like selinux to restrict access to /proc files, the same system could be used to restrict access to the /var/log files.
- yonilevy 9y agoI use the same approach when troubleshooting/getting into hairy code - git blame ("annotate" in IntelliJ) first, gives a lot of useful context!
- giis 9y agoMy first reaction after logging into unknown *nix machines. uptime uname -a w who df -h free -m cat /proc/cpuinfo mount top ps aux then check appropriate logs or dmesg etc
- Fnoord 9y agoYou say *nix machines, but some of your commands are Linux specific.
- freehunter 9y agoOnly two, 'free' and 'cat /proc/cpuinfo'. Everything else can work on both.
- StapleHorse 9y agoI do 'ls' as a reflex I don't know why. For network monitoring I like: nethogs nload -u K
- seanwilson 9y agoI stick to serverless setups and platforms like Heroku wherever I can. The whole idea of state being stored on a web server or having to SSH in to them to do any form of admin makes my skin crawl now.
- betaby 9y agoIt's not because 'there is no state' - you just outsourced state(s) to the third party.
- draw_down 9y agoThanks, professor
- seanwilson 9y agoI think it's more the case that typical e.g. VPS setups introduce state where you shouldn't have state at all. Web servers for example should typically have minimal state for scalability and robustness but for most VPS setups this is not the case. If the VPS got wiped it could take days of work to get up and running again, and if you wanted to scale horizontally you'll have some reengineering to do. You could try and replicate something like Heroku yourself to minimise state on your web servers but it'll take you a lot of time and won't be as robust.
- vermaden 9y agoOn FreeBSD systems there are two great commands that are not available on other systems. These are 'gstat' and 'top -m io' commands, both I/O related. Often in top/htop there is not much happening but the server is very slow, a lot of the times its because of I/O operations. The 'gstat' command will tell You (besides other useful statistics) how much the storage devices are loaded: # gstat dT: 1.001s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 1 9679 9614 4807 0.1 63 671 0.2 82.3| ada0 0 0 0 0 0.0 0 0 0.0 0.0| ada0p1 0 65 0 0 0.0 63 671 0.2 1.4| ada0p2 0 0 0 0 0.0 0 0 0.0 0.0| ada0p3 0 0 0 0 0.0 0 0 0.0 0.0| gpt/boot 0 65 0 0 0.0 63 671 0.2 1.5| gpt/sys 0 0 0 0 0.0 0 0 0.0 0.0| gpt/local 0 0 0 0 0.0 0 0 0.0 0.0| zvol/local/SWAP The 'top -m io' show which processes does how much I/O: # top -m io -o total 10 last pid: 51154; load averages: 0.31, 0.31, 0.28 up 3+18:01:00 14:58:15 54 processes: 1 running, 53 sleeping CPU: 2.4% user, 0.0% nice, 15.3% system, 5.3% interrupt, 77.1% idle Mem: 345M Active, 1236M Inact, 153M Laundry, 2158M Wired, 3903M Free ARC: 834M Total, 46M MFU, 295M MRU, 160K Anon, 5006K Header, 488M Other 67M Compressed, 274M Uncompressed, 4.08:1 Ratio Swap: 4096M Total, 4096M Free PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND 51021 vermaden 6005 16 6005 0 0 6005 99.92% dd 51154 vermaden 8 9 5 0 0 5 0.08% top 51036 vermaden 6 10 0 0 0 0 0.00% xterm 50907 vermaden 0 0 0 0 0 0 0.00% zsh 50905 vermaden 0 0 0 0 0 0 0.00% xterm 50815 vermaden 0 0 0 0 0 0 0.00% zsh 50813 vermaden 0 0 0 0 0 0 0.00% xterm 50755 vermaden 0 0 0 0 0 0 0.00% tail 41780 vermaden 0 0 0 0 0 0 0.00% leafpad 41255 vermaden 29 11 0 0 0 0 0.00% firefox Another command that is very useful on FreeBSD is 'vmstat -i' which show how much interrupts are happening: # vmstat -i interrupt total rate irq1: atkbd0 75135 0 irq9: acpi0 2575929 8 irq12: psm0 135060 0 irq16: ehci0 1532065 5 irq23: ehci1 3265677 10 cpu0:timer 102772345 317 cpu1:timer 94199942 291 irq264: vgapci0 370466 1 irq265: em0 7017 0 irq266: hdac0 1904427 6 irq268: iwn0 148690342 459 irq270: sdhci_pci0 147 0 irq272: ahci0 20875039 64 Total 376403591 1161 I always suffer when I have to debug Linux systems because of lack of these commands. Regards, vermaden
- jillesvangurp 9y agoI can recommend setting up audit beat & kibana or similar. Auditbeat is a recent addition to the Elastic beats agents and it sends audit logs for a lot of things, including ssh logins, system calls, and you can monitor changes to files/directories as well. So, you can flag boxes where people are poking around in /etc, and see which boxes are being accessed via ssh by which user. We have this and a few other elastic beats on most of our vms in amazon baked into the amis we use. So anything deployed by us starts sending lots of data to our logging cluster for metrics, auditing, internal application events, stacktraces from our docker infrastructure, syslogs, etc.
- coinerone 9y ago"w" & "last" are first for me. Then "top" and "uname"
- sporkenfang 9y agoThere are two things I don't like here: - the author seems to call out specific individuals, by name, to their team leads, based on where they were ssh'd in, instead of bringing up the issue with that person first and asking for the logic of why they were on the box doing the thing (sounds like a lot of assumptions combined with finger pointing) - it sounds like there's no well-configured monitoring or observability at the DC/rack/machine level involved, at all, which is surprising in a modern enterprise setup
- kbutler 9y agoAfter running w, “The next thing I'd do is to go get in touch with that person. It would be foolish to continue when the answer might be a few short chat messages away.”
- kbenson 9y ago> the author seems to call out specific individuals, by name, to their team leads, based on where they were ssh'd in, instead of bringing up the issue with that person first and asking for the logic of why they were on the box doing the thing (sounds like a lot of assumptions combined with finger pointing) I'm not sure why you would assume that. She specifically says "The next thing I'd do is to go get in touch with that person." Perhaps you're keying off "I've been able to track down some well-meaning but ultimately flawed attempts at fixing things that then blew up and became something much bigger. The folks who I pinged about it were amazed that I somehow had managed to "guess" that a specific member of their team had been poking at a specific box"? But keep in mind, that's specifically events that became a large issue. Is it not appropriate to notify management as to the cause of the issue? Either it's a first time mistake or not something the person may necessarily have known to look out for, in which case management should be lenient, or it's the latest in a string of events and management should possible take some other action. If nothing else, it allows management for that other team to say "hey, we don't need to be messing with this aspect of the server. Either contact the team whose responsibility it is and get them to do the work, or get them to sign off on it first."
- rachelbythebay 9y agoOn call person was on a plane. They did something using intermittent connectivity and made a mistake. Other person on the ground is helping until the first one lands. I tell #2 about a box touched by #1 and ask them to have a look. They did and they figured it out. Outage resolved. Why did you automatically assume I ratted them out to management? At no point does the story go there. I’m really curious, since misunderstandings like this can really poison a working environment when people think you’re doing things you’re not. I want to know what sent you down the wrong path here.
- spudlyo 9y agoI too instinctively run `w` whenever I log into a machine, and that instinct helped me land my current job. One hour long component of a (now deprecated) SRE interview loop was for the candidate to SSH into a series of EC2 instances and debug issues which got progressively harder as the interview wore on. I had wasted a substantial amount of time on the first and easiest problem by really overthinking it, and not trying the simplest of debugging techniques first. By the time I got to the final, hardest problem, I had just over 5 minutes remaining. The interviewer gave me a pass to skip it, but I was having fun, and really wanted to take a crack at it. The final problem was to try to figure out why logging into a particular machine with SSH was slow. While I sat waiting for a prompt, I had a number of thoughts. Is a reverse DNS lookup timing out? Is there a huge i/o load on the machine? Am I going to have to wire up strace to `sshd` and log in again? When I finally get to a shell prompt, I instinctively run `w` and it just hangs. I hit ^C, `strace` it, and discover that it's blocking on: fcntl64(5, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=0, len=0}) ... I look up a bit, and discover that file descriptor 5 is /var/run/utmp. So `w` is trying to get an advisory lock on utmp and failing. Then it hits me, `sshd` is likely also trying to acquire a lock on utmp, failing, and then eventually timing out. A little bit later, I've found and killed the the rogue program that held the lock, and SSH logins were fast again. Solving that last problem so quickly really boosted my spirits, and gave me the energy to push through the harder interviews that came later in the day. Thanks w!
- hitekker 9y agoGood memory and good story.
- da02 9y agoHow did you find the rogue program?
- eric_the_read 9y agoI would probably use lsof(8)
- 9y ago
- gyrgtyn 9y agoHow, after like 20 years goofing around on linux, have I never heard of `w` ?
- sah2ed 9y agoSame here, although I'm only 14 years in.
- icedchai 9y agoYou're both newbs. ;) My first Linux box ran kernel 0.99.10.
- spudlyo 9y agoDo you know about `comm`? Given two sorted files it will show you in three columns. * unique lines to file1 * unique lines to file2 * lines common to both files You can pass it various options to suppress any of the three columns. I once had an interview at Facebook where one of the problems was easily solved by `comm`. I found it funny that the interviewer, or anyone who reviewed the interview question, had never heard of it. I was a good sport about it though, I ended up writing a janky Perl script that roughly implemented `comm` to solve the problem, which (modulo Perl) was what they wanted me to do.
- gyrgtyn 9y ago`comm` at least sounds vaguely familiar. Thanks for reminding me; looks useful.
- theoh 9y agoIn that vein, I was asked about implementing n-way merge of logfiles in an interview once. The key insight/recollection they wanted was: use a heap to implement a priority queue. Not sure it was a great question as jumping to employ a data structure like that might suggest premature optimization.
- rthille 9y ago
- banned1 9y agoThis is the sort of article that I keep coming back to HN for. It's not politics. It's pure and simple technical advice with reasonable back and forth opinions from the community (other than the "if you want to impress me" guy). And nobody wants to take my 2A right, sorry, my right to use 2FA!
- lazyant 9y agoMy goto for initial troubleshooting a server is: uptime # uptime and CPU stress w # or better yet:last |head # who is/has been in netstat -tlpn # find server role df -h # out of disk space? grep kill /var/log/messages # out of memory? ps auxf # what's running htop # stressed? , look out for D (waiting on I/O typically) processes history # what has changed recently tail /var/log/application.log # anything interesting logged?
- fapjacks 9y agoI've wasted time not checking for inode availability, so I'd add a check for that to this list: df -hi
- JadeNB 9y ago> w # or better yet:last |head # who is/has been in The post argues that 'last' will not always give the desired information: > Obviously, I could also use 'last' to see who's been on the box recently, but this isn't the whole story. It's totally possible to "ssh root@box /path/to/command" and never start a login shell, which then leaves no trace in the lastlog, but then goes on to break something on the box. The syslog is how you'd find this.
- js2 9y agoRelated, "Linux Performance Analysis in 60,000 Milliseconds": http://techblog.netflix.com/2015/11/linux-performance-analysis-in-60s.html http://techblog.netflix.com/2015/11/linux-performance-analys...
- MPSimmons 9y agoI actually have a profile.d entry that runs w on login any time I open a shell on any machine in my infrastructure.
- mirekrusin 9y agoThere used to be time when the first thing I did was `sudo reboot` and it actually worked just fine many times.
- i_know_right 9y agoWhat year is this? Just kill the pod and wait for the scaling to kick in.