3 ms·

I had one of my EC2 instances (Ubuntu 14.04) lock up on Monday morning. I use it to run a Ruby on Rails app and to do a handful of ETL jobs. The website wasn'

I had one of my EC2 instances (Ubuntu 14.04) lock up on Monday morning. I use it to run a Ruby on Rails app and to do a handful of ETL jobs. The website wasn't able to load and I wasn't able to SSH into the box. So, I went into the AWS console and rebooted the instance. This has happened 3 times in the last 6 months. It looks like the CPU spikes up to almost 100% around the same time when the instance locks up. I'd love to be able to identify the root cause of the instance locking up. It seems like this article is more about the commands you'd run to assess the health of an active/working EC2 instance and not one that you're unable to SSH into. Any idea on how to identify the problem with my EC2 instance?

richardwhiuk 11y ago
Check the log files that got written before you killed the instance?
CTrox 11y ago
I can recommend atop[1] for that. It runs every ten minutes by default and writes lots of information to /var/log/atop/atop_YYYYMMDD. With that you can examine what happend before the crash, just open a file with atop -r /var/log/atop/atop_YYYYMMDD. [1] http://linux.die.net/man/1/atop http://linux.die.net/man/1/atop
- sciurus 11y ago
  A nice atop tutorial: https://lwn.net/Articles/387202/ https://lwn.net/Articles/387202/
perlgeek 11y ago
One thing we do at $work is collecting each server's static with https://collectd.org/ https://collectd.org/ and often the graphs are pretty revealing. Sometimes it's the disk activity that kills the server, sometimes it's the memory usage, or number of processes, or a disk running full. For things like databases, you can gather more specific information as well, for example number of queries and connections, failed transactions and the likes. If you have a hunch what it might be, you could try to counter-act, for example limiting memory, number of processes, or use cgroups to limit IO rate (see for examplehttp://unix.stackexchange.com/questions/48138/how-to-throttle-per-process-i-o-to-a-max-limit http://unix.stackexchange.com/questions/48138/how-to-throttl...). Also leave a ssh connection with something like htop or atop open, and look at it once it freezes up.

richardwhiuk 11y ago

Check the log files that got written before you killed the instance?

CTrox 11y ago

I can recommend atop[1] for that. It runs every ten minutes by default and writes lots of information to /var/log/atop/atop_YYYYMMDD. With that you can examine what happend before the crash, just open a file with atop -r /var/log/atop/atop_YYYYMMDD. [1] http://linux.die.net/man/1/atop http://linux.die.net/man/1/atop

sciurus 11y ago
A nice atop tutorial: https://lwn.net/Articles/387202/ https://lwn.net/Articles/387202/

perlgeek 11y ago

One thing we do at $work is collecting each server's static with https://collectd.org/ https://collectd.org/ and often the graphs are pretty revealing. Sometimes it's the disk activity that kills the server, sometimes it's the memory usage, or number of processes, or a disk running full. For things like databases, you can gather more specific information as well, for example number of queries and connections, failed transactions and the likes. If you have a hunch what it might be, you could try to counter-act, for example limiting memory, number of processes, or use cgroups to limit IO rate (see for examplehttp://unix.stackexchange.com/questions/48138/how-to-throttle-per-process-i-o-to-a-max-limit http://unix.stackexchange.com/questions/48138/how-to-throttl...). Also leave a ssh connection with something like htop or atop open, and look at it once it freezes up.