5 ms·

We ran into what we believe to be this very problem at SimpleGeo and spent a long time figuring out what the cause of the JVM locking up for tens of minutes at

by mihasya 15y ago

We ran into what we believe to be this very problem at SimpleGeo and spent a long time figuring out what the cause of the JVM locking up for tens of minutes at a time were; we believe this bug (or general class of bugs) to be responsible. Upgrading to 10.10 has caused the majority of the symptoms to disappear. My personal hunch is that there are probably startups and developers that dumped Cassandra due to "stability issues," which were in reality symptoms of this bug. There's obviously no way of confirming or denying this, so I'll reiterate that it's just a hunch.

gary4gar 15y ago
Instead of 10.10, I would suggest switching to Debian Or CentOS. Due its short support cycle, Maverick isn't suited for servers.
- imbriaco 15y ago
  The patch in the bug report seems solves the problem and allows you to stay on 10.04 LTS. It's a tiny patch and has been stable for us under production load.
- mihasya 15y ago
  You're neglecting the fact that which distro you choose has a large influence on the kernel version you get to run. With the amount of work going into stabilizing the kernel when running in a virtualized environment, chaining yourself to a slow-moving distro will cause the exact opposite of stability when running on Xen. See: this blog post.
  - moe 15y ago
    You're neglecting the fact that which distro you choose has a large influence on the kernel version you get to run That's nonsense. Most EC2 AMIs are linked to the amazon AKIs which are unrelated to whatever distro the AMI contains. Most of my debian instances run on a kernel tagged "fc8xen". The ability to chainload a self-compiled kernel on EC2 is a relatively recent invention (mid-2010) and I have yet to see a good reason to do that for linux. The article does unfortunately not mention which AKI(s) are affected, but it seems likely this bug was introduced because someone figured "newer is better" and went with the latest Ubuntu kernel instead of sticking to a proven amazon AKI.
    - mmalone 15y ago
      This bug affected several "supported" AMIs running 2.6.32 series kernels that we tested at SimpleGeo, including the official AMI released by Canonical. After we ran out of patience debugging this stuff we contacted Amazon and worked on the issue with a guy from their kernel team (who was really helpful, fwiw). He agreed that the behavior was bizarre and opened an upstream bug with Canonical [1]. You're sort of contradicting yourself here. You suggest that the distro you're running is independent of the kernel version you're running. But then you go on to claim that this bug was introduced by someone who was not running the default supported kernel. Are you saying that people should run the supported kernel, and be tied to whatever's supported upstream, or are you saying they should risk building their own? Clearly there are benefits and drawbacks either way. [1] https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/708920 https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/708...
      - moe 15y ago
        No, I'm saying you should run the kernels that Amazon provides and/or a time-tested one.
        mmalone 15y ago
        Amazon's official linux build, at this time, is a custom distribution that uses RPM and is different from the Ubuntu/Debian world in several significant ways (e.g., a new libc implementation). Migrating your WordPress blog to a new platform might be easy, but when you're managing hundreds of machines running thousands of packages that sort of change is not trivial. Running "time-tested" kernels is not really the best advise either in this case. Xen is a fairly new environment, and EC2's implementation has some quirks, so there's a pretty regular stream of bug fixes and other improvements in recent kernels that are often worth picking up. If I went to Canonical with a "time-tested" kernel bug they'd tell me to upgrade before they'd give any real support. When we talked to Amazon about switching to their AMIs they advised us that it was probably _not_ worth switching, that switching might not fix the problem, and that the AMIs we were running were widely used and supported. They made it clear that they work closely with Canonical and other providers to get high quality AMIs into their ecosystem. Long story short, the people who you admit know the most about the EC2 environment advised us that they weren't necessarily the best option, or at least not the only option, for good AMIs (sort of like how hardware manufacturers aren't the best option for an operating system). So the answers aren't really cut-and-dry here. Every time Amazon changes their dom0 there's a chance your "time-tested" kernel will stop working. And just because Amazon runs the infrastructure doesn't mean they're the best choice for a Linux distribution.
  - deleted 15y ago
    [deleted]
  - jerf 15y ago
    Recompiling a Linux kernel with a custom patch is in the core set of skills that I would expect any Linux admin to have. It's not that hard. It's not even that hard to take existing distro kernels and add a patch to them, and maintain your patch going forward as the distro continues their refinements. When you have Linux on your server, you own it. You can do whatever you want. The distro is the beginning of your power, not the end of it. If you're running a Linux server and you currently have the attitude that you are boxed in by your distro I recommend that you immediately dig into the relevant packaging system and learn enough to put your own patch on top of any existing software package, and recreate the package in the relevant manner (new RPMs, new .deb, whatever). (Yes, there's a cost/benefit tradeoff to each such patch you have to carry on, but there is still economic value merely in having the option.)
    - nupark2 15y ago
      It's 2011, and screwing around with kernel compilation is a waste of my time. It should 'just work'. The only time I recompile a kernel is when I'm working on kernel code. If UNIX distributions are doing their jobs, a sysadmin should never have to touch it.
      - jerf 15y ago
        It's 2011. Everything should just work. Alas.
        deleted 15y ago
        [deleted]
    - nowarninglabel 15y ago
      I really have to agree with the other reply that fooling around with re-compiling a kernel is definitely beyond what I would consider to be a rational expectation. You may not think it's that hard, and it may not be for someone whose sole job is to do just that, but for the vast majority of admins that are wearing many hats this is going to be a unneeded time sink.
- mmalone 15y ago
  Or even FreeBSD, which works on EC2 now. It uses a different technique for scheduling, so this issue will not affect it.
  - deleted 15y ago
    [deleted]
- dialtone 15y ago
  The importance of the short release cycle has decreased a lot for us since we moved to EC2. It's really simple to re-deploy the entire infrastructure on newer and supported images. Using puppet and cloudformation or whatever it is that you prefer you can upgrade with very little trouble.
saurik 15y ago
I would be careful with Maverick (or Lenny; Karmic was unaffected): there is a serious issue with it dropping interrupts on EC2. If you are going yo use Ubuntu on EC2, you /really/ want to be using Natty. https://bugs.launchpad.net/ubuntu/+source/linux/+bug/666211 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/666211
- dialtone 15y ago
  This is even more serious than the one in 10.04. While in 10.04 the bug would slow down the machine for a while in 10.10 with 2.6.35 kernel will simply hang the machine and requires a reboot (which can only be achieved by rebooting from the AWS console multiple times) of the instance to be "fixed". I'm in the process of upgrading all of our instances to Natty from 10.04 or younger. It's actually weird that this issue didn't get any attention whatsoever.
- mikeyk 15y ago
  Seconded. We had our 10.04 machines regularly, and become totally unresponsive. Then, we had even worse luck with Maverick--starting with the issue where it didn't recognize half the RAM on the 68GB instance type. Natty seems to be holding up the best so far.

by mihasya 15y ago

gary4gar 15y ago

Instead of 10.10, I would suggest switching to Debian Or CentOS. Due its short support cycle, Maverick isn't suited for servers.

imbriaco 15y ago
The patch in the bug report seems solves the problem and allows you to stay on 10.04 LTS. It's a tiny patch and has been stable for us under production load.
mihasya 15y ago
You're neglecting the fact that which distro you choose has a large influence on the kernel version you get to run. With the amount of work going into stabilizing the kernel when running in a virtualized environment, chaining yourself to a slow-moving distro will cause the exact opposite of stability when running on Xen. See: this blog post.
- moe 15y ago
  You're neglecting the fact that which distro you choose has a large influence on the kernel version you get to run That's nonsense. Most EC2 AMIs are linked to the amazon AKIs which are unrelated to whatever distro the AMI contains. Most of my debian instances run on a kernel tagged "fc8xen". The ability to chainload a self-compiled kernel on EC2 is a relatively recent invention (mid-2010) and I have yet to see a good reason to do that for linux. The article does unfortunately not mention which AKI(s) are affected, but it seems likely this bug was introduced because someone figured "newer is better" and went with the latest Ubuntu kernel instead of sticking to a proven amazon AKI.
  - mmalone 15y ago
    This bug affected several "supported" AMIs running 2.6.32 series kernels that we tested at SimpleGeo, including the official AMI released by Canonical. After we ran out of patience debugging this stuff we contacted Amazon and worked on the issue with a guy from their kernel team (who was really helpful, fwiw). He agreed that the behavior was bizarre and opened an upstream bug with Canonical [1]. You're sort of contradicting yourself here. You suggest that the distro you're running is independent of the kernel version you're running. But then you go on to claim that this bug was introduced by someone who was not running the default supported kernel. Are you saying that people should run the supported kernel, and be tied to whatever's supported upstream, or are you saying they should risk building their own? Clearly there are benefits and drawbacks either way. [1] https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/708920 https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/708...
    - moe 15y ago
      No, I'm saying you should run the kernels that Amazon provides and/or a time-tested one.
      - mmalone 15y ago
        Amazon's official linux build, at this time, is a custom distribution that uses RPM and is different from the Ubuntu/Debian world in several significant ways (e.g., a new libc implementation). Migrating your WordPress blog to a new platform might be easy, but when you're managing hundreds of machines running thousands of packages that sort of change is not trivial. Running "time-tested" kernels is not really the best advise either in this case. Xen is a fairly new environment, and EC2's implementation has some quirks, so there's a pretty regular stream of bug fixes and other improvements in recent kernels that are often worth picking up. If I went to Canonical with a "time-tested" kernel bug they'd tell me to upgrade before they'd give any real support. When we talked to Amazon about switching to their AMIs they advised us that it was probably _not_ worth switching, that switching might not fix the problem, and that the AMIs we were running were widely used and supported. They made it clear that they work closely with Canonical and other providers to get high quality AMIs into their ecosystem. Long story short, the people who you admit know the most about the EC2 environment advised us that they weren't necessarily the best option, or at least not the only option, for good AMIs (sort of like how hardware manufacturers aren't the best option for an operating system). So the answers aren't really cut-and-dry here. Every time Amazon changes their dom0 there's a chance your "time-tested" kernel will stop working. And just because Amazon runs the infrastructure doesn't mean they're the best choice for a Linux distribution.
- deleted 15y ago
  [deleted]
- jerf 15y ago
  Recompiling a Linux kernel with a custom patch is in the core set of skills that I would expect any Linux admin to have. It's not that hard. It's not even that hard to take existing distro kernels and add a patch to them, and maintain your patch going forward as the distro continues their refinements. When you have Linux on your server, you own it. You can do whatever you want. The distro is the beginning of your power, not the end of it. If you're running a Linux server and you currently have the attitude that you are boxed in by your distro I recommend that you immediately dig into the relevant packaging system and learn enough to put your own patch on top of any existing software package, and recreate the package in the relevant manner (new RPMs, new .deb, whatever). (Yes, there's a cost/benefit tradeoff to each such patch you have to carry on, but there is still economic value merely in having the option.)
  - nupark2 15y ago
    It's 2011, and screwing around with kernel compilation is a waste of my time. It should 'just work'. The only time I recompile a kernel is when I'm working on kernel code. If UNIX distributions are doing their jobs, a sysadmin should never have to touch it.
    - jerf 15y ago
      It's 2011. Everything should just work. Alas.
      - deleted 15y ago
        [deleted]
  - nowarninglabel 15y ago
    I really have to agree with the other reply that fooling around with re-compiling a kernel is definitely beyond what I would consider to be a rational expectation. You may not think it's that hard, and it may not be for someone whose sole job is to do just that, but for the vast majority of admins that are wearing many hats this is going to be a unneeded time sink.
mmalone 15y ago
Or even FreeBSD, which works on EC2 now. It uses a different technique for scheduling, so this issue will not affect it.
- deleted 15y ago
  [deleted]
dialtone 15y ago
The importance of the short release cycle has decreased a lot for us since we moved to EC2. It's really simple to re-deploy the entire infrastructure on newer and supported images. Using puppet and cloudformation or whatever it is that you prefer you can upgrade with very little trouble.

saurik 15y ago

I would be careful with Maverick (or Lenny; Karmic was unaffected): there is a serious issue with it dropping interrupts on EC2. If you are going yo use Ubuntu on EC2, you /really/ want to be using Natty. https://bugs.launchpad.net/ubuntu/+source/linux/+bug/666211 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/666211

dialtone 15y ago
This is even more serious than the one in 10.04. While in 10.04 the bug would slow down the machine for a while in 10.10 with 2.6.35 kernel will simply hang the machine and requires a reboot (which can only be achieved by rebooting from the AWS console multiple times) of the instance to be "fixed". I'm in the process of upgrading all of our instances to Natty from 10.04 or younger. It's actually weird that this issue didn't get any attention whatsoever.
mikeyk 15y ago
Seconded. We had our 10.04 machines regularly, and become totally unresponsive. Then, we had even worse luck with Maverick--starting with the issue where it didn't recognize half the RAM on the 68GB instance type. Natty seems to be holding up the best so far.