4 ms·

Most of this post-mortem has nothing to do with Ceph or Gentoo, and everything to do with competent system administration and change management. No one appears

Most of this post-mortem has nothing to do with Ceph or Gentoo, and everything to do with competent system administration and change management. No one appears to have stopped to consider the ramifications of unannounced upgrades to the fileserver or OSD's. Once you're frantically rebuilding world, its over. the lessons learned completely ignore communications between sysadmins and management or, god forbid, even customers. "do everything whenever you like" is not system administration. >If we notice something is going wrong with Ceph, we will not hesitate to shut down the cluster prematurely. once bitten..twice shy? how does this solve future problems in the HA filesystem for your customer? >We should not update Ceph on all machines at once. HA/HPC 101. rolling upgrades are fine, slow and steady, with lots of testing and a documented rollback procedure if things go screwy. large enough/critical systems often have dev test and prod >We will build glibc with debugging symbols. you will plan your OS upgrades separate from your Ceph upgrades and with properly communicated actions before, during, and after the events. >We will track Ceph releases more closely as opposed to...what? being a sysadmin, Im on more than a dozen release mailing lists. I track load balancer known issues and patches to make sure i dont roll out, say, an F5 firmware with known failure to properly proxy HTTP2

saganus 9y ago
To be fair, the author says right at the beginning: "Please remember that we are all unpaid volunteers who have our own studies and/or day jobs, and no one has had more experience with Ceph than what you get from reading the manual." So while you might have a valid point, it's hard to develop those skills if you are not a proper sysadmin. If you are just an unpaid volunteer, it means that most likely the environment you work in can't afford (or won't pay) for actual sysadmins whose paid job is to know these things. Maybe the discussion should be, why were there unpaid volunteers doing sysadmin work instead of an actual sysadmin, but I guess that is a normal occurrence in universities all over the world?
- kbenson 9y ago
  Considering sysadmins often have keys to the kingdom, unpaid volunteers for sysadmin work seems really risky to me. Want volunteers to design a webapp for you? Go ahead, your professional sysadmin will make sure they only get permission to what they need. It doesn't work well the other way around.
- batbomb 9y ago
  They also said: > First and foremost, having a multiple-day spanning downtime is completely unacceptable for a central service like this [...] Which goes to your last point a bit. It's not a normal occurrence to all universities, but to some. If it was me, I'd turn this thing off repeatedly until the people used it realized it's value and tried to dedicate a bit of manpower to it.
- amelius 9y ago
  Or perhaps the question should be why the gun didn't have basic/full protection against foot-shooting.
- sekh60 9y ago
  In defense of ceph the documentation is fantastic and ceph clusters have only gotten easier toanage with every release. I am a homelabber, not a sysadmin and I run a cluster and have only run into one or two issues that the docs or patch notes didn't cover. Also the mailing list is incredibly friendly and helpful. The IRC channel is not as useful iny experience though, seems like with many there are a ton of idlers. One does need to keep up with the release notes, mainly for the past couple releases, especially with the latest release, luminous which marked stable bluestore as a more performance alternative to filestore.
  - saganus 9y ago
    I don't even think Ceph is at fault here, since they are not the ones making a critical piece of infrastructure be the responsibility of unpaid volunteers. Even if said volunteers where seasoned sysadmins, I don't think they should be stressing over something so critical while being unpaid. Obviously you don't always get to be paid in money. Sometimes you feel like doing it for the greater good, or because you like the institution, or whatever the reason. However I still think as a university you should not rely on unpaid staff to handle critical systems because it's not a nice thing to do. I don't think most people enjoy stressing over a hobby.
- lathiat 9y ago
  These kind of volunteer positions are exactly how you learn before you have a real job. Worked great for me. And then had experience to actually get me into a real job.
peterwwillis 9y ago
I don't think people notice how difficult rollback is. Some people think because their app is stateless, rollback is trivial. But virtually every single piece of technology that has an indirect effect on your app does have state, and each one moves your app away from a known good state. Everything from firewall rules and network routes, to switch firmware, to DNS, to BIOS updates, to LDAP directory changes, compute and storage configuration changes, software updates, etc all affect whether you can roll back successfully. Do you know when your cloud provider updates its software, and how it will affect your app? If they change something and don't notify you, and it causes a problem, at what point will you notice when their latest changes were? Can they even roll back their change, and how long will that take? This was an example of one tier or service failing. But it could have easily been a firewall change, a DNS change, a time sync issue, etc. If you want to roll back to the last time your app worked, all of those need a rollback method, too.
- emmelaich 9y ago
  > I don't think people notice how difficult rollback is. Yep, while we always had a plan in release/upgrades for a fallback, everyone knew it would never happen. We had enough testing in qa/preprod that no matter what happened, a fallback would cost much more time and money than any methods (manual or otherwise) to rectify the problem.

saganus 9y ago

To be fair, the author says right at the beginning: "Please remember that we are all unpaid volunteers who have our own studies and/or day jobs, and no one has had more experience with Ceph than what you get from reading the manual." So while you might have a valid point, it's hard to develop those skills if you are not a proper sysadmin. If you are just an unpaid volunteer, it means that most likely the environment you work in can't afford (or won't pay) for actual sysadmins whose paid job is to know these things. Maybe the discussion should be, why were there unpaid volunteers doing sysadmin work instead of an actual sysadmin, but I guess that is a normal occurrence in universities all over the world?

kbenson 9y ago
Considering sysadmins often have keys to the kingdom, unpaid volunteers for sysadmin work seems really risky to me. Want volunteers to design a webapp for you? Go ahead, your professional sysadmin will make sure they only get permission to what they need. It doesn't work well the other way around.
batbomb 9y ago
They also said: > First and foremost, having a multiple-day spanning downtime is completely unacceptable for a central service like this [...] Which goes to your last point a bit. It's not a normal occurrence to all universities, but to some. If it was me, I'd turn this thing off repeatedly until the people used it realized it's value and tried to dedicate a bit of manpower to it.
amelius 9y ago
Or perhaps the question should be why the gun didn't have basic/full protection against foot-shooting.
sekh60 9y ago
In defense of ceph the documentation is fantastic and ceph clusters have only gotten easier toanage with every release. I am a homelabber, not a sysadmin and I run a cluster and have only run into one or two issues that the docs or patch notes didn't cover. Also the mailing list is incredibly friendly and helpful. The IRC channel is not as useful iny experience though, seems like with many there are a ton of idlers. One does need to keep up with the release notes, mainly for the past couple releases, especially with the latest release, luminous which marked stable bluestore as a more performance alternative to filestore.
- saganus 9y ago
  I don't even think Ceph is at fault here, since they are not the ones making a critical piece of infrastructure be the responsibility of unpaid volunteers. Even if said volunteers where seasoned sysadmins, I don't think they should be stressing over something so critical while being unpaid. Obviously you don't always get to be paid in money. Sometimes you feel like doing it for the greater good, or because you like the institution, or whatever the reason. However I still think as a university you should not rely on unpaid staff to handle critical systems because it's not a nice thing to do. I don't think most people enjoy stressing over a hobby.
lathiat 9y ago
These kind of volunteer positions are exactly how you learn before you have a real job. Worked great for me. And then had experience to actually get me into a real job.

peterwwillis 9y ago

I don't think people notice how difficult rollback is. Some people think because their app is stateless, rollback is trivial. But virtually every single piece of technology that has an indirect effect on your app does have state, and each one moves your app away from a known good state. Everything from firewall rules and network routes, to switch firmware, to DNS, to BIOS updates, to LDAP directory changes, compute and storage configuration changes, software updates, etc all affect whether you can roll back successfully. Do you know when your cloud provider updates its software, and how it will affect your app? If they change something and don't notify you, and it causes a problem, at what point will you notice when their latest changes were? Can they even roll back their change, and how long will that take? This was an example of one tier or service failing. But it could have easily been a firewall change, a DNS change, a time sync issue, etc. If you want to roll back to the last time your app worked, all of those need a rollback method, too.

emmelaich 9y ago
> I don't think people notice how difficult rollback is. Yep, while we always had a plan in release/upgrades for a fallback, everyone knew it would never happen. We had enough testing in qa/preprod that no matter what happened, a fallback would cost much more time and money than any methods (manual or otherwise) to rectify the problem.