6 ms·

Spotify migrate 5000 servers from Debian to Ubuntu

90 points by lmedinas 12y ago

jgrowl 12y ago
We're still running Debian Squeeze in production at one of the places I work. It had one advantage recently: The SSL package was so old that the heartbleed vulnerability hadn't yet been introduced, lol!
- zorpner 12y ago
  I have a client who was in the same position. They actually still have a few boxes on Etch... and amazingly, one Sarge box: $ cat /etc/issue Debian GNU/Linux 3.1 \n \ They're all running a very old hacked-up version of torque that no one remaining at the company can do anything with. At least they're walled off from everything else, I suppose?
tenfingers 12y ago
Ubuntu LTS is a joke, really. Any major issue found in any core package is usually forwarded "upstream" (which is Debian). Very few people in Ubuntu have, apparently, the knowledge to debug issues which go beyond simple repackaging of an updated copy. Even issues related strictly to Ubuntu (such as mount dependencies with upstart with complicated setups with multipath) do not get fixed if the issue is not resolved in newer releases and/or got already debugged in Debian. This is not what "support" in "long term support" is. Predictable release cycle in these terms is not worth much. For desktop systems it might be different, but then again in our lab (~50 users) most of our Ubuntu's users shifted away to Mint over the course of 3 years. Only one is still using Ubuntu. But of course I'm biased, I'm a Debian developer myself.
- aruggirello 12y ago
  Shifting away from desktop Ubuntu has probably nothing to do with release cycles, and everything to do with Unity. Kubuntu, Lubuntu and Xubuntu got very popular when Unity debuted :)
  - tenfingers 12y ago
    It also has somewhat to do with release cycles. If the package you're having problems with is an old release that is not maintained anymore upstream, and/or ubuntu shifted to another package, you can kiss your "support" goodbye in most cases. Now I'm a bit too pessimistic, because in Debian such cases do also exist, though it's much more rare due to the way the stable release is managed. And I have to say that for true long-term support, with support meaning=capability of debugging issues in-house for all the core stack they release (most importantly, the kernel), RedHat is probably the only real choice you have actually.
    - davidp 12y ago
      > for true long-term support, with support meaning=capability of debugging issues in-house for all the core stack they release Your definition of support doesn't match that used by enterprises like Spotify, i.e. the ones with real money on the line. They don't care whether the debugging is in-house or elsewhere; they just want the problem solved, and they want someone they can call who will go solve it. That involves paying someone directly (Canonical or other support companies), indirectly (paying your own employees to learn and manage things), or both. I'm guessing the ecosystem thing is in play here -- it's just easier to find Ubuntu-competent people/firms than Debian-competent ones. That said, the work done by the Debian team is very valuable (thank you), which it seems ought at some level be reflected economically. I have no familiarity with Debian's financial situation but the hosting and compute resources can't come for free. A reasonable business model might be for the Debian team, or a commercial branch or spinoff thereof, or even just some motivated entrepreneurs, to serve as a (potentially non-profit) service provider for the likes of Canonical. Seems like a natural evolution to me, but as I said I'm not familiar with Debian's economics. > If the package you're having problems with is an old release that is not maintained anymore upstream Wouldn't Debian suffer the same problem here, e.g. if some piece of software's maintainer gets hit by a bus? Or does Debian make the commitment to take "ownership", if necessary, of all the software it bundles? The latter seems logistically challenging but I am prepared to be (even more) impressed.
      - 12y ago
- opendais 12y ago
  > For desktop systems it might be different, but then again in our lab (~50 users) most of our Ubuntu's users shifted away to Mint over the course of 3 years. Only one is still using Ubuntu. That is due to the awful UI decisions Ubuntu made. Then again I run Ubuntu Gnome instead of Mint. > This is not what "support" in "long term support" is. Predictable release cycle in these terms is not worth much. That isn't related to the OP's argument. The OP is basically saying that predictable support cycles of 5 years is something Debian should do in the interests of keeping big players in their ecosystem instead of them drifting into Ubuntu's orbit. The quality of Ubuntu's LTS is irrelevant if people switch purely because of marketing. https://www.debian.org/News/2014/20140424 https://www.debian.org/News/2014/20140424 It was done for Debian 6 and I'm pretty sure the OP is trying to encourage Debian to continue that on Debian 7 & 8. However, with no guarantees, people will switch to Ubuntu since Ubuntu's quality is "good enough".
  - tenfingers 12y ago
    Sure, I understood the logic behind that. And having a more tight release cycle will also intrinsically help Debian, since maintaining backports/patching a system for a longer time is also more demanding. My main problem with these sentiments though is that Ubuntu still has marketing for some reason. This means that it's actually more likely that you can find your $commercial package pre-built for Ubuntu, while there's no official support for Debian. Obsolete releases for commercial software are actually helpful (less support required). Not to mention that in most cases there's zero effort in making the package exactly the same....
    - opendais 12y ago
      > My main problem with these sentiments though is that Ubuntu still has marketing for some reason. Yes, it tried to make Linux more "mainstream" and mostly succeeded. That is what happens when you have a ton of money you can throw at a problem. http://arstechnica.com/business/2014/01/ubuntu-maker-boosted-revenue-in-2013-but-doubled-loss-to-21-million/ http://arstechnica.com/business/2014/01/ubuntu-maker-boosted... It is still losing double digit millions a year. Debian can't match that kind of loss. I don't think Debian even has $20 million in a year in terms of donations? {not counting labor)
- gizmo686 12y ago
  My reading of LTS has always been that they will continue to provide the updates from upstream, whereas a release that has past end of life will recieve no updates.
- Alupis 12y ago
  At my company, we don't even consider Ubuntu as a viable server platform. It's RHEL/CentOS for just about everything. As you mentioned, the LTS Ubuntu versions are really not on the same level as the stable RHEL releases and support that is provided. Yes, Ubuntu is very popular... in both the Desktop and Server arena (mostly catalyzed by EC2 making Ubuntu the default Linux image, even though EC2 is built out of RHEL/CentOS boxes). However, most "big metal" servers are RHEL/CentOS or something else. In any event (and distro-fany-boyism aside ;-) ... one has to question the change from Debian Stable to Ubuntu LTS. The only thing I can think of, is they wanted a more "proper" company to call up for support... even though Debian are the guys Ubuntu calls up when they need support. At the end of the day, my 2 cents would be to get a distro that focuses on being a server distro first and forthright, not a distro that is a jack-of-all-trades.
  - throwawayaway 12y ago
    It's weird also, given that spotify were pushing systemd[0] hard, shuttleworth[1] conceded - but the version of LTS Ubuntu 14.04 [2] that spotify switched to is on Upstart. [0] http://www.muktware.com/2014/01/spotify-uses-debian-endorses-systemd-instead-upstarts-default/19587 http://www.muktware.com/2014/01/spotify-uses-debian-endorses... [1] http://www.markshuttleworth.com/archives/1316 http://www.markshuttleworth.com/archives/1316 [2] https://lists.debian.org/debian-publicity/2014/06/msg00014.html https://lists.debian.org/debian-publicity/2014/06/msg00014.h...
  - deleted 12y ago
    [deleted]
  - tenfingers 12y ago
    Indeed. We've been running RHEL kernels with a Debian userland for a looong time.
  - rlpb 12y ago
    > ...even though Debian are the guys Ubuntu calls up when they need support. Nonsense. Citation needed, please. Many key Ubuntu developers are also Debian developers, so I'm not sure what calling up Debian guys even means.
  - 12y ago
- rlpb 12y ago
  > Any major issue found in any core package is usually forwarded "upstream" (which is Debian). Of 4571 bugs sent to Debian from Ubuntu (and tagged as such), 3905 came with patches [1]. > Very few people in Ubuntu have, apparently, the knowledge to debug issues which go beyond simple repackaging of an updated copy. The data says otherwise. If the bug originates upstream, then it's considered just good manners to forward the bug report upstream, rather than maintain the patch just in Ubuntu. Otherwise upstreams scream loudly that Ubuntu doesn't play well with the community. You can't have it both ways. Do you want bug reports for bugs your packages, or don't you? Do you not want free QA from Ubuntu users? As for supplying fixes, that depends on priorities, and that applies to everyone (regardless of distribution). Better to forward the bug even if it isn't a priority to fix. But where the fix is a priority for Ubuntu users, a patch is generally sent to Debian, as the data shows. [1]: http://udd.debian.org/cgi-bin/ubuntu_usertag.cgi http://udd.debian.org/cgi-bin/ubuntu_usertag.cgi
- warbiscuit 12y ago
  Not to mention that Debian consistently supports "in-place" upgrades when moving between releases, minimizing admin time doing the upgrade, and actual server downtime. Whereas Ubuntu... I tried using it on a few servers, their complete lack of in-place support meant the servers had to be rebuilt from scratch on a near-yearly basis if we wanted the latest release. Just not feasible.
  - rlpb 12y ago
    > their complete lack of in-place support How so? In what way does Ubuntu not support this? Not to mention: if you're not at least using configuration management by now, you're doing it wrong. Rebuilding a server should be a completely automatic process.
    - warbiscuit 12y ago
      Typed too hastily, meant "in-place upgrades".
      - rlpb 12y ago
        ...which Ubuntu supports.
      - Alupis 12y ago
        In fairness, RHEL/CentOS did not support in-place upgrades between major releases until RHEL/CentOS 7 (just released). Prior to 7, it was recommend to always do a fresh install.
SudoAlex 12y ago
As someone who admins a bunch of Debian powered servers, it would've easier for me if the Debian LTS project had been announced last year instead of earlier this year. The 1 year gap between the release of the new version and the end of support for the old version just wasn't long enough. Sadly I'm also switching servers to Ubuntu because of it - along with a push towards using Docker containers powered by distros with long term support.
trustfundbaby 12y ago
TIL Spotify has 5000 servers ... somehow I expected they'd have more.
- hedwall 12y ago
  They have more. A guess would be about the double.
  - trustfundbaby 12y ago
    I see. Did you used to work there? Why were the other servers not upgraded?
    - functionalfish 12y ago
      Here are some actual #s, a few months old: https://www.youtube.com/watch?v=pts6F00GFuU#t=41 https://www.youtube.com/watch?v=pts6F00GFuU#t=41
    - hedwall 12y ago
      I'd love too, but no. But I have a close friend that works in their SRE team. I also meet lots of Spotify people at meetups and such as I live in Stockholm. My guess would be that they will not migrate their data-related machines (Postgres, Cassandra etc) as the reason for going to Ubuntu is partially for Docker and using Docker in their testing and staging environments (to begin with). The other one being that they don't have anyone maintaining an patched inhouse kernel for debian anymore (or they don't want to do that).
LaSombra 12y ago
One thing I don't understand is why companies don't pay Canonical for support? Is it such a bad thing to pay for Linux support? I wonder if companies that have chosen Ubuntu LTS think they are supporting Canonical and its developers just because they are running Ubuntu. This article doesn't mention Canonical at all. Support Debian developers is a great thing but Canonical probably spends tons of money and I rarely see companies saying "I chose Canonical and Ubuntu because...". DISCLAIMER: I work for a company that provides Linux support services EDIT: Typos
- opendais 12y ago
  > One thing I don't understand is why companies don't pay Canonical for support? Is it such a bad thing to pay for Linux support? I've never needed paid professional linux support. Or windows support. Or mac support. Or android support or any of that. I think the issue is...once you develop enough in-house Linux knowledge you don't really derive a benefit from paying for support. Tbh, the only kind of "linux support" I'd really want is overnight sysadmin support so I don't get calls at 3am. I suspect that isn't the kind of support you have in mind.
  - LaSombra 12y ago
    Nope. Not at all. Backporting upstream patches by request, enhancing upstream, help debugging difficult, multi-level issues and so on. I also took it for granted but seeing what I see today, I understand.
    - opendais 12y ago
      > Backporting upstream patches by request, enhancing upstream, help debugging difficult, multi-level issues and so on. I also took it for granted but seeing what I see today, I understand. I guess to me if you don't have the expertise in-house as a tech company, I think you have issues because you are outsourcing core competencies [the ability compile & maintain linux related software, the ability to debug complex issues that are likely part of generating your competitive advantage in someway].
skywhopper 12y ago
My team switched from Debian to Ubuntu a few years back for precisely for this reason: a predictable update schedule. Dealing with support isn't really my concern. I'm able and willing to workaround whatever edge-case issues arise, but when I deploy an OS on a box I need to know how long I can plan on the core OS elements being updatable. I need to be able to schedule mass OS upgrades months in advance. Ubuntu provides that capability, Debian has not. Though it sounds like Debian has woken up to this issue and is finally addressing it.

90 points by lmedinas 12y ago

We're still running Debian Squeeze in production at one of the places I work. It had one advantage recently: The SSL package was so old that the heartbleed vulnerability hadn't yet been introduced, lol!

zorpner 12y ago
I have a client who was in the same position. They actually still have a few boxes on Etch... and amazingly, one Sarge box: $ cat /etc/issue Debian GNU/Linux 3.1 \n \ They're all running a very old hacked-up version of torque that no one remaining at the company can do anything with. At least they're walled off from everything else, I suppose?

tenfingers 12y ago

Ubuntu LTS is a joke, really. Any major issue found in any core package is usually forwarded "upstream" (which is Debian). Very few people in Ubuntu have, apparently, the knowledge to debug issues which go beyond simple repackaging of an updated copy. Even issues related strictly to Ubuntu (such as mount dependencies with upstart with complicated setups with multipath) do not get fixed if the issue is not resolved in newer releases and/or got already debugged in Debian. This is not what "support" in "long term support" is. Predictable release cycle in these terms is not worth much. For desktop systems it might be different, but then again in our lab (~50 users) most of our Ubuntu's users shifted away to Mint over the course of 3 years. Only one is still using Ubuntu. But of course I'm biased, I'm a Debian developer myself.

aruggirello 12y ago
Shifting away from desktop Ubuntu has probably nothing to do with release cycles, and everything to do with Unity. Kubuntu, Lubuntu and Xubuntu got very popular when Unity debuted :)
- tenfingers 12y ago
  It also has somewhat to do with release cycles. If the package you're having problems with is an old release that is not maintained anymore upstream, and/or ubuntu shifted to another package, you can kiss your "support" goodbye in most cases. Now I'm a bit too pessimistic, because in Debian such cases do also exist, though it's much more rare due to the way the stable release is managed. And I have to say that for true long-term support, with support meaning=capability of debugging issues in-house for all the core stack they release (most importantly, the kernel), RedHat is probably the only real choice you have actually.
  - davidp 12y ago
    > for true long-term support, with support meaning=capability of debugging issues in-house for all the core stack they release Your definition of support doesn't match that used by enterprises like Spotify, i.e. the ones with real money on the line. They don't care whether the debugging is in-house or elsewhere; they just want the problem solved, and they want someone they can call who will go solve it. That involves paying someone directly (Canonical or other support companies), indirectly (paying your own employees to learn and manage things), or both. I'm guessing the ecosystem thing is in play here -- it's just easier to find Ubuntu-competent people/firms than Debian-competent ones. That said, the work done by the Debian team is very valuable (thank you), which it seems ought at some level be reflected economically. I have no familiarity with Debian's financial situation but the hosting and compute resources can't come for free. A reasonable business model might be for the Debian team, or a commercial branch or spinoff thereof, or even just some motivated entrepreneurs, to serve as a (potentially non-profit) service provider for the likes of Canonical. Seems like a natural evolution to me, but as I said I'm not familiar with Debian's economics. > If the package you're having problems with is an old release that is not maintained anymore upstream Wouldn't Debian suffer the same problem here, e.g. if some piece of software's maintainer gets hit by a bus? Or does Debian make the commitment to take "ownership", if necessary, of all the software it bundles? The latter seems logistically challenging but I am prepared to be (even more) impressed.
    - 12y ago
opendais 12y ago
> For desktop systems it might be different, but then again in our lab (~50 users) most of our Ubuntu's users shifted away to Mint over the course of 3 years. Only one is still using Ubuntu. That is due to the awful UI decisions Ubuntu made. Then again I run Ubuntu Gnome instead of Mint. > This is not what "support" in "long term support" is. Predictable release cycle in these terms is not worth much. That isn't related to the OP's argument. The OP is basically saying that predictable support cycles of 5 years is something Debian should do in the interests of keeping big players in their ecosystem instead of them drifting into Ubuntu's orbit. The quality of Ubuntu's LTS is irrelevant if people switch purely because of marketing. https://www.debian.org/News/2014/20140424 https://www.debian.org/News/2014/20140424 It was done for Debian 6 and I'm pretty sure the OP is trying to encourage Debian to continue that on Debian 7 & 8. However, with no guarantees, people will switch to Ubuntu since Ubuntu's quality is "good enough".
- tenfingers 12y ago
  Sure, I understood the logic behind that. And having a more tight release cycle will also intrinsically help Debian, since maintaining backports/patching a system for a longer time is also more demanding. My main problem with these sentiments though is that Ubuntu still has marketing for some reason. This means that it's actually more likely that you can find your $commercial package pre-built for Ubuntu, while there's no official support for Debian. Obsolete releases for commercial software are actually helpful (less support required). Not to mention that in most cases there's zero effort in making the package exactly the same....
  - opendais 12y ago
    > My main problem with these sentiments though is that Ubuntu still has marketing for some reason. Yes, it tried to make Linux more "mainstream" and mostly succeeded. That is what happens when you have a ton of money you can throw at a problem. http://arstechnica.com/business/2014/01/ubuntu-maker-boosted-revenue-in-2013-but-doubled-loss-to-21-million/ http://arstechnica.com/business/2014/01/ubuntu-maker-boosted... It is still losing double digit millions a year. Debian can't match that kind of loss. I don't think Debian even has $20 million in a year in terms of donations? {not counting labor)
gizmo686 12y ago
My reading of LTS has always been that they will continue to provide the updates from upstream, whereas a release that has past end of life will recieve no updates.
Alupis 12y ago
At my company, we don't even consider Ubuntu as a viable server platform. It's RHEL/CentOS for just about everything. As you mentioned, the LTS Ubuntu versions are really not on the same level as the stable RHEL releases and support that is provided. Yes, Ubuntu is very popular... in both the Desktop and Server arena (mostly catalyzed by EC2 making Ubuntu the default Linux image, even though EC2 is built out of RHEL/CentOS boxes). However, most "big metal" servers are RHEL/CentOS or something else. In any event (and distro-fany-boyism aside ;-) ... one has to question the change from Debian Stable to Ubuntu LTS. The only thing I can think of, is they wanted a more "proper" company to call up for support... even though Debian are the guys Ubuntu calls up when they need support. At the end of the day, my 2 cents would be to get a distro that focuses on being a server distro first and forthright, not a distro that is a jack-of-all-trades.
- throwawayaway 12y ago
  It's weird also, given that spotify were pushing systemd[0] hard, shuttleworth[1] conceded - but the version of LTS Ubuntu 14.04 [2] that spotify switched to is on Upstart. [0] http://www.muktware.com/2014/01/spotify-uses-debian-endorses-systemd-instead-upstarts-default/19587 http://www.muktware.com/2014/01/spotify-uses-debian-endorses... [1] http://www.markshuttleworth.com/archives/1316 http://www.markshuttleworth.com/archives/1316 [2] https://lists.debian.org/debian-publicity/2014/06/msg00014.html https://lists.debian.org/debian-publicity/2014/06/msg00014.h...
- deleted 12y ago
  [deleted]
- tenfingers 12y ago
  Indeed. We've been running RHEL kernels with a Debian userland for a looong time.
- rlpb 12y ago
  > ...even though Debian are the guys Ubuntu calls up when they need support. Nonsense. Citation needed, please. Many key Ubuntu developers are also Debian developers, so I'm not sure what calling up Debian guys even means.
- 12y ago
rlpb 12y ago
> Any major issue found in any core package is usually forwarded "upstream" (which is Debian). Of 4571 bugs sent to Debian from Ubuntu (and tagged as such), 3905 came with patches [1]. > Very few people in Ubuntu have, apparently, the knowledge to debug issues which go beyond simple repackaging of an updated copy. The data says otherwise. If the bug originates upstream, then it's considered just good manners to forward the bug report upstream, rather than maintain the patch just in Ubuntu. Otherwise upstreams scream loudly that Ubuntu doesn't play well with the community. You can't have it both ways. Do you want bug reports for bugs your packages, or don't you? Do you not want free QA from Ubuntu users? As for supplying fixes, that depends on priorities, and that applies to everyone (regardless of distribution). Better to forward the bug even if it isn't a priority to fix. But where the fix is a priority for Ubuntu users, a patch is generally sent to Debian, as the data shows. [1]: http://udd.debian.org/cgi-bin/ubuntu_usertag.cgi http://udd.debian.org/cgi-bin/ubuntu_usertag.cgi
warbiscuit 12y ago
Not to mention that Debian consistently supports "in-place" upgrades when moving between releases, minimizing admin time doing the upgrade, and actual server downtime. Whereas Ubuntu... I tried using it on a few servers, their complete lack of in-place support meant the servers had to be rebuilt from scratch on a near-yearly basis if we wanted the latest release. Just not feasible.
- rlpb 12y ago
  > their complete lack of in-place support How so? In what way does Ubuntu not support this? Not to mention: if you're not at least using configuration management by now, you're doing it wrong. Rebuilding a server should be a completely automatic process.
  - warbiscuit 12y ago
    Typed too hastily, meant "in-place upgrades".
    - rlpb 12y ago
      ...which Ubuntu supports.
    - Alupis 12y ago
      In fairness, RHEL/CentOS did not support in-place upgrades between major releases until RHEL/CentOS 7 (just released). Prior to 7, it was recommend to always do a fresh install.

SudoAlex 12y ago

As someone who admins a bunch of Debian powered servers, it would've easier for me if the Debian LTS project had been announced last year instead of earlier this year. The 1 year gap between the release of the new version and the end of support for the old version just wasn't long enough. Sadly I'm also switching servers to Ubuntu because of it - along with a push towards using Docker containers powered by distros with long term support.

trustfundbaby 12y ago

TIL Spotify has 5000 servers ... somehow I expected they'd have more.

hedwall 12y ago
They have more. A guess would be about the double.
- trustfundbaby 12y ago
  I see. Did you used to work there? Why were the other servers not upgraded?
  - functionalfish 12y ago
    Here are some actual #s, a few months old: https://www.youtube.com/watch?v=pts6F00GFuU#t=41 https://www.youtube.com/watch?v=pts6F00GFuU#t=41
  - hedwall 12y ago
    I'd love too, but no. But I have a close friend that works in their SRE team. I also meet lots of Spotify people at meetups and such as I live in Stockholm. My guess would be that they will not migrate their data-related machines (Postgres, Cassandra etc) as the reason for going to Ubuntu is partially for Docker and using Docker in their testing and staging environments (to begin with). The other one being that they don't have anyone maintaining an patched inhouse kernel for debian anymore (or they don't want to do that).

LaSombra 12y ago

One thing I don't understand is why companies don't pay Canonical for support? Is it such a bad thing to pay for Linux support? I wonder if companies that have chosen Ubuntu LTS think they are supporting Canonical and its developers just because they are running Ubuntu. This article doesn't mention Canonical at all. Support Debian developers is a great thing but Canonical probably spends tons of money and I rarely see companies saying "I chose Canonical and Ubuntu because...". DISCLAIMER: I work for a company that provides Linux support services EDIT: Typos

opendais 12y ago
> One thing I don't understand is why companies don't pay Canonical for support? Is it such a bad thing to pay for Linux support? I've never needed paid professional linux support. Or windows support. Or mac support. Or android support or any of that. I think the issue is...once you develop enough in-house Linux knowledge you don't really derive a benefit from paying for support. Tbh, the only kind of "linux support" I'd really want is overnight sysadmin support so I don't get calls at 3am. I suspect that isn't the kind of support you have in mind.
- LaSombra 12y ago
  Nope. Not at all. Backporting upstream patches by request, enhancing upstream, help debugging difficult, multi-level issues and so on. I also took it for granted but seeing what I see today, I understand.
  - opendais 12y ago
    > Backporting upstream patches by request, enhancing upstream, help debugging difficult, multi-level issues and so on. I also took it for granted but seeing what I see today, I understand. I guess to me if you don't have the expertise in-house as a tech company, I think you have issues because you are outsourcing core competencies [the ability compile & maintain linux related software, the ability to debug complex issues that are likely part of generating your competitive advantage in someway].

skywhopper 12y ago

My team switched from Debian to Ubuntu a few years back for precisely for this reason: a predictable update schedule. Dealing with support isn't really my concern. I'm able and willing to workaround whatever edge-case issues arise, but when I deploy an OS on a box I need to know how long I can plan on the core OS elements being updatable. I need to be able to schedule mass OS upgrades months in advance. Ubuntu provides that capability, Debian has not. Though it sounds like Debian has woken up to this issue and is finally addressing it.