6 ms·

Out of all the people commenting on #727708, The Spotify post is a rep from an organization with systems to administrate. Just like almost every other not-ctte

Out of all the people commenting on #727708, The Spotify post is a rep from an organization with systems to administrate. Just like almost every other not-ctte, not-init maintainer person on that list, so it makes sense for Spotify to comment, but not really worth giving undue weight to their post in the larger debate currently happening on debian-ctte. If you want a better picture of debian's init decision, the bug got punted to ctte to decide on, so check out the ctte ml archives for December [0] and January [1]. Russ Allbery and Ian Jackson are the most vocal CTTE members on the list and support systemd and upstart respectively. The thread "Bug#727708: init system other points, and conclusion" is where Ian [2] Russ [3] start making the case for their preliminary conclusions. Edit: If there are other things worth pointing out on the list put 'em here. I've really only been skimming the posts for the last few weeks. They're also considering openRC, but it appears not to be a real contender. Tollef Fog Heen is a debian systemd maintainer and has posts worth reading on both what systemd actually is and what the impact of different decisions. [0] https://lists.debian.org/debian-ctte/2013/12/ https://lists.debian.org/debian-ctte/2013/12/ [1] https://lists.debian.org/debian-ctte/2014/01/ https://lists.debian.org/debian-ctte/2014/01/ [2] https://lists.debian.org/debian-ctte/2013/12/msg00182.html https://lists.debian.org/debian-ctte/2013/12/msg00182.html [3] https://lists.debian.org/debian-ctte/2013/12/msg00234.html https://lists.debian.org/debian-ctte/2013/12/msg00234.html

rohansingh 13y ago
I should point out that while this is theoretically the official position of our infrastructure team here at Spotify, we've also got a lot of opinion and debate internally (like you have on any topic among a large team of engineers). One other fun problem Spotify is going to tackle is how we're going to handle systemd vs. upstart if/when we transition from straight-up Debian to Ubuntu.
- asdfaoeu 13y ago
  Why are you transitioning to Ubuntu?
  - rohansingh 13y ago
    We're currently on Debian Squeeze so we need to move to something as Squeeze is approaching end-of-life. If we stick with Debian, we'll find ourselves back in this situation fairly soon, since Debian's security team only supports oldstable for one year after a new stable is released (approximately — there isn't a set timeline available). Ubuntu LTS releases, on the other hand, give us predictability and five years of support. That makes me very happy from an infrastructure standpoint. And the more up-to-date packages in Ubuntu repos make our feature devs happier, versus having to maintain backports for modern software.
    - sitkack 13y ago
      Will those packages always be up to date even on the LTS Ubuntus? Will you be able to support Postgres 12 and Nginx 3 on those machines? I find it useful to deploy the application on the same OS it was dev'd on, but in a decoupled manner. The thing I despise the most is needless upgrades of infrastructure pieces of working apps (redis,python runtime,etc). Each should get there own static version. A couple jobs back, I would package my own JVM along with the application (in an rpm) as ops was being really slow to upgrade machines. The only thing I depend on for a box is libc.
      - kiallmacinnes 13y ago
        And how did you, or how do they now, handle security updates for this bundled JVM? Sure - Ops being slow to upgrade is annoying, Ops patching a security hole and you leaving it open could be devastating. There is a really good reason Debian forbids embedded copies of other packages, and why I despise "solutions" like Chef's OmniBus. These things are live grenades moments away from taking the whole ship down.
        sitkack 13y ago
        We would rebuild the rpm with a new JVM and redeploy the app after testing the fix. Ha! How many copies of Lua are living in applications that Debian doesn't have control over? There are probably package maintainers excising those as we speak. I don't want to get into a packaging philosophy war, not enough fun over text, needs to be face to face. This is a good read, http://vagabond.github.io/rants/2013/06/21/z_packagers-dont-know-best/ http://vagabond.github.io/rants/2013/06/21/z_packagers-dont-... on how over zealous single tree packaging can make a mess of things. I had to bundle in my JVM because ops wouldn't allow more than ONE on a machine. I wanted /opt/jvm/1.6.22 /opt/jvm/1.7.10 And I could symlink my apps to the one I needed. New apps could get new JVMs, old apps would continue to run just fine. But they wouldn't do this because it _broke_ Red Hat file system guidelines, for whatever definition of broke. Reuse can absolutely cause over coupling. I prefer to have tractable dependency graphs. Take a look at * http://nixos.org/nix/ http://nixos.org/nix/ * http://www.gobolinux.org http://www.gobolinux.org (defunct)
- the_mitsuhiko 13y ago
  > One other fun problem Spotify is going to tackle is how we're going to handle systemd vs. upstart if/when we transition from straight-up Debian to Ubuntu. After two years of upstart I am wholeheartedly regretting ever touching it. It's broken by design and currently in such a bad way that you actually might have to reboot the machine to fix it. Upstart is the main reason I'm considering dropping ubuntu for something else.
  - mixmastamyk 13y ago
    Works for me. What is the problem?
    - STRML 13y ago
      For example, there is a long-standing bug [1] where an improperly configured job using "expect fork" can cause upstart to become completely confused and unfixable without a reboot (or a really hacky ruby script [2]). As the comments discuss, it's not really always clear all the time when to use "expect fork" vs "expect daemon" and the rules change with a `script` block. So this hits people all the time, including me. It's these kinds of bugs in upstart that bite people all the time. I could go on about its obtuse configuration but I can declare unequivocally that upstart configurations are my least favorite part of system administration, bar none. `Start-stop-daemon` reduces some of the pain but is also difficult to debug. 1. https://bugs.launchpad.net/upstart/+bug/406397 https://bugs.launchpad.net/upstart/+bug/406397 2. https://github.com/ion1/workaround-upstart-snafu/ https://github.com/ion1/workaround-upstart-snafu/
fournm 13y ago
Russ's points are an incredibly well thought out and written discussion of systemd vs upstart and worth reading for anyone who needs background on the two.
dfc 13y ago
The recent discussion on lwn.net is also interesting: Positions forming in the Debian init system decision: http://lwn.net/Articles/578208/ http://lwn.net/Articles/578208/
- ekyo777 13y ago
  related: https://wiki.debian.org/Debate/initsystem/upstart https://wiki.debian.org/Debate/initsystem/upstart

rohansingh 13y ago

I should point out that while this is theoretically the official position of our infrastructure team here at Spotify, we've also got a lot of opinion and debate internally (like you have on any topic among a large team of engineers). One other fun problem Spotify is going to tackle is how we're going to handle systemd vs. upstart if/when we transition from straight-up Debian to Ubuntu.

asdfaoeu 13y ago
Why are you transitioning to Ubuntu?
- rohansingh 13y ago
  We're currently on Debian Squeeze so we need to move to something as Squeeze is approaching end-of-life. If we stick with Debian, we'll find ourselves back in this situation fairly soon, since Debian's security team only supports oldstable for one year after a new stable is released (approximately — there isn't a set timeline available). Ubuntu LTS releases, on the other hand, give us predictability and five years of support. That makes me very happy from an infrastructure standpoint. And the more up-to-date packages in Ubuntu repos make our feature devs happier, versus having to maintain backports for modern software.
  - sitkack 13y ago
    Will those packages always be up to date even on the LTS Ubuntus? Will you be able to support Postgres 12 and Nginx 3 on those machines? I find it useful to deploy the application on the same OS it was dev'd on, but in a decoupled manner. The thing I despise the most is needless upgrades of infrastructure pieces of working apps (redis,python runtime,etc). Each should get there own static version. A couple jobs back, I would package my own JVM along with the application (in an rpm) as ops was being really slow to upgrade machines. The only thing I depend on for a box is libc.
    - kiallmacinnes 13y ago
      And how did you, or how do they now, handle security updates for this bundled JVM? Sure - Ops being slow to upgrade is annoying, Ops patching a security hole and you leaving it open could be devastating. There is a really good reason Debian forbids embedded copies of other packages, and why I despise "solutions" like Chef's OmniBus. These things are live grenades moments away from taking the whole ship down.
      - sitkack 13y ago
        We would rebuild the rpm with a new JVM and redeploy the app after testing the fix. Ha! How many copies of Lua are living in applications that Debian doesn't have control over? There are probably package maintainers excising those as we speak. I don't want to get into a packaging philosophy war, not enough fun over text, needs to be face to face. This is a good read, http://vagabond.github.io/rants/2013/06/21/z_packagers-dont-know-best/ http://vagabond.github.io/rants/2013/06/21/z_packagers-dont-... on how over zealous single tree packaging can make a mess of things. I had to bundle in my JVM because ops wouldn't allow more than ONE on a machine. I wanted /opt/jvm/1.6.22 /opt/jvm/1.7.10 And I could symlink my apps to the one I needed. New apps could get new JVMs, old apps would continue to run just fine. But they wouldn't do this because it _broke_ Red Hat file system guidelines, for whatever definition of broke. Reuse can absolutely cause over coupling. I prefer to have tractable dependency graphs. Take a look at * http://nixos.org/nix/ http://nixos.org/nix/ * http://www.gobolinux.org http://www.gobolinux.org (defunct)
the_mitsuhiko 13y ago
> One other fun problem Spotify is going to tackle is how we're going to handle systemd vs. upstart if/when we transition from straight-up Debian to Ubuntu. After two years of upstart I am wholeheartedly regretting ever touching it. It's broken by design and currently in such a bad way that you actually might have to reboot the machine to fix it. Upstart is the main reason I'm considering dropping ubuntu for something else.
- mixmastamyk 13y ago
  Works for me. What is the problem?
  - STRML 13y ago
    For example, there is a long-standing bug [1] where an improperly configured job using "expect fork" can cause upstart to become completely confused and unfixable without a reboot (or a really hacky ruby script [2]). As the comments discuss, it's not really always clear all the time when to use "expect fork" vs "expect daemon" and the rules change with a `script` block. So this hits people all the time, including me. It's these kinds of bugs in upstart that bite people all the time. I could go on about its obtuse configuration but I can declare unequivocally that upstart configurations are my least favorite part of system administration, bar none. `Start-stop-daemon` reduces some of the pain but is also difficult to debug. 1. https://bugs.launchpad.net/upstart/+bug/406397 https://bugs.launchpad.net/upstart/+bug/406397 2. https://github.com/ion1/workaround-upstart-snafu/ https://github.com/ion1/workaround-upstart-snafu/

fournm 13y ago

Russ's points are an incredibly well thought out and written discussion of systemd vs upstart and worth reading for anyone who needs background on the two.

dfc 13y ago

The recent discussion on lwn.net is also interesting: Positions forming in the Debian init system decision: http://lwn.net/Articles/578208/ http://lwn.net/Articles/578208/

ekyo777 13y ago
related: https://wiki.debian.org/Debate/initsystem/upstart https://wiki.debian.org/Debate/initsystem/upstart