3 ms·

We‘ve been mostly happy users of Mesos, Apache aurora, and consul. It works pretty well for us (200+ engineers). We have maybe 5 people dedicated to keeping it

by gen220 6y ago

We‘ve been mostly happy users of Mesos, Apache aurora, and consul. It works pretty well for us (200+ engineers). We have maybe 5 people dedicated to keeping it all alive, and they’d be able to maintain the pieces of it that we use. Aurora configuration kinda sucks, but I think job configurations might just suck in general. That being said, it has been concerning to us to see the big institutional players move off of mesos and aurora. We wouldn’t like to be active maintainers, but we could. I think that’s the main difference between Mesos and K8s: with K8s, we wouldn’t want to maintain it, and we wouldn’t be able to (since it’s so large). Somehow mesos and aurora feel more manageable.

harrisonjackson 6y ago
5 dedicated folks = a lot of dollars. $500k-1M? Do those 5 folks keep things up reliably without burning out? Honestly curious - never used k8s or Mesos* in production and we have a much smaller team managing a much smaller infrastructure, I am sure. edit: meant to say mesos but applies to both
- gen220 6y ago
  Yea 500k-1m is probably correct. But, to be fair, they’re not dedicated to only keeping the lights on, they probably spend 20% (1 eng year per year) of their collective hours fighting fires, so I don’t think burnout is too much of an issue. The remainder of their time is spent maintaining libraries and web interfaces. They’re a pretty standard “platform engineering” team. In my perspective, it doesn’t matter what technology you go with (K8s or otherwise): a responsibly-managed your platform team requires at least one team with a full on-call rotation (i.e. at least 4 engineers), depending on how wide your golden path is.
ridv 6y ago
I'm a former Apache Aurora maintainer. Aurora has been (and continues to be) awesome for us and I'm so happy to hear other folks are still using it and it's working out for them. Funny that you mention the configuration part. At the most recent KubeCon in San Diego, CA, the folks at Reddit gave a talk in which they said they got sick and tired of dealing with yaml. They accidentally went on to recreate Pystachio as the remedy so I think you're right on the money with your statement. When the Project Management Committee (PMC) voted to put Aurora in the attic we were all super bummed but we just ran out of interested developers :(. The PMC agreed to kick off an "official" fork but so far it's just me maintaining it: https://github.com/aurora-scheduler/aurora https://github.com/aurora-scheduler/aurora
- gen220 6y ago
  Oh it’s super cool to see you in the wild! To clarify, I had a lot of qualifying thoughts running through my head when I said “kinda sucks” (hence the “kinda”!). :) I actually think managing aurora configs is way easier than managing yml files, and I agree that I think aurora configs were ahead of the game: having access to python in your config feels like a super power. I feel like we’ll converge on something that compiles aurora configs into yml files, prior to runtime. That being said, we’ve never been able to get good editor support for things like “go to definition”, with the whole “include” syntax. We have maybe 2-3k aurora config files, of which maybe 100 are shared boilerplate. Do you have any advice on this? I tell vim to treat them like python files, but pylint hates them :) We were bummed by the PMC decision too. I think some people at my company have considered becoming maintainers over the years, but, for the most part, everything “just works”, so we haven’t felt a selfish need to, so to speak. I actually think it’s a kind of unintuitive credit to your project, that it doesn’t require a horde of maintainers. That being said, I’ll set aside some time this weekend to take a look at some issues. :)
  - ridv 6y ago
    Oh no worries, no offence taken at all with the comment, configurations files tend to suck in general :). Pystachio was indeed very forward looking and the folks who worked on this at Twitter at the time deserve all the credit there. I think what you mention is a general problem I've encountered with IDEs when it comes to dealing with Python (esp. the "go to" issue you mention). Even when I've had to touch the Aurora client code, which is full on Python code, I've come out pulling my hair thanks to PyCharm acting wonky. > We have maybe 2-3k aurora config files Those are some big files! The boiler plate stuff is definitely something I've heard before from users but, unfortunately, there doesn't seem to be a better answer. When it comes to managing job configs, I'm pretty low on the pecking order in terms of knowledge since we ended up creating our own thrift client using Go to use with Aurora. (As a consequence, all our job definitions exist as Go code.) Stephan Erb (https://twitter.com/erbstephan/ https://twitter.com/erbstephan/) may have better advice in this case. Some of the Twitter folks may have good info too, but they've been radio silent for months. > I actually think it’s a kind of unintuitive credit to your project, that it doesn’t require a horde of maintainers. That's definitely a great point and a great compliment to the project. There's a lot of love that went into this project and I'd be ecstatic to get some new contributions, even if it something simple like fixing documentation or bumping up dependencies :D.

by gen220 6y ago

harrisonjackson 6y ago

5 dedicated folks = a lot of dollars. $500k-1M? Do those 5 folks keep things up reliably without burning out? Honestly curious - never used k8s or Mesos* in production and we have a much smaller team managing a much smaller infrastructure, I am sure. edit: meant to say mesos but applies to both

gen220 6y ago
Yea 500k-1m is probably correct. But, to be fair, they’re not dedicated to only keeping the lights on, they probably spend 20% (1 eng year per year) of their collective hours fighting fires, so I don’t think burnout is too much of an issue. The remainder of their time is spent maintaining libraries and web interfaces. They’re a pretty standard “platform engineering” team. In my perspective, it doesn’t matter what technology you go with (K8s or otherwise): a responsibly-managed your platform team requires at least one team with a full on-call rotation (i.e. at least 4 engineers), depending on how wide your golden path is.

ridv 6y ago

I'm a former Apache Aurora maintainer. Aurora has been (and continues to be) awesome for us and I'm so happy to hear other folks are still using it and it's working out for them. Funny that you mention the configuration part. At the most recent KubeCon in San Diego, CA, the folks at Reddit gave a talk in which they said they got sick and tired of dealing with yaml. They accidentally went on to recreate Pystachio as the remedy so I think you're right on the money with your statement. When the Project Management Committee (PMC) voted to put Aurora in the attic we were all super bummed but we just ran out of interested developers :(. The PMC agreed to kick off an "official" fork but so far it's just me maintaining it: https://github.com/aurora-scheduler/aurora https://github.com/aurora-scheduler/aurora

gen220 6y ago
Oh it’s super cool to see you in the wild! To clarify, I had a lot of qualifying thoughts running through my head when I said “kinda sucks” (hence the “kinda”!). :) I actually think managing aurora configs is way easier than managing yml files, and I agree that I think aurora configs were ahead of the game: having access to python in your config feels like a super power. I feel like we’ll converge on something that compiles aurora configs into yml files, prior to runtime. That being said, we’ve never been able to get good editor support for things like “go to definition”, with the whole “include” syntax. We have maybe 2-3k aurora config files, of which maybe 100 are shared boilerplate. Do you have any advice on this? I tell vim to treat them like python files, but pylint hates them :) We were bummed by the PMC decision too. I think some people at my company have considered becoming maintainers over the years, but, for the most part, everything “just works”, so we haven’t felt a selfish need to, so to speak. I actually think it’s a kind of unintuitive credit to your project, that it doesn’t require a horde of maintainers. That being said, I’ll set aside some time this weekend to take a look at some issues. :)
- ridv 6y ago
  Oh no worries, no offence taken at all with the comment, configurations files tend to suck in general :). Pystachio was indeed very forward looking and the folks who worked on this at Twitter at the time deserve all the credit there. I think what you mention is a general problem I've encountered with IDEs when it comes to dealing with Python (esp. the "go to" issue you mention). Even when I've had to touch the Aurora client code, which is full on Python code, I've come out pulling my hair thanks to PyCharm acting wonky. > We have maybe 2-3k aurora config files Those are some big files! The boiler plate stuff is definitely something I've heard before from users but, unfortunately, there doesn't seem to be a better answer. When it comes to managing job configs, I'm pretty low on the pecking order in terms of knowledge since we ended up creating our own thrift client using Go to use with Aurora. (As a consequence, all our job definitions exist as Go code.) Stephan Erb (https://twitter.com/erbstephan/ https://twitter.com/erbstephan/) may have better advice in this case. Some of the Twitter folks may have good info too, but they've been radio silent for months. > I actually think it’s a kind of unintuitive credit to your project, that it doesn’t require a horde of maintainers. That's definitely a great point and a great compliment to the project. There's a lot of love that went into this project and I'd be ecstatic to get some new contributions, even if it something simple like fixing documentation or bumping up dependencies :D.