17 ms·

Roadmap for CoreOS Integration with Red Hat OpenShift

159 points by davidmr 8y ago

davidmr 8y ago
The bits about the OpenShift integration are interesting I guess, but buried about halfway down is the news that they intend CoreOS' Container Linux to supplant their existing Fedora/RHEL Atomic Host, and Brandon Philips from CoreOS says that they're be continuing to base it around Ignition[0] I'm genuinely surprised at this. RH has put a ton of work into rpm-ostree for a long time. I guess there's a chance they'll meld it somehow with Ignition and Container Linux's Chrome OS bits when they turn it into Red Hat Container Linux or whatever it'll be called, but it's surprising to see Red Hat supporting a Linux distro not based of of RPMs and installed with kickstart/anaconda. [0] https://twitter.com/BrandonPhilips/status/993880972583092225 https://twitter.com/BrandonPhilips/status/993880972583092225
- puzzle 8y ago
  It's not clear to me if e.g. the Container Linux system will be still built using Gentoo's emerge or the RH tools. You seem to discount the latter, but I guess it depends on what they really mean with "based on Fedora and Red Hat Enterprise Linux sources". Are they going to rebuild e.g. the kernel using emerge, but from RH's source tarballs and collection of patch sets?
  - smarterclayton 8y ago
    Initially its going to be based on RHEL so we can quickly ramp up. Over time it’s very likely we get more aggressive with RH CoreOS and newer kernels, but those are all details being worked on. It’s very important for us to be able to benefit from existing engineering investments in the community and within RHEL, but there’s a lot of excitement about taking what CoreOS proved could be compelling and reinforcing that with the Red Hat engineering teams. Stay tuned for more.
- alexandre_m 8y ago
  Absolutely, CoreOS (container linux) not being killed and being integrated officially in a RedHat product was the most interesting piece.
- state_less 8y ago
  It's CoreOS (now called containerLinux) so they probably figure the end user will install containers instead of rpm packages. I haven't added anything to the base containerLinux image while using it, which is probably as intended.
  - frankharv 8y ago
    I am so glad they changed the name. TinyCore and Core Linux were around before CoreOS and I always thought they robbed the name. Defiantly made for some confusion.
    - justincormack 8y ago
      If you read the article they are changing it back to Red Hat CoreOS.
    - Already__Taken 8y ago
      container linux was always pretty much useless to search for help for.
      - riffic 8y ago
        Use quotation marks around your search terms to indicate a specific phrase.
    - mattdm 8y ago
      Although... Fedora Core predates them all....
- smarterclayton 8y ago
  The combo is going to be ignition + ostree + evolution of Omaha (update server) + more prototyping still being done. It won’t be using chrome OS bits. Edit: as of now this is what the team is thinking, still lots of room for changes as we refine down.
- emmelaich 8y ago
  Related: https://news.ycombinator.com/item?id=17000963 https://news.ycombinator.com/item?id=17000963 "Team Silverblue: Fedora as an image and container-based desktop OS" (using ostree)
- peterwwillis 8y ago
  This is why I won't advocate RH and similar vendors. The tech stack churn is too high. If it's not a widely adopted open source platform it's just going to eventually become bought out or die a lonely death with me having to migrate clusters to something else, or hang on to unsupported legacy systems for 10 years. Rather support a Frankenstein's monster of my own design.
  - cgwalters 8y ago
    Bear in mind that we're basically trying to do it both ways; We still have Red Hat Enterprise Linux, and I don't think anyone would involve the term "tech stack churn" there.
kev009 8y ago
The PR is written a bit ambiguously, is the Gentoo based container Linux going to be maintained or will it move to EL/Fedora?
- smarterclayton 8y ago
  Maintained. A new offering based on RHEL will be initially targeted for the supported scenarios under openshift while we work with the communities on how they want to evolve.
  - merb 8y ago
    So will there be a path to upgrade CoreOS to RedHat Container Linux?
    - cgwalters 8y ago
      If you're using Tectonic in particular, we're certainly aiming to have a nice path. Although there are a lot of details in the term "upgrade" - it may require reprovisioning, but a lot of the idea of carrying forward Ignition is that any early OS customization you've made still applies. If you're running a Kubernetes cluster, while Red Hat CoreOS will support automatic inplace updates just like existing CL, I'd say it's best practice to do periodic reprovisioning to flush out extra node state. For example in RHEL 7.5 we switched from devicemapper to overlayfs, but existing instances don't get automatically transitioned. If you're using k8s reprovisioning works well as all the containers just move off and then back on.
jzelinskie 8y ago
I'm a PM and engineer at CoreOS/Red Hat -- feel free to ask any questions and I'll do my best to answer. In the next few months, you should see an OpenShift that is built upon the same upgrade system as Tectonic which allows for more incremental buy-in to OpenShift PaaS functionality and a Linux distribution that leverages Ignition and immutability to provide the minimal environment needed to run Kubernetes/containers. My understanding is that Container Linux as is will be supported for years, but we will also be creating a new distro, RH CoreOS, that replaces the Gentoo build system with Fedora tooling. This shouldn't change much for users as they don't interact with the build system; they just consume the results of said system. I'd liken this scenario to the relationship between CentOS and RHEL, which are both maintained by Red Hat. Some details have yet to shake out; for example, I personally don't know if the resulting distro will leverage rpm-ostree, but we already have internal proof-of-concepts running OpenShift with Tectonic components on top of Container Linux. Please voice your opinions here and now! Nothing is set in stone and we're listening for the community to weigh in on these decisions as well.
- sytse 8y ago
  The operator framework seems really useful. Do you expect that eventually it or something like it will become part of Kubernetes?
  - jzelinskie 8y ago
    We're collaborating with the kubebuilder[0] project upstream, which is a subproject of SIG API Machinery that focuses on generating the best scaffolding for controllers. Myself and some Googlers also proposed creating a SIG focusing on platform extensions to Kubernetes, such as operators tooling[1]. The steering committee is currently not convinced that it merits a dedicated SIG, despite the many projects in the wild experimenting without organization. We're fully committed to taking well understood, community-accepted opinions from our tooling and upstreaming the work, if the community can agree on that aspect of the framework. A great example of this is the Application Definition Working Group, which has leveraged many ideas from our Operator Lifecycle Manager; our CRDs are practically the same! Now that things are open source, we should see things like these converge entirely. [0]: https://github.com/kubernetes-sigs/kubebuilder https://github.com/kubernetes-sigs/kubebuilder [1]: https://groups.google.com/d/msg/kubernetes-dev/RgwIQ9Dii-I/Q1_tTTnwBgAJ https://groups.google.com/d/msg/kubernetes-dev/RgwIQ9Dii-I/Q...
mattdaemon 8y ago
What are your plans for rkt? It’s great alternative in a docker-dominated ecosystem and much more solid architecture but haven’t seen much progress lately. Doesn’t seem to be getting a lot of love
- jzelinskie 8y ago
  Red Hat is backing CRI-O for an alternative OCI runtime. While we did pave the way for creating alternatives for Docker in Kubernetes, CoreOS never quite got rkt to 100% stability in Kubernetes. Personally, I love a lot of things about rkt, but the project's ultimate goal was to have standards, regardless of whether or not it was AppC. If you're still interested in rkt (it's great tech that we still use to this day to run kubelet for all Tectonic clusters), I recommended chatting to the awesome folks at Kinvolk[0]. They maintain rkt alongside CoreOS and support customers using it in production. [0]: https://kinvolk.io https://kinvolk.io
  - blixtra 8y ago
    Chris from Kinvolk here. As Jimmy mentioned, we've done a good chunk of the work on rkt with CoreOS and are happy to support customers using rkt, and have done so for CoreOS, BlaBlaCar, NASDAQ and others in the past. But we've chosen not to go the startup route, which means we can only really afford to work on rkt in the context of paid work. We're looking at doing more of this in the future through support contracts for Flatcar Linux[0], a fork of CoreOS' Container Linux, which includes rkt in the images, and through the contracts we get here and there from users looking for new features in, or support for, rkt directly. But rkt, as is, remains a great container runtime. It's our preferred runtime when running outside of Kubernetes, atm. The Kubernetes integration via rktlet[1] works well but does not have 100% functional parity with the default CRI implementation. It probably needs about 3 person-months of work to get there at this point. So yeah, it works well, but does indeed need a bit more love. If you're interested in helping out, get in touch. [0] https://www.flatcar-linux.org/ https://www.flatcar-linux.org/ [1] https://github.com/kubernetes-incubator/rktlet https://github.com/kubernetes-incubator/rktlet
  - madmulita 8y ago
actionowl 8y ago
"CoreOS technology to combine with Red Hat OpenShift to drive hybrid cloud-native services, will power fully-automated Linux container platform stack, from the operating system to application services, across the hybrid cloud" This is so overloaded with buzz words it took a few attempts to make any sense of it.
- erikb 8y ago
  Sounds cooler than "We will also create the AWS VMs for you, just give us the credentials", though.
joshberkus 8y ago
For anyone reading this thread who is at Red Hat Summit, we will have a Container Linux/Atomic BOF at 1pm today (May 10), in the BOF area on the 2nd floor.

159 points by davidmr 8y ago

The bits about the OpenShift integration are interesting I guess, but buried about halfway down is the news that they intend CoreOS' Container Linux to supplant their existing Fedora/RHEL Atomic Host, and Brandon Philips from CoreOS says that they're be continuing to base it around Ignition[0] I'm genuinely surprised at this. RH has put a ton of work into rpm-ostree for a long time. I guess there's a chance they'll meld it somehow with Ignition and Container Linux's Chrome OS bits when they turn it into Red Hat Container Linux or whatever it'll be called, but it's surprising to see Red Hat supporting a Linux distro not based of of RPMs and installed with kickstart/anaconda. [0] https://twitter.com/BrandonPhilips/status/993880972583092225 https://twitter.com/BrandonPhilips/status/993880972583092225

puzzle 8y ago
It's not clear to me if e.g. the Container Linux system will be still built using Gentoo's emerge or the RH tools. You seem to discount the latter, but I guess it depends on what they really mean with "based on Fedora and Red Hat Enterprise Linux sources". Are they going to rebuild e.g. the kernel using emerge, but from RH's source tarballs and collection of patch sets?
- smarterclayton 8y ago
  Initially its going to be based on RHEL so we can quickly ramp up. Over time it’s very likely we get more aggressive with RH CoreOS and newer kernels, but those are all details being worked on. It’s very important for us to be able to benefit from existing engineering investments in the community and within RHEL, but there’s a lot of excitement about taking what CoreOS proved could be compelling and reinforcing that with the Red Hat engineering teams. Stay tuned for more.
alexandre_m 8y ago
Absolutely, CoreOS (container linux) not being killed and being integrated officially in a RedHat product was the most interesting piece.
state_less 8y ago
It's CoreOS (now called containerLinux) so they probably figure the end user will install containers instead of rpm packages. I haven't added anything to the base containerLinux image while using it, which is probably as intended.
- frankharv 8y ago
  I am so glad they changed the name. TinyCore and Core Linux were around before CoreOS and I always thought they robbed the name. Defiantly made for some confusion.
  - justincormack 8y ago
    If you read the article they are changing it back to Red Hat CoreOS.
  - Already__Taken 8y ago
    container linux was always pretty much useless to search for help for.
    - riffic 8y ago
      Use quotation marks around your search terms to indicate a specific phrase.
  - mattdm 8y ago
    Although... Fedora Core predates them all....
smarterclayton 8y ago
The combo is going to be ignition + ostree + evolution of Omaha (update server) + more prototyping still being done. It won’t be using chrome OS bits. Edit: as of now this is what the team is thinking, still lots of room for changes as we refine down.
emmelaich 8y ago
Related: https://news.ycombinator.com/item?id=17000963 https://news.ycombinator.com/item?id=17000963 "Team Silverblue: Fedora as an image and container-based desktop OS" (using ostree)
peterwwillis 8y ago
This is why I won't advocate RH and similar vendors. The tech stack churn is too high. If it's not a widely adopted open source platform it's just going to eventually become bought out or die a lonely death with me having to migrate clusters to something else, or hang on to unsupported legacy systems for 10 years. Rather support a Frankenstein's monster of my own design.
- cgwalters 8y ago
  Bear in mind that we're basically trying to do it both ways; We still have Red Hat Enterprise Linux, and I don't think anyone would involve the term "tech stack churn" there.

kev009 8y ago

The PR is written a bit ambiguously, is the Gentoo based container Linux going to be maintained or will it move to EL/Fedora?

smarterclayton 8y ago
Maintained. A new offering based on RHEL will be initially targeted for the supported scenarios under openshift while we work with the communities on how they want to evolve.
- merb 8y ago
  So will there be a path to upgrade CoreOS to RedHat Container Linux?
  - cgwalters 8y ago
    If you're using Tectonic in particular, we're certainly aiming to have a nice path. Although there are a lot of details in the term "upgrade" - it may require reprovisioning, but a lot of the idea of carrying forward Ignition is that any early OS customization you've made still applies. If you're running a Kubernetes cluster, while Red Hat CoreOS will support automatic inplace updates just like existing CL, I'd say it's best practice to do periodic reprovisioning to flush out extra node state. For example in RHEL 7.5 we switched from devicemapper to overlayfs, but existing instances don't get automatically transitioned. If you're using k8s reprovisioning works well as all the containers just move off and then back on.

jzelinskie 8y ago

I'm a PM and engineer at CoreOS/Red Hat -- feel free to ask any questions and I'll do my best to answer. In the next few months, you should see an OpenShift that is built upon the same upgrade system as Tectonic which allows for more incremental buy-in to OpenShift PaaS functionality and a Linux distribution that leverages Ignition and immutability to provide the minimal environment needed to run Kubernetes/containers. My understanding is that Container Linux as is will be supported for years, but we will also be creating a new distro, RH CoreOS, that replaces the Gentoo build system with Fedora tooling. This shouldn't change much for users as they don't interact with the build system; they just consume the results of said system. I'd liken this scenario to the relationship between CentOS and RHEL, which are both maintained by Red Hat. Some details have yet to shake out; for example, I personally don't know if the resulting distro will leverage rpm-ostree, but we already have internal proof-of-concepts running OpenShift with Tectonic components on top of Container Linux. Please voice your opinions here and now! Nothing is set in stone and we're listening for the community to weigh in on these decisions as well.

sytse 8y ago
The operator framework seems really useful. Do you expect that eventually it or something like it will become part of Kubernetes?
- jzelinskie 8y ago
  We're collaborating with the kubebuilder[0] project upstream, which is a subproject of SIG API Machinery that focuses on generating the best scaffolding for controllers. Myself and some Googlers also proposed creating a SIG focusing on platform extensions to Kubernetes, such as operators tooling[1]. The steering committee is currently not convinced that it merits a dedicated SIG, despite the many projects in the wild experimenting without organization. We're fully committed to taking well understood, community-accepted opinions from our tooling and upstreaming the work, if the community can agree on that aspect of the framework. A great example of this is the Application Definition Working Group, which has leveraged many ideas from our Operator Lifecycle Manager; our CRDs are practically the same! Now that things are open source, we should see things like these converge entirely. [0]: https://github.com/kubernetes-sigs/kubebuilder https://github.com/kubernetes-sigs/kubebuilder [1]: https://groups.google.com/d/msg/kubernetes-dev/RgwIQ9Dii-I/Q1_tTTnwBgAJ https://groups.google.com/d/msg/kubernetes-dev/RgwIQ9Dii-I/Q...

mattdaemon 8y ago

What are your plans for rkt? It’s great alternative in a docker-dominated ecosystem and much more solid architecture but haven’t seen much progress lately. Doesn’t seem to be getting a lot of love

jzelinskie 8y ago
Red Hat is backing CRI-O for an alternative OCI runtime. While we did pave the way for creating alternatives for Docker in Kubernetes, CoreOS never quite got rkt to 100% stability in Kubernetes. Personally, I love a lot of things about rkt, but the project's ultimate goal was to have standards, regardless of whether or not it was AppC. If you're still interested in rkt (it's great tech that we still use to this day to run kubelet for all Tectonic clusters), I recommended chatting to the awesome folks at Kinvolk[0]. They maintain rkt alongside CoreOS and support customers using it in production. [0]: https://kinvolk.io https://kinvolk.io
- blixtra 8y ago
  Chris from Kinvolk here. As Jimmy mentioned, we've done a good chunk of the work on rkt with CoreOS and are happy to support customers using rkt, and have done so for CoreOS, BlaBlaCar, NASDAQ and others in the past. But we've chosen not to go the startup route, which means we can only really afford to work on rkt in the context of paid work. We're looking at doing more of this in the future through support contracts for Flatcar Linux[0], a fork of CoreOS' Container Linux, which includes rkt in the images, and through the contracts we get here and there from users looking for new features in, or support for, rkt directly. But rkt, as is, remains a great container runtime. It's our preferred runtime when running outside of Kubernetes, atm. The Kubernetes integration via rktlet[1] works well but does not have 100% functional parity with the default CRI implementation. It probably needs about 3 person-months of work to get there at this point. So yeah, it works well, but does indeed need a bit more love. If you're interested in helping out, get in touch. [0] https://www.flatcar-linux.org/ https://www.flatcar-linux.org/ [1] https://github.com/kubernetes-incubator/rktlet https://github.com/kubernetes-incubator/rktlet
- madmulita 8y ago

actionowl 8y ago

"CoreOS technology to combine with Red Hat OpenShift to drive hybrid cloud-native services, will power fully-automated Linux container platform stack, from the operating system to application services, across the hybrid cloud" This is so overloaded with buzz words it took a few attempts to make any sense of it.

erikb 8y ago
Sounds cooler than "We will also create the AWS VMs for you, just give us the credentials", though.

joshberkus 8y ago

For anyone reading this thread who is at Red Hat Summit, we will have a Container Linux/Atomic BOF at 1pm today (May 10), in the BOF area on the 2nd floor.