5 ms·

Our big project has moved from physical servers to Openshift. Its taken a lot of work, much more than expected. The best thing is that developers like it on the

by zz865 5y ago

Our big project has moved from physical servers to Openshift. Its taken a lot of work, much more than expected. The best thing is that developers like it on their resume, which is a bigger benefit than you'd think as we've kept some good people on the team. For users I see zero benefit. CI pipeline is just more complicated and probably slower. Cost wise it was cheaper for a while but now RedHat are bumping up licensing costs so now I think is about the same costs. Overall it seems like a waste of time, but has been interesting.

geerlingguy 5y ago
OpenShift is about 10x more complex than basic Docker / containers, and probably 2-4x more complex than plain old Kubernetes. I've seen more success from organizations running smaller K3s or K8s clusters (if they need the orchestration) or just running small apps via Docker/Docker Compose separately, using a CI system (even as simple as GitHub Actions) to manage deployments.
- zz865 5y ago
  Yeah its also a problem that our org has infrastructure teams that manage the openshift clusters and they are under resourced so dont help or often can't figure out how to fix problems. Linux sysadmins know what they're doing as the core infrastructure has been mostly the same for the last few decades.
mixermachine 5y ago
Moving from classic servers to containers you get: - Builds with fixed dependencies that never change. Rollback is easy - Easy deployment of a prod environment on a local machine - Fast deployment - Easy automation (use version X with config Y) With Kubernetes (or other derivates like Openshift) you get: - Auto scaling - Fail over - Better resource usage if multiple environments are executed - Abstraction of infrastructure - Zero downtime deployment (biggest point for my company, we deploy >3 times per week) There are applications that do not need Kubernetes or even containers, but is this list really nothing oO? I can imagine that if you use Kubernetes just like a classic cluster it could seem like an unnecesarry added complexity but you gain a lot of things.
- zz865 5y ago
  Yeah we have fixed usage so scaling or easy failover is not something we need.
  - mixermachine 5y ago
    Then I can understand you well. Kubernetes then just provides zero-downtime and additional complexity. When you already have something like a deployment window (like 2 am to 3 am) then ZDT also does not matter.
- TheDong 5y ago
  Each of those benefits are things I had before using containers or kubernetes, and were simpler. > Builds with fixed dependencies that never change. Rollback is easy Any good build system already did this, such as Bazel, or a Gemfile.lock. We'd just snapshot AMIs to keep OS dependencies fixed... which is what Docker images effectively do. If you re-docker-build the same Dockerfile, it's not like you get the same result of "apt-get install libxml" the next time either. > Easy deployment of a prod environment on a local machine How containers are deployed varies wildly between prod and the local machine. All the things that were hard before are still hard. Things like secrets and external dependencies still usually vary. If prod is a kubernetes environment, getting a suitable k8s environment setup locally sucks, especially since it will probably have a different ingress controller, load balancer setup, storage classes available, resource requests, etc. If prod is kubernetes and local is docker-compose, that honestly seems like just as much work to create a second way to run the stack than just using a bash script + "npm start" or "bundle exec rails server" or whatever. Either way, it's not really a prod environment. It's hard to run identical-to-prod environments locally, and those problems are related to secrets and clouds and such, not due to the lack of containers, in my experience. > Fast deployment In my experience, containers haven't sped up deployment. Let's say you use ubuntu for your host and container's OS. Before containers, this meant you had to download one version of libssl ever, and that was it. If there was an update to libz, that didn't require a new download of libssl. After containers, if you build your container for app1 last week, and your container for app2 today, the "FROM ubuntu" likely resolves to a different image. Both your apps now have different "ubuntu" layers, which probably have the same version of libssl, but deduplication of downloads only happens if the whole layer is identical. In essence, we went from downloading 1 copy of libssl (for the host OS only) to 3 copies (host OS + 2 containers w/ different ubuntu bases), and there's no deduplication. That by itself seems like it has to be slower since there's an inherent increase in network bandwidth that has to happen. Even if you have a shared base image, you're at least doubling the downloads of libssl since before you could use the host's copy only. All the items you listed under k8s are things I had before it, excluding "Abstraction of infrastructure". Frankly, if you have a well-made load balancer, it's hard not to have zero-downtime deployments and auto-scaling.
NomDePlum 5y ago
Completely different experience at the current organisation I'm at. Openshift has provided a stepping stone off of on-prem VMs and into the cloud. They just migrated the on-prem cluster to a AWS cluster. Next step is to move off Openshift and into cloud native where possible. Really it has provided a path to discontinue legacy services and build out a more modern system/service fairly quickly. I wasn't around when it was introduced so can't talk to the initial complexity/pain but the current team responsible for it is surprisingly small. The responsibility for everyday use has been delegated to all teams which gives them a capability to build and deploy frequently. That's a real enabler. Most of the things that make this work are not really technology based, good collaboration, sharing knowledge and practices and being able to ask and get help quickly but I do think Openshift itself isn't bad at all. It does appear to provide a very decent build, deploy and run platform. Was there particular problems, or just "all bad"?

by zz865 5y ago

geerlingguy 5y ago

OpenShift is about 10x more complex than basic Docker / containers, and probably 2-4x more complex than plain old Kubernetes. I've seen more success from organizations running smaller K3s or K8s clusters (if they need the orchestration) or just running small apps via Docker/Docker Compose separately, using a CI system (even as simple as GitHub Actions) to manage deployments.

zz865 5y ago
Yeah its also a problem that our org has infrastructure teams that manage the openshift clusters and they are under resourced so dont help or often can't figure out how to fix problems. Linux sysadmins know what they're doing as the core infrastructure has been mostly the same for the last few decades.

mixermachine 5y ago

Moving from classic servers to containers you get: - Builds with fixed dependencies that never change. Rollback is easy - Easy deployment of a prod environment on a local machine - Fast deployment - Easy automation (use version X with config Y) With Kubernetes (or other derivates like Openshift) you get: - Auto scaling - Fail over - Better resource usage if multiple environments are executed - Abstraction of infrastructure - Zero downtime deployment (biggest point for my company, we deploy >3 times per week) There are applications that do not need Kubernetes or even containers, but is this list really nothing oO? I can imagine that if you use Kubernetes just like a classic cluster it could seem like an unnecesarry added complexity but you gain a lot of things.

zz865 5y ago
Yeah we have fixed usage so scaling or easy failover is not something we need.
- mixermachine 5y ago
  Then I can understand you well. Kubernetes then just provides zero-downtime and additional complexity. When you already have something like a deployment window (like 2 am to 3 am) then ZDT also does not matter.
TheDong 5y ago
Each of those benefits are things I had before using containers or kubernetes, and were simpler. > Builds with fixed dependencies that never change. Rollback is easy Any good build system already did this, such as Bazel, or a Gemfile.lock. We'd just snapshot AMIs to keep OS dependencies fixed... which is what Docker images effectively do. If you re-docker-build the same Dockerfile, it's not like you get the same result of "apt-get install libxml" the next time either. > Easy deployment of a prod environment on a local machine How containers are deployed varies wildly between prod and the local machine. All the things that were hard before are still hard. Things like secrets and external dependencies still usually vary. If prod is a kubernetes environment, getting a suitable k8s environment setup locally sucks, especially since it will probably have a different ingress controller, load balancer setup, storage classes available, resource requests, etc. If prod is kubernetes and local is docker-compose, that honestly seems like just as much work to create a second way to run the stack than just using a bash script + "npm start" or "bundle exec rails server" or whatever. Either way, it's not really a prod environment. It's hard to run identical-to-prod environments locally, and those problems are related to secrets and clouds and such, not due to the lack of containers, in my experience. > Fast deployment In my experience, containers haven't sped up deployment. Let's say you use ubuntu for your host and container's OS. Before containers, this meant you had to download one version of libssl ever, and that was it. If there was an update to libz, that didn't require a new download of libssl. After containers, if you build your container for app1 last week, and your container for app2 today, the "FROM ubuntu" likely resolves to a different image. Both your apps now have different "ubuntu" layers, which probably have the same version of libssl, but deduplication of downloads only happens if the whole layer is identical. In essence, we went from downloading 1 copy of libssl (for the host OS only) to 3 copies (host OS + 2 containers w/ different ubuntu bases), and there's no deduplication. That by itself seems like it has to be slower since there's an inherent increase in network bandwidth that has to happen. Even if you have a shared base image, you're at least doubling the downloads of libssl since before you could use the host's copy only. All the items you listed under k8s are things I had before it, excluding "Abstraction of infrastructure". Frankly, if you have a well-made load balancer, it's hard not to have zero-downtime deployments and auto-scaling.

NomDePlum 5y ago

Completely different experience at the current organisation I'm at. Openshift has provided a stepping stone off of on-prem VMs and into the cloud. They just migrated the on-prem cluster to a AWS cluster. Next step is to move off Openshift and into cloud native where possible. Really it has provided a path to discontinue legacy services and build out a more modern system/service fairly quickly. I wasn't around when it was introduced so can't talk to the initial complexity/pain but the current team responsible for it is surprisingly small. The responsibility for everyday use has been delegated to all teams which gives them a capability to build and deploy frequently. That's a real enabler. Most of the things that make this work are not really technology based, good collaboration, sharing knowledge and practices and being able to ask and get help quickly but I do think Openshift itself isn't bad at all. It does appear to provide a very decent build, deploy and run platform. Was there particular problems, or just "all bad"?