5 ms·

I'm running both OpenShift and k3s in production, and there isnt that much that requires different treatment between the two. There are some specific OpenShift

I'm running both OpenShift and k3s in production, and there isnt that much that requires different treatment between the two. There are some specific OpenShift APIs (like routes which are terrible) and some quality of life improvements (service-ca signer) but nothing drastic.

blinkingled 5y ago
Have you tried running istio for example in an enterprise env - they needed you to install a RH specific older version and IIRC that wasn't just for support. I could list some more things if I recalled hard enough.
m3adow 5y ago
> like routes which are terrible Huh, interesting. What do you not like about routes? My team is providing an IaaS solution for internal developers in my company and a lot developers seemes to have less problems with Openshifts service exposition abstraction (Routes) in contrast to pure Kubernetes.
- freedomben 5y ago
  Routes are of the biggest things I miss when I'm on vanilla K8s. I don't see how anybody could prefer Ingress to Routes, but to each their own.
- nullify88 5y ago
  A big inconvenience is that for HTTP2 Routes or edge / re-encrypt Routes with a custom TLS certificate, the TLS certificate and keys must be inline in the Route resource instead of referencing a secret like Ingress resources do. I think this is a big oversight where Routes mix secrets and Ingress configuration together. It makes GitOps annoying because I don't want to treat the whole Route resource as a secret that needs to be encrypted or stored in vault. Do I also then treat Route resources as sensitive and deny some users access on the account they could contain private keys? I also have to worry about keeping the route updated before certificates expire instead of having it taken care of by cert-manager. So we use Traefik's IngressRoute.
  - m3adow 5y ago
    That I can relate to. For us, there's only one dev team which uses HTTP2 (financial industry, so HTTP2 is still seen as "new = beta") and encountered that problem. I have no idea how they solved it though.
    - nullify88 5y ago
      FWIW, although I've known for a while that OpenShift coverts ingress resources to routes, I just found out that the Ingress Controller sets up a watch on the secretref which keeps the inline TLS in the Route in sync. That could be enough for some people.
  - smarterclayton 5y ago
    For context, the reason Routes were designed to inline cert info (about 6-12mo before ingress) was that ingress controllers (which run on nodes) having the ability to read all secrets (which means the ingress controller has to be considered to be as powerful as any ServiceAccount on the cluster) was too scary. The alternative chosen was to prevent ingress controllers from seeing any secret but the ones the user decides to expose via the route. Later on, the pattern of referencing secrets in extensions became more common, and things like the NodeAuthorizer (which allows nodes to only read the secrets associated with the pods scheduled onto them) demonstrated a possible different pattern we could have chosen to implement (although nothing that can be efficiently implemented without changing kube itself today). Agree routes should have added a ref - that was feedback informing Gateway API, and once that hits GA and provides the best of both routes and ingress, we would probably suggest using that instead. Routes is mostly frozen for all but critical new features now though so we can ensure Gateway has everything we need to replace it while still providing the necessary forward compat. I would treat routes as sensitive. Note that within a namespace there is minimal cross user security (not part of the kube / openshift threat model), so giving namespace read access to a specific set of routes and only infrastructure users access to all routes, OR using a wildcard cert on the routers and keeping all key material out of the user’s space. On 4.x versions you could also create multiple ingress controllers and assign them to different namespaces, preventing leak between them.
- candiddevmike 5y ago
  Routes suck because they're basically the sameish API as ingress but now your developers have to maintain two kinds of manifests/helm templates depending on if they're targeting openshift or kubernetes.
  - nullify88 5y ago
    You can create Ingress resources on OpenShift and it will automatically create routes for you. You can customise the generated Route by using annotations on the Ingress resource. This has worked well for us because not all helm charts are OpenShift friendly but they usually do allow customising the ingress resource with annotations or we patch it in via Kustomize. https://docs.openshift.com/container-platform/4.8/networking/routes/route-configuration.html#nw-ingress-creating-a-route-via-an-ingress_route-configuration https://docs.openshift.com/container-platform/4.8/networking...
  - smarterclayton 5y ago
    When Gateway API lands we definitely plan to streamline everyone moving to that (since it’s the superset of Routes and Ingress and supports the features we didn’t want to lose going back to ingress). That should help. We have considered having a controller that mirrors both routes and ingress to a gateway http route (since gateway is similar to routes), but plans aren’t finalized yet.

blinkingled 5y ago

Have you tried running istio for example in an enterprise env - they needed you to install a RH specific older version and IIRC that wasn't just for support. I could list some more things if I recalled hard enough.

m3adow 5y ago

> like routes which are terrible Huh, interesting. What do you not like about routes? My team is providing an IaaS solution for internal developers in my company and a lot developers seemes to have less problems with Openshifts service exposition abstraction (Routes) in contrast to pure Kubernetes.

freedomben 5y ago
Routes are of the biggest things I miss when I'm on vanilla K8s. I don't see how anybody could prefer Ingress to Routes, but to each their own.
nullify88 5y ago
A big inconvenience is that for HTTP2 Routes or edge / re-encrypt Routes with a custom TLS certificate, the TLS certificate and keys must be inline in the Route resource instead of referencing a secret like Ingress resources do. I think this is a big oversight where Routes mix secrets and Ingress configuration together. It makes GitOps annoying because I don't want to treat the whole Route resource as a secret that needs to be encrypted or stored in vault. Do I also then treat Route resources as sensitive and deny some users access on the account they could contain private keys? I also have to worry about keeping the route updated before certificates expire instead of having it taken care of by cert-manager. So we use Traefik's IngressRoute.
- m3adow 5y ago
  That I can relate to. For us, there's only one dev team which uses HTTP2 (financial industry, so HTTP2 is still seen as "new = beta") and encountered that problem. I have no idea how they solved it though.
  - nullify88 5y ago
    FWIW, although I've known for a while that OpenShift coverts ingress resources to routes, I just found out that the Ingress Controller sets up a watch on the secretref which keeps the inline TLS in the Route in sync. That could be enough for some people.
- smarterclayton 5y ago
  For context, the reason Routes were designed to inline cert info (about 6-12mo before ingress) was that ingress controllers (which run on nodes) having the ability to read all secrets (which means the ingress controller has to be considered to be as powerful as any ServiceAccount on the cluster) was too scary. The alternative chosen was to prevent ingress controllers from seeing any secret but the ones the user decides to expose via the route. Later on, the pattern of referencing secrets in extensions became more common, and things like the NodeAuthorizer (which allows nodes to only read the secrets associated with the pods scheduled onto them) demonstrated a possible different pattern we could have chosen to implement (although nothing that can be efficiently implemented without changing kube itself today). Agree routes should have added a ref - that was feedback informing Gateway API, and once that hits GA and provides the best of both routes and ingress, we would probably suggest using that instead. Routes is mostly frozen for all but critical new features now though so we can ensure Gateway has everything we need to replace it while still providing the necessary forward compat. I would treat routes as sensitive. Note that within a namespace there is minimal cross user security (not part of the kube / openshift threat model), so giving namespace read access to a specific set of routes and only infrastructure users access to all routes, OR using a wildcard cert on the routers and keeping all key material out of the user’s space. On 4.x versions you could also create multiple ingress controllers and assign them to different namespaces, preventing leak between them.
candiddevmike 5y ago
Routes suck because they're basically the sameish API as ingress but now your developers have to maintain two kinds of manifests/helm templates depending on if they're targeting openshift or kubernetes.
- nullify88 5y ago
  You can create Ingress resources on OpenShift and it will automatically create routes for you. You can customise the generated Route by using annotations on the Ingress resource. This has worked well for us because not all helm charts are OpenShift friendly but they usually do allow customising the ingress resource with annotations or we patch it in via Kustomize. https://docs.openshift.com/container-platform/4.8/networking/routes/route-configuration.html#nw-ingress-creating-a-route-via-an-ingress_route-configuration https://docs.openshift.com/container-platform/4.8/networking...
- smarterclayton 5y ago
  When Gateway API lands we definitely plan to streamline everyone moving to that (since it’s the superset of Routes and Ingress and supports the features we didn’t want to lose going back to ingress). That should help. We have considered having a controller that mirrors both routes and ingress to a gateway http route (since gateway is similar to routes), but plans aren’t finalized yet.