7 ms·

I don't mean to hijack the thread, sincere concern. I've been trying to build a minimal kubernetes cluster in our lab to see what it would take to host this ki

I don't mean to hijack the thread, sincere concern. I've been trying to build a minimal kubernetes cluster in our lab to see what it would take to host this kind of infrastructure. It's not clear if we are allowed to use the public cloud yet. (We are a bank, yeah, I know) I've tried, at least: - kubeadm - rancher - canonical kubernetes - canonical kubernetes core - some random internet recipes And for some IaaS: - cloudfoundry - openstack - cloudstack - opennebula - ganeti Not one has worked out of the box in our environment. Every single one expects to have a direct connection to the internet. Any proxy in the middle creates havoc. I've been able to hammer some of this solutions until the cluster started and had some pods or VMs running, but it feels like this are not ready for production or not for 'secure' on-premise deployment.

cfontes 8y ago
https://github.com/apprenda/kismatic https://github.com/apprenda/kismatic It just works...
- madmulita 8y ago
  I have one Firefox tab opened with that repo, haven't tried it yet in part because I'm kind of loosing hope. Starting to believe we'll have to build something ourselves.
  - bryanlarsen 8y ago
    Yes, talking to the Apprenda guys I got the impression that's their specialty: air-gapped and otherwise isolated on-premises clusters. I'm sure they'd love to get you on a support contract.
    - ikester 8y ago
      I work for Apprenda. Sure, support contracts are an option but the Kismatic Enterprise Toolkit (KET) is completely Open Source and free. And you can get community support via the Slack channel: kismatic.slack.com
rb808 8y ago
openshift works for us.
- madmulita 8y ago
  I forgot to mention Openshift. The deployment worked, I guess, because we had something similar to an `iso` locally to do the full install. After that, if you are not using a local registry, you get to some of the same problems I mentioned in another response. Another problem we saw with Openshift was that it was not clear how we would be able to integrate the networking with our legacy infrastructure. If everything is working inside the cluster or, the rest of your infrastructure is kind of SOA, I guess it wouldn't be a problem, but as I mentioned... we are a bank :D
  - rb808 8y ago
    I didn't mention we have a dedicated team for this infrastructure that makes it work and who can help dev teams. Maybe that is a requirement.
    - madmulita 8y ago
      Hehe, I'm part of that team here, that's why we are trying to do it ourselves.
  - organsnyder 8y ago
    We had Red Hat consultants here for a couple of months helping our infra teams to get it stood up and functioning in our environment. It's far from trivial, especially once enterprise weirdness is figured in (I'm in healthcare, which I imaging is fairly similar to banking in this regard).
geofft 8y ago
I work on the OpenStack deployment for a hedge fund where all internet access goes through a proxy. (We have the same concern about public cloud although we're slowly figuring it out.) The initial deployment predated me, but there's nothing in our config that does anything special with OpenStack and internet access, and it works. We also have a Kubernetes deployment that also (AFAIK) just works. We're not able to `docker pull` very easily because of the proxy, but if we generate our own images they work fine. What is trying to connect to the public internet? (If you mean that they expect a direct connection between client machines and servers, then, yes, they probably do and that's not a scenario I've run into personally.)
- madmulita 8y ago
  The Openstack deployments first failed while trying to download different pieces of sotware. As each part/script seems to use a different tool to download or get packages, we had to chase each one to find where the proxies should be configured. Once the cluster started, again, you have to find which tool is trying to use the proxy and now can't connect with the rest of the cluster. Sorry I can't be more specific, I gave up with this some months ago. I don't think we have the scale for Openstack. Same problems, maybe in different steps, occurred with the other tools. To solve the `docker pull` problem, we are trying with Nexus to proxy the global registry. But given that some of the scripting/tools that deploy kubernetes do some magic starting things that we can't directly control, we can't configure docker to pull from a proxy (not that we have found how to tell the stupid thing how to avoid going to the global registry and only use our proxy/registry). Once again, when we finally managed to pull those images, we have to guess how to unconfigure proxies to be able to fully connect the cluster. The connection between clients and servers is another concern we haven't yet got to, but we can see it won't be a simple solution.
  - geofft 8y ago
    (Oh, I don't particularly recommend OpenStack unless you know you're a fit for its scale and also the specific things it's good at, yes.) I would expect some installer shell script not to work well, yes. If you can mirror repositories internally (or configure apt/yum/whatever to work with the proxy), that should be fine... we've deployed one OpenStack instance using (more or less) pip and one using Debian packages. Nothing in our running infrastructure attempts to connect to the external internet.
peterwwillis 8y ago
If you get to do an AWS trial, use Kops. It works really well. > Not one has worked out of the box in our environment. Every single one expects to have a direct connection to the internet. Any proxy in the middle creates havoc. Same. I had to go home and do half the setup disconnected from the VPN so I could download anything without the proxy screwing it up. Something about the google storage servers on IPv4 screws with connections. I'm still waiting for the god security team to fix it.
- madmulita 8y ago
  Yeah, at home or in my laptop I have everyone working (not that each worked out of the box :D )
cmdkeen 8y ago
Rancher has an "air gap" installation option, though requires you to be running your own on-premise registry (which you'll be doing if you're not sure you can use the public cloud. I'm busy deploying 1.6 at a financial institution, and thinking about the 2.1 upgrade to Kubernetes. The Rancher team have been great so far. https://rancher.com/docs/rancher/v2.x/en/installation/air-gap-installation/ https://rancher.com/docs/rancher/v2.x/en/installation/air-ga...
- madmulita 8y ago
  Noted. I didn't see this option. Thanks!
joseph 8y ago
I created https://github.com/cloudboss/keights https://github.com/cloudboss/keights because of the same issue, though it is for AWS only. I've been using it in an air gapped environment and only had to tweak two of the systemd unit dropins in the master CloudFormation template to add a proxy environment variables. The only reason I needed to do that is because two of the AWS services that are used don't yet have private VPC endpoints (cloudformation and autoscaling).
readams 8y ago
We run it in our proxy-only environment pretty regularly. You want docker to have proxy settings but not kubernetes. We use kubeadm and kubespray to install upstream. OpenShift is really broken with a proxy though and requires manual intervention to fix after the install, and even then not everything works.
ownagefool 8y ago
I've did this. It's because the kubelet is trying to downloading static pods (i.e. docker containers) from the internet. If you're using coreos, it's likely rkt is trying to download flannel (if you're using it) and the hyperkube container from the internet too. Good news, all this is configurable. Bad news, it's no longer really a turnkey solution. For the most part, bootstrapping kubernetes just from the hyperkube isn't that difficult. You just have to change the static manifests for the apiserver, controller, scheduler and the proxy to point to your private registry. You then need to change the kubelet(s) to use a private registry for the --pod-infra-container-image. This should probably be enough to get you up and running. With kubeadm (1.8+) you can use imageRepository: <private-registry> in kubeadm config file.
atgreen 8y ago
I work in the field at Red Hat, specifically with the banks (where you are, it seems). We get OpenShift, OpenStack, and more running in prod in both air-gapped, and ssl-proxied environments with full re-encrypt. Reach out if you need help!
spicyusername 8y ago
Here is a guide written by Redhat Consulting for installing a OpenShift cluster in a disconnected environment. http://v1.uncontained.io/playbooks/installation/#design-for-disconnected-environments http://v1.uncontained.io/playbooks/installation/#design-for-... This could be used to deploy OpenShift Origin, the upstream of the enterprise version
jacques_chester 8y ago
Like most of the others mentioned in your list, Cloud Foundry is designed to operate in disconnected environments. I worked on buildpacks in 2014, part of my job was to ensure that it worked in disconnected environments. These days every single version of Pivotal Application Service and all of our other software products are tested together in public and private cloud configurations. For every single release[0]. We already have many large private installations at the largest banks, insurers, telcos, healthcare and automotive companies in the world. And we are able to connect you with your peers in the industry to get their views on whether to pick Cloud Foundry or an alternative. I'm happy to put you in touch with our field folks (my email is jchester@pivotal.io) if you like. [0] https://content.pivotal.io/blog/you-deserve-a-continuously-integrated-platform-here-s-why-it-matters https://content.pivotal.io/blog/you-deserve-a-continuously-i...
elsonrodriguez 8y ago
If it's a non-transparent proxy, you're basically going to be fighting ghosts forever in regards to environment variables in/out of containers. Start the conversation now with your networking folks about a more transparent setup.

cfontes 8y ago

https://github.com/apprenda/kismatic https://github.com/apprenda/kismatic It just works...

madmulita 8y ago
I have one Firefox tab opened with that repo, haven't tried it yet in part because I'm kind of loosing hope. Starting to believe we'll have to build something ourselves.
- bryanlarsen 8y ago
  Yes, talking to the Apprenda guys I got the impression that's their specialty: air-gapped and otherwise isolated on-premises clusters. I'm sure they'd love to get you on a support contract.
  - ikester 8y ago
    I work for Apprenda. Sure, support contracts are an option but the Kismatic Enterprise Toolkit (KET) is completely Open Source and free. And you can get community support via the Slack channel: kismatic.slack.com

rb808 8y ago

openshift works for us.

madmulita 8y ago
I forgot to mention Openshift. The deployment worked, I guess, because we had something similar to an `iso` locally to do the full install. After that, if you are not using a local registry, you get to some of the same problems I mentioned in another response. Another problem we saw with Openshift was that it was not clear how we would be able to integrate the networking with our legacy infrastructure. If everything is working inside the cluster or, the rest of your infrastructure is kind of SOA, I guess it wouldn't be a problem, but as I mentioned... we are a bank :D
- rb808 8y ago
  I didn't mention we have a dedicated team for this infrastructure that makes it work and who can help dev teams. Maybe that is a requirement.
  - madmulita 8y ago
    Hehe, I'm part of that team here, that's why we are trying to do it ourselves.
- organsnyder 8y ago
  We had Red Hat consultants here for a couple of months helping our infra teams to get it stood up and functioning in our environment. It's far from trivial, especially once enterprise weirdness is figured in (I'm in healthcare, which I imaging is fairly similar to banking in this regard).

geofft 8y ago

I work on the OpenStack deployment for a hedge fund where all internet access goes through a proxy. (We have the same concern about public cloud although we're slowly figuring it out.) The initial deployment predated me, but there's nothing in our config that does anything special with OpenStack and internet access, and it works. We also have a Kubernetes deployment that also (AFAIK) just works. We're not able to `docker pull` very easily because of the proxy, but if we generate our own images they work fine. What is trying to connect to the public internet? (If you mean that they expect a direct connection between client machines and servers, then, yes, they probably do and that's not a scenario I've run into personally.)

madmulita 8y ago
The Openstack deployments first failed while trying to download different pieces of sotware. As each part/script seems to use a different tool to download or get packages, we had to chase each one to find where the proxies should be configured. Once the cluster started, again, you have to find which tool is trying to use the proxy and now can't connect with the rest of the cluster. Sorry I can't be more specific, I gave up with this some months ago. I don't think we have the scale for Openstack. Same problems, maybe in different steps, occurred with the other tools. To solve the `docker pull` problem, we are trying with Nexus to proxy the global registry. But given that some of the scripting/tools that deploy kubernetes do some magic starting things that we can't directly control, we can't configure docker to pull from a proxy (not that we have found how to tell the stupid thing how to avoid going to the global registry and only use our proxy/registry). Once again, when we finally managed to pull those images, we have to guess how to unconfigure proxies to be able to fully connect the cluster. The connection between clients and servers is another concern we haven't yet got to, but we can see it won't be a simple solution.
- geofft 8y ago
  (Oh, I don't particularly recommend OpenStack unless you know you're a fit for its scale and also the specific things it's good at, yes.) I would expect some installer shell script not to work well, yes. If you can mirror repositories internally (or configure apt/yum/whatever to work with the proxy), that should be fine... we've deployed one OpenStack instance using (more or less) pip and one using Debian packages. Nothing in our running infrastructure attempts to connect to the external internet.

peterwwillis 8y ago

If you get to do an AWS trial, use Kops. It works really well. > Not one has worked out of the box in our environment. Every single one expects to have a direct connection to the internet. Any proxy in the middle creates havoc. Same. I had to go home and do half the setup disconnected from the VPN so I could download anything without the proxy screwing it up. Something about the google storage servers on IPv4 screws with connections. I'm still waiting for the god security team to fix it.

madmulita 8y ago
Yeah, at home or in my laptop I have everyone working (not that each worked out of the box :D )

cmdkeen 8y ago

Rancher has an "air gap" installation option, though requires you to be running your own on-premise registry (which you'll be doing if you're not sure you can use the public cloud. I'm busy deploying 1.6 at a financial institution, and thinking about the 2.1 upgrade to Kubernetes. The Rancher team have been great so far. https://rancher.com/docs/rancher/v2.x/en/installation/air-gap-installation/ https://rancher.com/docs/rancher/v2.x/en/installation/air-ga...

madmulita 8y ago
Noted. I didn't see this option. Thanks!

joseph 8y ago

I created https://github.com/cloudboss/keights https://github.com/cloudboss/keights because of the same issue, though it is for AWS only. I've been using it in an air gapped environment and only had to tweak two of the systemd unit dropins in the master CloudFormation template to add a proxy environment variables. The only reason I needed to do that is because two of the AWS services that are used don't yet have private VPC endpoints (cloudformation and autoscaling).

readams 8y ago

We run it in our proxy-only environment pretty regularly. You want docker to have proxy settings but not kubernetes. We use kubeadm and kubespray to install upstream. OpenShift is really broken with a proxy though and requires manual intervention to fix after the install, and even then not everything works.

ownagefool 8y ago

I've did this. It's because the kubelet is trying to downloading static pods (i.e. docker containers) from the internet. If you're using coreos, it's likely rkt is trying to download flannel (if you're using it) and the hyperkube container from the internet too. Good news, all this is configurable. Bad news, it's no longer really a turnkey solution. For the most part, bootstrapping kubernetes just from the hyperkube isn't that difficult. You just have to change the static manifests for the apiserver, controller, scheduler and the proxy to point to your private registry. You then need to change the kubelet(s) to use a private registry for the --pod-infra-container-image. This should probably be enough to get you up and running. With kubeadm (1.8+) you can use imageRepository: <private-registry> in kubeadm config file.

atgreen 8y ago

I work in the field at Red Hat, specifically with the banks (where you are, it seems). We get OpenShift, OpenStack, and more running in prod in both air-gapped, and ssl-proxied environments with full re-encrypt. Reach out if you need help!

spicyusername 8y ago

Here is a guide written by Redhat Consulting for installing a OpenShift cluster in a disconnected environment. http://v1.uncontained.io/playbooks/installation/#design-for-disconnected-environments http://v1.uncontained.io/playbooks/installation/#design-for-... This could be used to deploy OpenShift Origin, the upstream of the enterprise version

jacques_chester 8y ago

Like most of the others mentioned in your list, Cloud Foundry is designed to operate in disconnected environments. I worked on buildpacks in 2014, part of my job was to ensure that it worked in disconnected environments. These days every single version of Pivotal Application Service and all of our other software products are tested together in public and private cloud configurations. For every single release[0]. We already have many large private installations at the largest banks, insurers, telcos, healthcare and automotive companies in the world. And we are able to connect you with your peers in the industry to get their views on whether to pick Cloud Foundry or an alternative. I'm happy to put you in touch with our field folks (my email is jchester@pivotal.io) if you like. [0] https://content.pivotal.io/blog/you-deserve-a-continuously-integrated-platform-here-s-why-it-matters https://content.pivotal.io/blog/you-deserve-a-continuously-i...

elsonrodriguez 8y ago

If it's a non-transparent proxy, you're basically going to be fighting ghosts forever in regards to environment variables in/out of containers. Start the conversation now with your networking folks about a more transparent setup.