3 ms·

There is also kubespray Ansible provisioning of Kubernetes, kops and kubeadm. Rolling Kubernetes on your own is quite hard though especially the networking par

by acd 7y ago

There is also kubespray Ansible provisioning of Kubernetes, kops and kubeadm. Rolling Kubernetes on your own is quite hard though especially the networking part so there should be a market for a company helping out with provisioning. Using public cloud can be very expensive for some work loads which were not written for the cloud due to higher resource usage. It is still quite hard to forecast cost of the cloud due to a lot of moving parts and micro billing for each item. With traditional service providers it can be easier to budget for the service expenses.

djsumdog 7y ago
I did a post on installing DC/OS a few years back. I was going to follow up with a K8s post, but between the official Ansible scripts, the (at the time) very alpha kubeadm, broken pod logging and WeaveNet that would constantly fail with no logs, I ended up giving up on that post: https://penguindreams.org/blog/installing-mesosphere-dcos-on-small-digital-ocean-droplets/ https://penguindreams.org/blog/installing-mesosphere-dcos-on... I remember the manual-ish server provisioning days of Autoyast and RedHat's Kickstart. It's not that difficult to bootstrap physical servers to get them to the point of being nodes in a Docker scheduling system. Rancher is another great tool for setting up K8s (which you can run against physical nodes once you get them past the initial install/bootstrap). I worked at one place where they took a hybrid approach. They had DC/OS running in the local VMWare data center (which we were migrating to from AWS to save costs), but there were nodes that could scale to AWS as well and you could label your deployments for which data center you needed them to run in. A lot of high volume loads we were able to move back to our data center, but we also had a kick-ass platform team and our AWS bill was well over $150k/month. The trouble is it's almost impossible to do a real cost comparison. There are just too many factors, and when you start going AWS, GCE or Azure, you start using all their other managed services that may not have self-hosted replacements, not to mention off-setting the costs of setting up master/slaves, backups, snapshots, redundancy and other operations tasks that come by default or with an additional price from a hosted solution.
ljm 7y ago
From my limited experience, you'd save a lot on the cloud if you replaced the default IngressController. Google Cloud, for instance, will spin up a fresh load balancer for every service by default and those cost money. Of course, it's not an easy task as a beginner to use nginx or traefik in its place, and to handle the complexity of that deploy.
- YawningAngel 7y ago
  Approximately 100% of the GKE users I know about use a reverse proxy behind the GCE Ingress. FWIW I'm not especially ops-y and configuring ambassador took me about a day.
neop1x 7y ago
It is better now with kubeadm. There are still lots of things to learn and lots of breaks which can happen but after months of learning I now have my hown cluster and it works perfectly. It mounts NFS store, also use local storage and I plan to play with OpenEBS. Nginx ingress + certbot (let's encrypt auto-renowal) works fine. Logging can be done with Fluentd and Graylog. Prometheus for performance measures. MetalLB as load-balancing replacement on L2. Calico for networking and networkPolicies. Cloud providers definitely have some added benefits, less friction. But 'do-it-yourself' is well possible as wrll and it helps you to understand more individual pieces K8S is made of and how to debug those.

by acd 7y ago

djsumdog 7y ago

I did a post on installing DC/OS a few years back. I was going to follow up with a K8s post, but between the official Ansible scripts, the (at the time) very alpha kubeadm, broken pod logging and WeaveNet that would constantly fail with no logs, I ended up giving up on that post: https://penguindreams.org/blog/installing-mesosphere-dcos-on-small-digital-ocean-droplets/ https://penguindreams.org/blog/installing-mesosphere-dcos-on... I remember the manual-ish server provisioning days of Autoyast and RedHat's Kickstart. It's not that difficult to bootstrap physical servers to get them to the point of being nodes in a Docker scheduling system. Rancher is another great tool for setting up K8s (which you can run against physical nodes once you get them past the initial install/bootstrap). I worked at one place where they took a hybrid approach. They had DC/OS running in the local VMWare data center (which we were migrating to from AWS to save costs), but there were nodes that could scale to AWS as well and you could label your deployments for which data center you needed them to run in. A lot of high volume loads we were able to move back to our data center, but we also had a kick-ass platform team and our AWS bill was well over $150k/month. The trouble is it's almost impossible to do a real cost comparison. There are just too many factors, and when you start going AWS, GCE or Azure, you start using all their other managed services that may not have self-hosted replacements, not to mention off-setting the costs of setting up master/slaves, backups, snapshots, redundancy and other operations tasks that come by default or with an additional price from a hosted solution.

ljm 7y ago

From my limited experience, you'd save a lot on the cloud if you replaced the default IngressController. Google Cloud, for instance, will spin up a fresh load balancer for every service by default and those cost money. Of course, it's not an easy task as a beginner to use nginx or traefik in its place, and to handle the complexity of that deploy.

YawningAngel 7y ago
Approximately 100% of the GKE users I know about use a reverse proxy behind the GCE Ingress. FWIW I'm not especially ops-y and configuring ambassador took me about a day.

neop1x 7y ago

It is better now with kubeadm. There are still lots of things to learn and lots of breaks which can happen but after months of learning I now have my hown cluster and it works perfectly. It mounts NFS store, also use local storage and I plan to play with OpenEBS. Nginx ingress + certbot (let's encrypt auto-renowal) works fine. Logging can be done with Fluentd and Graylog. Prometheus for performance measures. MetalLB as load-balancing replacement on L2. Calico for networking and networkPolicies. Cloud providers definitely have some added benefits, less friction. But 'do-it-yourself' is well possible as wrll and it helps you to understand more individual pieces K8S is made of and how to debug those.