5 ms·

I suppose we have to assume that we should use the right tool for the right job, and all that. And I'm sure that the Kubernetes folk know what they're doing. Bu

I suppose we have to assume that we should use the right tool for the right job, and all that. And I'm sure that the Kubernetes folk know what they're doing. But I definitely think that it's too complicated without a cutting-edge Kubernetes expert in place to manage it. And even then, it's just a building block for a larger system. I've tried maybe half a dozen times to get started for relatively small workloads – let's say between 5 to 50 servers. There seems to be a lack of any good documentation on how to get started – sure, I can spin up a couple of pods on a cluster or whatever, but then actually taking that to an easy-to-deploy, load-balanced, publicly-accessible infrastructure seems to be much harder. It's maybe a lack of some kind of official, blessed version of a "this is how you should build a modern infrastructure from container to deployment" guide. There are so many different moving parts that it's hard to figure out what the optimal strategy is, and I've not found much to make it easier. Maybe there's actually a niche for a nice and simple "infrastructure in a box" kind of product. I'd love something like Heroku that I could run on my own bare metal, or on AWS – the various solutions I've tried were all lacking. In the end I've fallen back to using the Hashicorp stack of Nomad and Consul. This seems to work in a much simpler way that I can actually wrap my brain around – I get a nice cluster running balanced across machines, deploying and scaling my applications as required, and it was super easy to set up with good documentation. I've been nervous about this approach because of how Kubernetes-mad the industry seems to have gone – so maybe it'll be worth another look when there are some more comprehensive solutions in place that make it easier to get started!

pm90 8y ago
I’ve worked as a developer on 2 major managed k8s providers. Using k8s is easy but operating it is not. The biggest reason is that every cloud provider is using different underlying technologies to deploy k8s clusters. See for example all the different kinds of ingress controllers.
- geerlingguy 8y ago
  And it doesn’t help that docs sometimes mention how you do things assuming you’re using Google Cloud, other times with examples for Google, AWS, Azure, or bare metal, other times minikube or some other smaller scale wrapper, and other times it’s not clear what kind of platform the commands should ‘just work’ on as-written. I had a ‘fun’ time figuring out the basics of things like ingress controllers in a bare metal setup because of this mishmash of docs.
icebraining 8y ago
Aren't the K8S "distributions" like OpenShift and Rancher the infrastructure in a box?
jacques_chester 8y ago
> I'd love something like Heroku that I could run on my own bare metal, or on AWS – the various solutions I've tried were all lacking. I've worked on or near to Cloud Foundry for about 4 years now, and Heroku was one of the inspirations for it, along with Borg. Is there something in particular it didn't do well enough?
- mwcampbell 8y ago
  I'm not the GP. But if I were setting up infrastructure for a fledgling startup today, I'd want a PaaS-in-a-box that worked like this: Given these inputs: * A minimum of 3 Linux servers: These could be VMs or bare-metal servers. They're all in the same data center. They each have 1 public IP address. Other than that, there's no network between them. Each of these nodes has at least 4 GB of RAM and a healthy amount of SSD storage. Note that I'm perfectly comfortable manually provisioning a fixed number of nodes, and manually adding more if truly necessary, because predictable cost is important. Edit: The platform should assume that each node has a bare installation of Ubuntu or CentOS. Don't make me install something custom. And ideally, don't be picky about the kernel, because some dedicated server hosts provide their own. Edit 2: All nodes should be treated equally. Any of them may become the master when needed. There should be no dedicated master; all nodes should be available to run applications. After all, resources are tight, and I want to get the most out of those three servers. * An API for creating and updating DNS records, e.g. Amazon Route 53, DNSimple, etc. * Edit: An off-site object store (e.g. Amazon S3, Backblaze B2) where the cluster can automatically send backups of all durable storage. The PaaS-in-a-box should give me an installation script to run on each node. During installation, I provide my DNS API credentials and the domain(s) I want the platform to manage. Edit: I'd also provide credentials for the off-site backup object storage. And I forgot that when installing on the second and subsequent nodes, I'd provide the public IP of an existing node during installation. And that's it. The cluster then manages itself, distributing durable storage among the nodes using something like Ceph for file storage, and DB-level replication for the supported database(s). HTTP/HTTPS traffic can be load-balanced among nodes using round-robin DNS, with unhealthy nodes automatically removed from the DNS records by one of the remaining nodes. If I need to run an outward-facing non-HTTP service, I should be able to reserve one or more ports for it and run it in either a round-robin or active/standby configuration with automatic failover. Perhaps this is a tall order. Is it impossible to do this on top of something as last-decade as a handful of manually provisioned dedicated servers, with only the public network between them (but in the same data center)? I hope not, because that kind of server, while now out of fashion, is attractive to a company on a shoestring budget that nevertheless doesn't want to compromise performance. Anyway, if a product meeting these requirements exists, I don't yet know about it. Cloud Foundry certainly doesn't market itself for that kind of deployment.

pm90 8y ago

I’ve worked as a developer on 2 major managed k8s providers. Using k8s is easy but operating it is not. The biggest reason is that every cloud provider is using different underlying technologies to deploy k8s clusters. See for example all the different kinds of ingress controllers.

geerlingguy 8y ago
And it doesn’t help that docs sometimes mention how you do things assuming you’re using Google Cloud, other times with examples for Google, AWS, Azure, or bare metal, other times minikube or some other smaller scale wrapper, and other times it’s not clear what kind of platform the commands should ‘just work’ on as-written. I had a ‘fun’ time figuring out the basics of things like ingress controllers in a bare metal setup because of this mishmash of docs.

icebraining 8y ago

Aren't the K8S "distributions" like OpenShift and Rancher the infrastructure in a box?

jacques_chester 8y ago

> I'd love something like Heroku that I could run on my own bare metal, or on AWS – the various solutions I've tried were all lacking. I've worked on or near to Cloud Foundry for about 4 years now, and Heroku was one of the inspirations for it, along with Borg. Is there something in particular it didn't do well enough?

mwcampbell 8y ago
I'm not the GP. But if I were setting up infrastructure for a fledgling startup today, I'd want a PaaS-in-a-box that worked like this: Given these inputs: * A minimum of 3 Linux servers: These could be VMs or bare-metal servers. They're all in the same data center. They each have 1 public IP address. Other than that, there's no network between them. Each of these nodes has at least 4 GB of RAM and a healthy amount of SSD storage. Note that I'm perfectly comfortable manually provisioning a fixed number of nodes, and manually adding more if truly necessary, because predictable cost is important. Edit: The platform should assume that each node has a bare installation of Ubuntu or CentOS. Don't make me install something custom. And ideally, don't be picky about the kernel, because some dedicated server hosts provide their own. Edit 2: All nodes should be treated equally. Any of them may become the master when needed. There should be no dedicated master; all nodes should be available to run applications. After all, resources are tight, and I want to get the most out of those three servers. * An API for creating and updating DNS records, e.g. Amazon Route 53, DNSimple, etc. * Edit: An off-site object store (e.g. Amazon S3, Backblaze B2) where the cluster can automatically send backups of all durable storage. The PaaS-in-a-box should give me an installation script to run on each node. During installation, I provide my DNS API credentials and the domain(s) I want the platform to manage. Edit: I'd also provide credentials for the off-site backup object storage. And I forgot that when installing on the second and subsequent nodes, I'd provide the public IP of an existing node during installation. And that's it. The cluster then manages itself, distributing durable storage among the nodes using something like Ceph for file storage, and DB-level replication for the supported database(s). HTTP/HTTPS traffic can be load-balanced among nodes using round-robin DNS, with unhealthy nodes automatically removed from the DNS records by one of the remaining nodes. If I need to run an outward-facing non-HTTP service, I should be able to reserve one or more ports for it and run it in either a round-robin or active/standby configuration with automatic failover. Perhaps this is a tall order. Is it impossible to do this on top of something as last-decade as a handful of manually provisioned dedicated servers, with only the public network between them (but in the same data center)? I hope not, because that kind of server, while now out of fashion, is attractive to a company on a shoestring budget that nevertheless doesn't want to compromise performance. Anyway, if a product meeting these requirements exists, I don't yet know about it. Cloud Foundry certainly doesn't market itself for that kind of deployment.