7 ms·

I doubt if someone wanting to run an app is the target audience. A lot of companies would love to have something like Heroku that could be hosted internally. A

by joseph 6y ago

I doubt if someone wanting to run an app is the target audience. A lot of companies would love to have something like Heroku that could be hosted internally. A platform team hosts it and development teams consume it. As it stands, they are stuck hand rolling their own poor implementations. Lots of wasted person-hours are happening in this space due to a lack of good, stable solutions that won't disappear (I don't know if this one qualifies).

bassdropvroom 6y ago
> As it stands, they are stuck hand rolling their own poor implementations It's either that or it's the monster that is Kubernetes.
- sofixa 6y ago
  Multiple projects like OpenFaaS and Knative try to bridge that moat. And then there's also Nomad, which is drastically simpler than k8s. Not Heroku-easier, but closer to docker-compose than Kubernetes. Self-plugging my article on the Nomad vs Kubernetes subject: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...
  - bassdropvroom 6y ago
    I have not had hands on experience with nomad, but as I understand it, it still requires a lot of plumbing. Something as simple as service discovery isn't, and requires a completely separate consul cluster. Similarly, there is no secrets management, which requires a vault cluster, which in turn requires a it's own consul cluster. I'm not sure what config management options there are. So now you're stuck with 3 consul clusters, a vault cluster, whatever you choose for config management, and a nomad cluster. Feels like you didn't gain much from the simplicity of nomad. In addition to that, knowing Hashicorp's pricing, I bet that set up would run you north of a million a year for enterprise setup.
    - sofixa 6y ago
      Nope. Nomad relies on Consul for Service Discovery and K/V storage, and Vault for secrets, indeed ( Vault can use a variety of backends, including an integrated Raft-based one, Consul, object storage, etc.). One tool that does one thing well, which integrates with other tools that their thing well. I vastly prefer having three simple Raft-based clusters to manage than the "everything and the kitchen sink" approach Kubernetes takes, with results like base64 encoded for "secrets". And as someone doing both, Nomad+Consul+Vault are drastically easier on day one and day two. They're also usable outside of Nomad ( you can have bare metal machines outside of Nomad using Vault secrets and Consul for SD and K/V), and you can link multiple regions together. You do indeed need some basic config management to configure the clusters. Ansible seems to have won that race sadly, and there are available playbooks. > In addition to that, knowing Hashicorp's pricing, I bet that set up would run you north of a million a year for enterprise setup They've changed up their pricing structure and there are more tiers and add-ons, so i doubt it ( basic Vault cluster was in the 5 figured range per year), if you need support. It's not like enterprise support for Kubernetes would come cheap either, especially if you do it the recommended way with multiple clusters and all that.
      - debarshri 6y ago
        You don't have to deal with operational aspects of consult, nomad and vault of that matter if you choose the managed kubernetes cluster of the cloud provider. If you are talking about container orchestration onpremise - the experience I had with kubernetes is terrible. There 1000 things that can go wrong. Onpremise, I would recommend using k3s which is super simple to setup or microk8s with comes with ubuntu 20.04. I am not sure what benefits you would get with nomad as compared to the lightweight kubernetes distros. One of the benefits of these lightweight solutions is that, you will basically find a helm chart for any serious application out there where as in nomad you might have to figure it out how to deploy. For instances deploying cassandra, you will find a helm chart, but nomad, you might find blog that does it or figure it out yourself. To my best of knowledge that how it is, may be I am wrong?
        sofixa 6y ago
  - linkdd 6y ago
    > And then there's also Nomad, which is drastically simpler than k8s That's normal, since Nomad is only about deployment, where k8s is about full cluster management.
    - debarshri 6y ago
      Is it though? I though nomad was also an orchestrator. Let me look up.
      - linkdd 6y ago
        https://www.nomadproject.io/docs/nomad-vs-kubernetes https://www.nomadproject.io/docs/nomad-vs-kubernetes > Kubernetes aims to provide all the features needed to run Docker-based applications including cluster management, scheduling, service discovery, monitoring, secrets management and more. > Nomad only aims to focus on cluster management and scheduling and is designed with the Unix philosophy of having a small scope while composing with tools like Consul for service discovery/service mesh and Vault for secret management.
      - sofixa 6y ago
        Yep, it's an orchestrator that can orchestrate pretty much anything ( Docker, QEMU, Firecracker, LXC, etc.)
        debarshri 6y ago
        Yup that's my understanding.
- mikepurvis 6y ago
  I'm still relatively green, so there are likely a bunch of Kubernetes nightmare scenarios I haven't encountered, but I recently stood up microk8s to provide workers for Jenkins and GitLab CI, and I thought the ergonomics of it were great— easy to get going, easy to deploy stuff with the integrated helm3, easy to access the dashboard and get metrics out of it. I'm sure there's still a gap to be bridged there between that and a PaaS which you literally just add as a git remote. But I don't think it's huge.
  - bassdropvroom 6y ago
    Microk8s is great for local development, but it's operational complexity doesn't come close to run k8s in a production environment. Habing been on both sides of the isle, in my opinion, K8s has great ux for consumers, but for is a nightmare for ops teams who maintain it. For a self-hosted version anyway.
    - mikepurvis 6y ago
      That's good to know— I've heard others say that it's fine if you just have a cluster of a few nodes, private cloud, that kind of thing, particularly if it's "throwaway" compute like CI workers, as opposed to something genuinely high availability. Now, all that said, Canonical certainly advertises microk8s as being production-ready, production-grade, and suitable for use in production environments, for example in [1]. It definitely seems like it's meant to be far more serious than, say, minikube, which explicitly is just for local development. Can you speak to specific limitations with microk8s, or point to resources which go into more depth on this? [1]: https://microk8s.io/high-availability https://microk8s.io/high-availability
      - debarshri 6y ago
        Great question, I have been research around the same topic for past 6-8 months. The problem is they have been advertising as production grade very recently. I am not sure what limitation you will hit. Having said that, k3s running in production has same issue that you will see in managed kubernetes cluster in a cloud provider.
joshmanders 6y ago
I've learned this too, and that's why at my company (https://primcloud.com https://primcloud.com) we're building obviously a PaaS for those who want Heroku/Netlify experience, but also building it in a way that we plan to package it up and offer an enterprise solution where you can install it on your own infrastructure, like GitHub Enterprise. This allows you to have the same experience but be in full control.
- debarshri 6y ago
  I love the way you are shamelessly plugging you product in the discussion. Is it container orchestration underneath? How does the app get deployed?
  - joshmanders 6y ago
    Haha not trying to shamelessly plug, but it's hard to talk about the subject including features we're building without actually mentioning it. Yes its container orchestration. We're built on top of kubernetes. Our idea of the enterprise version is just deploy your own kubernetes cluster, then install our helm chart or whatever and it bootstraps and sets up the platform on your cluster.
dustymcp 6y ago
I disagree most cloudservices give it Away for free like azure devops

by joseph 6y ago

bassdropvroom 6y ago

> As it stands, they are stuck hand rolling their own poor implementations It's either that or it's the monster that is Kubernetes.

sofixa 6y ago
Multiple projects like OpenFaaS and Knative try to bridge that moat. And then there's also Nomad, which is drastically simpler than k8s. Not Heroku-easier, but closer to docker-compose than Kubernetes. Self-plugging my article on the Nomad vs Kubernetes subject: https://atodorov.me/2021/02/27/why-you-should-take-a-look-at-nomad-before-jumping-on-kubernetes/ https://atodorov.me/2021/02/27/why-you-should-take-a-look-at...
- bassdropvroom 6y ago
  I have not had hands on experience with nomad, but as I understand it, it still requires a lot of plumbing. Something as simple as service discovery isn't, and requires a completely separate consul cluster. Similarly, there is no secrets management, which requires a vault cluster, which in turn requires a it's own consul cluster. I'm not sure what config management options there are. So now you're stuck with 3 consul clusters, a vault cluster, whatever you choose for config management, and a nomad cluster. Feels like you didn't gain much from the simplicity of nomad. In addition to that, knowing Hashicorp's pricing, I bet that set up would run you north of a million a year for enterprise setup.
  - sofixa 6y ago
    Nope. Nomad relies on Consul for Service Discovery and K/V storage, and Vault for secrets, indeed ( Vault can use a variety of backends, including an integrated Raft-based one, Consul, object storage, etc.). One tool that does one thing well, which integrates with other tools that their thing well. I vastly prefer having three simple Raft-based clusters to manage than the "everything and the kitchen sink" approach Kubernetes takes, with results like base64 encoded for "secrets". And as someone doing both, Nomad+Consul+Vault are drastically easier on day one and day two. They're also usable outside of Nomad ( you can have bare metal machines outside of Nomad using Vault secrets and Consul for SD and K/V), and you can link multiple regions together. You do indeed need some basic config management to configure the clusters. Ansible seems to have won that race sadly, and there are available playbooks. > In addition to that, knowing Hashicorp's pricing, I bet that set up would run you north of a million a year for enterprise setup They've changed up their pricing structure and there are more tiers and add-ons, so i doubt it ( basic Vault cluster was in the 5 figured range per year), if you need support. It's not like enterprise support for Kubernetes would come cheap either, especially if you do it the recommended way with multiple clusters and all that.
    - debarshri 6y ago
      You don't have to deal with operational aspects of consult, nomad and vault of that matter if you choose the managed kubernetes cluster of the cloud provider. If you are talking about container orchestration onpremise - the experience I had with kubernetes is terrible. There 1000 things that can go wrong. Onpremise, I would recommend using k3s which is super simple to setup or microk8s with comes with ubuntu 20.04. I am not sure what benefits you would get with nomad as compared to the lightweight kubernetes distros. One of the benefits of these lightweight solutions is that, you will basically find a helm chart for any serious application out there where as in nomad you might have to figure it out how to deploy. For instances deploying cassandra, you will find a helm chart, but nomad, you might find blog that does it or figure it out yourself. To my best of knowledge that how it is, may be I am wrong?
      - sofixa 6y ago
- linkdd 6y ago
  > And then there's also Nomad, which is drastically simpler than k8s That's normal, since Nomad is only about deployment, where k8s is about full cluster management.
  - debarshri 6y ago
    Is it though? I though nomad was also an orchestrator. Let me look up.
    - linkdd 6y ago
      https://www.nomadproject.io/docs/nomad-vs-kubernetes https://www.nomadproject.io/docs/nomad-vs-kubernetes > Kubernetes aims to provide all the features needed to run Docker-based applications including cluster management, scheduling, service discovery, monitoring, secrets management and more. > Nomad only aims to focus on cluster management and scheduling and is designed with the Unix philosophy of having a small scope while composing with tools like Consul for service discovery/service mesh and Vault for secret management.
    - sofixa 6y ago
      Yep, it's an orchestrator that can orchestrate pretty much anything ( Docker, QEMU, Firecracker, LXC, etc.)
      - debarshri 6y ago
        Yup that's my understanding.
mikepurvis 6y ago
I'm still relatively green, so there are likely a bunch of Kubernetes nightmare scenarios I haven't encountered, but I recently stood up microk8s to provide workers for Jenkins and GitLab CI, and I thought the ergonomics of it were great— easy to get going, easy to deploy stuff with the integrated helm3, easy to access the dashboard and get metrics out of it. I'm sure there's still a gap to be bridged there between that and a PaaS which you literally just add as a git remote. But I don't think it's huge.
- bassdropvroom 6y ago
  Microk8s is great for local development, but it's operational complexity doesn't come close to run k8s in a production environment. Habing been on both sides of the isle, in my opinion, K8s has great ux for consumers, but for is a nightmare for ops teams who maintain it. For a self-hosted version anyway.
  - mikepurvis 6y ago
    That's good to know— I've heard others say that it's fine if you just have a cluster of a few nodes, private cloud, that kind of thing, particularly if it's "throwaway" compute like CI workers, as opposed to something genuinely high availability. Now, all that said, Canonical certainly advertises microk8s as being production-ready, production-grade, and suitable for use in production environments, for example in [1]. It definitely seems like it's meant to be far more serious than, say, minikube, which explicitly is just for local development. Can you speak to specific limitations with microk8s, or point to resources which go into more depth on this? [1]: https://microk8s.io/high-availability https://microk8s.io/high-availability
    - debarshri 6y ago
      Great question, I have been research around the same topic for past 6-8 months. The problem is they have been advertising as production grade very recently. I am not sure what limitation you will hit. Having said that, k3s running in production has same issue that you will see in managed kubernetes cluster in a cloud provider.

joshmanders 6y ago

I've learned this too, and that's why at my company (https://primcloud.com https://primcloud.com) we're building obviously a PaaS for those who want Heroku/Netlify experience, but also building it in a way that we plan to package it up and offer an enterprise solution where you can install it on your own infrastructure, like GitHub Enterprise. This allows you to have the same experience but be in full control.

debarshri 6y ago
I love the way you are shamelessly plugging you product in the discussion. Is it container orchestration underneath? How does the app get deployed?
- joshmanders 6y ago
  Haha not trying to shamelessly plug, but it's hard to talk about the subject including features we're building without actually mentioning it. Yes its container orchestration. We're built on top of kubernetes. Our idea of the enterprise version is just deploy your own kubernetes cluster, then install our helm chart or whatever and it bootstraps and sets up the platform on your cluster.

dustymcp 6y ago

I disagree most cloudservices give it Away for free like azure devops