4 ms·

My last company had hundreds possibly thousands of LXC containers, and we orchestrated everything via saltstack (which is similar to ansible or puppet if you ar

My last company had hundreds possibly thousands of LXC containers, and we orchestrated everything via saltstack (which is similar to ansible or puppet if you aren't familiar). The justification was that we needed our SaaS to also work on-prem for financial companies and government entities, and thus we could not count on kubernetes or any specific cloud vendor to be available, so we rolled our own orchestration built on ubuntu hosts and (for some stupid reasons) centos LXC containers running on top of those. LXC is a good system and all, but what we were doing with it was a total nightmare, we were basically trying to reinvent kubernetes using traditional configuration management and LXC with two flavors of linux and a million dependencies.

thedougd 4y ago
I can't speak to supporting these, but there are a few vendors that offer k8s in a box for this exact use case. Replicated is the first that comes to my mind. I've used this one as a customer. It worked fine but felt it necessary to do a bit of poking under the hood to help it understand things like our AZs in a private cloud: https://www.replicated.com/kubernetes/ https://www.replicated.com/kubernetes/
- locusofself 4y ago
  ya I'm sure there are some good options especially nowadays. But now think about getting signoff to use that software at VISA, US Bank, or the US military, for extremely sensitive info. Sometimes its just the policies that limit what tools you can use.
  - thedougd 4y ago
    I hear you. I was at one of those banks and everything was miserable. For reference, Mulesoft uses (or use to) use Replicated for one of their products that is popular with banks.
0xbadcafebee 4y ago
> we needed our SaaS to also work on-prem for financial companies and government entities, and thus we could not count on kubernetes or any specific cloud vendor ....? kubernetes is not a cloud vendor, it runs on any Linux distribution, and it's FOSS.... ?!
- hobs 4y ago
  And that's why the sentence contains an or - because they are not a cloud provider but you cant guarantee a local corp is gonna run kube.
  - tonnydourado 4y ago
    A previous employer had the same issue, and solved it by writing the application to kubernetes and deploying minikube to the clients that didn't have their own kubernetes cluster. To be fair, the situation ended up being: bunch of clients running minikube, 1 major client with their own kubernetes cluster, and a lot of other clients using the multi tenant, cloud hosted, offering. Made it a bit of a pain to support that one client with a cluster we could not control, but it was one of our biggest clients, so, them the breaks.
- locusofself 4y ago
  I mean that bringing kubernetes to an on-premises customer, just to run our software, was (probably correctly) deemed to be too much complexity. In the end it was complex enough as it was. 5 years ago kubernetes on-prem was not particularly easy, it probably still isn't, and doing it for a customer who just wants to run our software is a lot of support burden.
  - ncrmro 4y ago
    K3os makes on prem quite nice these days

thedougd 4y ago

I can't speak to supporting these, but there are a few vendors that offer k8s in a box for this exact use case. Replicated is the first that comes to my mind. I've used this one as a customer. It worked fine but felt it necessary to do a bit of poking under the hood to help it understand things like our AZs in a private cloud: https://www.replicated.com/kubernetes/ https://www.replicated.com/kubernetes/

locusofself 4y ago
ya I'm sure there are some good options especially nowadays. But now think about getting signoff to use that software at VISA, US Bank, or the US military, for extremely sensitive info. Sometimes its just the policies that limit what tools you can use.
- thedougd 4y ago
  I hear you. I was at one of those banks and everything was miserable. For reference, Mulesoft uses (or use to) use Replicated for one of their products that is popular with banks.

0xbadcafebee 4y ago

> we needed our SaaS to also work on-prem for financial companies and government entities, and thus we could not count on kubernetes or any specific cloud vendor ....? kubernetes is not a cloud vendor, it runs on any Linux distribution, and it's FOSS.... ?!

hobs 4y ago
And that's why the sentence contains an or - because they are not a cloud provider but you cant guarantee a local corp is gonna run kube.
- tonnydourado 4y ago
  A previous employer had the same issue, and solved it by writing the application to kubernetes and deploying minikube to the clients that didn't have their own kubernetes cluster. To be fair, the situation ended up being: bunch of clients running minikube, 1 major client with their own kubernetes cluster, and a lot of other clients using the multi tenant, cloud hosted, offering. Made it a bit of a pain to support that one client with a cluster we could not control, but it was one of our biggest clients, so, them the breaks.
locusofself 4y ago
I mean that bringing kubernetes to an on-premises customer, just to run our software, was (probably correctly) deemed to be too much complexity. In the end it was complex enough as it was. 5 years ago kubernetes on-prem was not particularly easy, it probably still isn't, and doing it for a customer who just wants to run our software is a lot of support burden.
- ncrmro 4y ago
  K3os makes on prem quite nice these days