8 ms·

I had an excellent time working with kubernetes and I am practically a one person company. Kubernetes frees my mind from so many things that now I hate to work

I had an excellent time working with kubernetes and I am practically a one person company. Kubernetes frees my mind from so many things that now I hate to work without it. Couple of those things include: - automated ssl - centralized logging - super easy scaling up and down (going from 1 instance to 2 instance is a headache manually) - centralized auth (have a service which doesnt have built in auth?) - super easy recovery (containers recreated/volumes attached automatically/apps just work unless you have db type of application which shouldn't be just restarted, which is rare) - Smooth CI/CD (gogs/jenkins/private docker image repository) As for the "execute query" example, why is it such a headache ? I just "kubectl ssh <container>" and I am in. > I know a bio scientist who spent Super obscure example. Kubernetes is definitely not for non-tech people. And I didn't pick k8s overnight. Spent a year doing playful POCs before deploying in a production environment. If the only thing thats stopping you from using k8s is the learning curve, I suggest you to go ahead and learn it. Its a huge boon.

bthornbury 8y ago
The learning curve is sharp. The amount of things happening you don't have awareness of is also worrisome (to me). Source: I am also using Kubernetes in production, migrating off it soon.
- austinjp 8y ago
  It would be interesting if you could briefly say your reasons for moving away from K8s and what you're moving to.
  - bthornbury 8y ago
    Sure. The largest reason is cost. My small deployment (3 nodes) is running around $100 / mo on AWS (That's my app, nginx, redis, and postgres). It doesn't even need 3 nodes, I don't recall if 3 is the minimum, but realistically I only need one (for now). For larger projects this is probably a non-issue. Second largest reason, is that really I have no idea what is going on on these nodes, and I probably never will. Magically my services run when I have the correct configurations. Not to say that's always a bad thing, but I've found it difficult to determine the default level of security as a result of this. A third reason is the learning curve. This is less of an issue because I've invested the time to learn already. But like, the first time I tried to get traffic ingressed was painful. As to what I'm moving to, I migrated one of my websites to a simple rsync + docker-compose setup and am pretty happy with it. In the past, I ran ansible on a 50 node cluster and it worked really well.
    - rconti 8y ago
      I'm not really clear on how moving off Kubernetes saves you money in this scenario. Seems like the most likely source of the cost is the AWS, which is the non-free part. I'm just learning k8s, so I feel you on the learning curve.
      - bthornbury 8y ago
        Kubernetes has a minimum node count. Moving to one node, saves cost, by a factor of 3. Not to mention all of the other resources it creates (load balancers).
sklivvz1971 8y ago
> As for the "execute query" example, why is it such a headache ? I just "kubectl ssh <container>" and I am in. It's slow as hell. > I had an excellent time working with kubernetes I'm happy for you, seriously, and I'm not claiming general validity of my experience. What I am disputing is the blog post (which does claim general validity).
reacharavindh 8y ago
Thanks for sharing your experience. I question myself whether it is necessary to hide my operational needs behind a behemoth of complexity like Kubernetes. The list of conveniences you mentioned sounds like magic you get from Kubernetes. What if there is a problem with any of them? Missing logs? Inappropriate scaling? Auth failures? or worse, failure of Auth system? Easy recovery? what if there were failures to checkpoint/backup/snapshot containers? CI/CD is good regardless of whether you use Kubernetes or not. EDIT: The question is, if you have any of these problems, why is it better to get your head into how Kubernetes deals with those operations & tools rather than dealing with well-defined unix tools that specialise in doing these jobs. syslog instead of however Kubernetes gathers the logs, reading FreeIPA docs instead of Kubernetes auth system logs? My point is, that to deal with all of the conveniences you mentioned, you need to know their details anyway. Why rely on Kubernetes abstraction if there is no such need? (I'm not trying to being snarky. I'm genuinely curious why you think it is a good idea. If you convince me otherwise, perhaps I would start adopting Kubernetes as well.) I run my cluster(I'm a sysadmin) with a couple of OpenBSD server that runs redundant DNS and DHCP. a CentOS 7 box that runs FreeIPA as central Auth. an OpenBSD server that acts as public facing SSH server. about 20 nodes, all provisioned using kickstart files, and then configured using Ansible. They run databases, web servers, batch jobs, Git, etc. A single server that runs ELK stack for log analysis. A single server that runs Prometheus for quick aggregated monitoring. Do you think I should switch over to Kubernetes for any benefits?
- merb 8y ago
  how often do you update your nodes?!
  - StudentStuff 8y ago
    It sounds like the prior commenter Ansiblized their environment, they likely update the 20+ machines often since its not much of an effort.
  - reacharavindh 8y ago
    All nodes are bare-metal servers that run CentOS 7, and are configured strictly via Ansible. If a node experiences a hardware failure, we just pick another spare server and run our Ansible playbook on it + a script for infrastructure changes like DNS & DHCP. Our workload is not strictly attached to physical resource. So, we have our updates scheduled for every week, with the condition that updates run when there are no user tasks scheduled on them. The other nodes that serve a specific purpose - like database servers, app servers etc, get updated regularly for security updates as & when they are available and necessary, and checked for version upgrades (with scheduled downtime) once every 4 months.
brightball 8y ago
Honestly, this is why so many people just use Heroku.
marmaduke 8y ago
> I am practically a one person company > Spent a year > go ahead and learn it. Its a huge boon. for green field deployment with a 1 year R&D budget for devops, sure K8s is great choice perhaps, but for the rest of us (even in tech)?
konradb 8y ago
Would it be possible to elaborate on centralized auth with an example? I've done a small amount of playing around with k8s but I'd not heard of this specific use case.
base698 8y ago
To play devils advocate, why not use AppEngine or Heroku, that does most of these things and you don't have to manually setup each of the apps with long YAML configs involving resource constraints and restart policies?
- ownagefool 8y ago
  I gather it's simply less popular because they're proprietary, opaque and opinionated. I suspect we're going to continue to see layers built on top of kubernetes, such as gitlabs autodevops, where for the simple things, you don't need to write any of that, but you can still inspect what they create, and jump off the rails for things that are a bit more complex.
  - dgruesso 8y ago
    Hi, GitLab PM here, thanks for the mention. We're indeed trying to make use of kubernetes inside GitLab as simple as possible. And just to add to your comment, you can try out kubernetes integration (https://docs.gitlab.com/ee/user/project/clusters/ https://docs.gitlab.com/ee/user/project/clusters/) independent of auto devops and vice-versa (https://docs.gitlab.com/ee/topics/autodevops/ https://docs.gitlab.com/ee/topics/autodevops/). Both of these are available in our core offering.

bthornbury 8y ago

The learning curve is sharp. The amount of things happening you don't have awareness of is also worrisome (to me). Source: I am also using Kubernetes in production, migrating off it soon.

austinjp 8y ago
It would be interesting if you could briefly say your reasons for moving away from K8s and what you're moving to.
- bthornbury 8y ago
  Sure. The largest reason is cost. My small deployment (3 nodes) is running around $100 / mo on AWS (That's my app, nginx, redis, and postgres). It doesn't even need 3 nodes, I don't recall if 3 is the minimum, but realistically I only need one (for now). For larger projects this is probably a non-issue. Second largest reason, is that really I have no idea what is going on on these nodes, and I probably never will. Magically my services run when I have the correct configurations. Not to say that's always a bad thing, but I've found it difficult to determine the default level of security as a result of this. A third reason is the learning curve. This is less of an issue because I've invested the time to learn already. But like, the first time I tried to get traffic ingressed was painful. As to what I'm moving to, I migrated one of my websites to a simple rsync + docker-compose setup and am pretty happy with it. In the past, I ran ansible on a 50 node cluster and it worked really well.
  - rconti 8y ago
    I'm not really clear on how moving off Kubernetes saves you money in this scenario. Seems like the most likely source of the cost is the AWS, which is the non-free part. I'm just learning k8s, so I feel you on the learning curve.
    - bthornbury 8y ago
      Kubernetes has a minimum node count. Moving to one node, saves cost, by a factor of 3. Not to mention all of the other resources it creates (load balancers).

sklivvz1971 8y ago

> As for the "execute query" example, why is it such a headache ? I just "kubectl ssh <container>" and I am in. It's slow as hell. > I had an excellent time working with kubernetes I'm happy for you, seriously, and I'm not claiming general validity of my experience. What I am disputing is the blog post (which does claim general validity).

reacharavindh 8y ago

Thanks for sharing your experience. I question myself whether it is necessary to hide my operational needs behind a behemoth of complexity like Kubernetes. The list of conveniences you mentioned sounds like magic you get from Kubernetes. What if there is a problem with any of them? Missing logs? Inappropriate scaling? Auth failures? or worse, failure of Auth system? Easy recovery? what if there were failures to checkpoint/backup/snapshot containers? CI/CD is good regardless of whether you use Kubernetes or not. EDIT: The question is, if you have any of these problems, why is it better to get your head into how Kubernetes deals with those operations & tools rather than dealing with well-defined unix tools that specialise in doing these jobs. syslog instead of however Kubernetes gathers the logs, reading FreeIPA docs instead of Kubernetes auth system logs? My point is, that to deal with all of the conveniences you mentioned, you need to know their details anyway. Why rely on Kubernetes abstraction if there is no such need? (I'm not trying to being snarky. I'm genuinely curious why you think it is a good idea. If you convince me otherwise, perhaps I would start adopting Kubernetes as well.) I run my cluster(I'm a sysadmin) with a couple of OpenBSD server that runs redundant DNS and DHCP. a CentOS 7 box that runs FreeIPA as central Auth. an OpenBSD server that acts as public facing SSH server. about 20 nodes, all provisioned using kickstart files, and then configured using Ansible. They run databases, web servers, batch jobs, Git, etc. A single server that runs ELK stack for log analysis. A single server that runs Prometheus for quick aggregated monitoring. Do you think I should switch over to Kubernetes for any benefits?

merb 8y ago
how often do you update your nodes?!
- StudentStuff 8y ago
  It sounds like the prior commenter Ansiblized their environment, they likely update the 20+ machines often since its not much of an effort.
- reacharavindh 8y ago
  All nodes are bare-metal servers that run CentOS 7, and are configured strictly via Ansible. If a node experiences a hardware failure, we just pick another spare server and run our Ansible playbook on it + a script for infrastructure changes like DNS & DHCP. Our workload is not strictly attached to physical resource. So, we have our updates scheduled for every week, with the condition that updates run when there are no user tasks scheduled on them. The other nodes that serve a specific purpose - like database servers, app servers etc, get updated regularly for security updates as & when they are available and necessary, and checked for version upgrades (with scheduled downtime) once every 4 months.

brightball 8y ago

Honestly, this is why so many people just use Heroku.

marmaduke 8y ago

> I am practically a one person company > Spent a year > go ahead and learn it. Its a huge boon. for green field deployment with a 1 year R&D budget for devops, sure K8s is great choice perhaps, but for the rest of us (even in tech)?

konradb 8y ago

Would it be possible to elaborate on centralized auth with an example? I've done a small amount of playing around with k8s but I'd not heard of this specific use case.

base698 8y ago

To play devils advocate, why not use AppEngine or Heroku, that does most of these things and you don't have to manually setup each of the apps with long YAML configs involving resource constraints and restart policies?

ownagefool 8y ago
I gather it's simply less popular because they're proprietary, opaque and opinionated. I suspect we're going to continue to see layers built on top of kubernetes, such as gitlabs autodevops, where for the simple things, you don't need to write any of that, but you can still inspect what they create, and jump off the rails for things that are a bit more complex.
- dgruesso 8y ago
  Hi, GitLab PM here, thanks for the mention. We're indeed trying to make use of kubernetes inside GitLab as simple as possible. And just to add to your comment, you can try out kubernetes integration (https://docs.gitlab.com/ee/user/project/clusters/ https://docs.gitlab.com/ee/user/project/clusters/) independent of auto devops and vice-versa (https://docs.gitlab.com/ee/topics/autodevops/ https://docs.gitlab.com/ee/topics/autodevops/). Both of these are available in our core offering.