29 ms·

We use k8s at $JOB for literally no reason other than some engineer decided he wanted to learn k8s. It's the worst, I don't even get to see my server logs beca

We use k8s at $JOB for literally no reason other than some engineer decided he wanted to learn k8s. It's the worst, I don't even get to see my server logs because "that would mean giving you access to the entire thing". I'm not a k8s person but surely that has to be missing something. We could literally make do with a cloudflare 5$ plan and have left to work with. With better integration, and better DX.

tayo42 3y ago
That's not a k8s thing. It's an idealistic DevOps thing. You'd probably deal with that in alot of places.
- amtamt 3y ago
  That for sure is not "idealistic devops", but someone missing fixing RBAC to let devs have access to logs of their workload.
  - tayo42 3y ago
    Maybe without details or knowing exactly what he means. But I've worked with people that decided you don't get to see any logs on a server except for a select few through some web ui or a log aggregator.
    - eddd-ddde 3y ago
      I know what you are talking about, unfortunately it's way worse than that. The same guy that runs the k8s is the guy that "reviews" my PRs. It's also the same guy that can't get eslint into our PR process, and the same guy that merges code from other people that just out right brick the server because it had some bad syntax. I _wish_ my problems were caused from too strict standards and procedures. But really is just a lack of understanding and cooperation.
    - amtamt 3y ago
      There are production systems handling PII (finance, healthcare etc), where logs are treated as radioactive and only people with trust, training and protective gears should be touching them. This is the case where these are legal/ compliance requirements. Non-prod of those systems can have wide open access to logs, unless someone decides to import prod data for "testing" in non-prod. While I agree access to logs may seem too restrictive, there are some very solid reasons for that... and then we have some cargo-cult and some lack of time/ understanding which causes too restrictive log access where it might not be required.
ActionHank 3y ago
Current project I work on sees: * everything running in docker containers, but you aren't supposed to run that locally, only local dev\test is through unit and component tests. * all k8s is maintained by a separate group pulling in those images through a seemingly very over engineered solution * when it is all combined in a deployment, things break, or more scary, they don't and you need to manually test and hope it's working. Issues take too long to track down and resolve and the reasons are very obvious to everyone but technical leadership who are constantly high fiving each other over their amazing creation that is already collapsing under it's own weight.
vbezhenar 3y ago
> It's the worst, I don't even get to see my server logs because "that would mean giving you access to the entire thing". I'm not a k8s person but surely that has to be missing something. I'm pretty sure it's possible to configure access to logs, not to the entire thing (whatever that means). He's probably lazy and does not want to bother. Besides, you should have centralized logging using loki or something similar. Kubernetes logging is not enough for any reasonable use-case.
- eddd-ddde 3y ago
  I know, but we don't get any. I legit had to do logging on my end to a _database_ because they weren't a fan of something like say new relic or similar.
- josephcsible 3y ago
  Isn't that just because Kubernetes makes it unreasonably hard to configure access to logs like that?
  - throaway2342551 3y ago
    No, it is simple to setup.
  - ongy 3y ago
    It's really not that hard if you got proper users. But doing proper users on k8s is hard. I suspect they just run with admin credentials and no real way to generate users.
    - vbezhenar 3y ago
      For small cluster you can just create service account for a user, create token for it and write it in the kubeconfig. Then assign role to this service account and that's about it. The main issue with this approach is that you can't organize those "users" into a groups. But for a small number of users you can just create all rolebindings and be done with it.
ufmace 3y ago
I dunno if k8s is actually right for your company, but it most certainly is possible to configure it to give engineers limited access, including viewing logs, pod status, services and deployments, etc, but not reading sensitive data or updating anything. In my opinion, deploying production services using devops technology nobody on the team has much understanding of is what's a bad idea, not any particular technology.
5Qn8mNbc2FNCiVV 3y ago
> that would mean giving you access to the entire thing There's clearly a skill issue at play here and your intuition is correct

tayo42 3y ago

That's not a k8s thing. It's an idealistic DevOps thing. You'd probably deal with that in alot of places.

amtamt 3y ago
That for sure is not "idealistic devops", but someone missing fixing RBAC to let devs have access to logs of their workload.
- tayo42 3y ago
  Maybe without details or knowing exactly what he means. But I've worked with people that decided you don't get to see any logs on a server except for a select few through some web ui or a log aggregator.
  - eddd-ddde 3y ago
    I know what you are talking about, unfortunately it's way worse than that. The same guy that runs the k8s is the guy that "reviews" my PRs. It's also the same guy that can't get eslint into our PR process, and the same guy that merges code from other people that just out right brick the server because it had some bad syntax. I _wish_ my problems were caused from too strict standards and procedures. But really is just a lack of understanding and cooperation.
  - amtamt 3y ago
    There are production systems handling PII (finance, healthcare etc), where logs are treated as radioactive and only people with trust, training and protective gears should be touching them. This is the case where these are legal/ compliance requirements. Non-prod of those systems can have wide open access to logs, unless someone decides to import prod data for "testing" in non-prod. While I agree access to logs may seem too restrictive, there are some very solid reasons for that... and then we have some cargo-cult and some lack of time/ understanding which causes too restrictive log access where it might not be required.

ActionHank 3y ago

Current project I work on sees: * everything running in docker containers, but you aren't supposed to run that locally, only local dev\test is through unit and component tests. * all k8s is maintained by a separate group pulling in those images through a seemingly very over engineered solution * when it is all combined in a deployment, things break, or more scary, they don't and you need to manually test and hope it's working. Issues take too long to track down and resolve and the reasons are very obvious to everyone but technical leadership who are constantly high fiving each other over their amazing creation that is already collapsing under it's own weight.

vbezhenar 3y ago

> It's the worst, I don't even get to see my server logs because "that would mean giving you access to the entire thing". I'm not a k8s person but surely that has to be missing something. I'm pretty sure it's possible to configure access to logs, not to the entire thing (whatever that means). He's probably lazy and does not want to bother. Besides, you should have centralized logging using loki or something similar. Kubernetes logging is not enough for any reasonable use-case.

eddd-ddde 3y ago
I know, but we don't get any. I legit had to do logging on my end to a _database_ because they weren't a fan of something like say new relic or similar.
josephcsible 3y ago
Isn't that just because Kubernetes makes it unreasonably hard to configure access to logs like that?
- throaway2342551 3y ago
  No, it is simple to setup.
- ongy 3y ago
  It's really not that hard if you got proper users. But doing proper users on k8s is hard. I suspect they just run with admin credentials and no real way to generate users.
  - vbezhenar 3y ago
    For small cluster you can just create service account for a user, create token for it and write it in the kubeconfig. Then assign role to this service account and that's about it. The main issue with this approach is that you can't organize those "users" into a groups. But for a small number of users you can just create all rolebindings and be done with it.

ufmace 3y ago

I dunno if k8s is actually right for your company, but it most certainly is possible to configure it to give engineers limited access, including viewing logs, pod status, services and deployments, etc, but not reading sensitive data or updating anything. In my opinion, deploying production services using devops technology nobody on the team has much understanding of is what's a bad idea, not any particular technology.

5Qn8mNbc2FNCiVV 3y ago

> that would mean giving you access to the entire thing There's clearly a skill issue at play here and your intuition is correct