4 ms·

We don't have sysadmins anymore "because of the cloud"

by monksy 7y ago

mbreedlove 7y ago
"Sysadmins? No, we just have devops now!"
- dvfjsdhgfv 7y ago
  Many managers use "devops" as an excuse to put a lot of burden on a small team, then this team is doing their best to automate managing a large number of machines but it's physically impossible to delve deeper into details and polish things, hence mishaps are bound to happen. And don't get me started on on what is happening inside containers.
  - deleted 7y ago
    [deleted]
  - tylfin 7y ago
    Sorry but I'd like to get you started on what is happening inside containers ;P Specifically can you go into more details about what worries you with containers. Is it insecure images with out of date software, or risky applications inside the containers? Something else?
    - mschuster91 7y ago
      Let's imagine your JIRA is insecure, someone owns it and obtains RCE, then does a privilege escalation on the host, whoops suddenly all services are accessible whereas that would have required more steps and owning in the old one-vm/bare metal-server-per-service model.
      - arpa 7y ago
        Escaping properly namespaced/pivot_root'ed environment and owning a host is non-trivial too.
bcrosby95 7y ago
Equally important, we don't have network admins. It would be physically impossible to expose our search database like this to the open internet. Extra layers of protection are great. It reminds me of companies I've worked with before that accidentally had a production site pointed to a dev database. Why the hell is that even physically possible with your network setup?
- mschuster91 7y ago
  Welcome to the wonderful world of Kubernetes (or, for that matter, any Docker orchestration solution, such as DC/OS). Anything can reach anything, provided you know the naming schema... and there's no easy way to fix it on anything that is not AWS/Azure/GCP, not without losing all the benefits of a self hosted k8s cluster in the first place. Openstack at least provides ways to isolate machines, but that's VM-level only and truly an ultimate PITA to set up.
  - arpa 7y ago
    Not entirely true, there are network security policies (on select few CNI providers) and other means of segregation using good old iptables (although probably need to update alternatives for iptables to point to iptables-legacy for them to work).
    - mschuster91 7y ago
      Ew. iptables (or any other way of messing around with the black magic that Docker and the orchestrators do to provide intra-container networking) is one thing only and that is a nice way to shoot yourself in the head while aiming at your legs.
      - arpa 7y ago
        To be entirely honest, if you know what you're doing and how lets say kube-proxy works in essence, things get pretty easy and simple. If you start every configuration of firewall with iptables -F, you're gonna have an interesting time. However, if you spend some time around these beasts, they are pretty well and logically built and it is trivial to coexist and modify your chains without touching those managed by docker/k8s. There is no black magic/and or wrong with the way they manage the rules. I'm much more angry at proper iptables being moved to iptables-legacy and systemd messing around with my resolv.conf :)

We don't have sysadmins anymore "because of the cloud"

by monksy 7y ago

We don't have sysadmins anymore "because of the cloud"

mbreedlove 7y ago

"Sysadmins? No, we just have devops now!"

dvfjsdhgfv 7y ago
Many managers use "devops" as an excuse to put a lot of burden on a small team, then this team is doing their best to automate managing a large number of machines but it's physically impossible to delve deeper into details and polish things, hence mishaps are bound to happen. And don't get me started on on what is happening inside containers.
- deleted 7y ago
  [deleted]
- tylfin 7y ago
  Sorry but I'd like to get you started on what is happening inside containers ;P Specifically can you go into more details about what worries you with containers. Is it insecure images with out of date software, or risky applications inside the containers? Something else?
  - mschuster91 7y ago
    Let's imagine your JIRA is insecure, someone owns it and obtains RCE, then does a privilege escalation on the host, whoops suddenly all services are accessible whereas that would have required more steps and owning in the old one-vm/bare metal-server-per-service model.
    - arpa 7y ago
      Escaping properly namespaced/pivot_root'ed environment and owning a host is non-trivial too.

bcrosby95 7y ago

Equally important, we don't have network admins. It would be physically impossible to expose our search database like this to the open internet. Extra layers of protection are great. It reminds me of companies I've worked with before that accidentally had a production site pointed to a dev database. Why the hell is that even physically possible with your network setup?

mschuster91 7y ago
Welcome to the wonderful world of Kubernetes (or, for that matter, any Docker orchestration solution, such as DC/OS). Anything can reach anything, provided you know the naming schema... and there's no easy way to fix it on anything that is not AWS/Azure/GCP, not without losing all the benefits of a self hosted k8s cluster in the first place. Openstack at least provides ways to isolate machines, but that's VM-level only and truly an ultimate PITA to set up.
- arpa 7y ago
  Not entirely true, there are network security policies (on select few CNI providers) and other means of segregation using good old iptables (although probably need to update alternatives for iptables to point to iptables-legacy for them to work).
  - mschuster91 7y ago
    Ew. iptables (or any other way of messing around with the black magic that Docker and the orchestrators do to provide intra-container networking) is one thing only and that is a nice way to shoot yourself in the head while aiming at your legs.
    - arpa 7y ago
      To be entirely honest, if you know what you're doing and how lets say kube-proxy works in essence, things get pretty easy and simple. If you start every configuration of firewall with iptables -F, you're gonna have an interesting time. However, if you spend some time around these beasts, they are pretty well and logically built and it is trivial to coexist and modify your chains without touching those managed by docker/k8s. There is no black magic/and or wrong with the way they manage the rules. I'm much more angry at proper iptables being moved to iptables-legacy and systemd messing around with my resolv.conf :)