7 ms·

Ok, wow. Talk about mixed feelings towards the article... the author is clearly knowledgeable, but it is so unclear why in the world does he opt for this kind o

by arpa 7y ago

Ok, wow. Talk about mixed feelings towards the article... the author is clearly knowledgeable, but it is so unclear why in the world does he opt for this kind of torture. 1. Why kubernetes for development? For most cases you don't need the whole orchestration over multiple nodes overhead, docker-compose is just fine. Incidentally, your push/pull problem is solved as you pull the dependencies and BUILD your app locally. 2. Want to have someplace to test your kube configurations? That's the ops part of devops. Set up a proper testing cluster instead of trying to cram everything in your devenv. Kube config should be in a separate project anyways. 3. No strace/bash. Look, man, this isn't even kubes' problem. That's how you build your images. Have separate dev images that use prod image as a base. Deploy dev images to testing env. Prod images to staging/prod. Build dev images locally. 4. Network debugging. That's difficult on any system. If you don't have access to the underlying host, you are screwed either way - call your devops for support. Anything else (internal ips, svc endpoints, ingresses) is trivial to debug on kube. However, we're talking about development... just don't use kube for development, period. Use docker-compose. Which, actually solves all your problems... right tools for the right job, or suffer the self-inflicted consequences.

echelon 7y ago
> 1. Why kubernetes for development? For most cases you don't need the whole orchestration over multiple nodes overhead, docker-compose is just fine. Incidentally, your push/pull problem is solved as you pull the dependencies and BUILD your app locally. Then it's not hermetic and repeatable. We strive for reproducibility between envs, because it reduces the chance that we're relying on incorrect assumptions about environmental behaviors. > 2. Want to have someplace to test your kube configurations? That's the ops part of devops. Set up a proper testing cluster instead of trying to cram everything in your devenv. Kube config should be in a separate project anyways. Is your test env just for you, or is there a chance you could break other engineers? What if they need to leverage the services your team provides in order to test what they're building, but your services are currently broken as a result of your own testing? Develop in development. Keep staging/test accessible for tests that are minimally intrusive. > 3. No strace/bash. Look, man, this isn't even kubes' problem. That's how you build your images. Have separate dev images that use prod image as a base. Deploy dev images to staging env. Build dev images locally. I agree, but it still sucks. We need to get better at this. There should be a means of injecting tooling. Or having it sit on an outer layer only privileged users can execute. But then you also don't want to ship those bytes... It's a problem. > 4. Network debugging. That's difficult on any system. If you don't have access to the underlying host, you are screwed either way - call your devops for support. Anything else (internal ips, svc endpoints, ingresses) is trivial to debug on kube. However, we're talking about development... just don't use kube for development, period. Use docker-compose. Which, actually solves all your problems... right tools for the right job, or suffer the self-inflicted consequences. It's dangerous to get away from a real production config. It's easy to work yourself into a world where things don't work the same. Prod could talk over abstract sockets to an envoy control plane with additional smarts for retry or forwarding behavior, and you might miss some of this behavior in development if you swapped out grpc ingress.
- pojzon 7y ago
  I think your team is missing a real testing pipeline that validates your changes against a production-like config on a test-kube.. Like someone higher said: Develop using a docker-compose, when you think you are done -> run a test pipeline with your versions. That pipeline should be able to validate whether you did not break anything. Dev_Ops means you do both but doesnt exactly mean you have to do both at the same time.. Most of the issues author described were rether self-imposed.
- arpa 7y ago
  > Then it's not hermetic and repeatable. in my experience, the core difference between local env and production in kubernetes is configuration of baremetal nodes and the firewall between them. In essence, if your docker-compose does not cover the logical equivalent of your cluster... well, you have architecture problems, and you should probably not solve them just by throwing kubernetes at it. Either that, or, covering your next point > Is your test env just for you, or is there a chance you could break other engineers? if your system requires significantly more resources or for some other reason you just can not bring the whole system up via docker-compose and you just really need to have a staging-level-thing-in-development, build a separate cluster for development. Leverage kubernetes namespaces so that each developer could have his/her own little namespace-sandbox-kube. I've seen that working pretty well in practice. > There should be a means of injecting tooling i agree that https://github.com/kubernetes/kubernetes/issues/30656 https://github.com/kubernetes/kubernetes/issues/30656 should have been closed long ago, but, again, this is not something kube should (or could) solve. Consider the simple example of your base images - one is using alpine, the other - debian. Injecting tooling is very different on these two. This could be in theory solved by injecting statically-linked executables, but, boy, this is another can of worms i'd rather leave in the hands of end-users... > It's dangerous to get away from a real production config. you will get away from a real production config. Your actual cluster nodes have a different configuration already. And, in my experience, that is more of an issue regarding network than additional ingress smarts. And, again, you can run a local version of the ingress as well if you need to replicate its' behavior (ingress is usually just managed nginx anyway, if you're not into exotics like Kong)... And if your control planes' smarts are undocumented and really require a full blown grpc-ingress with all the bells and whistles that kube brings, well... something's rotten in Denmark then...
cbanek 7y ago
I agree with you that many of these are self inflicted problems. I particularly like your answer to #3, which I think I'm going to have to start adopting! One part that I agree with the author is that local kubernetes clusters or even bare metal clusters are hard. I love being able to just spin up a cluster in GKE and have it just work. It really does just work, especially for ingress and storage. Get one piece working at a time, then you can move it to its final location and do whatever else needs to happen to make that work. It's valuable to have it working in the cloud even if prod is not, since that makes for easier testing/CI. As for your #4, it is hard, although I've seen some success with running a pod that has tcpdump in it as a sidecar. Example here: https://medium.com/@xxradar/how-to-tcpdump-effectively-in-kubernetes-part-1-a1546b683d2f https://medium.com/@xxradar/how-to-tcpdump-effectively-in-ku... (haven't used these directions personally, but it looks very similar to what I've seen). Some networking problems will only happen in kubernetes, things like timeout configuration and nginx stuff will just be different, so it's good to be able to do this. One other note is for logs, just use the helm charts to install kibana/fluentd/elasticsearch into your cluster. It's really easy, and then you don't have to worry about which pod is making the logs, and by default it is tagged with the pod name and other k8s metadata.
- arpa 7y ago
  thank you. As for #4, i've heard good things about https://istio.io https://istio.io - supposedly it does provide nice means to gain introspection to the traffic between services.
- merb 7y ago
  > that local kubernetes clusters or even bare metal clusters are hard what's so hard about k3s, kubespray or minikube? the only problem k8s has, that without a linux dev machine you are going through the vm pain. but anything else is just simple. btw. using mac sucks when trying to develop with local k8s. windows is better/okish, but it's best to run directly on linux.
  - cassianoleal 7y ago
    Why is it worse on the mac than on Windows? Legit question here, I haven't run Windows in many years so wouldn't know. I find developing on k8s on the mac just fine. Most of the time I either just run the Docker for Desktop k8s feature, or fire up a kind cluster. kind fits really well with VS Code's Remote - Containers extension for a completely isolated, all batteries included dev experience.
dragonsh 7y ago
1. Kubernetes is when your service needs google kind of load which 90% of systems don't. Just trade it for simple VM or use LXD containers. Don't jump on next hype cycle. If something works for Google does not mean will work for you. 2. Don't spend too much on it just use traditional knowledge and, you can still use bare-metal, VM or LXD container. Focus on application not on programming a tool designed to solve problems for Google size deployment. 3. Kubernetes is a completely different way of doing things. So if you want to leverage old cluster and distributed system knowledge you can use VM or LXD containers. Kubernetes is an overkill. 4. Abandon kubernetes instead of complaining, just use simple systems.
- arpa 7y ago
  for development, i agree. For production/testing/staging - hmmmm, it depends. I have taken a particular liking to the self-healing/HA properties of kubernetes. You just swap the dead nodes out without significant impact on the production... it's awesome.
  - sametmax 7y ago
    The trick is to understand that most products don't need to have 99% uptime. Your business will be fine if you are offline once in a while. Hell, the stock market goes offline every night. Those last % of uptime are very, very expensive at every level (money, people, logistic...). They better be worth it.
    - tinco 7y ago
      There's a difference between going offline at a set time for a set amount of time, and going offline randomly for unpredictable amounts of time. HA is not just about getting that sixth 9, it's also about the peace of mind that you are insulated from a range of problems. Not only would I not worry about our SQL service going down, I also don't have to worry about what I have to do when a node goes down, because that's automated.
      - sametmax 7y ago
arkh 7y ago
> docker-compose is just fine Not when part of your dev use macOS and another part uses linux: gotta love how docker manages file permissions between host and containers.
- arpa 7y ago
  docker on mac is a pain either way, since it sits in a linux vm and talks to mac fs via nfs. And as for permissions... well, it's been pretty much covered long time now - e.g. https://www.berthon.eu/2018/containers-volumes-and-file-permissions/ https://www.berthon.eu/2018/containers-volumes-and-file-perm... hope this makes your life a bit easier.

by arpa 7y ago

echelon 7y ago

> 1. Why kubernetes for development? For most cases you don't need the whole orchestration over multiple nodes overhead, docker-compose is just fine. Incidentally, your push/pull problem is solved as you pull the dependencies and BUILD your app locally. Then it's not hermetic and repeatable. We strive for reproducibility between envs, because it reduces the chance that we're relying on incorrect assumptions about environmental behaviors. > 2. Want to have someplace to test your kube configurations? That's the ops part of devops. Set up a proper testing cluster instead of trying to cram everything in your devenv. Kube config should be in a separate project anyways. Is your test env just for you, or is there a chance you could break other engineers? What if they need to leverage the services your team provides in order to test what they're building, but your services are currently broken as a result of your own testing? Develop in development. Keep staging/test accessible for tests that are minimally intrusive. > 3. No strace/bash. Look, man, this isn't even kubes' problem. That's how you build your images. Have separate dev images that use prod image as a base. Deploy dev images to staging env. Build dev images locally. I agree, but it still sucks. We need to get better at this. There should be a means of injecting tooling. Or having it sit on an outer layer only privileged users can execute. But then you also don't want to ship those bytes... It's a problem. > 4. Network debugging. That's difficult on any system. If you don't have access to the underlying host, you are screwed either way - call your devops for support. Anything else (internal ips, svc endpoints, ingresses) is trivial to debug on kube. However, we're talking about development... just don't use kube for development, period. Use docker-compose. Which, actually solves all your problems... right tools for the right job, or suffer the self-inflicted consequences. It's dangerous to get away from a real production config. It's easy to work yourself into a world where things don't work the same. Prod could talk over abstract sockets to an envoy control plane with additional smarts for retry or forwarding behavior, and you might miss some of this behavior in development if you swapped out grpc ingress.

pojzon 7y ago
I think your team is missing a real testing pipeline that validates your changes against a production-like config on a test-kube.. Like someone higher said: Develop using a docker-compose, when you think you are done -> run a test pipeline with your versions. That pipeline should be able to validate whether you did not break anything. Dev_Ops means you do both but doesnt exactly mean you have to do both at the same time.. Most of the issues author described were rether self-imposed.
arpa 7y ago
> Then it's not hermetic and repeatable. in my experience, the core difference between local env and production in kubernetes is configuration of baremetal nodes and the firewall between them. In essence, if your docker-compose does not cover the logical equivalent of your cluster... well, you have architecture problems, and you should probably not solve them just by throwing kubernetes at it. Either that, or, covering your next point > Is your test env just for you, or is there a chance you could break other engineers? if your system requires significantly more resources or for some other reason you just can not bring the whole system up via docker-compose and you just really need to have a staging-level-thing-in-development, build a separate cluster for development. Leverage kubernetes namespaces so that each developer could have his/her own little namespace-sandbox-kube. I've seen that working pretty well in practice. > There should be a means of injecting tooling i agree that https://github.com/kubernetes/kubernetes/issues/30656 https://github.com/kubernetes/kubernetes/issues/30656 should have been closed long ago, but, again, this is not something kube should (or could) solve. Consider the simple example of your base images - one is using alpine, the other - debian. Injecting tooling is very different on these two. This could be in theory solved by injecting statically-linked executables, but, boy, this is another can of worms i'd rather leave in the hands of end-users... > It's dangerous to get away from a real production config. you will get away from a real production config. Your actual cluster nodes have a different configuration already. And, in my experience, that is more of an issue regarding network than additional ingress smarts. And, again, you can run a local version of the ingress as well if you need to replicate its' behavior (ingress is usually just managed nginx anyway, if you're not into exotics like Kong)... And if your control planes' smarts are undocumented and really require a full blown grpc-ingress with all the bells and whistles that kube brings, well... something's rotten in Denmark then...

cbanek 7y ago

I agree with you that many of these are self inflicted problems. I particularly like your answer to #3, which I think I'm going to have to start adopting! One part that I agree with the author is that local kubernetes clusters or even bare metal clusters are hard. I love being able to just spin up a cluster in GKE and have it just work. It really does just work, especially for ingress and storage. Get one piece working at a time, then you can move it to its final location and do whatever else needs to happen to make that work. It's valuable to have it working in the cloud even if prod is not, since that makes for easier testing/CI. As for your #4, it is hard, although I've seen some success with running a pod that has tcpdump in it as a sidecar. Example here: https://medium.com/@xxradar/how-to-tcpdump-effectively-in-kubernetes-part-1-a1546b683d2f https://medium.com/@xxradar/how-to-tcpdump-effectively-in-ku... (haven't used these directions personally, but it looks very similar to what I've seen). Some networking problems will only happen in kubernetes, things like timeout configuration and nginx stuff will just be different, so it's good to be able to do this. One other note is for logs, just use the helm charts to install kibana/fluentd/elasticsearch into your cluster. It's really easy, and then you don't have to worry about which pod is making the logs, and by default it is tagged with the pod name and other k8s metadata.

arpa 7y ago
thank you. As for #4, i've heard good things about https://istio.io https://istio.io - supposedly it does provide nice means to gain introspection to the traffic between services.
merb 7y ago
> that local kubernetes clusters or even bare metal clusters are hard what's so hard about k3s, kubespray or minikube? the only problem k8s has, that without a linux dev machine you are going through the vm pain. but anything else is just simple. btw. using mac sucks when trying to develop with local k8s. windows is better/okish, but it's best to run directly on linux.
- cassianoleal 7y ago
  Why is it worse on the mac than on Windows? Legit question here, I haven't run Windows in many years so wouldn't know. I find developing on k8s on the mac just fine. Most of the time I either just run the Docker for Desktop k8s feature, or fire up a kind cluster. kind fits really well with VS Code's Remote - Containers extension for a completely isolated, all batteries included dev experience.

dragonsh 7y ago

1. Kubernetes is when your service needs google kind of load which 90% of systems don't. Just trade it for simple VM or use LXD containers. Don't jump on next hype cycle. If something works for Google does not mean will work for you. 2. Don't spend too much on it just use traditional knowledge and, you can still use bare-metal, VM or LXD container. Focus on application not on programming a tool designed to solve problems for Google size deployment. 3. Kubernetes is a completely different way of doing things. So if you want to leverage old cluster and distributed system knowledge you can use VM or LXD containers. Kubernetes is an overkill. 4. Abandon kubernetes instead of complaining, just use simple systems.

arpa 7y ago
for development, i agree. For production/testing/staging - hmmmm, it depends. I have taken a particular liking to the self-healing/HA properties of kubernetes. You just swap the dead nodes out without significant impact on the production... it's awesome.
- sametmax 7y ago
  The trick is to understand that most products don't need to have 99% uptime. Your business will be fine if you are offline once in a while. Hell, the stock market goes offline every night. Those last % of uptime are very, very expensive at every level (money, people, logistic...). They better be worth it.
  - tinco 7y ago
    There's a difference between going offline at a set time for a set amount of time, and going offline randomly for unpredictable amounts of time. HA is not just about getting that sixth 9, it's also about the peace of mind that you are insulated from a range of problems. Not only would I not worry about our SQL service going down, I also don't have to worry about what I have to do when a node goes down, because that's automated.
    - sametmax 7y ago

arkh 7y ago

> docker-compose is just fine Not when part of your dev use macOS and another part uses linux: gotta love how docker manages file permissions between host and containers.

arpa 7y ago
docker on mac is a pain either way, since it sits in a linux vm and talks to mac fs via nfs. And as for permissions... well, it's been pretty much covered long time now - e.g. https://www.berthon.eu/2018/containers-volumes-and-file-permissions/ https://www.berthon.eu/2018/containers-volumes-and-file-perm... hope this makes your life a bit easier.