4 ms·

Evolution of code deployment tools at Mixpanel

45 points by i0exception 5y ago

Vendan 5y ago
Hey, I'm one of the engineers on the DevInfra team at Mixpanel, and worked on the Argo implementation. AMA, and we are hiring in general, and for the DevInfra team in specific (feel free to email me, just gotta solve the "puzzle" in my profile :D)
- gravypod 5y ago
  Have you considered using Bazel to build an artifact that describes your deployment configuration? This is something that I've been thinking about trying to implement for some time. Essentially I want to have a way to support defining service configurations like this: service_binary( name = "foobar", srcs = ["foobar.jsonnet"], deps = ["//jsonnet/service.jsonnet"], images = ["//a:image"], ) Then make it possible to create a class of `*_test` rules that would start up a bunch of docker containers in the same network topology they would exist in prod, run your test in the same netns, then clean everything up. It could look something like this: cc_integration_test( name = "...", services = ["//a", "//b"], srcs = ["main.cc"] ) There's some BazelCon talks about people doing similar stuff but not actually open sourcing their code. P.S. if you use rules_docker please feel free to open a PR to add your company to our README: https://github.com/bazelbuild/rules_docker/#adopters https://github.com/bazelbuild/rules_docker/#adopters > service_binary
  - Vendan 5y ago
    We've looked at it, but we have services that aren't using rules_docker to build their images, so for now at least, we'd rather have everything use as similar of a deployment process as possible. (and we are using rules_docker, and love it! Added a pr, gotta look into the CLA stuff (would like to do it from employer side, but that will have to wait until monday at least))
  - sa46 5y ago
    With something like service_binary, how do you wire up services? For example, If main.cc depended on Redis, how would you get the host and port of the Redis dependency from main.cc?
    - gravypod 5y ago
      You can do some amazing things based on convention. In kube it's pretty common to use Services for service discovery. Essentially all a Service is is a smart DNS name that points to a collection of pods (netns). If you want to mirror that setup you can create a virtual bridge network setup the same docker-compose does which would let you have a container with a hostname of `database` and then `ping database` would resolve to the correct container. To match kube's conventions you could do this: https://kubernetes.io/docs/concepts/services-networking/service/#dns https://kubernetes.io/docs/concepts/services-networking/serv... Another thing you could do is generate environment variables similar to what kube does: https://kubernetes.io/docs/concepts/services-networking/service/#environment-variables https://kubernetes.io/docs/concepts/services-networking/serv...
- postpawl 5y ago
  Off-topic, I know… but do you think it was a good idea to disable SSL cert verification by default in your python client recently? You also did it in a way that ignores all insecure cert warnings for requests from other libraries too: https://github.com/mixpanel/mixpanel-python/pull/102 https://github.com/mixpanel/mixpanel-python/pull/102
  - Vendan 5y ago
    That's not an area I'm involved with, but I'll poke the people that are... sorry :(
cagenut 5y ago
I'm seeing this "we started out with 2 and now there's dozens" pattern with k8s everywhere, and just like in this example the cause is a one-two-punch combo of proliferation of service complexity and geographic replication demands (be they legal or performance). kube was originally meant to abstract the datacenter, is there anyone who's abstracting the globe/internet yet? I'd love a geo-kube abstraction where based on annotations or something I could provision deployments/pods to arbitrary geographic locations. I want a scale-to-zero node-pool in every region, and one kube master api for the world.
- gravypod 5y ago
  There's active work on a standard called kubefed [0] that is being worked on. > I want a scale-to-zero node-pool in every region, and one kube master api for the world. Personally, I'd generalize this to: "I want to describe the reliability requirements and configuration for my software and have an automated system solve for where, how many, when, and how to route to it" I want to have something where I can say "I need to have high availability, lowest latency, and X GB of RAM and Y cores" and have a system automatically schedule me wherever compute is cheapest while also intelligently routing traffic to my servers based on client origins. [0] - https://github.com/kubernetes-sigs/kubefed https://github.com/kubernetes-sigs/kubefed
  - cagenut 5y ago
    does it have to be mulitple federated clusters? can etcd and kubelet not just handle 100ms? (or be adapted to).
    - p_l 5y ago
      While laten ies can get pretty good these days, I'd rather not plan on having 100ms ceiling for multiple DCs without dedicated lines and considerably close geographic locations.
      - cagenut 5y ago
        us east-west coast is in the 60's and us to western eu is in the 70s. hell I'd be content to run the etcd cluster across a tighter range, us-east-1/us-east-2/ca-central-1 are all within 30ms of each other. so I guess its really about kublet and whatever comes after that I'm not thinking of (networking).

45 points by i0exception 5y ago

Hey, I'm one of the engineers on the DevInfra team at Mixpanel, and worked on the Argo implementation. AMA, and we are hiring in general, and for the DevInfra team in specific (feel free to email me, just gotta solve the "puzzle" in my profile :D)

gravypod 5y ago
Have you considered using Bazel to build an artifact that describes your deployment configuration? This is something that I've been thinking about trying to implement for some time. Essentially I want to have a way to support defining service configurations like this: service_binary( name = "foobar", srcs = ["foobar.jsonnet"], deps = ["//jsonnet/service.jsonnet"], images = ["//a:image"], ) Then make it possible to create a class of `*_test` rules that would start up a bunch of docker containers in the same network topology they would exist in prod, run your test in the same netns, then clean everything up. It could look something like this: cc_integration_test( name = "...", services = ["//a", "//b"], srcs = ["main.cc"] ) There's some BazelCon talks about people doing similar stuff but not actually open sourcing their code. P.S. if you use rules_docker please feel free to open a PR to add your company to our README: https://github.com/bazelbuild/rules_docker/#adopters https://github.com/bazelbuild/rules_docker/#adopters > service_binary
- Vendan 5y ago
  We've looked at it, but we have services that aren't using rules_docker to build their images, so for now at least, we'd rather have everything use as similar of a deployment process as possible. (and we are using rules_docker, and love it! Added a pr, gotta look into the CLA stuff (would like to do it from employer side, but that will have to wait until monday at least))
- sa46 5y ago
  With something like service_binary, how do you wire up services? For example, If main.cc depended on Redis, how would you get the host and port of the Redis dependency from main.cc?
  - gravypod 5y ago
    You can do some amazing things based on convention. In kube it's pretty common to use Services for service discovery. Essentially all a Service is is a smart DNS name that points to a collection of pods (netns). If you want to mirror that setup you can create a virtual bridge network setup the same docker-compose does which would let you have a container with a hostname of `database` and then `ping database` would resolve to the correct container. To match kube's conventions you could do this: https://kubernetes.io/docs/concepts/services-networking/service/#dns https://kubernetes.io/docs/concepts/services-networking/serv... Another thing you could do is generate environment variables similar to what kube does: https://kubernetes.io/docs/concepts/services-networking/service/#environment-variables https://kubernetes.io/docs/concepts/services-networking/serv...
postpawl 5y ago
Off-topic, I know… but do you think it was a good idea to disable SSL cert verification by default in your python client recently? You also did it in a way that ignores all insecure cert warnings for requests from other libraries too: https://github.com/mixpanel/mixpanel-python/pull/102 https://github.com/mixpanel/mixpanel-python/pull/102
- Vendan 5y ago
  That's not an area I'm involved with, but I'll poke the people that are... sorry :(

cagenut 5y ago

I'm seeing this "we started out with 2 and now there's dozens" pattern with k8s everywhere, and just like in this example the cause is a one-two-punch combo of proliferation of service complexity and geographic replication demands (be they legal or performance). kube was originally meant to abstract the datacenter, is there anyone who's abstracting the globe/internet yet? I'd love a geo-kube abstraction where based on annotations or something I could provision deployments/pods to arbitrary geographic locations. I want a scale-to-zero node-pool in every region, and one kube master api for the world.

gravypod 5y ago
There's active work on a standard called kubefed [0] that is being worked on. > I want a scale-to-zero node-pool in every region, and one kube master api for the world. Personally, I'd generalize this to: "I want to describe the reliability requirements and configuration for my software and have an automated system solve for where, how many, when, and how to route to it" I want to have something where I can say "I need to have high availability, lowest latency, and X GB of RAM and Y cores" and have a system automatically schedule me wherever compute is cheapest while also intelligently routing traffic to my servers based on client origins. [0] - https://github.com/kubernetes-sigs/kubefed https://github.com/kubernetes-sigs/kubefed
- cagenut 5y ago
  does it have to be mulitple federated clusters? can etcd and kubelet not just handle 100ms? (or be adapted to).
  - p_l 5y ago
    While laten ies can get pretty good these days, I'd rather not plan on having 100ms ceiling for multiple DCs without dedicated lines and considerably close geographic locations.
    - cagenut 5y ago
      us east-west coast is in the 60's and us to western eu is in the 70s. hell I'd be content to run the etcd cluster across a tighter range, us-east-1/us-east-2/ca-central-1 are all within 30ms of each other. so I guess its really about kublet and whatever comes after that I'm not thinking of (networking).