3 ms·

May I ask how you do it? I've thought about using minikube for local development myself as a way of being able to reuse helm charts so instead of having one se

May I ask how you do it? I've thought about using minikube for local development myself as a way of being able to reuse helm charts so instead of having one setup for dev (perhaps docker compose or simply the stack standard way) and one for staging/production (helm), it's all the same helm chart with just different values. Another reason is to be able to encapsulate dev environments better. Right now, every stack I use have different ways to do it (pythons virtualenv for example). If I were using minikube, everything would be using the same encapsulation. I've tried setting it up, and it works fine, using host mapping so files are still reloaded when files change locally on and so on, but something I haven't solved yet is that in this setup, the only way stack specific packages/modules/whatever is installed, is inside the container, which is a problem because my editor/IDE requires the packages/modules to be available locally for it to be able to do "intellisense". Have you and your team delt with the same problem or other problems? How have you solved it? Any tips and thoughts are much appreciated!

hosh 9y ago
No problem. I wrote Matsuri as a way to manage this. https://github.com/matsuri-rb/matsuri https://github.com/matsuri-rb/matsuri This was written with K8S 1.1 in mind, and later somewhat updated to 1.4. It was primarily intended to solve: (1) Consolidate interfaces for working with dev and production (or other environments such as staging, demo, etc.) (2) It implements this by being a manifest generator and feeding the manifest into `kubectl`. This was designed and written before ConfigSet, so some of what Matsuri solved is now solvable with ConfigSet. Matsuri is written in Ruby, and so it takes advantages of class inheritance and method override to achieve what it does. The class definitions used generate the manifest is broken down into composable chunks, which can be overridden at any point. In this way, I can have a base template for say, a web pod for dev, staging, and production. In dev, I would then method-override what is returned for volume mounts and volumes. In that way, I can specify source mounting and SSH agent sock mounting for dev ... but not for staging or production. In another example, I can use method override to augment the environment variables passed into the container in order to capture the uid/gid so that the container can reconfigure itself to use those. There are helper methods to generate things like, correct format of ports or volumes. None of those are necessary if you already know how to format it. Likewise, since a manifest is just a class definition, replication controller, replicasets, and deployments all reference a pod and use those to create the pod template spec. I then have an extra resource called an "app" that does not map to K8S. They define a bundle of K8S resources or other apps that can be converged together. That allows someone to quickly setup a dev environment with all the moving pieces. Those also include hooks for you to do things like, define the command for shelling (with correct uid/gid) or console (which depends; redis console is invoked differently than a rails console). This allows a consolidation ... for example, to log into my dev rails console, I would do something like: bin/dev console myapp And to do that on production: bin/production console myapp (Powerful, but I suspect there will be some asks around security here). I had flirted with the idea of allowing plugins or packages ... and then Helm was announced. I figured more people will use that than Matsuri. But one possibility might be to get Matsuri to work with Helm. There are a lot of rough spots with Matsuri. The command-line interface could be done a lot better (right now, it assumes you will always be in the root of your platform repo). There are no documentation or examples for Matsuri. It does not implement all of the resources. It also helps a lot if you know Ruby. Infrastructure is not my full-time role on my team. Being a small team and early stage, we have had to wear multiple hats. I had not had the energy or time to document and promote this. If you want to chat about this further, feel free to email me at hosheng.hsiao at gmail.
- hosh 9y ago
  On the stack-specific modules -- we're using Rails. The gems are installed on a host mount. None of us are using an IDE that needs to reference where it is, but if we did, we could point it to that path. One of the headaches we run into comes from using Alpine as a baseimage. They are compiled against muslc instead of libc. You always have to install gems from inside the container -- but we can do that with host mounting and then executing the install from inside the container. That hasn't been a problem until we tried to install google-api gem and found out grpc doesn't play well with muslc.
  - fredsir 9y ago
    Thanks for the write-up!

hosh 9y ago

No problem. I wrote Matsuri as a way to manage this. https://github.com/matsuri-rb/matsuri https://github.com/matsuri-rb/matsuri This was written with K8S 1.1 in mind, and later somewhat updated to 1.4. It was primarily intended to solve: (1) Consolidate interfaces for working with dev and production (or other environments such as staging, demo, etc.) (2) It implements this by being a manifest generator and feeding the manifest into `kubectl`. This was designed and written before ConfigSet, so some of what Matsuri solved is now solvable with ConfigSet. Matsuri is written in Ruby, and so it takes advantages of class inheritance and method override to achieve what it does. The class definitions used generate the manifest is broken down into composable chunks, which can be overridden at any point. In this way, I can have a base template for say, a web pod for dev, staging, and production. In dev, I would then method-override what is returned for volume mounts and volumes. In that way, I can specify source mounting and SSH agent sock mounting for dev ... but not for staging or production. In another example, I can use method override to augment the environment variables passed into the container in order to capture the uid/gid so that the container can reconfigure itself to use those. There are helper methods to generate things like, correct format of ports or volumes. None of those are necessary if you already know how to format it. Likewise, since a manifest is just a class definition, replication controller, replicasets, and deployments all reference a pod and use those to create the pod template spec. I then have an extra resource called an "app" that does not map to K8S. They define a bundle of K8S resources or other apps that can be converged together. That allows someone to quickly setup a dev environment with all the moving pieces. Those also include hooks for you to do things like, define the command for shelling (with correct uid/gid) or console (which depends; redis console is invoked differently than a rails console). This allows a consolidation ... for example, to log into my dev rails console, I would do something like: bin/dev console myapp And to do that on production: bin/production console myapp (Powerful, but I suspect there will be some asks around security here). I had flirted with the idea of allowing plugins or packages ... and then Helm was announced. I figured more people will use that than Matsuri. But one possibility might be to get Matsuri to work with Helm. There are a lot of rough spots with Matsuri. The command-line interface could be done a lot better (right now, it assumes you will always be in the root of your platform repo). There are no documentation or examples for Matsuri. It does not implement all of the resources. It also helps a lot if you know Ruby. Infrastructure is not my full-time role on my team. Being a small team and early stage, we have had to wear multiple hats. I had not had the energy or time to document and promote this. If you want to chat about this further, feel free to email me at hosheng.hsiao at gmail.

hosh 9y ago
On the stack-specific modules -- we're using Rails. The gems are installed on a host mount. None of us are using an IDE that needs to reference where it is, but if we did, we could point it to that path. One of the headaches we run into comes from using Alpine as a baseimage. They are compiled against muslc instead of libc. You always have to install gems from inside the container -- but we can do that with host mounting and then executing the install from inside the container. That hasn't been a problem until we tried to install google-api gem and found out grpc doesn't play well with muslc.
- fredsir 9y ago
  Thanks for the write-up!