6 ms·

Show HN: Miniboss, versatile local container management with Python

0xbadcafebee 5y ago
Why not fork docker-compose v1? It's Python! https://github.com/docker/compose/tree/1.29.2/compose https://github.com/docker/compose/tree/1.29.2/compose You could keep the YAML format and extend it, and still write compose files as pure Python.
- afroisalreadyin 5y ago
  I actually dug around in the source code of docker-compose to figure out how to properly work with networks. However the fundamental ideas of the two projects are different (yaml -> containers vs. Python code -> containers), so starting off from docker-compose just wasn't right. It would have simply complicated things.
  - freeqaz 5y ago
    Have you thought about taking an approach similar to the AWS CDK with CloudFormation? Aka the whole "infrastructure as code" movement? I'm not sure how that would work with the whole "lifecycle hooks" you've mentioned a few times, but maybe it would be easier than trying to compete with Docker Compose yourself by simply wrapping it. I posted in another comment here too but we wrote a bunch of code to deal with programmatic generation of Docker Compose files, and it was really sweet to use! I've honestly thought about making that code[0] a stand-alone library because of how valuable it was. 0: https://github.com/lunasec-io/lunasec/blob/master/js/sdks/packages/cli/src/docker-compose/lunasec-stack.ts https://github.com/lunasec-io/lunasec/blob/master/js/sdks/pa...
- ingvul 5y ago
  It's easier (and usually more rewarding) to start something from scratch than to fork.
  - 0xbadcafebee 5y ago
    I think you just described the origin of all new software :)
dec0dedab0de 5y ago
This looks great, I am so sick of YAML. While it's probably fine for most of the use cases you're predicting, if someone who isn't quite familiar with Python's gotchas decides to subclass, or modify one of the items at runtime, it will be a tricky thing to troubleshoot. Even though that is a rite of passage for Python developers, it's probably best to not have examples that could set them up for weird bugs. edit: I was talking about mutable types at the top level of a class, and somehow left that out.
- afroisalreadyin 5y ago
  Good point. I was thinking of adding `docker-compose` compatibility, so that you can use a `docker-compose.yml` file to define services, and then add event hooks etc. with decorators, or something similar. You would then also have access to the attributes of the services.
  - dec0dedab0de 5y ago
    I meant to say that having mutable types defined at the top level of a class was a problem, I must have backspaced over the most important part while I was figuring out to say it. Classes in and of themselves are great, but having lists and dicts defined at the top level can cause problems.
    - afroisalreadyin 5y ago
      Aha, I get it. I never thought about the attributes of a service definition getting modified, but you're right, it might happen. I'll try to figure out whether using immutable data structures for attributes would work out.
      - awild 5y ago
        I'll second the use of immutable data structures. Had to do some real nasty work of debugging last week because someone accidentally mutated a configuration object causing persistence issues much deeper in the stack. Would've been impossible with frozendicts.
freeqaz 5y ago
I've definitely experienced the problems of using Docker Compose for local dev. Oh, that file changed? Control-C to kill the whole cluster. Wait until the DB flushes. Start it again. And if you just want to restart one service? Well you have to run 4 commands[0] in order to properly do it... And what the author was saying about needing to "template" the YAML files is also true. In the past, when shipping an Open Source library for others to use, we had to wrap Docker Compose in a CLI that generates the YAML file and then invokes Docker Compose for you. (See it here[1][2][3]) It sucks and this library definitely solves a real problem! 0: https://stackoverflow.com/questions/31466428/how-to-restart-a-single-container-with-docker-compose https://stackoverflow.com/questions/31466428/how-to-restart-... 1: The code: https://github.com/lunasec-io/lunasec/blob/master/js/sdks/packages/cli/src/docker-compose/lunasec-stack.ts https://github.com/lunasec-io/lunasec/blob/master/js/sdks/pa... 2: The package it lives in: https://github.com/lunasec-io/lunasec/tree/master/js/sdks/packages/cli https://github.com/lunasec-io/lunasec/tree/master/js/sdks/pa... 3: The docs for the wrapped CLI: https://www.lunasec.io/docs/pages/overview/demo-app/overview https://www.lunasec.io/docs/pages/overview/demo-app/overview
- afroisalreadyin 5y ago
  These are exactly the kinds of issues that drove me to work on miniboss. My top grievance has been the "waiting for service available" issue, which is generally solved by wrapping the dependent code in a `wait-for-it.sh` bash script. A rather hacky solution for a rather simple problem.
  - freeqaz 5y ago
    Hahaha yes! I had to add all of that to our code as well. I also had to write a "wait-for-file.sh" script for a mechanism to pass config values. (We use LocalStack to host an AWS env locally and the S3 bucket name is random.) In Docker Compose v2 they had a more sophisticated "depends on" model with health checks, but they removed it in v3. The docs say something about Docker Swarm so I'm guessing they removed that in order to try to make v3 syntax work for clusters too. The real tragedy here is that Kubernetes won, but we're still stuck with Docker Compose trying to be the same thing. I hope your tool can help change this annoying status quo!
kfk 5y ago
Can this be adapted to the Kubernetes ecosystem?
- afroisalreadyin 5y ago
  I have no idea, as I'm a bit lost when it comes to Kubernetes tooling. When I wrote a Kubernetes tutorial (https://okigiveup.net/tutorials/a-tutorial-introduction-to-kubernetes/ https://okigiveup.net/tutorials/a-tutorial-introduction-to-k...) nearly four years ago, you had kubectl and YAML files, and that was it. These days there is a whole hierarchy of tools, which probably cover the same ground with miniboss and much more. So yes, it could be adapted, but I'm pretty sure it wouldn't be necessary.
mataug 5y ago
Would it be correct to assume that something similar can be done to replace helm.sh with kubernetes ?
- qbasic_forever 5y ago
  Check out Pulumi, it's a similar idea of python (or other languages) to generate container orchestration config for Kubernetes or other systems: https://github.com/pulumi/pulumi https://github.com/pulumi/pulumi
- ramaro 5y ago
  Checkout kapitan, with its Kadet input type, you can generate YAML with a semi-opinionated Python based DSL: https://kapitan.dev/compile/#kadet https://kapitan.dev/compile/#kadet I am the author and have used it at a FAANG previously and now more recently at my startup to manage large, complex kubernetes and terraform managed infrastructure.
roomey 5y ago
Is there anyway to use this with kompose convert so it can be deployed to production in a kubernetes cluster?
- afroisalreadyin 5y ago
  I have no idea what kompose convert does, honestly. Principally, one could print out any format from miniboss, but it depends on how much you want to trust a tool written for local deployments with kubernetes stuff.
  - roomey 5y ago
    It takes your docker-compose.yaml and creates a bunch of kubernetes yamls that can just be deployed without touching them. I've managed to never touch a Kubernetes yaml thanks to it.
blondin 5y ago
so it's like terraform or pulumi but scoped down to docker and using python?
- afroisalreadyin 5y ago
  I would say it's more similar to Pulumi, as the main aim is to avoid markup (or a simplified programming language, like HCL in the case of terraform). The other two are correct though: Only for Docker and in Python.
emacsen 5y ago
This looks potentially very interesting indeed! Thanks for sharing. How tied is it to the Docker daemon? Instead of Docker, could one use podman, which doesn't need root.
- afroisalreadyin 5y ago
  That's a good question, I think only this class would need to be reimplemented for podman: https://github.com/afroisalreadyinu/miniboss/blob/main/miniboss/docker_client.py https://github.com/afroisalreadyinu/miniboss/blob/main/minib... . The question is of course whether there are any incompatibilities in terms of networking etc.
  - emacsen 5y ago
    The biggest difference/issue is that podman doesn't provide a daemon of events on its own. It's docker-compose compatibility is tied up in essentially re-implementing/emulating that, which partially defeats the purpose. My main need for the daemon component is https://github.com/nginx-proxy/acme-companion https://github.com/nginx-proxy/acme-companion which listens to docker events to know when a new service is available to proxy.

Why not fork docker-compose v1? It's Python! https://github.com/docker/compose/tree/1.29.2/compose https://github.com/docker/compose/tree/1.29.2/compose You could keep the YAML format and extend it, and still write compose files as pure Python.

afroisalreadyin 5y ago
I actually dug around in the source code of docker-compose to figure out how to properly work with networks. However the fundamental ideas of the two projects are different (yaml -> containers vs. Python code -> containers), so starting off from docker-compose just wasn't right. It would have simply complicated things.
- freeqaz 5y ago
  Have you thought about taking an approach similar to the AWS CDK with CloudFormation? Aka the whole "infrastructure as code" movement? I'm not sure how that would work with the whole "lifecycle hooks" you've mentioned a few times, but maybe it would be easier than trying to compete with Docker Compose yourself by simply wrapping it. I posted in another comment here too but we wrote a bunch of code to deal with programmatic generation of Docker Compose files, and it was really sweet to use! I've honestly thought about making that code[0] a stand-alone library because of how valuable it was. 0: https://github.com/lunasec-io/lunasec/blob/master/js/sdks/packages/cli/src/docker-compose/lunasec-stack.ts https://github.com/lunasec-io/lunasec/blob/master/js/sdks/pa...
ingvul 5y ago
It's easier (and usually more rewarding) to start something from scratch than to fork.
- 0xbadcafebee 5y ago
  I think you just described the origin of all new software :)

dec0dedab0de 5y ago

This looks great, I am so sick of YAML. While it's probably fine for most of the use cases you're predicting, if someone who isn't quite familiar with Python's gotchas decides to subclass, or modify one of the items at runtime, it will be a tricky thing to troubleshoot. Even though that is a rite of passage for Python developers, it's probably best to not have examples that could set them up for weird bugs. edit: I was talking about mutable types at the top level of a class, and somehow left that out.

afroisalreadyin 5y ago
Good point. I was thinking of adding `docker-compose` compatibility, so that you can use a `docker-compose.yml` file to define services, and then add event hooks etc. with decorators, or something similar. You would then also have access to the attributes of the services.
- dec0dedab0de 5y ago
  I meant to say that having mutable types defined at the top level of a class was a problem, I must have backspaced over the most important part while I was figuring out to say it. Classes in and of themselves are great, but having lists and dicts defined at the top level can cause problems.
  - afroisalreadyin 5y ago
    Aha, I get it. I never thought about the attributes of a service definition getting modified, but you're right, it might happen. I'll try to figure out whether using immutable data structures for attributes would work out.
    - awild 5y ago
      I'll second the use of immutable data structures. Had to do some real nasty work of debugging last week because someone accidentally mutated a configuration object causing persistence issues much deeper in the stack. Would've been impossible with frozendicts.

freeqaz 5y ago

I've definitely experienced the problems of using Docker Compose for local dev. Oh, that file changed? Control-C to kill the whole cluster. Wait until the DB flushes. Start it again. And if you just want to restart one service? Well you have to run 4 commands[0] in order to properly do it... And what the author was saying about needing to "template" the YAML files is also true. In the past, when shipping an Open Source library for others to use, we had to wrap Docker Compose in a CLI that generates the YAML file and then invokes Docker Compose for you. (See it here[1][2][3]) It sucks and this library definitely solves a real problem! 0: https://stackoverflow.com/questions/31466428/how-to-restart-a-single-container-with-docker-compose https://stackoverflow.com/questions/31466428/how-to-restart-... 1: The code: https://github.com/lunasec-io/lunasec/blob/master/js/sdks/packages/cli/src/docker-compose/lunasec-stack.ts https://github.com/lunasec-io/lunasec/blob/master/js/sdks/pa... 2: The package it lives in: https://github.com/lunasec-io/lunasec/tree/master/js/sdks/packages/cli https://github.com/lunasec-io/lunasec/tree/master/js/sdks/pa... 3: The docs for the wrapped CLI: https://www.lunasec.io/docs/pages/overview/demo-app/overview https://www.lunasec.io/docs/pages/overview/demo-app/overview

afroisalreadyin 5y ago
These are exactly the kinds of issues that drove me to work on miniboss. My top grievance has been the "waiting for service available" issue, which is generally solved by wrapping the dependent code in a `wait-for-it.sh` bash script. A rather hacky solution for a rather simple problem.
- freeqaz 5y ago
  Hahaha yes! I had to add all of that to our code as well. I also had to write a "wait-for-file.sh" script for a mechanism to pass config values. (We use LocalStack to host an AWS env locally and the S3 bucket name is random.) In Docker Compose v2 they had a more sophisticated "depends on" model with health checks, but they removed it in v3. The docs say something about Docker Swarm so I'm guessing they removed that in order to try to make v3 syntax work for clusters too. The real tragedy here is that Kubernetes won, but we're still stuck with Docker Compose trying to be the same thing. I hope your tool can help change this annoying status quo!

kfk 5y ago

Can this be adapted to the Kubernetes ecosystem?

afroisalreadyin 5y ago
I have no idea, as I'm a bit lost when it comes to Kubernetes tooling. When I wrote a Kubernetes tutorial (https://okigiveup.net/tutorials/a-tutorial-introduction-to-kubernetes/ https://okigiveup.net/tutorials/a-tutorial-introduction-to-k...) nearly four years ago, you had kubectl and YAML files, and that was it. These days there is a whole hierarchy of tools, which probably cover the same ground with miniboss and much more. So yes, it could be adapted, but I'm pretty sure it wouldn't be necessary.

mataug 5y ago

Would it be correct to assume that something similar can be done to replace helm.sh with kubernetes ?

qbasic_forever 5y ago
Check out Pulumi, it's a similar idea of python (or other languages) to generate container orchestration config for Kubernetes or other systems: https://github.com/pulumi/pulumi https://github.com/pulumi/pulumi
ramaro 5y ago
Checkout kapitan, with its Kadet input type, you can generate YAML with a semi-opinionated Python based DSL: https://kapitan.dev/compile/#kadet https://kapitan.dev/compile/#kadet I am the author and have used it at a FAANG previously and now more recently at my startup to manage large, complex kubernetes and terraform managed infrastructure.

roomey 5y ago

Is there anyway to use this with kompose convert so it can be deployed to production in a kubernetes cluster?

afroisalreadyin 5y ago
I have no idea what kompose convert does, honestly. Principally, one could print out any format from miniboss, but it depends on how much you want to trust a tool written for local deployments with kubernetes stuff.
- roomey 5y ago
  It takes your docker-compose.yaml and creates a bunch of kubernetes yamls that can just be deployed without touching them. I've managed to never touch a Kubernetes yaml thanks to it.

blondin 5y ago

so it's like terraform or pulumi but scoped down to docker and using python?

afroisalreadyin 5y ago
I would say it's more similar to Pulumi, as the main aim is to avoid markup (or a simplified programming language, like HCL in the case of terraform). The other two are correct though: Only for Docker and in Python.

emacsen 5y ago

This looks potentially very interesting indeed! Thanks for sharing. How tied is it to the Docker daemon? Instead of Docker, could one use podman, which doesn't need root.

afroisalreadyin 5y ago
That's a good question, I think only this class would need to be reimplemented for podman: https://github.com/afroisalreadyinu/miniboss/blob/main/miniboss/docker_client.py https://github.com/afroisalreadyinu/miniboss/blob/main/minib... . The question is of course whether there are any incompatibilities in terms of networking etc.
- emacsen 5y ago
  The biggest difference/issue is that podman doesn't provide a daemon of events on its own. It's docker-compose compatibility is tied up in essentially re-implementing/emulating that, which partially defeats the purpose. My main need for the daemon component is https://github.com/nginx-proxy/acme-companion https://github.com/nginx-proxy/acme-companion which listens to docker events to know when a new service is available to proxy.