3 ms·
Ask HN: Would you know a “Python idempotent incremental update manager”?
I maintain a fleet of edge devices, including their embedded software written in python. There is a release every two weeks or so, which usually consists on pulling a new set of docker images.
My current pain comes from the fact that, sometimes, I have to update the docker-compose.yml on the unit, or add a library, or change something outside of the unit.
I am currently writing a (python) script and a documented process for every update, which is gently becoming a pain in the xxx for the units which are not available so often, and for which I have a few updates late.
I would love to have this pile of scripts ran through automatically with a higher level command, and create the skeleton of a new migration script on demand.
Hence the question: how do you manage such remote incremental update manager ? I am looking for something as simple as alembic, but obviously to run scripts not DB migrations.
- Someone 4y agoYou wrote “sometimes, I have to update the docker-compose.yml on the unit, or add a library, or change something outside of the unit.” and “I am looking for something as simple as alembic, but obviously to run scripts not DB migrations.” I don’t see how having to run scripts follows from the first. Can’t you put the stuff that lives outside docker images in a git repo and pull changes from that? Initial install would be a git clone and updates a git pull (maybe with a reboot to make sure nothing is running that’s using outdated stuff)
- ebreton 4y agoThe repo will provide a way to pull new scripts for new updates. But I would still need the framework to iterate through all of them
- Someone 4y agoI think you either didn’t mention some requirement, or are too focused on “I need to run scripts to do these updates” because that’s what you use now. To do “sometimes, I have to update the docker-compose.yml on the unit, or add a library, or change something outside of the unit.”, I don’t see what else than git pull you would need, if you put docker-compose.yml and the libraries in the repo. rsync or other sync programs also should work.
- yuppie_scum 4y agoDependabot?
- ebreton 4y agoThanks for the idea ! I will try pyinfra first, which provides a way to connect to the fleet and then to execute a whole set of scripts. Whereas Dependabot's perimeter looks more contained on dependencies update.
- fermigier 4y agoNot sure I understand the problem precisely, a couple of thoughts from what I understand: 1) Aren't (Docker/OCI) containers supposed to be immutable (at least, as a best practice) ? 2) Have you looked at Pyinfra and would it help solve your problem ?
- ebreton 4y ago1) yes, docker containers are immutable. The pain comes more from updates that are around them 2) Trying pyinfra. Thanks for pointing it to me ! Looks promising indeed :)
- ebreton 4y agoMy feedback for those who helped me or are interested in following up :) Pyinfra was the perfect match. I have used their "operations" to pile up the actions that come with the next release: - A new package (apt) - an updtated pip dependency - checking a file - and docker-compose actions to update all containers I was able define my "inventory" easily, which allows me to update my whole fleet with one single command. One caveat here, the timeout is not properly configurable with a ProxyJump. Discussion opened on stackoverflow. Pyinfra idempotency and dry-run approaches allow me to run the update as many times as I want. For the next release, would I need to change anything different, I will simply add a new operation. The beauty of it is that all operations are strictly executed in the order I define them, which mean that any devices that would not have been updated with the first release, would be still updated appropriately with the second release. Cheers !