6 ms·

Pyinfra automates infrastructure super fast at scale

147 points by gjvc 4y ago

turtlebits 4y ago
Does this have any advantages over Ansible? It seems to have some config management, but not as robust or well documented. Ansible also does parallel remote execution.
- dal 4y ago
  I guess one would be that it is python and not Ansible.
- karlicoss 4y ago
  Yes. - you use an actual programming language (which you're likely to already know) instead of desperately figuring out how to simulate a for loop in yaml or to replace a part of a string, or whatnot - you have all Python static analysis tools at your disposal. mypy/pylint etc can make your deploys less error prone - you can easily implement a custom DSL in python, so deploys ends up more declarative and with less copy pasting - it's easy to implement custom arbitrary operations (since you're working in python) -- and again you can reuse variables, etc instead of having to use some weird templating or pass them in command line
  - usrme 4y ago
    At what point are you just re-inventing what Ansible already does if you're going down the DSL path? I sometimes feel like I'm in the minority, but I truly do like Ansible and if you slap Mitogen[1] in front of it makes the whole process even more enjoyable! I'm honestly trying to understand whether this hating on Ansible is more of a bandwagon thing (similar to how people shit on the Python dependency management ecosystem) and stems from having bad experiences that have completely turned them off of it. Honest questions in response to your points: 1. What are you deploying where you would need static analysis? 2. If you did end up writing your own DSL on top of Python at what point would you concede that it's probably better to use someone's else creation as opposed to forever iterating on your own thing for the sake of using your own thing? 3. Why would you want "custom arbitrary operations" and how custom are they that Ansible and Jinja2 can't do that for you in usage patterns that are far more common than your own creation would be? --- [1]: https://mitogen.networkgenomics.com/ansible_detailed.html https://mitogen.networkgenomics.com/ansible_detailed.html
    - mdeeks 4y ago
      Does mitogen really only support up to Ansible 2.9? Thats a couple years old isn't it? I was really excited for it until I saw that :(
      - PaywallBuster 4y ago
        Have releases up to 2.13 https://github.com/willhallonline/docker-ansible https://github.com/willhallonline/docker-ansible Still multiple major releases behind but still Also I wonder how often the new releases are used (market share?) since so many people moved to cloud/k8s
      - usrme 4y ago
        I'm on 'ansible==5.5.0' / 'ansible-core==2.12.3' and am using Mitogen with no issues whatsoever.
    - traverseda 4y ago
      One example where I personally ran into trouble with ansible was in dealing with SSL certificates. In production we use letsencrypt to generate SSL certificates as you'd expect, but for testing we were using self-signed certificates. This was surprisingly complicated under the ansible model, basically I ended up writing a reusable playbook that worked more or less as a function, and running a hook at the end to actually create the keys. Due to the specifics of our app depending on what kind you needed the actual certificate paths got the certificates were different. So I ended up doing some crazy hacks passing around state instead of just writing a function.
      - Terretta 4y ago
        Feels as though perhaps this should have dropped from playbook into a module? https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_general.html https://docs.ansible.com/ansible/latest/dev_guide/developing... Then just writing a function.
        traverseda 4y ago
  - blondin 4y ago
    so pulumi[1]? [1]: https://github.com/pulumi/pulumi https://github.com/pulumi/pulumi
    - latchkey 4y ago
      As much as I want to love pulumi, their ssh story seems missing. It feels like pulumi is dedicated towards managing cloud instances through cloud apis, while I'd want to manage bare metal instances through ssh (or a local daemon). I'd love correction on this.
      - pbecotte 4y ago
        no, you're right. Terraform/Pulumi are much more focused on provisioning resources, while Ansible has its sweet spot in configuring a machine once its running.
catstack 4y ago
Does any one have any links to some good writes ups on this being used in a bigger environment? Anyone want to post their experience using it here?
- mdaniel 4y ago
  There was more commentary about it when it was posted in 2020: https://news.ycombinator.com/item?id=23487178 https://news.ycombinator.com/item?id=23487178 (although I didn't go through them all looking for endorsements)
- Fizzadar 4y ago
  I wrote pyinfra so am biased but we used it at my previous company on a fleet of roughly 1.5k physical boxes (including deploys spanning all of them at once) extensively, so it scales to that level well. We had probably 30 groups within that each with their own deployments as well and all tied together using a single Jenkins box. I would love to test beyond that but haven’t had the opportunity sadly! Performance and scale is a headline feature so will always look to push this further.
gautamdivgi 4y ago
What’s the difference between pyinfra and fabric? Both seem to be similar.
- BiteCode_dev 4y ago
  idempotence
mdeeks 4y ago
How is the performance with larger deploy scripts? Ansible really struggles with hundreds of hosts and for simple things like uploading a dozen files to each host. The benchmarks are promising but the test really only does four things. https://docs.pyinfra.com/en/2.x/performance.html https://docs.pyinfra.com/en/2.x/performance.html
- Fizzadar 4y ago
  Way back when I started pyinfra my only goal was performance I recall turning a 40 minute ansible run on 500ish hosts into 5 minutes. Big caveat: anecdotal and pretty old information there. It’s been too long since I did perf testing but I still believe pyinfra Will significantly outperform ansible for the same tasks, you’ll have to try it ;) (also if perf is not good please raise an issue perf is absolutely a feature I wish to maintain)
reifyx 4y ago
I haven't used this for large deployments but I use it for my personal server and it works perfectly. Almost everything is built in and I can easily write my own custom operations when I need them. Documentation is good and the operations are well designed. Only downside is I couldn't make it work with my SSH agent, but that might be a problem with Paramiko and not Pyinfra.
- Fizzadar 4y ago
  > Only downside is I couldn't make it work with my SSH agent, but that might be a problem with Paramiko and not Pyinfra. Interested to hear more about this, might be something that can be added to paramiko (or on top of).
dang 4y ago
Related: Show HN: pyinfra v2 - https://news.ycombinator.com/item?id=30999030 https://news.ycombinator.com/item?id=30999030 - April 2022 (2 comments) Pyinfra v2.0 Released - https://news.ycombinator.com/item?id=30973976 https://news.ycombinator.com/item?id=30973976 - April 2022 (3 comments) Show HN: Pyinfra v1.4 - https://news.ycombinator.com/item?id=26983266 https://news.ycombinator.com/item?id=26983266 - April 2021 (3 comments) Pyinfra – automate infrastructure super fast at scale - https://news.ycombinator.com/item?id=23487178 https://news.ycombinator.com/item?id=23487178 - June 2020 (64 comments) Pyinfra v0.2 - https://news.ycombinator.com/item?id=12956784 https://news.ycombinator.com/item?id=12956784 - Nov 2016 (2 comments)
Fizzadar 4y ago
Hey all, I (& many amazing contributors) built pyinfra! Crazy to see it on the homepage I’ll do my best to answer any questions!
- dolni 4y ago
  Does pyinfra play nicely with devices that provide an SSH interface, but aren't traditional Unix boxes? I use Ansible currently to manage the config on a Ubiquiti EdgeRouter. Using actual Python would be even better.
  - Fizzadar 4y ago
    Yes! I know of people that have connected to systems without any traditional shell/unix stuff. You can change the shell (or have none) used if needed using the global `_shell_executable` global arg (https://docs.pyinfra.com/en/2.x/arguments.html#shell-control-features https://docs.pyinfra.com/en/2.x/arguments.html#shell-control...). Most operations rely on various Linux/similar tools but the `server.shell` operation plus shell flag above should get you connecting and executing commands. Please do reach out if this doesn’t work for your setup!
  - jononor 4y ago
    Interested in the same usecase, but for Mikrotik routers.
- ngcc_hk 4y ago
  Read a few pages I gave up. May I ask - what is its use case? - security - charge model - self hosting Sorry if miss a faq on the site ?
- inhumantsar 4y ago
  Late reply but I'm curious if the focus is idempotent execution (like ansible) or desired state (like puppet). I'm guessing the former since it's agentless.
ebreton 4y ago
How would you use pyinfra to manage a fleet of edge devices ? The main difference with a park of servers being that the edge devices are mounted on vehicles and therefore switching on and off with their own rhythm each. We cutrently get an alert from each device when an update is needed, ssh into it, and docker-compose the latest services.
- emrahayanoglu 4y ago
  Nice question! I am also wondering how it helps for the IoT devices... Up to now, What we found out is that Pull way makes more sense instead of Push way for the fleet of more than 500Ks devices in the field. We use Ansible Pull, but would also considering of such a way through GitOps approach. Not sure, if it would be also provided via Pyinfra to get such stuff via Pull way.
v3ss0n 4y ago
That the tool i am looking for a while. I used fabric before but it's too old and limited, no longer maintained. Plumini is focused on cloud, not on dedicated or bare metal servers. Why I don't like Ansible is : YAML is a bitch to maintain and python is much more dynamic and flexible. So this is the real thing. I am going to use it for my DevOps life. Keep it up!!
zach_garwood 4y ago
I use pyinfra for my personal projects and I love it. Compared to Ansible it's a breath of fresh air. Ansible is very straightforward to start with, but quickly devolves whenever you step off the beaten path, and you end up with a tangled mess of unintelligible YAML after a while. Pyinfra is just Python. If you can read Python, you can understand what the deployment module is doing.
darthrupert 4y ago
It seems like Pyinfra is sort of declarative in the same style as Ansible. Is this a correct assumption?
rkwasny 4y ago
Pyinfra is great, simple and no DSL! just pure python

147 points by gjvc 4y ago

Does this have any advantages over Ansible? It seems to have some config management, but not as robust or well documented. Ansible also does parallel remote execution.

dal 4y ago
I guess one would be that it is python and not Ansible.
karlicoss 4y ago
Yes. - you use an actual programming language (which you're likely to already know) instead of desperately figuring out how to simulate a for loop in yaml or to replace a part of a string, or whatnot - you have all Python static analysis tools at your disposal. mypy/pylint etc can make your deploys less error prone - you can easily implement a custom DSL in python, so deploys ends up more declarative and with less copy pasting - it's easy to implement custom arbitrary operations (since you're working in python) -- and again you can reuse variables, etc instead of having to use some weird templating or pass them in command line
- usrme 4y ago
  At what point are you just re-inventing what Ansible already does if you're going down the DSL path? I sometimes feel like I'm in the minority, but I truly do like Ansible and if you slap Mitogen[1] in front of it makes the whole process even more enjoyable! I'm honestly trying to understand whether this hating on Ansible is more of a bandwagon thing (similar to how people shit on the Python dependency management ecosystem) and stems from having bad experiences that have completely turned them off of it. Honest questions in response to your points: 1. What are you deploying where you would need static analysis? 2. If you did end up writing your own DSL on top of Python at what point would you concede that it's probably better to use someone's else creation as opposed to forever iterating on your own thing for the sake of using your own thing? 3. Why would you want "custom arbitrary operations" and how custom are they that Ansible and Jinja2 can't do that for you in usage patterns that are far more common than your own creation would be? --- [1]: https://mitogen.networkgenomics.com/ansible_detailed.html https://mitogen.networkgenomics.com/ansible_detailed.html
  - mdeeks 4y ago
    Does mitogen really only support up to Ansible 2.9? Thats a couple years old isn't it? I was really excited for it until I saw that :(
    - PaywallBuster 4y ago
      Have releases up to 2.13 https://github.com/willhallonline/docker-ansible https://github.com/willhallonline/docker-ansible Still multiple major releases behind but still Also I wonder how often the new releases are used (market share?) since so many people moved to cloud/k8s
    - usrme 4y ago
      I'm on 'ansible==5.5.0' / 'ansible-core==2.12.3' and am using Mitogen with no issues whatsoever.
  - traverseda 4y ago
    One example where I personally ran into trouble with ansible was in dealing with SSL certificates. In production we use letsencrypt to generate SSL certificates as you'd expect, but for testing we were using self-signed certificates. This was surprisingly complicated under the ansible model, basically I ended up writing a reusable playbook that worked more or less as a function, and running a hook at the end to actually create the keys. Due to the specifics of our app depending on what kind you needed the actual certificate paths got the certificates were different. So I ended up doing some crazy hacks passing around state instead of just writing a function.
    - Terretta 4y ago
      Feels as though perhaps this should have dropped from playbook into a module? https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_general.html https://docs.ansible.com/ansible/latest/dev_guide/developing... Then just writing a function.
      - traverseda 4y ago
- blondin 4y ago
  so pulumi[1]? [1]: https://github.com/pulumi/pulumi https://github.com/pulumi/pulumi
  - latchkey 4y ago
    As much as I want to love pulumi, their ssh story seems missing. It feels like pulumi is dedicated towards managing cloud instances through cloud apis, while I'd want to manage bare metal instances through ssh (or a local daemon). I'd love correction on this.
    - pbecotte 4y ago
      no, you're right. Terraform/Pulumi are much more focused on provisioning resources, while Ansible has its sweet spot in configuring a machine once its running.

catstack 4y ago

Does any one have any links to some good writes ups on this being used in a bigger environment? Anyone want to post their experience using it here?

mdaniel 4y ago
There was more commentary about it when it was posted in 2020: https://news.ycombinator.com/item?id=23487178 https://news.ycombinator.com/item?id=23487178 (although I didn't go through them all looking for endorsements)
Fizzadar 4y ago
I wrote pyinfra so am biased but we used it at my previous company on a fleet of roughly 1.5k physical boxes (including deploys spanning all of them at once) extensively, so it scales to that level well. We had probably 30 groups within that each with their own deployments as well and all tied together using a single Jenkins box. I would love to test beyond that but haven’t had the opportunity sadly! Performance and scale is a headline feature so will always look to push this further.

gautamdivgi 4y ago

What’s the difference between pyinfra and fabric? Both seem to be similar.

BiteCode_dev 4y ago
idempotence

mdeeks 4y ago

How is the performance with larger deploy scripts? Ansible really struggles with hundreds of hosts and for simple things like uploading a dozen files to each host. The benchmarks are promising but the test really only does four things. https://docs.pyinfra.com/en/2.x/performance.html https://docs.pyinfra.com/en/2.x/performance.html

Fizzadar 4y ago
Way back when I started pyinfra my only goal was performance I recall turning a 40 minute ansible run on 500ish hosts into 5 minutes. Big caveat: anecdotal and pretty old information there. It’s been too long since I did perf testing but I still believe pyinfra Will significantly outperform ansible for the same tasks, you’ll have to try it ;) (also if perf is not good please raise an issue perf is absolutely a feature I wish to maintain)

reifyx 4y ago

I haven't used this for large deployments but I use it for my personal server and it works perfectly. Almost everything is built in and I can easily write my own custom operations when I need them. Documentation is good and the operations are well designed. Only downside is I couldn't make it work with my SSH agent, but that might be a problem with Paramiko and not Pyinfra.

Fizzadar 4y ago
> Only downside is I couldn't make it work with my SSH agent, but that might be a problem with Paramiko and not Pyinfra. Interested to hear more about this, might be something that can be added to paramiko (or on top of).

dang 4y ago

Related: Show HN: pyinfra v2 - https://news.ycombinator.com/item?id=30999030 https://news.ycombinator.com/item?id=30999030 - April 2022 (2 comments) Pyinfra v2.0 Released - https://news.ycombinator.com/item?id=30973976 https://news.ycombinator.com/item?id=30973976 - April 2022 (3 comments) Show HN: Pyinfra v1.4 - https://news.ycombinator.com/item?id=26983266 https://news.ycombinator.com/item?id=26983266 - April 2021 (3 comments) Pyinfra – automate infrastructure super fast at scale - https://news.ycombinator.com/item?id=23487178 https://news.ycombinator.com/item?id=23487178 - June 2020 (64 comments) Pyinfra v0.2 - https://news.ycombinator.com/item?id=12956784 https://news.ycombinator.com/item?id=12956784 - Nov 2016 (2 comments)

Fizzadar 4y ago

Hey all, I (& many amazing contributors) built pyinfra! Crazy to see it on the homepage I’ll do my best to answer any questions!

dolni 4y ago
Does pyinfra play nicely with devices that provide an SSH interface, but aren't traditional Unix boxes? I use Ansible currently to manage the config on a Ubiquiti EdgeRouter. Using actual Python would be even better.
- Fizzadar 4y ago
  Yes! I know of people that have connected to systems without any traditional shell/unix stuff. You can change the shell (or have none) used if needed using the global `_shell_executable` global arg (https://docs.pyinfra.com/en/2.x/arguments.html#shell-control-features https://docs.pyinfra.com/en/2.x/arguments.html#shell-control...). Most operations rely on various Linux/similar tools but the `server.shell` operation plus shell flag above should get you connecting and executing commands. Please do reach out if this doesn’t work for your setup!
- jononor 4y ago
  Interested in the same usecase, but for Mikrotik routers.
ngcc_hk 4y ago
Read a few pages I gave up. May I ask - what is its use case? - security - charge model - self hosting Sorry if miss a faq on the site ?
inhumantsar 4y ago
Late reply but I'm curious if the focus is idempotent execution (like ansible) or desired state (like puppet). I'm guessing the former since it's agentless.

ebreton 4y ago

How would you use pyinfra to manage a fleet of edge devices ? The main difference with a park of servers being that the edge devices are mounted on vehicles and therefore switching on and off with their own rhythm each. We cutrently get an alert from each device when an update is needed, ssh into it, and docker-compose the latest services.

emrahayanoglu 4y ago
Nice question! I am also wondering how it helps for the IoT devices... Up to now, What we found out is that Pull way makes more sense instead of Push way for the fleet of more than 500Ks devices in the field. We use Ansible Pull, but would also considering of such a way through GitOps approach. Not sure, if it would be also provided via Pyinfra to get such stuff via Pull way.

v3ss0n 4y ago

That the tool i am looking for a while. I used fabric before but it's too old and limited, no longer maintained. Plumini is focused on cloud, not on dedicated or bare metal servers. Why I don't like Ansible is : YAML is a bitch to maintain and python is much more dynamic and flexible. So this is the real thing. I am going to use it for my DevOps life. Keep it up!!

zach_garwood 4y ago

I use pyinfra for my personal projects and I love it. Compared to Ansible it's a breath of fresh air. Ansible is very straightforward to start with, but quickly devolves whenever you step off the beaten path, and you end up with a tangled mess of unintelligible YAML after a while. Pyinfra is just Python. If you can read Python, you can understand what the deployment module is doing.

darthrupert 4y ago

It seems like Pyinfra is sort of declarative in the same style as Ansible. Is this a correct assumption?

rkwasny 4y ago

Pyinfra is great, simple and no DSL! just pure python