7 ms·

PyPI Migrated to New Infrastructure

52 points by donaldstufft 13y ago

SEJeff 13y ago
They are using salt to configure their infrastructure now instead of chef, interesting! https://github.com/saltstack/salt https://github.com/saltstack/salt
- jonesetc 13y ago
  They felt dirty using a ruby project.
  - eberfreitas 13y ago
    Ansible would be another awesome python solution to use in this case.
    - donpdonp 13y ago
      as a solo developer doing my own devops, ansible has been fantastic for its near-zero infrastructure overhead.
  - jbeja 13y ago
    Why you have the notion that they despite ruby?
    - deleted 13y ago
      [deleted]
- steder 13y ago
  Being a Pythonista and working with Chef for the last year I think that this is makes some sense. We've been recently contemplating the same migration. Of course, this case the number of servers being managed is low enough that there isn't really a going to be much of a performance difference between Chef or Salt so this is likely just the preference of the current maintainers.
- rdtsc 13y ago
  +1 for Salt. I really like it. I came to it not as a replacement of puppet or chef or other mature solutions but as a better "log in with ssh, run commands and dump some some config scripts" system. Ansible was another contender but Windows support in Salt won over (and Salt now also has ssh-only mode I hear).
- stonesixone 13y ago
  Their reason for using Salt (from the Distutils mailing list [1]): "> Can you say a little about the choice to use Salt instead of Chef? I don't really care either way, but am just curious. Is it because Salt is written in Python, or were there other reasons (functionality, etc)? "I’d need to ask Ernest to be sure, but I believe it was mostly that he was more familiar with it. The fact that it was written in Python was a bonus as well ;) I don’t think that there was anything that Chef was missing or that Salt had over Chef, just familiarity of the person who did most of the work. I’ll double check with Ernest to make sure there wasn’t another reason :)" [1] https://mail.python.org/pipermail/distutils-sig/2014-January/023527.html https://mail.python.org/pipermail/distutils-sig/2014-January...
  - donaldstufft 13y ago
    I have actually now asked Ernest, and that was basically the reason. Familiarity. He did most of the work and was more familiar so it enabled him to complete the work quicker. We still do use Chef on the python.org infra, just not for PyPI itself.
e12e 13y ago
Well, great news for pypi. As for using salt -- I thought: "Well, I guess it's time I had another look, they probably have their security story together now, after all.". So I have a look at the "Masterless Quickstart" and "Installation" pages [1,2] and find: wget -O - http://bootstrap.saltstack.org | sudo sh And the server answering on 443 on that host doesn't have a certificate for that domain, or the bootstrap script (clearly it's not configured for a ssl vhost by that that name/id -- through SNI or otherwise). So, just run some code downloaded over plain http as root on all my servers? Great plan! edit: I think I'll stick with Ansible for now. edit2: For those wanting a look at the salt states, from the thread[3]: > Where are the states stored? https://github.com/python/pypi-salt https://github.com/python/pypi-salt 1: http://docs.saltstack.com/topics/tutorials/quickstart.html http://docs.saltstack.com/topics/tutorials/quickstart.html 2: http://docs.saltstack.com/topics/installation/index.html http://docs.saltstack.com/topics/installation/index.html 3: https://mail.python.org/pipermail/distutils-sig/2014-January/023531.html https://mail.python.org/pipermail/distutils-sig/2014-January...
- jyap 13y ago
  On the Installation page you linked to it clearly mentions: PLATFORM-SPECIFIC INSTALLATION INSTRUCTIONS These guides go into detail how to install salt on a given platform. A QuickStart is just that. Something which serves its purpose as a QuickStart. If you have security concerns then read the source code or test it out in a VM.
  - e12e 13y ago
    It doesn't change the fact that running code that's downloaded over http,unverified by any kind of signature, as root -- is a very bad idea, and just having it there in official documentation for package whose purpose is managing systems gives me a very poor impression of the projects security goals.
- andreasvc 13y ago
  The alternative is: 1. get tarball 2. do full code audit 3. run code However, almost no one will do step 2, so running code piped from wget is effectively equivalent with doing step 1 and 3. https provides some level of security, but a determined attacker can make sure you don't get warnings from their MITM attack, which gets us back to having to do step 2.
mrweasel 13y ago
I'm impressed how little infrastructure they where make due with previously. PyPI hasn't been too bad considering the backend.

52 points by donaldstufft 13y ago

They are using salt to configure their infrastructure now instead of chef, interesting! https://github.com/saltstack/salt https://github.com/saltstack/salt

jonesetc 13y ago
They felt dirty using a ruby project.
- eberfreitas 13y ago
  Ansible would be another awesome python solution to use in this case.
  - donpdonp 13y ago
    as a solo developer doing my own devops, ansible has been fantastic for its near-zero infrastructure overhead.
- jbeja 13y ago
  Why you have the notion that they despite ruby?
  - deleted 13y ago
    [deleted]
steder 13y ago
Being a Pythonista and working with Chef for the last year I think that this is makes some sense. We've been recently contemplating the same migration. Of course, this case the number of servers being managed is low enough that there isn't really a going to be much of a performance difference between Chef or Salt so this is likely just the preference of the current maintainers.
rdtsc 13y ago
+1 for Salt. I really like it. I came to it not as a replacement of puppet or chef or other mature solutions but as a better "log in with ssh, run commands and dump some some config scripts" system. Ansible was another contender but Windows support in Salt won over (and Salt now also has ssh-only mode I hear).
stonesixone 13y ago
Their reason for using Salt (from the Distutils mailing list [1]): "> Can you say a little about the choice to use Salt instead of Chef? I don't really care either way, but am just curious. Is it because Salt is written in Python, or were there other reasons (functionality, etc)? "I’d need to ask Ernest to be sure, but I believe it was mostly that he was more familiar with it. The fact that it was written in Python was a bonus as well ;) I don’t think that there was anything that Chef was missing or that Salt had over Chef, just familiarity of the person who did most of the work. I’ll double check with Ernest to make sure there wasn’t another reason :)" [1] https://mail.python.org/pipermail/distutils-sig/2014-January/023527.html https://mail.python.org/pipermail/distutils-sig/2014-January...
- donaldstufft 13y ago
  I have actually now asked Ernest, and that was basically the reason. Familiarity. He did most of the work and was more familiar so it enabled him to complete the work quicker. We still do use Chef on the python.org infra, just not for PyPI itself.

e12e 13y ago

Well, great news for pypi. As for using salt -- I thought: "Well, I guess it's time I had another look, they probably have their security story together now, after all.". So I have a look at the "Masterless Quickstart" and "Installation" pages [1,2] and find: wget -O - http://bootstrap.saltstack.org | sudo sh And the server answering on 443 on that host doesn't have a certificate for that domain, or the bootstrap script (clearly it's not configured for a ssl vhost by that that name/id -- through SNI or otherwise). So, just run some code downloaded over plain http as root on all my servers? Great plan! edit: I think I'll stick with Ansible for now. edit2: For those wanting a look at the salt states, from the thread[3]: > Where are the states stored? https://github.com/python/pypi-salt https://github.com/python/pypi-salt 1: http://docs.saltstack.com/topics/tutorials/quickstart.html http://docs.saltstack.com/topics/tutorials/quickstart.html 2: http://docs.saltstack.com/topics/installation/index.html http://docs.saltstack.com/topics/installation/index.html 3: https://mail.python.org/pipermail/distutils-sig/2014-January/023531.html https://mail.python.org/pipermail/distutils-sig/2014-January...

jyap 13y ago
On the Installation page you linked to it clearly mentions: PLATFORM-SPECIFIC INSTALLATION INSTRUCTIONS These guides go into detail how to install salt on a given platform. A QuickStart is just that. Something which serves its purpose as a QuickStart. If you have security concerns then read the source code or test it out in a VM.
- e12e 13y ago
  It doesn't change the fact that running code that's downloaded over http,unverified by any kind of signature, as root -- is a very bad idea, and just having it there in official documentation for package whose purpose is managing systems gives me a very poor impression of the projects security goals.
andreasvc 13y ago
The alternative is: 1. get tarball 2. do full code audit 3. run code However, almost no one will do step 2, so running code piped from wget is effectively equivalent with doing step 1 and 3. https provides some level of security, but a determined attacker can make sure you don't get warnings from their MITM attack, which gets us back to having to do step 2.

mrweasel 13y ago

I'm impressed how little infrastructure they where make due with previously. PyPI hasn't been too bad considering the backend.