6 ms·

Be aware that PyPI isn't a great way to get accurate statistics. One of the reasons is when tox tests via a matrix, Python 2.7 is usually the first python liste

Be aware that PyPI isn't a great way to get accurate statistics. One of the reasons is when tox tests via a matrix, Python 2.7 is usually the first python listed, and if it fails then Python > 3 will not be tested. Also, it'll reuse libraries from the pip download cache. I don't doubt that python 2.7 accounts for the majority of the downloads, but I sincerely doubt 2.7 usage is at ~85%. 2.6 usage is probably almost exclusively RHEL/CentOS 5/6 system pythons and not representative of actual projects.

viraptor 11y ago
Also it's important to remember that system-wide packages get installed once, while pypi stuff will get redownloaded a lot. It means that for example users of recent arch, ubuntu and other distros which install python3 by default are more likely to have some of the dependencies already there, while anyone installing just python2 base package will download deps from pypi. Also pip itself got upgraded to use caching by default not that long ago. All distributions defaulting to python2 will still have old pip which redownloads your packages on every installation (`tox -r` for example). More recent distros will give you tox which caches the downloads locally (which lowers the numbers). Pypi also doesn't count private repos. If openstack started testing commits using pypi repository and turned off caching, I'm guessing it would add at least 10% to python 3.4. TL;DR: pypi stats are interesting, but don't rely on them to give you precise answer.
- voltagex_ 11y ago
  Debian Popcon is interesting too, but don't rely on it either: https://qa.debian.org/popcon-graph.php?packages=python3.4%2C+python2.7&show_installed=on&want_legend=on&want_ticks=on&from_date=&to_date=&hlght_date=&date_fmt=%25Y-%25m&beenhere=1 https://qa.debian.org/popcon-graph.php?packages=python3.4%2C...
shadowmint 11y ago
Be aware that PyPI isn't a great way to get accurate statistics. You can't just ignore a bunch of statistics because they aren't convenient to you; that's not how maths works. Statistical sampling is based on the premise that your sample is randomized from the total population; I would argue that the stats from pypi represent a reasonable random sample of python users around the world. Certainly, you can argue otherwise... if you care to provide some meaningful alternative statistics?
- scrollaway 11y ago
  GP has argued otherwise. PyPI sampling is not random. It is heavily influenced by automatic downloads and such and completely ignores that most simple 2.x libs / non-3.3-tagged libs will work just fine on python 3. In other words, it is neither random nor representative of python 3 usage, failing the premise you yourself mentioned. There's no easy way to determine Python 3 usage overall. Your best bet is to go around polling companies of all sizes as well as indie devs. What you will find is that: 1. Some amount of people still use Python 2.x because they don't know any better 2. Some amount of people want to use 3.x, but still use Python 2.x because of incompatible dependencies. 3. Some amount of people (companies mostly) still use Python 2.x because a migration to 3.x is currently too costly. 1 -> This is partly caused by FUD. Hating on Python 3 has become a popular past time of people who don't know how to be productive otherwise. It's also caused by the very high amount of tutorials etc around the web that "require" 2.x because, for example, they have an unparenthesized print call (relevant XKCD: 353). Arguably this is one of the big failures of the 3.x push. 2 -> That number is constantly diminishing as more dependencies become available on 3.x. The most recent big win is Scrapy, a massively popular scraping framework, which just finished its 3.x port a couple weeks ago. It's an ongoing process - we are in a much better state than we used to be, the Py3WOS is looking almost entirely green. 3 -> This number is mostly irrelevant as it tends to be in closed ecosystems. So maybe CloudShareBuzzr Inc. is using Python 2.7 internally, but they don't produce anything open source so it doesn't affect the rest of the ecosystem. As new, major features are constantly added to Python 3.x (venv in 3.3, pip in 3.4, async def in 3.5), eventually the cost of switching will be worth it.
  - shadowmint 11y ago
    ...it is neither random nor representative of python 3 usage, failing the premise you yourself mentioned. I'm not arguing the py2 vs py3 band wagon; all I'm saying is that if you choose to dispute the validity of statistics you need to do it properly, not just with 'those stats aren't any good because [arbitrary speculative reason 1] [arbitrary speculative reason 2]'. Can you actually justify these assertions? Or are you just speculating because you don't like the results? Your best bet is to go around polling companies of all sizes as well as indie devs. What you will find is that: Oh what? Come on. Now I can't take you seriously at all. You know what they're going to say already, without even asking? You're just speculating. Stats > guesses. Even if the pypi stats aren't completely reflective of the entire ecosystem, I'm going to argue that they give a pretty decent impression of it, regardless of the mild bias that you've listed above but can't quantify.

viraptor 11y ago

Also it's important to remember that system-wide packages get installed once, while pypi stuff will get redownloaded a lot. It means that for example users of recent arch, ubuntu and other distros which install python3 by default are more likely to have some of the dependencies already there, while anyone installing just python2 base package will download deps from pypi. Also pip itself got upgraded to use caching by default not that long ago. All distributions defaulting to python2 will still have old pip which redownloads your packages on every installation (`tox -r` for example). More recent distros will give you tox which caches the downloads locally (which lowers the numbers). Pypi also doesn't count private repos. If openstack started testing commits using pypi repository and turned off caching, I'm guessing it would add at least 10% to python 3.4. TL;DR: pypi stats are interesting, but don't rely on them to give you precise answer.

voltagex_ 11y ago
Debian Popcon is interesting too, but don't rely on it either: https://qa.debian.org/popcon-graph.php?packages=python3.4%2C+python2.7&show_installed=on&want_legend=on&want_ticks=on&from_date=&to_date=&hlght_date=&date_fmt=%25Y-%25m&beenhere=1 https://qa.debian.org/popcon-graph.php?packages=python3.4%2C...

shadowmint 11y ago

Be aware that PyPI isn't a great way to get accurate statistics. You can't just ignore a bunch of statistics because they aren't convenient to you; that's not how maths works. Statistical sampling is based on the premise that your sample is randomized from the total population; I would argue that the stats from pypi represent a reasonable random sample of python users around the world. Certainly, you can argue otherwise... if you care to provide some meaningful alternative statistics?

scrollaway 11y ago
GP has argued otherwise. PyPI sampling is not random. It is heavily influenced by automatic downloads and such and completely ignores that most simple 2.x libs / non-3.3-tagged libs will work just fine on python 3. In other words, it is neither random nor representative of python 3 usage, failing the premise you yourself mentioned. There's no easy way to determine Python 3 usage overall. Your best bet is to go around polling companies of all sizes as well as indie devs. What you will find is that: 1. Some amount of people still use Python 2.x because they don't know any better 2. Some amount of people want to use 3.x, but still use Python 2.x because of incompatible dependencies. 3. Some amount of people (companies mostly) still use Python 2.x because a migration to 3.x is currently too costly. 1 -> This is partly caused by FUD. Hating on Python 3 has become a popular past time of people who don't know how to be productive otherwise. It's also caused by the very high amount of tutorials etc around the web that "require" 2.x because, for example, they have an unparenthesized print call (relevant XKCD: 353). Arguably this is one of the big failures of the 3.x push. 2 -> That number is constantly diminishing as more dependencies become available on 3.x. The most recent big win is Scrapy, a massively popular scraping framework, which just finished its 3.x port a couple weeks ago. It's an ongoing process - we are in a much better state than we used to be, the Py3WOS is looking almost entirely green. 3 -> This number is mostly irrelevant as it tends to be in closed ecosystems. So maybe CloudShareBuzzr Inc. is using Python 2.7 internally, but they don't produce anything open source so it doesn't affect the rest of the ecosystem. As new, major features are constantly added to Python 3.x (venv in 3.3, pip in 3.4, async def in 3.5), eventually the cost of switching will be worth it.
- shadowmint 11y ago
  ...it is neither random nor representative of python 3 usage, failing the premise you yourself mentioned. I'm not arguing the py2 vs py3 band wagon; all I'm saying is that if you choose to dispute the validity of statistics you need to do it properly, not just with 'those stats aren't any good because [arbitrary speculative reason 1] [arbitrary speculative reason 2]'. Can you actually justify these assertions? Or are you just speculating because you don't like the results? Your best bet is to go around polling companies of all sizes as well as indie devs. What you will find is that: Oh what? Come on. Now I can't take you seriously at all. You know what they're going to say already, without even asking? You're just speculating. Stats > guesses. Even if the pypi stats aren't completely reflective of the entire ecosystem, I'm going to argue that they give a pretty decent impression of it, regardless of the mild bias that you've listed above but can't quantify.