20 ms·

Python 3 in Science: the great migration has begun

86 points by ngoldbaum 11y ago

tzs 11y ago
> Firstly, most users are using either Python 2.7 or 3.4 That was an amusing statement. His data shows 81% using 2.7, so that 3.4 could be replaced with any version and the statement would remain true.
- sfilipov 11y ago
  The usage of 3.4 is 16% and combined with the 81% for 2.7 that gives us 97%. You can't replace 3.4 with "any version" and keep the statement true.
  - coldtea 11y ago
    Check his use of "most" and "or", given that 2.7 is already "most", and 3.4 is 5 times less used.
  - Walkman 11y ago
    "Firstly, most users are using either Python 2.7 or 100" That is 81% (0% version 100) using Python 2.7 or 100. AFAIK anything over 50% is most :)
- mapt 11y ago
  An amusingly redundant statement judged by the rules of formal logic. A highly informative and not-problematic-at-all statement judged by the rules of human conversation.
  - coldtea 11y ago
    No, it's a highly problematic statement by the rules of human conversation, the same way shady media or advertisers play tricks with statistics to misrepresent an issue or product. In fact, whether intensional or not, it's a trick with statistics itself. In that sentense he's presenting the huge majority and a much smaller minority selection as equals in what "most" people use.
- po 11y ago
  Both 2.7 and 3.4 have a special status in that they are the latest of their branches. They will always be included. The author is writing for people who are interested in what they have to do to support pytyhon 3. People want to know: I already have to support 2.7 and 3.4 to 'do the right thing' according to the plan. Is that sufficient or should I worry about supporting more? This is very similar to the process web developers go through in deciding which browsers to support and when to cut off support.
skierscott 11y ago
I'm a scientific user with no incentive to switch; I'm still on Python 2.7. But the release of the @ matrix multiplication operator in Python 3.5 gives me strong incentive to switch. I'll be switching as soon as Anaconda support Python 3.5.
zf00002 11y ago
Is the high % of mac users still on 2.7 partly because that is what OS X ships with?
jcadam 11y ago
I'm currently working on a new project using Python 2.6. You see, I'm creating an analysis tool that absolutely must run on an old, isolated (no internet connection) RedHat system, and have access to only a very limited number of packages I can install. Which, in my case, means Python 2.6, fairly old versions of NumPy and SciPy, and Tk for the GUI. This is actually my first significant project in Python, so naturally I wanted to use the latest and greatest (learning opportunity and all), but no such luck.
- mappu 11y ago
  I've never quite been convinced by this argument. If you can deploy and run .py files, surely you could deploy and run a local copy of python3? It's not like it needs root. The target system not having an internet connection is a best-case for a local python3 package, since the lower attack surface makes package security updates less urgent.
lucb1e 11y ago
Perhaps off topic, but this was the most interesting to me: http://astrofrog.github.io/images/survey_plots/os.svg http://astrofrog.github.io/images/survey_plots/os.svg People really use Linux in scientific communities? Like, non-computer people? In the Netherlands Linux usage (or anything other than Windows and OS X), even among university software engineering students, is almost non-existent. Three out of seventy students I know use it (4%), including myself.
- ngoldbaum 11y ago
  Linux is very common in the physical sciences. Back in the 80s and 90s a Unix box from Sun Microsystems might have been more common, but Linux has been firmly in place for more than a decade now. The audience for this survey is also probably somewhat biased, since it was mostly promoted on twitter. That said, none of my colleagues use Windows. I'd say 75% Mac with the rest various flavors of Linux.
  - pjmlp 11y ago
    Currently I am doing some consulting work in life sciences for well known multinationals. On my specific case, Linux is only used in their HPC clusters and for hosting some DB servers. All researchers use Windows systems as their desktops and control systems for their automated robotic systems.
- santaclaus 11y ago
  I know a ton of (mech, civil, aerospace) engineers who use Linux, including quite a few engineering graduate graduate students who use emacs form the command line as their primary development environment.
- jarboot 11y ago
  I have a friend just starting to get into astronomy research, and he had to learn how to use basic Linux CLI. A ton of tools are written explicitly for Linux!
coldtea 11y ago
>But this is all wrong – we should be teaching new users to use Python 3! New users won't thank you if you teach them Python 2 and they have to migrate all their scripts to Python 3 in a few years... Believe me, they won't have to. Python 2 will stay with us for a loooong time, and most NEW web stuff AND scientific work is done with it. >However, the Python developers have now stated that there will be no Python 2.8 release. Essentially, no new features are going to be added to Python 2. In fact, after 2020 (which is not so far in the future), Python 2 will no longer be supported. Yeah, let's see how this works out. After all it's the same guys who said that Python 3 transition would end with 3 being the default choice by 2014.
- coliveira 11y ago
  Whatever the result of this, the transition Python 2->3 will remain as one of the most troubled ever for a mainstream programming language (of course, nothing beats Perl 5 on this area). If you consider that Java is on version 8 and C++14 is here, we can see that even old languages have done this in a much more organized way.
  - lqdc13 11y ago
    Isn't Python the only one with breaking changes out of this group?
    - unscaled 11y ago
      Which group are you referring to? Mainstream languages? The languages mentioned above? Perl 6 was designed from start to be very breaking, but even putting Perl 6 aside, Ruby had quite a few breaking changes going from 1.8 to 1.9 - and that was a merely point release. Breaking changes are tough, but many libraries with a large codebase managed to update to Python 3 and even maintain dual-version support using compatible features, compatibility libraries and automated conversion tools. For the everyday users things would be much simpler, and hence the main reasons for converting is not "too many breaking changes" but rather "not enough compelling changes".
Alex3917 11y ago
Python 3.2 is the latest version that is PyPy compatible, so support for 3.2 shouldn't be dropped if that's a concern.
krychu 11y ago
> The main reason for Python 2 users to not switch to Python 3 is the lack of motivation/killer features. We need to therefore be more proactive in encouraging people to switch to Python 3 by (a) making sure that any new users are always directed to the latest Python 3 version, and (b) releasing, in the near future, new major versions of packages for Python 3 only, while maintaining long term bugfix support for Python 2 versions. That's the evil right there. Read carefully what's written. "The main reason for Python 2 users to not switch to Python 3 is the lack of motivation/killer features". Which says that users are familiar with what Python 3 has to offer but consider it not good enough. Why would you then jump to a conclusion that you have to be more proactive in directing people to switch to Python 3. Or even better, to stop adding features to what 81% of people use.
- tormeh 11y ago
  Directing people to the newest version is unsolved. If you type "apt-get install python [or pypy]" you get 2.7. This is obviously a problem. Basically, everything here suggests that 2.7 is the official version and 3 is some kind of risky beta.
  - broodbucket 11y ago
    It's anecdotal, but Arch has had Python 3 as default for over a year. I think Fedora is coming close to this too.
    - keenerd 11y ago
      Almost five years[1] in fact. To correct some other common misunderstandings elsewhere, it is not correct to say that python3's "changes are extremely breaking." It is easy to write code that runs under both 2 and 3, and 2to3 can convert 99% of code I've thrown at it[2]. The last one percent is straight up bad code that does crazy things like override the list() function with a variable. The other pain point is hybrid python-C libraries that monkey around with Python internals - but those often have trouble keeping up with mere point releases. Given everything done to make the transition easy, if a library doesn't work under python3 by now it is a code smell. Either the library does weird things under the hood or it is practically unsupported. Regarding "lack of killer features", there are some pretty compelling ones[3]. I've been writing all my code in py3 (or at least making it support both 2 and 3) for years now. [1] https://www.archlinux.org/news/python-is-now-python-3/ https://www.archlinux.org/news/python-is-now-python-3/ [2] About two dozen packages I maintain and a bunch of personal projects. [3] http://asmeurer.github.io/python3-presentation/python3-presentation.pdf http://asmeurer.github.io/python3-presentation/python3-prese...
JulianWasTaken 11y ago
The scientific community should focus on moving to PyPy and helping to excise CPython C extensions from more scientific libraries. That's at least a change that will benefit them.
- coliveira 11y ago
  The scientific community will use whatever makes it easier to achieve its own goals. The idea that you need to use something because it is cooler, more open, fancier, etc., works only in the programming community, where software itself is the end goal.
- ngoldbaum 11y ago
  At a minimum, that would require largely rewriting scipy, pandas, and any other piece of software that uses the CPython C extension interface rather than the FFI. That's a pretty tall order...
  - filmor 11y ago
    SciPy and Pandas use Cython, so if that can ever be made runnable semi-directly in PyPy, those two are dealt with. Only NumPy uses the CPython API directly but this is being worked on as NumPyPy (see http://morepypy.blogspot.de/2015/02/linalg-support-in-pypynumpy.html http://morepypy.blogspot.de/2015/02/linalg-support-in-pypynu...).
    - maxerickson 11y ago
      SciPy uses Cython, but it also uses other C interfaces. The most definitive statement I found in a couple minutes is here: http://www.scipy.org/scipylib/faq.html#how-can-scipy-be-fast-if-it-is-written-in-an-interpreted-language-like-python http://www.scipy.org/scipylib/faq.html#how-can-scipy-be-fast... SciPy uses a variety of methods to generate “wrappers” around these algorithms so that they can be used in Python. Some wrappers were generated by hand coding them in C. The rest were generated using either SWIG or f2py. Some of the newer contributions to SciPy are either written entirely or wrapped with Cython.
nchammas 11y ago
On a related note, Apache Spark recently landed Python 3 support in master, which will be released as part of Spark 1.4 in the next month or so. [0] [0] https://issues.apache.org/jira/browse/SPARK-4897 https://issues.apache.org/jira/browse/SPARK-4897
has2k1 11y ago
Does this[1] type of plot have a name? [1] http://astrofrog.github.io/images/survey_plots/python_vs_experience.svg http://astrofrog.github.io/images/survey_plots/python_vs_exp...
- po 11y ago
  hmm... I would describe it as something like a '2D histogram square-bin plot' a 'categorical plot' or perhaps a variant of a 'mosaic plot' using color density instead of area. http://www.math.yorku.ca/SCS/sugi/sugi17-paper.html http://www.math.yorku.ca/SCS/sugi/sugi17-paper.html http://en.wikipedia.org/wiki/Mosaic_plot http://en.wikipedia.org/wiki/Mosaic_plot
  - has2k1 11y ago
    Thanks, that is a lot more than I hoped for yet very appropriate considering what I plan to do with it.
- peatmoss 11y ago
  In all honesty, I'd call it a table. This plot stood out for me as being unnecessarily a plot. It's tabular data, plain and simple. The x and y axes are simply used to order the margins of the table. They've added colors, which I guess are a little plotty, though I'm not sure the color adds or detracts here.
analog31 11y ago
I'm a scientific user, and work in a R&D environment for scientific instrumentation. I'm probably the most advanced Python user at my site, but have inspired a number of colleagues to give it a try. We use it for pretty much everything but developing our actual commercial software. Perhaps an added dimension worth studying is that Python is so widely used by people who have little chance of even understanding the differences, aside from the print() function. I'm not sure that I could clearly articulate them myself. I migrated to 3.4, just so I could find out and then tell people from my own experience that 3.4 is not broken, and that it supports a sufficiency of packages. In other words I'm using myself as a guinea pig. At the same time, I offer to help people by maintaining my own programs in a way that lets them run on 2.7 or 3.4 systems, e.g., with some kind of "if version > 3" verbiage. Given that I know my audience, I can tell them confidently that they will not get hung up by the 2 vs 3 dichotomy, and that I will help them if it ever becomes an issue. I think the benefits of Python, including the giant ecosystem of packages, outweigh the risks of choosing the wrong version for most of us. Edit: One more thing for beginners, a lot of us have working Python installations on our computers, but we don't quite know how or why. A useful migration tool might be a program that analyzes your 2.x installation, tells you what you've got, and maybe even offers to build an identical 3.x installation for you.
watersb 11y ago
Only 10% Windows? Has ESO switched to Linux?
- frozenport 11y ago
  And half are astrophysics?
BuckRogers 11y ago
Why can't those who want to use Python3 just use 3? Why do they have to push the rest of us over? This read as war against the silent majority to me. The transition has failed. Python is now 2 separate communities and we only have the 'leadership team' to thank for that. Until they make more compromises this will continue. That ~17% userbase is your new Python3 community.
- Animats 11y ago
  There's a definite "screw the 2.x users" attitude in the Python community. Report a bug in 2.7 and see what happens. This attitude is encouraged by Python's little tin god.[1] The big excuse for Python 3 is that it does Unicode. Python has done Unicode since Python 2.6; it just wasn't the default. I've been writing all-Unicode Python for about five years. The killer problem is that many package developers, faced with the 2->3 conversion, abandoned their packages. New packages were written by others to replace them. Users are thus forced to convert to using different packages, packages with different bugs and a much smaller user base. I wrote previously on converting a medium-sized production system from Python 2 to Python 3. I was finding bugs in third-party Python 3 packages, bugs so blatant that they would have been found years ago if the packages were being widely used. This was in the web space; the article indicates similar problems in the scientific space. [1] https://www.python.org/dev/peps/pep-0404/ https://www.python.org/dev/peps/pep-0404/
  - BuckRogers 11y ago
    Worse it becomes an emotional plea, "do the right thing". As if not switching sides to 3.x is a moral issue. And the myth that 3.x is in our best interests or "inevitable". The author is fooled into this even though his own numbers show they can't crack 20% even though 3.0 was released in late 2008. Using those numbers, even if the Python3 community doubles in another 7 years they'll still be at 34%. I think that would be an optimistic outlook as many people are simply finding something else to use instead of move to 3. I'd be as interested in Ruby 3.0 or Go as I would Python3. Many programmers for some reason cannot resist technical churn. To me that's what Python3 is. All things considered not better, not worse. Just different. But with the library issue. My guess is Pyston will catch on and maintain the 2.x line after 2020. Pyston may end up the innovation that Py3 wasn't, but without breaking everyone's code. LLVM JIT compiler with C extension support. It's music to my ears and transitioning from being a Python programmer to a Pyston programmer sounds good. I suspect by 2020 it'll be ready as a spiritual 2.8, and the 80%+ will move to that. Only this core dev team could've achieved defeat from the jaws of success by shooting itself in the face.
jbdigriz 11y ago
I'm assuming you chose science for the basis of this analysis because scientists would be more likely to provide thoughtful replies, if any reply at all correct? I did like the analysis, but once the proselytizing kept popping up, it started to lose value and actually stoke some some of the existing tensions on this matter (we should stop teaching Python 2??) So now we have to come down from the clouds into reality. For one, having limited yourself to scientific fields (sort of) you have literally chosen the tip of the iceberg. Let's ignore the fact that the stats clearly show abysmal adoption of Python 3 among those who make a living by being on the cutting edge and can most afford to adopt. But 786 replies?? And we should stop Python 2 for THAT (even though ~700 in that set aren't using it)??? If you venture into the commercial world, this entire analysis becomes irrelevant. My field of finance and derivative trading is one of the big up and comers in the Python space for a number of reasons but likely to costs - rapid prototyping, powerful testing, ease of training, etc. And I can say with a fair degree of confidence that there are more Python developers at my firm than this entire set - and all of them use Python 2.6. Or at least they did until a multi year upgrade migration finally brought us to Python 2.7 just this year ( and cost millions to achieve). Most are still using 32 bit, develop in Windows (on in house built IDEs) and have some split of deployment between Window client side projects (GUIs) and Linux server deployment. This is not to say we are simpletons - we have vast and complex systems, globally distributed compute grids with tens of thousands of cores (and equivalent disaster recovery clusters), reactive caching graphs, time shifting time series, cutting edge data stores manipulating massive data sets in the tens of billions of rows and basically run huge swaths of the HUNDREDS OF TRILLIONS in notional derivatives that drive the entire global economy. These systems have BILLION dollar budgets. And mine is one firm out of thousands employing tens (if not hundreds) of thousands of well compensated software developers. If my point isn't abundantly clear by now, then to put it frankly: WE (and many other industries like ours) ARE the Python community. And it shifts all your stats to the point where Python 3 isn't even statistically significant. So perhaps it would be wise to work with the elephant in the room instead of trying to pull the rug out from under him. Because elephants are big and don't like that shit. Moreover, as a multitude of examples can attest to, any features left out of Python 2.X ultimately just get developed in house and become closed source, with a huge inefficiency in developing the wheel again and again but also locking up the critical knowledge on the best way to do it, backed by proven production deployments with big money at stake. I don't want to disway this sort of analytical review, but if it's going to be science, it should bear at least some resemblance to it.
deckiedan 11y ago
Python 2.7 is an extremely productive language, for a lot of us. I want to move to 3.x at some point, but there's very little I can't do with 2.7, and it is the default on OSX, and many linux platforms. The one killer feature that would make me move almost straight away would be making it a lot faster. But since I have pypy, I'm much more interested in installing that as my 'extra python version' than I am in going to 3.x straight away.
jincheker 11y ago
My conclusion is: if there were no Python 3, Python would be much better

86 points by ngoldbaum 11y ago

tzs 11y ago

> Firstly, most users are using either Python 2.7 or 3.4 That was an amusing statement. His data shows 81% using 2.7, so that 3.4 could be replaced with any version and the statement would remain true.

sfilipov 11y ago
The usage of 3.4 is 16% and combined with the 81% for 2.7 that gives us 97%. You can't replace 3.4 with "any version" and keep the statement true.
- coldtea 11y ago
  Check his use of "most" and "or", given that 2.7 is already "most", and 3.4 is 5 times less used.
- Walkman 11y ago
  "Firstly, most users are using either Python 2.7 or 100" That is 81% (0% version 100) using Python 2.7 or 100. AFAIK anything over 50% is most :)
mapt 11y ago
An amusingly redundant statement judged by the rules of formal logic. A highly informative and not-problematic-at-all statement judged by the rules of human conversation.
- coldtea 11y ago
  No, it's a highly problematic statement by the rules of human conversation, the same way shady media or advertisers play tricks with statistics to misrepresent an issue or product. In fact, whether intensional or not, it's a trick with statistics itself. In that sentense he's presenting the huge majority and a much smaller minority selection as equals in what "most" people use.
po 11y ago
Both 2.7 and 3.4 have a special status in that they are the latest of their branches. They will always be included. The author is writing for people who are interested in what they have to do to support pytyhon 3. People want to know: I already have to support 2.7 and 3.4 to 'do the right thing' according to the plan. Is that sufficient or should I worry about supporting more? This is very similar to the process web developers go through in deciding which browsers to support and when to cut off support.

skierscott 11y ago

I'm a scientific user with no incentive to switch; I'm still on Python 2.7. But the release of the @ matrix multiplication operator in Python 3.5 gives me strong incentive to switch. I'll be switching as soon as Anaconda support Python 3.5.

zf00002 11y ago

Is the high % of mac users still on 2.7 partly because that is what OS X ships with?

jcadam 11y ago

I'm currently working on a new project using Python 2.6. You see, I'm creating an analysis tool that absolutely must run on an old, isolated (no internet connection) RedHat system, and have access to only a very limited number of packages I can install. Which, in my case, means Python 2.6, fairly old versions of NumPy and SciPy, and Tk for the GUI. This is actually my first significant project in Python, so naturally I wanted to use the latest and greatest (learning opportunity and all), but no such luck.

mappu 11y ago
I've never quite been convinced by this argument. If you can deploy and run .py files, surely you could deploy and run a local copy of python3? It's not like it needs root. The target system not having an internet connection is a best-case for a local python3 package, since the lower attack surface makes package security updates less urgent.

lucb1e 11y ago

Perhaps off topic, but this was the most interesting to me: http://astrofrog.github.io/images/survey_plots/os.svg http://astrofrog.github.io/images/survey_plots/os.svg People really use Linux in scientific communities? Like, non-computer people? In the Netherlands Linux usage (or anything other than Windows and OS X), even among university software engineering students, is almost non-existent. Three out of seventy students I know use it (4%), including myself.

ngoldbaum 11y ago
Linux is very common in the physical sciences. Back in the 80s and 90s a Unix box from Sun Microsystems might have been more common, but Linux has been firmly in place for more than a decade now. The audience for this survey is also probably somewhat biased, since it was mostly promoted on twitter. That said, none of my colleagues use Windows. I'd say 75% Mac with the rest various flavors of Linux.
- pjmlp 11y ago
  Currently I am doing some consulting work in life sciences for well known multinationals. On my specific case, Linux is only used in their HPC clusters and for hosting some DB servers. All researchers use Windows systems as their desktops and control systems for their automated robotic systems.
santaclaus 11y ago
I know a ton of (mech, civil, aerospace) engineers who use Linux, including quite a few engineering graduate graduate students who use emacs form the command line as their primary development environment.
jarboot 11y ago
I have a friend just starting to get into astronomy research, and he had to learn how to use basic Linux CLI. A ton of tools are written explicitly for Linux!

coldtea 11y ago

>But this is all wrong – we should be teaching new users to use Python 3! New users won't thank you if you teach them Python 2 and they have to migrate all their scripts to Python 3 in a few years... Believe me, they won't have to. Python 2 will stay with us for a loooong time, and most NEW web stuff AND scientific work is done with it. >However, the Python developers have now stated that there will be no Python 2.8 release. Essentially, no new features are going to be added to Python 2. In fact, after 2020 (which is not so far in the future), Python 2 will no longer be supported. Yeah, let's see how this works out. After all it's the same guys who said that Python 3 transition would end with 3 being the default choice by 2014.

coliveira 11y ago
Whatever the result of this, the transition Python 2->3 will remain as one of the most troubled ever for a mainstream programming language (of course, nothing beats Perl 5 on this area). If you consider that Java is on version 8 and C++14 is here, we can see that even old languages have done this in a much more organized way.
- lqdc13 11y ago
  Isn't Python the only one with breaking changes out of this group?
  - unscaled 11y ago
    Which group are you referring to? Mainstream languages? The languages mentioned above? Perl 6 was designed from start to be very breaking, but even putting Perl 6 aside, Ruby had quite a few breaking changes going from 1.8 to 1.9 - and that was a merely point release. Breaking changes are tough, but many libraries with a large codebase managed to update to Python 3 and even maintain dual-version support using compatible features, compatibility libraries and automated conversion tools. For the everyday users things would be much simpler, and hence the main reasons for converting is not "too many breaking changes" but rather "not enough compelling changes".

Alex3917 11y ago

Python 3.2 is the latest version that is PyPy compatible, so support for 3.2 shouldn't be dropped if that's a concern.

krychu 11y ago

> The main reason for Python 2 users to not switch to Python 3 is the lack of motivation/killer features. We need to therefore be more proactive in encouraging people to switch to Python 3 by (a) making sure that any new users are always directed to the latest Python 3 version, and (b) releasing, in the near future, new major versions of packages for Python 3 only, while maintaining long term bugfix support for Python 2 versions. That's the evil right there. Read carefully what's written. "The main reason for Python 2 users to not switch to Python 3 is the lack of motivation/killer features". Which says that users are familiar with what Python 3 has to offer but consider it not good enough. Why would you then jump to a conclusion that you have to be more proactive in directing people to switch to Python 3. Or even better, to stop adding features to what 81% of people use.

tormeh 11y ago
Directing people to the newest version is unsolved. If you type "apt-get install python [or pypy]" you get 2.7. This is obviously a problem. Basically, everything here suggests that 2.7 is the official version and 3 is some kind of risky beta.
- broodbucket 11y ago
  It's anecdotal, but Arch has had Python 3 as default for over a year. I think Fedora is coming close to this too.
  - keenerd 11y ago
    Almost five years[1] in fact. To correct some other common misunderstandings elsewhere, it is not correct to say that python3's "changes are extremely breaking." It is easy to write code that runs under both 2 and 3, and 2to3 can convert 99% of code I've thrown at it[2]. The last one percent is straight up bad code that does crazy things like override the list() function with a variable. The other pain point is hybrid python-C libraries that monkey around with Python internals - but those often have trouble keeping up with mere point releases. Given everything done to make the transition easy, if a library doesn't work under python3 by now it is a code smell. Either the library does weird things under the hood or it is practically unsupported. Regarding "lack of killer features", there are some pretty compelling ones[3]. I've been writing all my code in py3 (or at least making it support both 2 and 3) for years now. [1] https://www.archlinux.org/news/python-is-now-python-3/ https://www.archlinux.org/news/python-is-now-python-3/ [2] About two dozen packages I maintain and a bunch of personal projects. [3] http://asmeurer.github.io/python3-presentation/python3-presentation.pdf http://asmeurer.github.io/python3-presentation/python3-prese...

JulianWasTaken 11y ago

The scientific community should focus on moving to PyPy and helping to excise CPython C extensions from more scientific libraries. That's at least a change that will benefit them.

coliveira 11y ago
The scientific community will use whatever makes it easier to achieve its own goals. The idea that you need to use something because it is cooler, more open, fancier, etc., works only in the programming community, where software itself is the end goal.
ngoldbaum 11y ago
At a minimum, that would require largely rewriting scipy, pandas, and any other piece of software that uses the CPython C extension interface rather than the FFI. That's a pretty tall order...
- filmor 11y ago
  SciPy and Pandas use Cython, so if that can ever be made runnable semi-directly in PyPy, those two are dealt with. Only NumPy uses the CPython API directly but this is being worked on as NumPyPy (see http://morepypy.blogspot.de/2015/02/linalg-support-in-pypynumpy.html http://morepypy.blogspot.de/2015/02/linalg-support-in-pypynu...).
  - maxerickson 11y ago
    SciPy uses Cython, but it also uses other C interfaces. The most definitive statement I found in a couple minutes is here: http://www.scipy.org/scipylib/faq.html#how-can-scipy-be-fast-if-it-is-written-in-an-interpreted-language-like-python http://www.scipy.org/scipylib/faq.html#how-can-scipy-be-fast... SciPy uses a variety of methods to generate “wrappers” around these algorithms so that they can be used in Python. Some wrappers were generated by hand coding them in C. The rest were generated using either SWIG or f2py. Some of the newer contributions to SciPy are either written entirely or wrapped with Cython.

nchammas 11y ago

On a related note, Apache Spark recently landed Python 3 support in master, which will be released as part of Spark 1.4 in the next month or so. [0] [0] https://issues.apache.org/jira/browse/SPARK-4897 https://issues.apache.org/jira/browse/SPARK-4897

has2k1 11y ago

Does this[1] type of plot have a name? [1] http://astrofrog.github.io/images/survey_plots/python_vs_experience.svg http://astrofrog.github.io/images/survey_plots/python_vs_exp...

po 11y ago
hmm... I would describe it as something like a '2D histogram square-bin plot' a 'categorical plot' or perhaps a variant of a 'mosaic plot' using color density instead of area. http://www.math.yorku.ca/SCS/sugi/sugi17-paper.html http://www.math.yorku.ca/SCS/sugi/sugi17-paper.html http://en.wikipedia.org/wiki/Mosaic_plot http://en.wikipedia.org/wiki/Mosaic_plot
- has2k1 11y ago
  Thanks, that is a lot more than I hoped for yet very appropriate considering what I plan to do with it.
peatmoss 11y ago
In all honesty, I'd call it a table. This plot stood out for me as being unnecessarily a plot. It's tabular data, plain and simple. The x and y axes are simply used to order the margins of the table. They've added colors, which I guess are a little plotty, though I'm not sure the color adds or detracts here.

analog31 11y ago

I'm a scientific user, and work in a R&D environment for scientific instrumentation. I'm probably the most advanced Python user at my site, but have inspired a number of colleagues to give it a try. We use it for pretty much everything but developing our actual commercial software. Perhaps an added dimension worth studying is that Python is so widely used by people who have little chance of even understanding the differences, aside from the print() function. I'm not sure that I could clearly articulate them myself. I migrated to 3.4, just so I could find out and then tell people from my own experience that 3.4 is not broken, and that it supports a sufficiency of packages. In other words I'm using myself as a guinea pig. At the same time, I offer to help people by maintaining my own programs in a way that lets them run on 2.7 or 3.4 systems, e.g., with some kind of "if version > 3" verbiage. Given that I know my audience, I can tell them confidently that they will not get hung up by the 2 vs 3 dichotomy, and that I will help them if it ever becomes an issue. I think the benefits of Python, including the giant ecosystem of packages, outweigh the risks of choosing the wrong version for most of us. Edit: One more thing for beginners, a lot of us have working Python installations on our computers, but we don't quite know how or why. A useful migration tool might be a program that analyzes your 2.x installation, tells you what you've got, and maybe even offers to build an identical 3.x installation for you.

watersb 11y ago

Only 10% Windows? Has ESO switched to Linux?

frozenport 11y ago
And half are astrophysics?

BuckRogers 11y ago

Why can't those who want to use Python3 just use 3? Why do they have to push the rest of us over? This read as war against the silent majority to me. The transition has failed. Python is now 2 separate communities and we only have the 'leadership team' to thank for that. Until they make more compromises this will continue. That ~17% userbase is your new Python3 community.

Animats 11y ago
There's a definite "screw the 2.x users" attitude in the Python community. Report a bug in 2.7 and see what happens. This attitude is encouraged by Python's little tin god.[1] The big excuse for Python 3 is that it does Unicode. Python has done Unicode since Python 2.6; it just wasn't the default. I've been writing all-Unicode Python for about five years. The killer problem is that many package developers, faced with the 2->3 conversion, abandoned their packages. New packages were written by others to replace them. Users are thus forced to convert to using different packages, packages with different bugs and a much smaller user base. I wrote previously on converting a medium-sized production system from Python 2 to Python 3. I was finding bugs in third-party Python 3 packages, bugs so blatant that they would have been found years ago if the packages were being widely used. This was in the web space; the article indicates similar problems in the scientific space. [1] https://www.python.org/dev/peps/pep-0404/ https://www.python.org/dev/peps/pep-0404/
- BuckRogers 11y ago
  Worse it becomes an emotional plea, "do the right thing". As if not switching sides to 3.x is a moral issue. And the myth that 3.x is in our best interests or "inevitable". The author is fooled into this even though his own numbers show they can't crack 20% even though 3.0 was released in late 2008. Using those numbers, even if the Python3 community doubles in another 7 years they'll still be at 34%. I think that would be an optimistic outlook as many people are simply finding something else to use instead of move to 3. I'd be as interested in Ruby 3.0 or Go as I would Python3. Many programmers for some reason cannot resist technical churn. To me that's what Python3 is. All things considered not better, not worse. Just different. But with the library issue. My guess is Pyston will catch on and maintain the 2.x line after 2020. Pyston may end up the innovation that Py3 wasn't, but without breaking everyone's code. LLVM JIT compiler with C extension support. It's music to my ears and transitioning from being a Python programmer to a Pyston programmer sounds good. I suspect by 2020 it'll be ready as a spiritual 2.8, and the 80%+ will move to that. Only this core dev team could've achieved defeat from the jaws of success by shooting itself in the face.

jbdigriz 11y ago

I'm assuming you chose science for the basis of this analysis because scientists would be more likely to provide thoughtful replies, if any reply at all correct? I did like the analysis, but once the proselytizing kept popping up, it started to lose value and actually stoke some some of the existing tensions on this matter (we should stop teaching Python 2??) So now we have to come down from the clouds into reality. For one, having limited yourself to scientific fields (sort of) you have literally chosen the tip of the iceberg. Let's ignore the fact that the stats clearly show abysmal adoption of Python 3 among those who make a living by being on the cutting edge and can most afford to adopt. But 786 replies?? And we should stop Python 2 for THAT (even though ~700 in that set aren't using it)??? If you venture into the commercial world, this entire analysis becomes irrelevant. My field of finance and derivative trading is one of the big up and comers in the Python space for a number of reasons but likely to costs - rapid prototyping, powerful testing, ease of training, etc. And I can say with a fair degree of confidence that there are more Python developers at my firm than this entire set - and all of them use Python 2.6. Or at least they did until a multi year upgrade migration finally brought us to Python 2.7 just this year ( and cost millions to achieve). Most are still using 32 bit, develop in Windows (on in house built IDEs) and have some split of deployment between Window client side projects (GUIs) and Linux server deployment. This is not to say we are simpletons - we have vast and complex systems, globally distributed compute grids with tens of thousands of cores (and equivalent disaster recovery clusters), reactive caching graphs, time shifting time series, cutting edge data stores manipulating massive data sets in the tens of billions of rows and basically run huge swaths of the HUNDREDS OF TRILLIONS in notional derivatives that drive the entire global economy. These systems have BILLION dollar budgets. And mine is one firm out of thousands employing tens (if not hundreds) of thousands of well compensated software developers. If my point isn't abundantly clear by now, then to put it frankly: WE (and many other industries like ours) ARE the Python community. And it shifts all your stats to the point where Python 3 isn't even statistically significant. So perhaps it would be wise to work with the elephant in the room instead of trying to pull the rug out from under him. Because elephants are big and don't like that shit. Moreover, as a multitude of examples can attest to, any features left out of Python 2.X ultimately just get developed in house and become closed source, with a huge inefficiency in developing the wheel again and again but also locking up the critical knowledge on the best way to do it, backed by proven production deployments with big money at stake. I don't want to disway this sort of analytical review, but if it's going to be science, it should bear at least some resemblance to it.

deckiedan 11y ago

Python 2.7 is an extremely productive language, for a lot of us. I want to move to 3.x at some point, but there's very little I can't do with 2.7, and it is the default on OSX, and many linux platforms. The one killer feature that would make me move almost straight away would be making it a lot faster. But since I have pypy, I'm much more interested in installing that as my 'extra python version' than I am in going to 3.x straight away.

jincheker 11y ago

My conclusion is: if there were no Python 3, Python would be much better