11 ms·

I must say I'm always a little annoyed by the split in Python community around the scientific stack vs the rest. There are two ecosystems, two packaging tools e

by fijal 10y ago

I must say I'm always a little annoyed by the split in Python community around the scientific stack vs the rest. There are two ecosystems, two packaging tools etc (which is fine), but the insistence that there are no other python users that are worth considering (both sides are guilty) is really frustrating.

twa927 10y ago
Hi, I recognize you as a PyPy developer. Do you think it's fair to write that "no one uses PyPy"?. I feel there are quite many companies using PyPy, mostly for web backend. You don't see many usages in other usages like Linux distros or ad-hoc scripts, even when the C extensions are not used, because 1) In 95% of cases, the performance is good enough, 2) the warmup slowdown dwarfs the overall gains. Anyway, I think PyPy doesn't get the attention it deserves - the performance gains are fantastic and the vision of the Python ecosystem as Python-only packages with occasional lightweight C libraries integrated with cffi looks very nice to me.
- fijal 10y ago
  I mean, it's obviously unfair. Even in the scientific community someone uses PyPy. As far as download statistics go, I think between 0.5-1% of downloads from PyPI are for PyPy. This is far from "majority" but far from "noone" as well. For what is worth enough big players use PyPy that the consulting company that I run can stay afloat, so I somehow doubt that's the case :-)
  - nbadg 10y ago
    From an outside perspective with no skin in the game (other than using Python profusely and just generally wanting it to be the "best" possible Python it can be) I wouldn't be so quick to call it obviously unfair. The exact phrasing was: > PyPy is ten years old at this point, but to a first approximation, no one is using it. Using your 0.5% - 1.0% PyPI metric as an approximation for how many people are actually using pypy, I think it's reasonable to say that. I personally would definitely have phrased it differently, but if >=99% of the market isn't using PyPy, then it is far, far, far removed from the "mainstream" market. In fairness, the Python community is, on the whole, large enough to give you a reasonably sustainable niche, but it pales in comparison to the market itself. It's a bit like comparing Facebook to CouchSurfing, to be honest. You know about it, maybe even have some friends who have done it, and hey, it's still 3 million people, but in comparison to Facebook... no one uses it. And that's not a value judgment or anything, it's just a scale of comparison kind of thing. That being said, I share your frustration about the deep schism between web vs scientific in the Python community, and I think it's really a shame for the community as a whole. Unfortunately I'm not sure that's going to change very soon: the web side of things has a tremendous amount of momentum (and money) invested in "their" architecture, and at the same time, the scientific side of things has far, far less patience for pain points in their language tooling. Put differently, if your background and job is programming, you're more likely to view "dealing with this programming problem" as actual work, but if your background and job is "I have this data, and I need to analyze it", then "fiddling with this programming problem" is, at best, a frustrating distraction from your actual task. And I think both arenas need to have a better appreciation for the other: as programmers, our tools generally really do suck for everyone, we're just used to it; as data scientists, we are woefully under-aware of how difficult these programming problems can be. Some more unity would be tremendously beneficial for all. And then on top of it all, there's this whole group of weirdos using Python for stuff like desktop applications (I happen to be in this camp). Good luck finding a packaging and deployment solution there!
- synparb 10y ago
  While I still think it a bit unfair, the exact quote is "...but to a first approximation, no one is using it". So I think (assuming fijal's numbers of 0.5-1% are roughly correct) it is not completely unreasonable to say that PyPy adoption is very limited to date, which was the point of what he was saying (although I think it could have been said more diplomatically).
  - twa927 10y ago
    Yes, some people wondered if PyPy is "the future of Python" - but it didn't take over the Python ecosystem. If the initiatives mentioned in the article will yield a 100%-CPython-compatible implementation that is always faster and doesn't have big warmup slowdown, it could be "future of Python".
    - stuaxo 10y ago
      I've been testing it Gtk stuff each release for a while now, it seems really close to working with the things I need (CPython libs). About 3 years ago, I couldn't have guessed it would come that far + this is as a fan/lurker of their mailing list :)
      - twa927 10y ago
        I'm also surprised that it mostly works with extensions written for CPython. Unfortunately there's a big difference between working in 99.9% of cases vs 100%...
        sanxiyn 10y ago
        Actual target is not 100%, but CPython interversion agreement, because CPython versions (say 3.4 and 3.5) are not 100% compatible with each other. I think PyPy can reach that target some day.
- twblalock 10y ago
  > the vision of the Python ecosystem as Python-only packages with occasional lightweight C libraries integrated with cffi looks very nice to me. I have the opposite view. I'd rather have all of the important libraries written in a language with a safer type system than Python's. C's type system has some safety issues, but it's still safer than Python's. If Python 3's optional type hints could actually be enforced at runtime, I'd be sold on using Python for pretty much everything.
  - rer 10y ago
    How is Python's type system less safe than C's?
  - nnq 10y ago
    > C's type system has some safety issues, but it's still safer than Python's. Uhm. An error in Python code code can lead to a runtime crash with an exception or a bad result ...most of the time. An error in C can lead to anything from an impossibly hard to diagnose memory leak to an exploitable buffer overflow... most of the time. C may be be "statically typed", but I think the number of people capable of writing (and maintaining!) secure C code is incredibly low. Heck, even the openssl devs failed at this at least once. And the fact that C is "static" and Python is "dynamic" does not make C's type system safe. Heck, even js's type "system" is safer than C's if when you use the word safe you refer to, you know, security.
  - wolf550e 10y ago
    A language with undefined behavior is not suitable for new code in 2016 if you have alternatives.
    - twblalock 10y ago
      Python has undefined behavior because it's implementations are written in languages that do. Every implementation of it is written in a language with undefined behavior such as C, or in languages which are themselves implemented in languages with undefined behavior (e.g. Jython, which runs on the JVM, which is written in C and C++). There is no guarantee that an implementation of Python is bug-free, so there is no guarantee that any Python problem is free of undefined behavior. So, if you write a Python library in C or in Python, you'll have undefined behavior either way. Undefined behavior is not a good criteria for choosing which language to use.
- Animats 10y ago
  PyPy finally got $200K of funding, from Mozilla.[1] They're going to support Python 3.5. PyPy ought to be able to use a version of NumPy written in Python. If someone can pound Python's little tin god into making Python's new type annotations have sane semantics (as in enforcing the typing), then NumPy in Python could come up to NumPy speed. The amount of C code used with Python declined somewhat with Python 3, because the C interface changed and things had to be reimplemented. There's pymysql, for example, which is a database connector in pure Python. No more need for the C version. [1] https://morepypy.blogspot.com/2016/08/pypy-gets-funding-from-mozilla-for.html https://morepypy.blogspot.com/2016/08/pypy-gets-funding-from...
  - claudiug 10y ago
    I wish this will happen to ruby. For me, and this is personal, ruby without jruby and truffle is dead
smallnamespace 10y ago
It seems the scientific bits of Python have been moving at a much faster development pace though. Do you have any insight as to why? Personally, I started using Anaconda for everything and just drop down into pip when I find a package that isn't in there already, and it's been working very well.
- synparb 10y ago
  You should also check out https://conda-forge.github.io/ https://conda-forge.github.io/, which is building high quality, community curated conda packages for a lot of stuff that isn't available from the main Continuum channel.
- fijal 10y ago
  I'm not really commenting on the relative merits of the technology or who should use what. It's about the lack of admission that there is other stuff and other requirements
- drauh 10y ago
  For scientific work, I find it worth the effort to compile from source using Intel compilers + MKL (or a compiler + math library that can optimize well for your particular hardware platform).
  - wclax04 10y ago
    aren't all of the default conda channel packages compiled with MKL now?
- stuaxo 10y ago
  Hm, I would consider using Anaconda if it worked for Gtk libraries (GEGL etc) on all 3 platforms (yeah I know this is hard outside Anaconda too).
dagss 10y ago
Former Cython developer here. To get some perspective to this, consider the alternatives for a scientific programmer: MATLAB, R, FORTRAN, Mathematica, or if you're hip, Julia -- all specifically made for scientific programming with 0% general purpose/web/etc. development going on. So I would say that the scientific Python community has been doing extremely well in terms of even using a language that isn't designed ground up for scientific computing. I could write a lot about why that is (and how some of the CS and IT crowd doesn't "get" scientific computing..) -- I'll refrain, I just wanted to say that were you see something and get frustrated, I see the same picture and think it's actually an incredible success, to bring so many scientists onto at least the same ball park as other programmers, even if they are still playing their own game.
- p4wnc6 10y ago
  Slight aside -- I am a long-time user of Cython and would love to contribute, but poking around the source code feels daunting -- like there is a huge learning curve to be able to contribute, and that other contributors will only view newcomers as an annoyance who will drain their time with questions. Given all this, what's the best way for someone to pick up and start contributing to Cython?
  - dagss 10y ago
    It definitely is daunting; in particular because Cython isn't so much a goal as itself as a tool that developers are motivated to work on it because they use it in some other project they care about (e.g., lxml, Sage, ...). So it's been pragmatic development. If you don't already, subscribe to and start to follow cython-devel. First step is probably repeat the question there for more up-to-date info than what I can give (I don't even follow any longer). Think about what you want to achieve/change in Cython. A new feature may be easier than a bugfix, though I'm not sure how many "low hanging features" are left at this point.... Anyway, make sure you understand what the change would involve in changes in the C code. Write a testcase that uses Cython the way you would want it to work (elicit the failure/bug/feature); look at the C code that Cython generates, and make sure you understand why that C code is the wrong code and that you know how you'd want it to look. (Understanding the generated C code at this level and read it almost fluently may take a little bit of getting used to but is an absolute requirement for working with Cython -- eventually you look past all the __pyx_ everywhere). Then somehow try to beat the Cython codebase into submission in generating that C code... as you repeat this process you'll gradually learn the code. It seems like Robert Bradshaw and Stefan Behnel are still around, they are very capable and friendly people and I learned a lot of what I know about programming from them, they were very welcoming to me as a new contributor.
radarsat1 10y ago
> two packaging tools wait, what? I always use pip and/or the ubuntu repositories to install scipy. Is there another package manager for it?
- RodericDay 10y ago
  Anaconda wants you to do as much as possible via the `conda` tool. Maybe that's what they're referring to.
- calycosa 10y ago
  I imagine OP is referring to anaconda[0], which is frequently used in scientific contexts and comes with numpy, scipy and other packages included out of the box as well as its own package manager. 0. https://www.continuum.io/ https://www.continuum.io/
  - synparb 10y ago
    The actual package distribution/management (it is also a replacement for virtualenv) is conda (http://conda.pydata.org/ http://conda.pydata.org/). Since it can be confusing, the difference between conda, anaconda and miniconda is described here: http://conda.pydata.org/docs/faq.html#general-questions http://conda.pydata.org/docs/faq.html#general-questions
  - xigency 10y ago
    There is something upsetting about this website, and it may just be that this company is obviously for-profit, but I don't think it needs as much stock photography.
    - synparb 10y ago
      Conda is 100% free and open source, but is developed by Continuum, which is a company (but one that contributes heavily to many of the key components of the open-source pydata/scipy ecosystem, and whose CEO basically created numpy/scipy for the community at the expense of his academic career). Personally I have had nothing but good interactions with people from Continuum and the software they develop. They (as a company and as the individuals in it) do an incredible amount for the open source community and we all benefit from that. They seem to have worked out a sustainable model for running a company and working in/contributing to open source.
GFK_of_xmaspast 10y ago
Without the scientific stuff, python is just one of a bunch of modern-ish scripting languages out there.

by fijal 10y ago

twa927 10y ago

Hi, I recognize you as a PyPy developer. Do you think it's fair to write that "no one uses PyPy"?. I feel there are quite many companies using PyPy, mostly for web backend. You don't see many usages in other usages like Linux distros or ad-hoc scripts, even when the C extensions are not used, because 1) In 95% of cases, the performance is good enough, 2) the warmup slowdown dwarfs the overall gains. Anyway, I think PyPy doesn't get the attention it deserves - the performance gains are fantastic and the vision of the Python ecosystem as Python-only packages with occasional lightweight C libraries integrated with cffi looks very nice to me.

fijal 10y ago
I mean, it's obviously unfair. Even in the scientific community someone uses PyPy. As far as download statistics go, I think between 0.5-1% of downloads from PyPI are for PyPy. This is far from "majority" but far from "noone" as well. For what is worth enough big players use PyPy that the consulting company that I run can stay afloat, so I somehow doubt that's the case :-)
- nbadg 10y ago
  From an outside perspective with no skin in the game (other than using Python profusely and just generally wanting it to be the "best" possible Python it can be) I wouldn't be so quick to call it obviously unfair. The exact phrasing was: > PyPy is ten years old at this point, but to a first approximation, no one is using it. Using your 0.5% - 1.0% PyPI metric as an approximation for how many people are actually using pypy, I think it's reasonable to say that. I personally would definitely have phrased it differently, but if >=99% of the market isn't using PyPy, then it is far, far, far removed from the "mainstream" market. In fairness, the Python community is, on the whole, large enough to give you a reasonably sustainable niche, but it pales in comparison to the market itself. It's a bit like comparing Facebook to CouchSurfing, to be honest. You know about it, maybe even have some friends who have done it, and hey, it's still 3 million people, but in comparison to Facebook... no one uses it. And that's not a value judgment or anything, it's just a scale of comparison kind of thing. That being said, I share your frustration about the deep schism between web vs scientific in the Python community, and I think it's really a shame for the community as a whole. Unfortunately I'm not sure that's going to change very soon: the web side of things has a tremendous amount of momentum (and money) invested in "their" architecture, and at the same time, the scientific side of things has far, far less patience for pain points in their language tooling. Put differently, if your background and job is programming, you're more likely to view "dealing with this programming problem" as actual work, but if your background and job is "I have this data, and I need to analyze it", then "fiddling with this programming problem" is, at best, a frustrating distraction from your actual task. And I think both arenas need to have a better appreciation for the other: as programmers, our tools generally really do suck for everyone, we're just used to it; as data scientists, we are woefully under-aware of how difficult these programming problems can be. Some more unity would be tremendously beneficial for all. And then on top of it all, there's this whole group of weirdos using Python for stuff like desktop applications (I happen to be in this camp). Good luck finding a packaging and deployment solution there!
synparb 10y ago
While I still think it a bit unfair, the exact quote is "...but to a first approximation, no one is using it". So I think (assuming fijal's numbers of 0.5-1% are roughly correct) it is not completely unreasonable to say that PyPy adoption is very limited to date, which was the point of what he was saying (although I think it could have been said more diplomatically).
- twa927 10y ago
  Yes, some people wondered if PyPy is "the future of Python" - but it didn't take over the Python ecosystem. If the initiatives mentioned in the article will yield a 100%-CPython-compatible implementation that is always faster and doesn't have big warmup slowdown, it could be "future of Python".
  - stuaxo 10y ago
    I've been testing it Gtk stuff each release for a while now, it seems really close to working with the things I need (CPython libs). About 3 years ago, I couldn't have guessed it would come that far + this is as a fan/lurker of their mailing list :)
    - twa927 10y ago
      I'm also surprised that it mostly works with extensions written for CPython. Unfortunately there's a big difference between working in 99.9% of cases vs 100%...
      - sanxiyn 10y ago
        Actual target is not 100%, but CPython interversion agreement, because CPython versions (say 3.4 and 3.5) are not 100% compatible with each other. I think PyPy can reach that target some day.
twblalock 10y ago
> the vision of the Python ecosystem as Python-only packages with occasional lightweight C libraries integrated with cffi looks very nice to me. I have the opposite view. I'd rather have all of the important libraries written in a language with a safer type system than Python's. C's type system has some safety issues, but it's still safer than Python's. If Python 3's optional type hints could actually be enforced at runtime, I'd be sold on using Python for pretty much everything.
- rer 10y ago
  How is Python's type system less safe than C's?
- nnq 10y ago
  > C's type system has some safety issues, but it's still safer than Python's. Uhm. An error in Python code code can lead to a runtime crash with an exception or a bad result ...most of the time. An error in C can lead to anything from an impossibly hard to diagnose memory leak to an exploitable buffer overflow... most of the time. C may be be "statically typed", but I think the number of people capable of writing (and maintaining!) secure C code is incredibly low. Heck, even the openssl devs failed at this at least once. And the fact that C is "static" and Python is "dynamic" does not make C's type system safe. Heck, even js's type "system" is safer than C's if when you use the word safe you refer to, you know, security.
- wolf550e 10y ago
  A language with undefined behavior is not suitable for new code in 2016 if you have alternatives.
  - twblalock 10y ago
    Python has undefined behavior because it's implementations are written in languages that do. Every implementation of it is written in a language with undefined behavior such as C, or in languages which are themselves implemented in languages with undefined behavior (e.g. Jython, which runs on the JVM, which is written in C and C++). There is no guarantee that an implementation of Python is bug-free, so there is no guarantee that any Python problem is free of undefined behavior. So, if you write a Python library in C or in Python, you'll have undefined behavior either way. Undefined behavior is not a good criteria for choosing which language to use.
Animats 10y ago
PyPy finally got $200K of funding, from Mozilla.[1] They're going to support Python 3.5. PyPy ought to be able to use a version of NumPy written in Python. If someone can pound Python's little tin god into making Python's new type annotations have sane semantics (as in enforcing the typing), then NumPy in Python could come up to NumPy speed. The amount of C code used with Python declined somewhat with Python 3, because the C interface changed and things had to be reimplemented. There's pymysql, for example, which is a database connector in pure Python. No more need for the C version. [1] https://morepypy.blogspot.com/2016/08/pypy-gets-funding-from-mozilla-for.html https://morepypy.blogspot.com/2016/08/pypy-gets-funding-from...
- claudiug 10y ago
  I wish this will happen to ruby. For me, and this is personal, ruby without jruby and truffle is dead

smallnamespace 10y ago

It seems the scientific bits of Python have been moving at a much faster development pace though. Do you have any insight as to why? Personally, I started using Anaconda for everything and just drop down into pip when I find a package that isn't in there already, and it's been working very well.

synparb 10y ago
You should also check out https://conda-forge.github.io/ https://conda-forge.github.io/, which is building high quality, community curated conda packages for a lot of stuff that isn't available from the main Continuum channel.
fijal 10y ago
I'm not really commenting on the relative merits of the technology or who should use what. It's about the lack of admission that there is other stuff and other requirements
drauh 10y ago
For scientific work, I find it worth the effort to compile from source using Intel compilers + MKL (or a compiler + math library that can optimize well for your particular hardware platform).
- wclax04 10y ago
  aren't all of the default conda channel packages compiled with MKL now?
stuaxo 10y ago
Hm, I would consider using Anaconda if it worked for Gtk libraries (GEGL etc) on all 3 platforms (yeah I know this is hard outside Anaconda too).

dagss 10y ago

Former Cython developer here. To get some perspective to this, consider the alternatives for a scientific programmer: MATLAB, R, FORTRAN, Mathematica, or if you're hip, Julia -- all specifically made for scientific programming with 0% general purpose/web/etc. development going on. So I would say that the scientific Python community has been doing extremely well in terms of even using a language that isn't designed ground up for scientific computing. I could write a lot about why that is (and how some of the CS and IT crowd doesn't "get" scientific computing..) -- I'll refrain, I just wanted to say that were you see something and get frustrated, I see the same picture and think it's actually an incredible success, to bring so many scientists onto at least the same ball park as other programmers, even if they are still playing their own game.

p4wnc6 10y ago
Slight aside -- I am a long-time user of Cython and would love to contribute, but poking around the source code feels daunting -- like there is a huge learning curve to be able to contribute, and that other contributors will only view newcomers as an annoyance who will drain their time with questions. Given all this, what's the best way for someone to pick up and start contributing to Cython?
- dagss 10y ago
  It definitely is daunting; in particular because Cython isn't so much a goal as itself as a tool that developers are motivated to work on it because they use it in some other project they care about (e.g., lxml, Sage, ...). So it's been pragmatic development. If you don't already, subscribe to and start to follow cython-devel. First step is probably repeat the question there for more up-to-date info than what I can give (I don't even follow any longer). Think about what you want to achieve/change in Cython. A new feature may be easier than a bugfix, though I'm not sure how many "low hanging features" are left at this point.... Anyway, make sure you understand what the change would involve in changes in the C code. Write a testcase that uses Cython the way you would want it to work (elicit the failure/bug/feature); look at the C code that Cython generates, and make sure you understand why that C code is the wrong code and that you know how you'd want it to look. (Understanding the generated C code at this level and read it almost fluently may take a little bit of getting used to but is an absolute requirement for working with Cython -- eventually you look past all the __pyx_ everywhere). Then somehow try to beat the Cython codebase into submission in generating that C code... as you repeat this process you'll gradually learn the code. It seems like Robert Bradshaw and Stefan Behnel are still around, they are very capable and friendly people and I learned a lot of what I know about programming from them, they were very welcoming to me as a new contributor.

radarsat1 10y ago

> two packaging tools wait, what? I always use pip and/or the ubuntu repositories to install scipy. Is there another package manager for it?

RodericDay 10y ago
Anaconda wants you to do as much as possible via the `conda` tool. Maybe that's what they're referring to.
calycosa 10y ago
I imagine OP is referring to anaconda[0], which is frequently used in scientific contexts and comes with numpy, scipy and other packages included out of the box as well as its own package manager. 0. https://www.continuum.io/ https://www.continuum.io/
- synparb 10y ago
  The actual package distribution/management (it is also a replacement for virtualenv) is conda (http://conda.pydata.org/ http://conda.pydata.org/). Since it can be confusing, the difference between conda, anaconda and miniconda is described here: http://conda.pydata.org/docs/faq.html#general-questions http://conda.pydata.org/docs/faq.html#general-questions
- xigency 10y ago
  There is something upsetting about this website, and it may just be that this company is obviously for-profit, but I don't think it needs as much stock photography.
  - synparb 10y ago
    Conda is 100% free and open source, but is developed by Continuum, which is a company (but one that contributes heavily to many of the key components of the open-source pydata/scipy ecosystem, and whose CEO basically created numpy/scipy for the community at the expense of his academic career). Personally I have had nothing but good interactions with people from Continuum and the software they develop. They (as a company and as the individuals in it) do an incredible amount for the open source community and we all benefit from that. They seem to have worked out a sustainable model for running a company and working in/contributing to open source.

GFK_of_xmaspast 10y ago

Without the scientific stuff, python is just one of a bunch of modern-ish scripting languages out there.