4 ms·

Fijal, I'm not sure you even understand my perspective. None of your comments in response have given me confidence that you do. I have no disagreement with y

Fijal, I'm not sure you even understand my perspective. None of your comments in response have given me confidence that you do. I have no disagreement with you about how far "PyPy could go". Obviously, we could re-create the entire Python ecosystem including the scientific stack under its run-time. I just question your understanding of how expensive and time-consuming that would be. My official position is that I think it's possible to get the benefits of PyPy (i.e. fast Python loops) in different ways that don't also toss out years of extension modules in the process and whose answer to current users about the features they rely on in CPython not working under PyPy is "just port it" or "just use ctypes".

fijal 15y ago
Well. Fast Python loops in CPython has been tried before and failed. I seriously don't see a way of getting a working JIT that really optimizes a lot of code out there and native support for CPython extensions. There will be some side that suffers. Also, numpy is fairly special as it does have a good potential to be optimized by the JIT in ways that are not quite possible using C or Cython.
synparb 15y ago
From having read the various replies to this and similar threads, I think a basic problem (as Travis pointed out) is that Fijal has been (1) highly dismissive of numpy and particularly cython, which a lot of really smart people have put a lot of time and effort into and created an amazing scientific community within python, and (2) confrontational with people who are trying to lend an alternative opinion which has been informed by a lot of experience in the scientific python world. PyPy might eventually be a wholly superior alternative for scientific computing with Python, but it would be good to acknowledge the insight of those who are 'in the trenches' even if you go a different route, because part of the success of scientific python has been the community. I know very little about the development aspect of PyPy or Numpy, but I know that at this moment in time Numpy/Scipy/Cython have revolutionized how I do research on a day to day basis. It seems unfortunate that there seems to be such animosity surrounding this issue.
- wesm 15y ago
  My thoughts exactly.
fijal 15y ago
Note: this reply is on the wrong level, I cannot reply correctly. I don't think I'm highly dismissive about numpy/scipy/cython community. I wouldn't be implementing all this stuff if I didn't think those APIs are good and they're the future of scientific computing, they just lack a reasonable replacement for C. If Python is to surpress Fortran on the scientific field, it really does need a way to express fast algorithms in Python and I don't think CPython can provide that. Personally, I don't like Cython as a way to speed up Python, because it sacrifices the beauty of the language in favor of performance. I think investing time in the Python VM is a much better spent time, but this is a very personal opinion and I won't blame people who thing otherwise. I think Cython is a better way to call to C than all other options that exist right now (like using CPython C API or ctypes), but this is yet entirely different than using it for speedups. The proposed way so far has been "everything must be 100% backwards compatible, otherwise it won't work". This is all well and good, but I'm not aware about a way to make things both 100% compatible and fast, so we decided to break with some compatibility like CPython C API or reusing most of what's implemented in C in numpy. I argued many times for those decisions, but in short -- there has to be a bit of breakage before we can make a leap. This is not based on dismissal of other people's opinions, especially those that spent tons of time with the scientific community, it's just that I don't see a way forward that does not introduce some breakage.
- pwang 15y ago
  "...they just lack a reasonable replacement for C" Actually, Fijal, I have a question - and this is related to our previous Skype discussion as well. I know that a lot of the work on PyPy has been on the JIT, but have you guys really ever pursued the idea of just building PyPy as a front end for LLVM? All your type inferencing logic would probably be a lot more code than what a traditional LLVM front-end normally consists of, but you'd get to leverage all of the massive community efforts on LLVM's optimized code generation and other backend optimizers. Just a thought..
  - kingkilr 15y ago
    LLVM has been looked at, as both a backened for RPython as well as for the JIT, I've written down some of the reasons it's not appropriate for PyPy (particularly WRT the JIT) here: http://www.quora.com/LLVM/Is-LLVM-not-good-for-interpreted-languages-Why http://www.quora.com/LLVM/Is-LLVM-not-good-for-interpreted-l...

fijal 15y ago

Well. Fast Python loops in CPython has been tried before and failed. I seriously don't see a way of getting a working JIT that really optimizes a lot of code out there and native support for CPython extensions. There will be some side that suffers. Also, numpy is fairly special as it does have a good potential to be optimized by the JIT in ways that are not quite possible using C or Cython.

synparb 15y ago

From having read the various replies to this and similar threads, I think a basic problem (as Travis pointed out) is that Fijal has been (1) highly dismissive of numpy and particularly cython, which a lot of really smart people have put a lot of time and effort into and created an amazing scientific community within python, and (2) confrontational with people who are trying to lend an alternative opinion which has been informed by a lot of experience in the scientific python world. PyPy might eventually be a wholly superior alternative for scientific computing with Python, but it would be good to acknowledge the insight of those who are 'in the trenches' even if you go a different route, because part of the success of scientific python has been the community. I know very little about the development aspect of PyPy or Numpy, but I know that at this moment in time Numpy/Scipy/Cython have revolutionized how I do research on a day to day basis. It seems unfortunate that there seems to be such animosity surrounding this issue.

wesm 15y ago
My thoughts exactly.

fijal 15y ago

Note: this reply is on the wrong level, I cannot reply correctly. I don't think I'm highly dismissive about numpy/scipy/cython community. I wouldn't be implementing all this stuff if I didn't think those APIs are good and they're the future of scientific computing, they just lack a reasonable replacement for C. If Python is to surpress Fortran on the scientific field, it really does need a way to express fast algorithms in Python and I don't think CPython can provide that. Personally, I don't like Cython as a way to speed up Python, because it sacrifices the beauty of the language in favor of performance. I think investing time in the Python VM is a much better spent time, but this is a very personal opinion and I won't blame people who thing otherwise. I think Cython is a better way to call to C than all other options that exist right now (like using CPython C API or ctypes), but this is yet entirely different than using it for speedups. The proposed way so far has been "everything must be 100% backwards compatible, otherwise it won't work". This is all well and good, but I'm not aware about a way to make things both 100% compatible and fast, so we decided to break with some compatibility like CPython C API or reusing most of what's implemented in C in numpy. I argued many times for those decisions, but in short -- there has to be a bit of breakage before we can make a leap. This is not based on dismissal of other people's opinions, especially those that spent tons of time with the scientific community, it's just that I don't see a way forward that does not introduce some breakage.

pwang 15y ago
"...they just lack a reasonable replacement for C" Actually, Fijal, I have a question - and this is related to our previous Skype discussion as well. I know that a lot of the work on PyPy has been on the JIT, but have you guys really ever pursued the idea of just building PyPy as a front end for LLVM? All your type inferencing logic would probably be a lot more code than what a traditional LLVM front-end normally consists of, but you'd get to leverage all of the massive community efforts on LLVM's optimized code generation and other backend optimizers. Just a thought..
- kingkilr 15y ago
  LLVM has been looked at, as both a backened for RPython as well as for the JIT, I've written down some of the reasons it's not appropriate for PyPy (particularly WRT the JIT) here: http://www.quora.com/LLVM/Is-LLVM-not-good-for-interpreted-languages-Why http://www.quora.com/LLVM/Is-LLVM-not-good-for-interpreted-l...