5 ms·
If all it was used for was scripting it would be fine, but what is strange is that Python has emerged as a major language for complex data analysis. So I am som
by zmmmmm 5y ago
If all it was used for was scripting it would be fine, but what is strange is that Python has emerged as a major language for complex data analysis. So I am sometimes confronted with partial solutions in Python that solve a piece of the puzzle that is purely numerical (so all runs on numpy etc) but then extending it to actually solve the real problem ends up either extremely awkward (the whole architecture getting bent around Python's limitations) or being a complete rewrite. It's frustrating that the ecosystem for certain problems is so bent towards a language that is unsuitable for large parts of the job. (mainly, I would add, because it sucks the oxygen away from alternatives that would be better in the longer term).
- xscott 5y ago> it sucks the oxygen away from alternatives that would be better in the longer term It's always frustrated me that the computing world isn't as it should be. Python as a language seems to have found a sweet spot for accessibility to a lot of people, but CPython the interpreter steals all the air from PyPy, which is a vastly better implementation.
- seunosewa 5y agoPyPy has a GIL too though.
- xscott 5y agoThe PyPy FAQ addresses this. To my eyes, it looks like this is partly to provide the same promises CPython does for C extensions, and partly because their transactional memory approach hasn't gotten the attention it would need to succeed. I think both of those show CPython stealing the air from PyPy. https://doc.pypy.org/en/latest/faq.html#does-pypy-have-a-gil-why https://doc.pypy.org/en/latest/faq.html#does-pypy-have-a-gil...
- fastasucan 5y agoYou cant both claim that it is major language for complex data analysis and that it is "unsuitable for large parts of the job". If it is a major language for complex data analysis it is apparently not unsuitable. It can be not optimal/not the best/bad but it cannot be unsuitable, because then your first statement isn't true. Which one is it?
- lmm 5y agoIt's a great language for exploratory prototyping of complex data analysis and a terrible language for robust implementation of complex data analysis. Well-managed organisations prototype in Python but take the time to rewrite the prototype for production use; poorly-managed ones skip step 2.
- maccard 5y agoWhat language do you rewrite it in so?
- chalst 5y agoIt's typically pretty easy to turn Python+numpy into performant Julia.
- rofrol 5y agoI remember reading this comment: >I recently tried to migrate my R code to Julia. Even though I already knew R data.table is faster than DataFrames.jl, I was totally blown away by how slow Julia is. So I quickly gave up. I think I will have to write unavoidable hard loop in cpp, which I really don't want to do... https://news.ycombinator.com/item?id=27404910 https://news.ycombinator.com/item?id=27404910
- pjmlp 5y agoApparently it is working for some, https://juliacomputing.com/case-studies/ https://juliacomputing.com/case-studies/
- lmm 5y agoIn the case I was thinking of, mainly Scala (we were a very JVM-oriented place and used Spark clusters for our big data stuff).
- pharmakom 5y agoDoes this seem really inefficient to anyone else? Why is there not a language that can do the prototype and offer a high performance ceiling? Julia seems like a candidate.
- maccard 5y agoIf you're doing complex data analysis in python, the actual number crunching isn't happening in python it's happening in a C extension. Using numpy or tensor flow, you're really just calling C code.
- grumpyprole 5y agoAnd Fortran code (many parts of numpy). C doesn't have a complete monopoly on high-performance code.
- Qi_ 5y agoWhat is the advantage of Fortran over only C in numpy’s case specifically?
- staticassertion 5y agoFortran has the more optimized libraries (blas) and is also capable of optimizations that C isn't (guarantees that pointers dont alias).
- qsort 5y agoModern C has the restrict keyword for that. There isn't any competitive advantage left for Fortran over C or C++, the only reason why Fortran is still part of the modern numeric stack is that BLAS, LAPACK, QUADPACK and friends run the freaking world and nobody is ever going to rewrite them to C without a compelling reason to do so.
- staticassertion 5y agoI'm aware of the restrict keyword, though its usage is not typical. It enables an optimization that fortran has by default, that was my point. And yes, I already said that packages like blas have had massive amounts of work put into them and have lasting power.
- d0mine 5y agoCould you provide a specific example where GIL prevents "complex data analysis"?