5 ms·

Are people actually doing science with Python or are they talking about doing science? There's so much buggy low quality stuff in that space that I'd write a s

by rhlap 5y ago

Are people actually doing science with Python or are they talking about doing science? There's so much buggy low quality stuff in that space that I'd write a serious application in C or C++ from scratch. It would be a custom application, sure, but not everything needs to be general. Also, I find Lisp much more natural for mathematical reasoning.

beforeolives 5y ago
> Are people actually doing science with Python or are they talking about doing science? People are actually doing it. And a lot of it too. Both in terms of data science (as a broad term that can mean a bunch of different things) and in terms of computation for specific scientific fields like physics or biology.
gustavo-fring 5y ago
Yeah, they're very much doing it. Pandas is huge, libraries like Spacy, NetworkX, etc exist. It's a massive and good ecosystem. Python is the goto for scientific computing in most of the sciences for newer students I'd hazard a guess over the older R and Julia. This will be blindingly obvious if you work in that area. Yes, you can do it in another language, but you're missing out on a lot of stuff that is just done and is state of the art and is fast because the speedy parts aren't in Python. The complaints about parens for lisp are superficial, but it's my experience the same same goes for whitespace in Python. They just don't matter.
- tluyben2 5y ago
  > and is fast because the speedy parts aren't in Python. Having worked months with a slew of senior data scientists, this was a bit painful. Python is so slow and those data scientists were very good at coming up with solutions for the issues of the company, but the implementations (using Spacy, Pandas and other libs) had enough Python in them to make them not practical for the company use case. Nice prototypes which I then had to fix them or even rewrite to C/C++(we worked Rust as well to try it out) to make them usable in the company data pipeline. I think companies are burning millions (billions in total?) on depressingly slow solutions in this space by throwing massive power at it all to make them complete their computations before the sun dies out. Example: we needed a specific keyword extraction algorithm for multiple languages; my colleague used Spacy and Python to create it. It took a couple of seconds per page of text; we needed max a few ms on modern hardware. He spent quite a lot of time rewriting and changing it, but never got it under 1s per page on xlarge aws instances. My version takes a few ms on average executing the same algorithm but in optimised c/c++. Sure we could've spun up a lot more instances, but my rewrite was far cheaper than that, even in the first month.
  - jgilias 5y ago
    But that's fine, no? I mean, it's a pretty common workflow where the people close to the science part of something write a prototype in their language/ecosystem of choice, and then the engineering side is in charge of taking the prototype implementation and making it performant enough for production use. Finding people who know both, data science, and low level programming languages well enough to be able to implement data science applications directly for production is pretty hard, I'm sure. In either case, I much prefer prototypes in Python than, say, Matlab. To speed things up I once rewrote an internal Scipy function to a version that allowed me to use it in vectorized code on my end. If the prototype is in Matlab, the optimization and integration possibilities are much more limited due to licensing, toolboxes, and the closed ecosystem in general.
    - andi999 5y ago
      Also I think it is good to be able to use the python code as testcases/validation on smaller datasets for the C code.
    - tluyben2 5y ago
      Yes, I guess it is fine if that is the flow. I just didn't expect it upfront (my bad).
      - jgilias 5y ago
        Yeah, if it's actually OK or not depends a lot on the particulars. Like, if it's not actually your job, and the data people were supposed to produce production ready stuff themselves, and then you have to go out of your way to actually make it work, then it's not OK. But that's more to do with how organizations function, not technical merits of the involved programming languages.
  - pydry 5y ago
    Python performance is something that I see about 15-20x as much in discussions about python than I do wrestling with real life problems.
7thaccount 5y ago
Very much actively doing so on my end where nearly all work in the industry is in Python, with some Matlab, C, C++, and Julia sprinkled in. Python is a great high level language for basically everything, but hardcore low latency apps. I can parse text, connect to databases, do sparse matrix computations on massive matrices, calculate network flows, generate large node-graph diagrams, use a Python based API to connect to any vendor software I've seen, do any kind of statistical analysis thing I need with pandas, amazing and free IDE allows me to use a REPL, code editor, and data structure viewer with ease, Python notebooks for education...etc etc. I've frequently found that I can rewrite a vendor's 10k line C++ program in a few pages of Python as the built-in Python data structures make text parsing extremely flexible and simple.
- enriquto 5y ago
  > do sparse matrix computations on massive matrices This is completely impossible to do in the Python language, unless you resort to external tooling written in C or Fortran. Sure, you can call these codes from Python, as you can call them from any other language.
  - qsort 5y ago
    The ecosystem matters. I'm a developer and not a scientist, but having everything inside an environment that's at least workable is a huge boon. Of course you could call the same functions from the ffi of any other language, but nobody does that for the same reason that nobody writes web applications in C. I hate python, as far as I'm concerned it's a nightmare hell of a language that does everything wrong, and yet it's probably the language I use the most due to its sheer convenience and massive ecosystem.
  - odonnellryan 5y ago
    This is really splitting hairs isn't it? Plenty of languages are not bootstrapped. Isn't that essentially the same thing?
    - enriquto 5y ago
golergka 5y ago
Yes. Python is pretty much the main tool in biology, for example. C or C++ would be abysmal for similar exploratory scientific takes.

by rhlap 5y ago

beforeolives 5y ago

> Are people actually doing science with Python or are they talking about doing science? People are actually doing it. And a lot of it too. Both in terms of data science (as a broad term that can mean a bunch of different things) and in terms of computation for specific scientific fields like physics or biology.

gustavo-fring 5y ago

Yeah, they're very much doing it. Pandas is huge, libraries like Spacy, NetworkX, etc exist. It's a massive and good ecosystem. Python is the goto for scientific computing in most of the sciences for newer students I'd hazard a guess over the older R and Julia. This will be blindingly obvious if you work in that area. Yes, you can do it in another language, but you're missing out on a lot of stuff that is just done and is state of the art and is fast because the speedy parts aren't in Python. The complaints about parens for lisp are superficial, but it's my experience the same same goes for whitespace in Python. They just don't matter.

tluyben2 5y ago
> and is fast because the speedy parts aren't in Python. Having worked months with a slew of senior data scientists, this was a bit painful. Python is so slow and those data scientists were very good at coming up with solutions for the issues of the company, but the implementations (using Spacy, Pandas and other libs) had enough Python in them to make them not practical for the company use case. Nice prototypes which I then had to fix them or even rewrite to C/C++(we worked Rust as well to try it out) to make them usable in the company data pipeline. I think companies are burning millions (billions in total?) on depressingly slow solutions in this space by throwing massive power at it all to make them complete their computations before the sun dies out. Example: we needed a specific keyword extraction algorithm for multiple languages; my colleague used Spacy and Python to create it. It took a couple of seconds per page of text; we needed max a few ms on modern hardware. He spent quite a lot of time rewriting and changing it, but never got it under 1s per page on xlarge aws instances. My version takes a few ms on average executing the same algorithm but in optimised c/c++. Sure we could've spun up a lot more instances, but my rewrite was far cheaper than that, even in the first month.
- jgilias 5y ago
  But that's fine, no? I mean, it's a pretty common workflow where the people close to the science part of something write a prototype in their language/ecosystem of choice, and then the engineering side is in charge of taking the prototype implementation and making it performant enough for production use. Finding people who know both, data science, and low level programming languages well enough to be able to implement data science applications directly for production is pretty hard, I'm sure. In either case, I much prefer prototypes in Python than, say, Matlab. To speed things up I once rewrote an internal Scipy function to a version that allowed me to use it in vectorized code on my end. If the prototype is in Matlab, the optimization and integration possibilities are much more limited due to licensing, toolboxes, and the closed ecosystem in general.
  - andi999 5y ago
    Also I think it is good to be able to use the python code as testcases/validation on smaller datasets for the C code.
  - tluyben2 5y ago
    Yes, I guess it is fine if that is the flow. I just didn't expect it upfront (my bad).
    - jgilias 5y ago
      Yeah, if it's actually OK or not depends a lot on the particulars. Like, if it's not actually your job, and the data people were supposed to produce production ready stuff themselves, and then you have to go out of your way to actually make it work, then it's not OK. But that's more to do with how organizations function, not technical merits of the involved programming languages.
- pydry 5y ago
  Python performance is something that I see about 15-20x as much in discussions about python than I do wrestling with real life problems.

7thaccount 5y ago

Very much actively doing so on my end where nearly all work in the industry is in Python, with some Matlab, C, C++, and Julia sprinkled in. Python is a great high level language for basically everything, but hardcore low latency apps. I can parse text, connect to databases, do sparse matrix computations on massive matrices, calculate network flows, generate large node-graph diagrams, use a Python based API to connect to any vendor software I've seen, do any kind of statistical analysis thing I need with pandas, amazing and free IDE allows me to use a REPL, code editor, and data structure viewer with ease, Python notebooks for education...etc etc. I've frequently found that I can rewrite a vendor's 10k line C++ program in a few pages of Python as the built-in Python data structures make text parsing extremely flexible and simple.

enriquto 5y ago
> do sparse matrix computations on massive matrices This is completely impossible to do in the Python language, unless you resort to external tooling written in C or Fortran. Sure, you can call these codes from Python, as you can call them from any other language.
- qsort 5y ago
  The ecosystem matters. I'm a developer and not a scientist, but having everything inside an environment that's at least workable is a huge boon. Of course you could call the same functions from the ffi of any other language, but nobody does that for the same reason that nobody writes web applications in C. I hate python, as far as I'm concerned it's a nightmare hell of a language that does everything wrong, and yet it's probably the language I use the most due to its sheer convenience and massive ecosystem.
- odonnellryan 5y ago
  This is really splitting hairs isn't it? Plenty of languages are not bootstrapped. Isn't that essentially the same thing?
  - enriquto 5y ago

golergka 5y ago

Yes. Python is pretty much the main tool in biology, for example. C or C++ would be abysmal for similar exploratory scientific takes.