6 ms·

I like Lisp, and I'm not a fan of e.g. Python's whitespace sensitivity. That said, for niches such as ML and data science, I find you just can't beat the Python

I like Lisp, and I'm not a fan of e.g. Python's whitespace sensitivity. That said, for niches such as ML and data science, I find you just can't beat the Python ecosystem.

rhlap 6y ago
Are people actually doing science with Python or are they talking about doing science? There's so much buggy low quality stuff in that space that I'd write a serious application in C or C++ from scratch. It would be a custom application, sure, but not everything needs to be general. Also, I find Lisp much more natural for mathematical reasoning.
- beforeolives 6y ago
  > Are people actually doing science with Python or are they talking about doing science? People are actually doing it. And a lot of it too. Both in terms of data science (as a broad term that can mean a bunch of different things) and in terms of computation for specific scientific fields like physics or biology.
- gustavo-fring 6y ago
  Yeah, they're very much doing it. Pandas is huge, libraries like Spacy, NetworkX, etc exist. It's a massive and good ecosystem. Python is the goto for scientific computing in most of the sciences for newer students I'd hazard a guess over the older R and Julia. This will be blindingly obvious if you work in that area. Yes, you can do it in another language, but you're missing out on a lot of stuff that is just done and is state of the art and is fast because the speedy parts aren't in Python. The complaints about parens for lisp are superficial, but it's my experience the same same goes for whitespace in Python. They just don't matter.
  - tluyben2 6y ago
    > and is fast because the speedy parts aren't in Python. Having worked months with a slew of senior data scientists, this was a bit painful. Python is so slow and those data scientists were very good at coming up with solutions for the issues of the company, but the implementations (using Spacy, Pandas and other libs) had enough Python in them to make them not practical for the company use case. Nice prototypes which I then had to fix them or even rewrite to C/C++(we worked Rust as well to try it out) to make them usable in the company data pipeline. I think companies are burning millions (billions in total?) on depressingly slow solutions in this space by throwing massive power at it all to make them complete their computations before the sun dies out. Example: we needed a specific keyword extraction algorithm for multiple languages; my colleague used Spacy and Python to create it. It took a couple of seconds per page of text; we needed max a few ms on modern hardware. He spent quite a lot of time rewriting and changing it, but never got it under 1s per page on xlarge aws instances. My version takes a few ms on average executing the same algorithm but in optimised c/c++. Sure we could've spun up a lot more instances, but my rewrite was far cheaper than that, even in the first month.
    - jgilias 6y ago
      But that's fine, no? I mean, it's a pretty common workflow where the people close to the science part of something write a prototype in their language/ecosystem of choice, and then the engineering side is in charge of taking the prototype implementation and making it performant enough for production use. Finding people who know both, data science, and low level programming languages well enough to be able to implement data science applications directly for production is pretty hard, I'm sure. In either case, I much prefer prototypes in Python than, say, Matlab. To speed things up I once rewrote an internal Scipy function to a version that allowed me to use it in vectorized code on my end. If the prototype is in Matlab, the optimization and integration possibilities are much more limited due to licensing, toolboxes, and the closed ecosystem in general.
      - andi999 6y ago
        Also I think it is good to be able to use the python code as testcases/validation on smaller datasets for the C code.
      - tluyben2 6y ago
        Yes, I guess it is fine if that is the flow. I just didn't expect it upfront (my bad).
        jgilias 6y ago
        Yeah, if it's actually OK or not depends a lot on the particulars. Like, if it's not actually your job, and the data people were supposed to produce production ready stuff themselves, and then you have to go out of your way to actually make it work, then it's not OK. But that's more to do with how organizations function, not technical merits of the involved programming languages.
    - pydry 6y ago
      Python performance is something that I see about 15-20x as much in discussions about python than I do wrestling with real life problems.
- 7thaccount 6y ago
  Very much actively doing so on my end where nearly all work in the industry is in Python, with some Matlab, C, C++, and Julia sprinkled in. Python is a great high level language for basically everything, but hardcore low latency apps. I can parse text, connect to databases, do sparse matrix computations on massive matrices, calculate network flows, generate large node-graph diagrams, use a Python based API to connect to any vendor software I've seen, do any kind of statistical analysis thing I need with pandas, amazing and free IDE allows me to use a REPL, code editor, and data structure viewer with ease, Python notebooks for education...etc etc. I've frequently found that I can rewrite a vendor's 10k line C++ program in a few pages of Python as the built-in Python data structures make text parsing extremely flexible and simple.
  - enriquto 6y ago
    > do sparse matrix computations on massive matrices This is completely impossible to do in the Python language, unless you resort to external tooling written in C or Fortran. Sure, you can call these codes from Python, as you can call them from any other language.
    - qsort 6y ago
      The ecosystem matters. I'm a developer and not a scientist, but having everything inside an environment that's at least workable is a huge boon. Of course you could call the same functions from the ffi of any other language, but nobody does that for the same reason that nobody writes web applications in C. I hate python, as far as I'm concerned it's a nightmare hell of a language that does everything wrong, and yet it's probably the language I use the most due to its sheer convenience and massive ecosystem.
    - odonnellryan 6y ago
      This is really splitting hairs isn't it? Plenty of languages are not bootstrapped. Isn't that essentially the same thing?
      - enriquto 6y ago
- golergka 6y ago
  Yes. Python is pretty much the main tool in biology, for example. C or C++ would be abysmal for similar exploratory scientific takes.
chalst 6y ago
> for niches such as ML and data science, I find you just can't beat the Python ecosystem There definitely are many areas where Python is best, but for ML and data science, Julia is (i) very competitive in library coverage, (ii) more performant and flexible, and (iii) has a very good Python bridge if it's needed. I can imagine there are niches within ML and data science where what you need are Python-only libraries, you don't miss anything restricting yourself to the numpy type hierarchy and there's no advantage to calling the libraries from Julia, but I'm curious to check if that is what you actually meant and if so, what you are doing.
- SatvikBeri 6y ago
  After using python for data science for 3 years (since founding the current startup), I mostly switched to Julia about 4 months ago. So far the only Python libraries I've really missed are boto3 and sqlalchemy for sql generation – both of which can easily be called in Julia using PyCall. I think people often underestimate just how much faster Julia is than numpy, I've consistently seen performance improvements on the order of 10x-30x when porting code.
pjmlp 6y ago
Many of those libraries are implemented in C++, any language can call them.

I like Lisp, and I'm not a fan of e.g. Python's whitespace sensitivity. That said, for niches such as ML and data science, I find you just can't beat the Python ecosystem.

rhlap 6y ago

Are people actually doing science with Python or are they talking about doing science? There's so much buggy low quality stuff in that space that I'd write a serious application in C or C++ from scratch. It would be a custom application, sure, but not everything needs to be general. Also, I find Lisp much more natural for mathematical reasoning.

beforeolives 6y ago
> Are people actually doing science with Python or are they talking about doing science? People are actually doing it. And a lot of it too. Both in terms of data science (as a broad term that can mean a bunch of different things) and in terms of computation for specific scientific fields like physics or biology.
gustavo-fring 6y ago
Yeah, they're very much doing it. Pandas is huge, libraries like Spacy, NetworkX, etc exist. It's a massive and good ecosystem. Python is the goto for scientific computing in most of the sciences for newer students I'd hazard a guess over the older R and Julia. This will be blindingly obvious if you work in that area. Yes, you can do it in another language, but you're missing out on a lot of stuff that is just done and is state of the art and is fast because the speedy parts aren't in Python. The complaints about parens for lisp are superficial, but it's my experience the same same goes for whitespace in Python. They just don't matter.
- tluyben2 6y ago
  > and is fast because the speedy parts aren't in Python. Having worked months with a slew of senior data scientists, this was a bit painful. Python is so slow and those data scientists were very good at coming up with solutions for the issues of the company, but the implementations (using Spacy, Pandas and other libs) had enough Python in them to make them not practical for the company use case. Nice prototypes which I then had to fix them or even rewrite to C/C++(we worked Rust as well to try it out) to make them usable in the company data pipeline. I think companies are burning millions (billions in total?) on depressingly slow solutions in this space by throwing massive power at it all to make them complete their computations before the sun dies out. Example: we needed a specific keyword extraction algorithm for multiple languages; my colleague used Spacy and Python to create it. It took a couple of seconds per page of text; we needed max a few ms on modern hardware. He spent quite a lot of time rewriting and changing it, but never got it under 1s per page on xlarge aws instances. My version takes a few ms on average executing the same algorithm but in optimised c/c++. Sure we could've spun up a lot more instances, but my rewrite was far cheaper than that, even in the first month.
  - jgilias 6y ago
    But that's fine, no? I mean, it's a pretty common workflow where the people close to the science part of something write a prototype in their language/ecosystem of choice, and then the engineering side is in charge of taking the prototype implementation and making it performant enough for production use. Finding people who know both, data science, and low level programming languages well enough to be able to implement data science applications directly for production is pretty hard, I'm sure. In either case, I much prefer prototypes in Python than, say, Matlab. To speed things up I once rewrote an internal Scipy function to a version that allowed me to use it in vectorized code on my end. If the prototype is in Matlab, the optimization and integration possibilities are much more limited due to licensing, toolboxes, and the closed ecosystem in general.
    - andi999 6y ago
      Also I think it is good to be able to use the python code as testcases/validation on smaller datasets for the C code.
    - tluyben2 6y ago
      Yes, I guess it is fine if that is the flow. I just didn't expect it upfront (my bad).
      - jgilias 6y ago
        Yeah, if it's actually OK or not depends a lot on the particulars. Like, if it's not actually your job, and the data people were supposed to produce production ready stuff themselves, and then you have to go out of your way to actually make it work, then it's not OK. But that's more to do with how organizations function, not technical merits of the involved programming languages.
  - pydry 6y ago
    Python performance is something that I see about 15-20x as much in discussions about python than I do wrestling with real life problems.
7thaccount 6y ago
Very much actively doing so on my end where nearly all work in the industry is in Python, with some Matlab, C, C++, and Julia sprinkled in. Python is a great high level language for basically everything, but hardcore low latency apps. I can parse text, connect to databases, do sparse matrix computations on massive matrices, calculate network flows, generate large node-graph diagrams, use a Python based API to connect to any vendor software I've seen, do any kind of statistical analysis thing I need with pandas, amazing and free IDE allows me to use a REPL, code editor, and data structure viewer with ease, Python notebooks for education...etc etc. I've frequently found that I can rewrite a vendor's 10k line C++ program in a few pages of Python as the built-in Python data structures make text parsing extremely flexible and simple.
- enriquto 6y ago
  > do sparse matrix computations on massive matrices This is completely impossible to do in the Python language, unless you resort to external tooling written in C or Fortran. Sure, you can call these codes from Python, as you can call them from any other language.
  - qsort 6y ago
    The ecosystem matters. I'm a developer and not a scientist, but having everything inside an environment that's at least workable is a huge boon. Of course you could call the same functions from the ffi of any other language, but nobody does that for the same reason that nobody writes web applications in C. I hate python, as far as I'm concerned it's a nightmare hell of a language that does everything wrong, and yet it's probably the language I use the most due to its sheer convenience and massive ecosystem.
  - odonnellryan 6y ago
    This is really splitting hairs isn't it? Plenty of languages are not bootstrapped. Isn't that essentially the same thing?
    - enriquto 6y ago
golergka 6y ago
Yes. Python is pretty much the main tool in biology, for example. C or C++ would be abysmal for similar exploratory scientific takes.

chalst 6y ago

> for niches such as ML and data science, I find you just can't beat the Python ecosystem There definitely are many areas where Python is best, but for ML and data science, Julia is (i) very competitive in library coverage, (ii) more performant and flexible, and (iii) has a very good Python bridge if it's needed. I can imagine there are niches within ML and data science where what you need are Python-only libraries, you don't miss anything restricting yourself to the numpy type hierarchy and there's no advantage to calling the libraries from Julia, but I'm curious to check if that is what you actually meant and if so, what you are doing.

SatvikBeri 6y ago
After using python for data science for 3 years (since founding the current startup), I mostly switched to Julia about 4 months ago. So far the only Python libraries I've really missed are boto3 and sqlalchemy for sql generation – both of which can easily be called in Julia using PyCall. I think people often underestimate just how much faster Julia is than numpy, I've consistently seen performance improvements on the order of 10x-30x when porting code.

pjmlp 6y ago

Many of those libraries are implemented in C++, any language can call them.