4 ms·

Data scientists are hired for a certain set of skills, and software engineers are hired for a different set. That’s the reality of specialization. I don’t often

by oivey 4y ago

Data scientists are hired for a certain set of skills, and software engineers are hired for a different set. That’s the reality of specialization. I don’t often see data scientists complaining that software engineers are bad at statistics. I’m not sure why so many software engineers feel the need to complain that data scientists are bad at software engineering. At my job I work both sides of the fence. Sometimes I write research code in Python/Julia/Matlab. Sometimes I write production code in C++/CUDA. The choice of production language is primarily due to performance needs rather than code quality, though. I’ll repeat myself but slightly differently: the main reason Python is used over, say, Ruby is because Ruby lacks a comparatively good library for working with arrays. It subsequently didn’t develop all the ecosystem for ML and scientific computing (SciPy, Matplotlib, sklearn, skimage, Torch, etc). It’s basically never going to catch up. And when it comes down to it, Ruby is basically as ill-suited as Python, Clojure, Elixir, and most other languages when it comes to doing something like automatic differentiation on a GPU. And again, even so, Python is still the best choice among those because so many resources have been poured into PyTorch, Matplotlib, NumPy, etc.

z9znz 4y ago
I'm not attacking data scientists. I have no objection to how they solve their problems. My point is that their choice of Python (because of the libraries they use, or the training they got in school?) bleeds out and causes Python to be used in places where it doesn't fit so well. If the libraries you mention were written for Ruby (and they absolutely could have been... probably with fewer defects and fewer lines of code), then we wouldn't be having this discussion. There is no use of any of those libraries that someone could demonstrate which could not be equally or better demonstrated in one of the other languages you mention. Therefore, the only argument for using Python is that it's what those libraries are written in. In fact, with a little effort an API could be written and wrapped around those libraries to make them available to other languages, decoupling the bigger projects from the library-specific logic. This is how it should be done at any reasonable company anyway. The data science wizards build their specific systems in whatever language they need, and that gets built into a minimal service. But things often start highly coupled, meaning the special code in Python gets business logic and persistence stuff built around it also in Python.
- oivey 4y ago
  > My point is that their choice of Python (because of the libraries they use, or the training they got in school?) bleeds out and causes Python to be used in places where it doesn't fit so well. They use those libraries because there aren’t alternatives in the languages you prefer. Millions of man hours went into developing the Python ecosystem. > If the libraries you mention were written for Ruby (and they absolutely could have been... probably with fewer defects and fewer lines of code), then we wouldn't be having this discussion. This probably isn’t true because the core of NumPy is C. A theoretical NumRs would be the same. The issue with both languages is that they’re slow, so you have to resort to C wrappers. NumRs might be incrementally better at best. It wouldn’t be a revolution. > There is no use of any of those libraries that someone could demonstrate which could not be equally or better demonstrated in one of the other languages you mention. Therefore, the only argument for using Python is that it's what those libraries are written in. But no one wrote these libraries in Ruby. Someone had to do it, and no one did. Someone did it in Python. It would be impossible to justify recreating NumPy in Ruby today. If I want to do some simple image processing, Fourier transforms, simple optimization, and plotting in Python it’s trivial. If I want to do it in Ruby it is a ton of work. > In fact, with a little effort an API could be written and wrapped around those libraries to make them available to other languages, decoupling the bigger projects from the library-specific logic. This is how it should be done at any reasonable company anyway. The data science wizards build their specific systems in whatever language they need, and that gets built into a minimal service. But things often start highly coupled, meaning the special code in Python gets business logic and persistence stuff built around it also in Python. This isn’t a company scale problem. The Python numerical computing stack has developed over 20 years with enormous investment from companies and universities. Code could maybe be shared via an API, but someone would have to do it. It would have to be done very carefully to guarantee performance. If it were easy you’d expect to see significant code reuse between, for example, Ruby and Java web frameworks. I guess you could say this API does exist, and it’s called C FFI interfaces. Sane companies do use this, but it still sucks pretty bad. Python’s lead isn’t unassailable, but any language that is just another C wrapper isn’t going to unseat it. Whatever does it will need to be compiled. I’d bet on it being dynamic with type inference, too, because static type checking almost never helps with correctness issues in numerical code while adding significant cognitive overhead. If it isn’t Julia, Julia is at least the blueprint.

by oivey 4y ago

z9znz 4y ago

I'm not attacking data scientists. I have no objection to how they solve their problems. My point is that their choice of Python (because of the libraries they use, or the training they got in school?) bleeds out and causes Python to be used in places where it doesn't fit so well. If the libraries you mention were written for Ruby (and they absolutely could have been... probably with fewer defects and fewer lines of code), then we wouldn't be having this discussion. There is no use of any of those libraries that someone could demonstrate which could not be equally or better demonstrated in one of the other languages you mention. Therefore, the only argument for using Python is that it's what those libraries are written in. In fact, with a little effort an API could be written and wrapped around those libraries to make them available to other languages, decoupling the bigger projects from the library-specific logic. This is how it should be done at any reasonable company anyway. The data science wizards build their specific systems in whatever language they need, and that gets built into a minimal service. But things often start highly coupled, meaning the special code in Python gets business logic and persistence stuff built around it also in Python.

oivey 4y ago
> My point is that their choice of Python (because of the libraries they use, or the training they got in school?) bleeds out and causes Python to be used in places where it doesn't fit so well. They use those libraries because there aren’t alternatives in the languages you prefer. Millions of man hours went into developing the Python ecosystem. > If the libraries you mention were written for Ruby (and they absolutely could have been... probably with fewer defects and fewer lines of code), then we wouldn't be having this discussion. This probably isn’t true because the core of NumPy is C. A theoretical NumRs would be the same. The issue with both languages is that they’re slow, so you have to resort to C wrappers. NumRs might be incrementally better at best. It wouldn’t be a revolution. > There is no use of any of those libraries that someone could demonstrate which could not be equally or better demonstrated in one of the other languages you mention. Therefore, the only argument for using Python is that it's what those libraries are written in. But no one wrote these libraries in Ruby. Someone had to do it, and no one did. Someone did it in Python. It would be impossible to justify recreating NumPy in Ruby today. If I want to do some simple image processing, Fourier transforms, simple optimization, and plotting in Python it’s trivial. If I want to do it in Ruby it is a ton of work. > In fact, with a little effort an API could be written and wrapped around those libraries to make them available to other languages, decoupling the bigger projects from the library-specific logic. This is how it should be done at any reasonable company anyway. The data science wizards build their specific systems in whatever language they need, and that gets built into a minimal service. But things often start highly coupled, meaning the special code in Python gets business logic and persistence stuff built around it also in Python. This isn’t a company scale problem. The Python numerical computing stack has developed over 20 years with enormous investment from companies and universities. Code could maybe be shared via an API, but someone would have to do it. It would have to be done very carefully to guarantee performance. If it were easy you’d expect to see significant code reuse between, for example, Ruby and Java web frameworks. I guess you could say this API does exist, and it’s called C FFI interfaces. Sane companies do use this, but it still sucks pretty bad. Python’s lead isn’t unassailable, but any language that is just another C wrapper isn’t going to unseat it. Whatever does it will need to be compiled. I’d bet on it being dynamic with type inference, too, because static type checking almost never helps with correctness issues in numerical code while adding significant cognitive overhead. If it isn’t Julia, Julia is at least the blueprint.