6 ms·

> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost. Tried putting R into production

> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost. Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas. C++ is out for different reasons I suspect. As this touches on, “modellers” (and data scientists/engineers) ought to be decent developers, but a lot of them are not, and in my experience actively refuse to learn any of these skills, the comfort zone is “jupyter notebooks” and that’s it. Getting them to write C++ (disregarding language debates), a language that is unequivocally more difficult and fraught with complexity than Python is basically a non-starter. Do I wish it was different? Yep. Do I wish there was some more variety in the “language ecosystem” so it’s more than just the “lowest common denominator Python” dominance? Absolutely.

bigger_cheese 4y ago
I work in an Industrial smelter (I'm a Materials/Chem Eng I look at outputs of models don't have a lot to do with developing them) the way the data pipelines are setup here the models are deployed as something called a "pickle file" I think this is something very specific to Python - maybe the equivalent would be a .dll file in C++ world. From point of view of the integration process the pickle file is just a black box input data flows in, response flows out of it. If something changes with the model (trained it on a new dataset etc.) then it's just a matter of drop in a new pickle file and it all keeps working. I don't know if you can do the equivalent with packaging R code like this. All of the tooling is setup around Python and I think this is pretty industry standard we use Azure and I'm pretty sure this is all out of the box stuff.
- GeneralMayhem 4y ago
  Pickles and .dlls solve completely different purposes. A .dll, which stands for "dynamic link library", is code - it's how you package parts of your implementation in certain environments (Windows). A pickle file is data - it's a way of serializing out the contents of the state of a data structure, using some magic built into Python reflection, and then loading it back later. Pickling is very convenient for its purpose. It's also... very bad in production. The problem is that the serialization format is derived from the implementation and not very stable, so if the implementation changes - sometimes in subtle ways like upgrading your Python version - the serialized data isn't readable anymore. You're almost always better off either using an explicit data serialization format (JSON with schemas, protocol buffers, etc.) and writing your own logic for saving and restoring to/from it. > the pickle file is just a black box input data flows in, response flows out of it There's 100% more to it than that - there's got to be something that's evaluating the data in the pickle file and actually using it to compute the response. That substrate layer probably doesn't change often, but it should have its own versioning and deployment story.
  - bigger_cheese 4y ago
    > There's 100% more to it than that - there's got to be something that's evaluating the data in the pickle file and actually using it to compute the response. I suspect that is the Microsoft Azure ML part I'm not sure. As I said I don't write the models, I just come along afterwards and review the results. I don't know anything about Python.
  - fbdab103 4y ago
    >the serialization format is derived from the implementation and not very stable, so if the implementation changes - sometimes in subtle ways like upgrading your Python version - the serialized data isn't readable anymore. Could be wrong, but I do not believe this is true anymore. There are pickle versions, but I believe the format has been standardized such that you should be able to freely move between Python releases.
    - jononor 4y ago
      The base serialization is stable and compatible across Python versions. However pickle files can serialize entire Python objects, including from custom and third party code. Compatibility of serialization then depends on the versions of the code that implements those objects, which is often not well managed.
      - fbdab103 4y ago
        Ah that's fair. I was only considering serializing native types (for which there are much better options).
- goosedragons 4y ago
  I think the R equivalent of a pickle file is just an object(s) that's been saved with "save" or "saveRDS" to a file. These can then be loaded and could contain a trained model.
- kgwgk 4y ago
  > something called a "pickle file" I think this is something very specific to Python - maybe the equivalent would be a .dll file in C++ world. The equivalent would be a .dat file :-)
fbdab103 4y ago
>Tried putting R into production recently? It’s a frustrating and brittle experience. Oh man, can I sign onto this rant. While Python has spent a decade+ trying-and-failing to standardize on one of a dozen tools to properly manage dependencies, at least it is trying. R is still global-namespace, no-pinning by default. Sorta-kinda you can squint where renv is going, but still needs a lot of development. I had some hopes of Julia stealing the R mindshare and righting some of the more egregious wrongs in that ecosystem, but (as an outsider) it feels like Julia has lost a lot of steam.
- FridgeSeal 4y ago
  > Oh man, can I sign onto this rant. Please, be my guest hahaha. I think my issue with so many of these tools is that they’re fundamentally meant to be used interactively. R (renv/etc) doesn’t have a package manager separate from the runtime, so you can’t just “yarn/pip/cargo/etc install” and have your dependencies setup prior to your script running, you have to invoke manual R commands and hope everything goes off ok, it doesn’t spontaneously pick a different directory to install your stuff in, or have a tantrum about permissions because it wants to install in some root dir.
- lukego 4y ago
  I find it really painless to put bits of R into production using the Nix support: https://nixos.org/manual/nixpkgs/stable/#r https://nixos.org/manual/nixpkgs/stable/#r The same framework can provide an identical-to-production R or RStudio environment for development. I have a harder time with Julia since its tooling is more opinionated. I don't know a better solution than wrapping it in a Docker container.
  - fbdab103 4y ago
    If the cure for R is that I need to jump fully into the Nix ecosystem, I am not sure that is a solution. Disappointing about Julia. Only played with it a small bit, but I had assumed it had identified dependency management as a huge problem that needed to be addressed.
    - 4y ago
Tcepsa 4y ago
I believe that is one of the points the author of the article is trying to make: that the data scientists ought to be able to work in whatever language lets them develop the model most quickly and then hand it over to a software developer to make it work well in code, so that each can focus on doing what they do best and both aspects can be done very well.
andy800 4y ago
> wish there was some more variety Kotlin is coming up on the outside. Late to the party, but the data science ecosystem is progressing nicely.
- hdespiritu 4y ago
  This is an interesting observation. Some of the data scientists I work with have been using Kotlin to define "analytic grammars", and is the first example (in my limited Kotlin awareness) i've seen of Kotlin outside of Android development. It seems that Roman Elizarov (Kotlin Project Lead) has identified the opportunity for a better language ecosystem to enter the data science space https://discuss.kotlinlang.org/t/ai-and-deep-learning-define-the-future-of-programming-will-kotlin-fly-or-die/2264/5 https://discuss.kotlinlang.org/t/ai-and-deep-learning-define...
- pjmlp 4y ago
  CERN started diving into Python back in 2002, I was part of one of the first trainings that were taking place. Same applies to many other research institutions. Good luck catching up with a 20 years old ecosystem, with a language that only gets used mainly thanks to Google's sponsorhip for Android apps.
  - andy800 4y ago
    Kind of ignorant... given that Kotlin is JVM-compatible, it works with libraries even more than 20 years old. Do I care if it is at parity with Python? Hell no I don't, as long as it allows me to accomplish the tasks I need -- and it does, faster, easier, with fewer errors -- then I'm very happy.

bigger_cheese 4y ago

I work in an Industrial smelter (I'm a Materials/Chem Eng I look at outputs of models don't have a lot to do with developing them) the way the data pipelines are setup here the models are deployed as something called a "pickle file" I think this is something very specific to Python - maybe the equivalent would be a .dll file in C++ world. From point of view of the integration process the pickle file is just a black box input data flows in, response flows out of it. If something changes with the model (trained it on a new dataset etc.) then it's just a matter of drop in a new pickle file and it all keeps working. I don't know if you can do the equivalent with packaging R code like this. All of the tooling is setup around Python and I think this is pretty industry standard we use Azure and I'm pretty sure this is all out of the box stuff.

GeneralMayhem 4y ago
Pickles and .dlls solve completely different purposes. A .dll, which stands for "dynamic link library", is code - it's how you package parts of your implementation in certain environments (Windows). A pickle file is data - it's a way of serializing out the contents of the state of a data structure, using some magic built into Python reflection, and then loading it back later. Pickling is very convenient for its purpose. It's also... very bad in production. The problem is that the serialization format is derived from the implementation and not very stable, so if the implementation changes - sometimes in subtle ways like upgrading your Python version - the serialized data isn't readable anymore. You're almost always better off either using an explicit data serialization format (JSON with schemas, protocol buffers, etc.) and writing your own logic for saving and restoring to/from it. > the pickle file is just a black box input data flows in, response flows out of it There's 100% more to it than that - there's got to be something that's evaluating the data in the pickle file and actually using it to compute the response. That substrate layer probably doesn't change often, but it should have its own versioning and deployment story.
- bigger_cheese 4y ago
  > There's 100% more to it than that - there's got to be something that's evaluating the data in the pickle file and actually using it to compute the response. I suspect that is the Microsoft Azure ML part I'm not sure. As I said I don't write the models, I just come along afterwards and review the results. I don't know anything about Python.
- fbdab103 4y ago
  >the serialization format is derived from the implementation and not very stable, so if the implementation changes - sometimes in subtle ways like upgrading your Python version - the serialized data isn't readable anymore. Could be wrong, but I do not believe this is true anymore. There are pickle versions, but I believe the format has been standardized such that you should be able to freely move between Python releases.
  - jononor 4y ago
    The base serialization is stable and compatible across Python versions. However pickle files can serialize entire Python objects, including from custom and third party code. Compatibility of serialization then depends on the versions of the code that implements those objects, which is often not well managed.
    - fbdab103 4y ago
      Ah that's fair. I was only considering serializing native types (for which there are much better options).
goosedragons 4y ago
I think the R equivalent of a pickle file is just an object(s) that's been saved with "save" or "saveRDS" to a file. These can then be loaded and could contain a trained model.
kgwgk 4y ago
> something called a "pickle file" I think this is something very specific to Python - maybe the equivalent would be a .dll file in C++ world. The equivalent would be a .dat file :-)

fbdab103 4y ago

>Tried putting R into production recently? It’s a frustrating and brittle experience. Oh man, can I sign onto this rant. While Python has spent a decade+ trying-and-failing to standardize on one of a dozen tools to properly manage dependencies, at least it is trying. R is still global-namespace, no-pinning by default. Sorta-kinda you can squint where renv is going, but still needs a lot of development. I had some hopes of Julia stealing the R mindshare and righting some of the more egregious wrongs in that ecosystem, but (as an outsider) it feels like Julia has lost a lot of steam.

FridgeSeal 4y ago
> Oh man, can I sign onto this rant. Please, be my guest hahaha. I think my issue with so many of these tools is that they’re fundamentally meant to be used interactively. R (renv/etc) doesn’t have a package manager separate from the runtime, so you can’t just “yarn/pip/cargo/etc install” and have your dependencies setup prior to your script running, you have to invoke manual R commands and hope everything goes off ok, it doesn’t spontaneously pick a different directory to install your stuff in, or have a tantrum about permissions because it wants to install in some root dir.
lukego 4y ago
I find it really painless to put bits of R into production using the Nix support: https://nixos.org/manual/nixpkgs/stable/#r https://nixos.org/manual/nixpkgs/stable/#r The same framework can provide an identical-to-production R or RStudio environment for development. I have a harder time with Julia since its tooling is more opinionated. I don't know a better solution than wrapping it in a Docker container.
- fbdab103 4y ago
  If the cure for R is that I need to jump fully into the Nix ecosystem, I am not sure that is a solution. Disappointing about Julia. Only played with it a small bit, but I had assumed it had identified dependency management as a huge problem that needed to be addressed.
  - 4y ago

Tcepsa 4y ago

I believe that is one of the points the author of the article is trying to make: that the data scientists ought to be able to work in whatever language lets them develop the model most quickly and then hand it over to a software developer to make it work well in code, so that each can focus on doing what they do best and both aspects can be done very well.

andy800 4y ago

> wish there was some more variety Kotlin is coming up on the outside. Late to the party, but the data science ecosystem is progressing nicely.

hdespiritu 4y ago
This is an interesting observation. Some of the data scientists I work with have been using Kotlin to define "analytic grammars", and is the first example (in my limited Kotlin awareness) i've seen of Kotlin outside of Android development. It seems that Roman Elizarov (Kotlin Project Lead) has identified the opportunity for a better language ecosystem to enter the data science space https://discuss.kotlinlang.org/t/ai-and-deep-learning-define-the-future-of-programming-will-kotlin-fly-or-die/2264/5 https://discuss.kotlinlang.org/t/ai-and-deep-learning-define...
pjmlp 4y ago
CERN started diving into Python back in 2002, I was part of one of the first trainings that were taking place. Same applies to many other research institutions. Good luck catching up with a 20 years old ecosystem, with a language that only gets used mainly thanks to Google's sponsorhip for Android apps.
- andy800 4y ago
  Kind of ignorant... given that Kotlin is JVM-compatible, it works with libraries even more than 20 years old. Do I care if it is at parity with Python? Hell no I don't, as long as it allows me to accomplish the tasks I need -- and it does, faster, easier, with fewer errors -- then I'm very happy.