4 ms·

I've mostly seen the opposite - pandas and jupyter notebooks shipped directly to production because the data scientists and AI guys didn't know how to do anythi

by umvi 2y ago

I've mostly seen the opposite - pandas and jupyter notebooks shipped directly to production because the data scientists and AI guys didn't know how to do anything but python. As a result, the solutions were not performant and often had lots of runtime crashes due to python's more loose typing

Spivak 2y ago
I'm not sure how you're going to fix this given the data science tools are in Python. Are you gonna implement a half-broken 20% of numpy/scipy for a one-off program and then try to port? These libraries hide a lot of complexity and implementing even a few operators is a project.
- supermatt 2y ago
  i dont think they are complaining about the libraries, but rather the scratchpads and notebooks that people use for ideation and evaluation being moved directly into a production environment because the authors don't have the experience or time to build more structured, efficient and maintainable code.
- grahamjameson 2y ago
  Rust + Polars comes to mind.
philote 2y ago
If they're shipping notebooks to production and having so many crashes, I'd question that they even know how to do Python.
- Narhem 2y ago
  When do you ship notebooks to production? Jupiter was never meant for external clients.
  - pclmulqdq 2y ago
    While terrifying, it is not uncommon to see python notebooks make it to production.
    - EuAndreh 2y ago
      Oh god why. I thought (and hoped) that GP didn't actually mean this. I see how a team or an organization can eventually get to this point. It just saddens me that they got there.
      - nsonha 2y ago
        why not, many data related tasks are rather ad-hoc, it's a waste of time to make a long lasting software out of every ad-hoc request
    - Narhem 2y ago
      Seen the same with exposed matlab web apps. Not just the Python guys, kind of shocking how much group think exists on any platform.
  - mathgeek 2y ago
    Quite a number of AI edtech sites use notebooks in production for assignments, as an example of when it happens.
  - umvi 2y ago
    Usually what happens is: 1. "We got it all working in the notebook" 2. "Great, ship it" 3. (data scientist takes the notebook code almost verbatim, wraps it in a basic CLI or HTTP API and it gets shipped off in a docker container for other services to consume)
    - Narhem 2y ago
      In practice that’s not too crazy. It’s fast easy to debug and works with most CI/CD tools. If you ever want to work in teams that kind of setup works extremely well.

by umvi 2y ago

Spivak 2y ago

I'm not sure how you're going to fix this given the data science tools are in Python. Are you gonna implement a half-broken 20% of numpy/scipy for a one-off program and then try to port? These libraries hide a lot of complexity and implementing even a few operators is a project.

supermatt 2y ago
i dont think they are complaining about the libraries, but rather the scratchpads and notebooks that people use for ideation and evaluation being moved directly into a production environment because the authors don't have the experience or time to build more structured, efficient and maintainable code.
grahamjameson 2y ago
Rust + Polars comes to mind.

philote 2y ago

If they're shipping notebooks to production and having so many crashes, I'd question that they even know how to do Python.

Narhem 2y ago
When do you ship notebooks to production? Jupiter was never meant for external clients.
- pclmulqdq 2y ago
  While terrifying, it is not uncommon to see python notebooks make it to production.
  - EuAndreh 2y ago
    Oh god why. I thought (and hoped) that GP didn't actually mean this. I see how a team or an organization can eventually get to this point. It just saddens me that they got there.
    - nsonha 2y ago
      why not, many data related tasks are rather ad-hoc, it's a waste of time to make a long lasting software out of every ad-hoc request
  - Narhem 2y ago
    Seen the same with exposed matlab web apps. Not just the Python guys, kind of shocking how much group think exists on any platform.
- mathgeek 2y ago
  Quite a number of AI edtech sites use notebooks in production for assignments, as an example of when it happens.
- umvi 2y ago
  Usually what happens is: 1. "We got it all working in the notebook" 2. "Great, ship it" 3. (data scientist takes the notebook code almost verbatim, wraps it in a basic CLI or HTTP API and it gets shipped off in a docker container for other services to consume)
  - Narhem 2y ago
    In practice that’s not too crazy. It’s fast easy to debug and works with most CI/CD tools. If you ever want to work in teams that kind of setup works extremely well.