5 ms·

>how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions Probably not all that muc

>how much Python spaghetti probably tanks OpenAI's and other "big ML" shops' infra when they should've trusted in zero copy solutions Probably not all that much. All of the Python numeric computing frameworks (Numpy, PyTorch, TensorFlow, etc.) are basically just wrappers for lower level C++/C/Fortran code. Unless you’re doing something boneheaded and converting framework-native tensors to Python objects, passing tensors around within a framework essentially just passes a pointer around, which has marginal overhead even when encapsulated in a bloated Python object. Indeed, a huge number of PyTorch operations are explicitly zero copy: https://pytorch.org/docs/stable/tensor_view.html https://pytorch.org/docs/stable/tensor_view.html

oceanplexian 4y ago
It’s not that the performance is the issue, it’s that it’s unmaintainable and prone to break. Exceptions aren’t handled right, dependencies are a disaster (Proprietary NVIDIA drivers+CUDA+PyTorch+ the various versions of stuff are a complete disaster) This leads to all sorts of bugs and breaking changes that are cool in an academic or hobbyist setting but a total headache on a large production system.
- eastWestMath 4y ago
  I was in a PLT group in grad school going into robotics. I could spend all day ranting about how Python is just completely unsuitable for professional software development. Even something like F# would be an enormous improvement.
- rfoo 4y ago
  The stack is very volatile and unmaintainable because it doesn't need to be maintainable. Exactly why we have unmaintainable software in other domains. During the last 10 years there are ALWAYS totally new model architecture with new operations (or in case of CV new bizarre uses of Conv). By the time you get your performant perfectly maintainable masterpiece ready it's not needed anymore. The stack optimizes for flexibility and iteration speed naturally, just like why people use Rails. In fact I'd love to see that Transformer really dominates. We can then start to converge on software. And compute-wise transformers are really simple, too!
  - nunobrito 4y ago
    Still a poor excuse. Had they written this in Java and things wouldn't be so difficult both on performance and maintainability. Never understood why people think that indented languages are any simpler when in fact they bring all kinds of trouble for getting things done.
    - revelio 4y ago
      There's a Java ML library called Tribuo that might be worth looking at.
      - dopidopHN 4y ago
        Thanks, the boring aspect of Java is appealing here.
    - rfoo 4y ago
      There's deeplearning4j (from Theano days!), go figure why it didn't take off.
  - wootland 4y ago
    Does this mean it would be easy to move off Python all together? It seems like the problem stems from everyone using pytorch at the base layer. How realistic is it recreate those apis in another, more modern language. Coding in Rust, Go... then distributing a single binary vs. pip hell seems like it would be worth it.
    - sroussey 4y ago
      Go would be interesting for the reason you could send an executable. I’d love for JS/TS to dominate as well. Use ‘bun bun’ to send an executable if need be, but also use in in web backends.
    - rfoo 4y ago
      Check https://pytorch.org/tutorials/advanced/cpp_frontend.html https://pytorch.org/tutorials/advanced/cpp_frontend.html You can easily build a standalone binary (well, it would be GiB+ if you use CUDA... but that's the cost of statically linking cu*), had you coded your model and training loop in C++. It then happily runs everywhere as long as a NVIDIA GPU driver is available (don't need to install CUDA). Protip: Your AI research team REALLY DON'T WANT TO DO THIS BECAUSE THEY LOVE PYTHON. Having Python, even with the dependency management shit, is a feature, not a bug. (if you want Rust / Go and don't want to wrapping libtorch/tf then you have a lot of work to do but yeah it's possible. also there are model compiler guys [1] where the promise is model.py in model.o out you just link it with your code) [1] https://mlc.ai https://mlc.ai
- CurrentB 4y ago
  Yeah, I've been using python for the first time in a while to try out some of the llm stuff and I can't believe how bad the dependency hell is. It's probably particularly bad due to the pace of change in this field. But I spend an hour getting dependencies fixed every time I touch anything. 80% of the Google Collabs I find are just outright broken. I wish there were other viable non python options to try out these things.
  - bboygravity 4y ago
    No idea what a Google Collab is, but does the code come with an environment or at least a specifications of which packages and versions to use (requirements.txt)? It sounds unnecessarily weird to me that people would share Python code that simply doesn't work out at all out of the box.
    - version_five 4y ago
      Its rarely as easy as sharing a requirements.txt. There are lots of things that can still break - for examples you get weird situations where different modules require different versions of a third module. Or all the Cuda toolkit version issues thsy seem to come up with gpu stuff. When we share python, we tend to share a docker image, and even this isn't foolproof. A big problem I think is that it doesn't incentivize building something portable. And it's very hard to test across different machines. Add to that all the different practices re virtual environments, venv, conda, etc, everyone tries to install the dependencies differently or is starting from some nonstandard state. It's a mess.
      - pablo1107 4y ago
        Maybe using Nix it's a better experience for creating such an environment where you depending also on system utilities.
        superkuh 4y ago
        Everyone is using llama.cpp because we reject the idea of giving up on system libraries like nix does. That kind of tomfoolery (at least in the desktop context) is only required when you use software projects that use libraries/languages which break forwards compatibility every 3 years. If you just write straight c++ (without c++xx, or anything like it) you can compile the code on machines from decades ago if you want.
miraculixx 4y ago
What a bad take! Python is not the cause of dependency hell. Deep dependency trees are. The only way to deal with this is to use seperate environments and to carefully specify the exact requirements. Those who claim some language would be a magical fix clearly lack experience in multiple languages.
- wnoise 4y ago
  It's true nothing forces or forbids this, but some languages/toolings/communities/ecosystems encourage that more than others though.

oceanplexian 4y ago

It’s not that the performance is the issue, it’s that it’s unmaintainable and prone to break. Exceptions aren’t handled right, dependencies are a disaster (Proprietary NVIDIA drivers+CUDA+PyTorch+ the various versions of stuff are a complete disaster) This leads to all sorts of bugs and breaking changes that are cool in an academic or hobbyist setting but a total headache on a large production system.

eastWestMath 4y ago
I was in a PLT group in grad school going into robotics. I could spend all day ranting about how Python is just completely unsuitable for professional software development. Even something like F# would be an enormous improvement.
rfoo 4y ago
The stack is very volatile and unmaintainable because it doesn't need to be maintainable. Exactly why we have unmaintainable software in other domains. During the last 10 years there are ALWAYS totally new model architecture with new operations (or in case of CV new bizarre uses of Conv). By the time you get your performant perfectly maintainable masterpiece ready it's not needed anymore. The stack optimizes for flexibility and iteration speed naturally, just like why people use Rails. In fact I'd love to see that Transformer really dominates. We can then start to converge on software. And compute-wise transformers are really simple, too!
- nunobrito 4y ago
  Still a poor excuse. Had they written this in Java and things wouldn't be so difficult both on performance and maintainability. Never understood why people think that indented languages are any simpler when in fact they bring all kinds of trouble for getting things done.
  - revelio 4y ago
    There's a Java ML library called Tribuo that might be worth looking at.
    - dopidopHN 4y ago
      Thanks, the boring aspect of Java is appealing here.
  - rfoo 4y ago
    There's deeplearning4j (from Theano days!), go figure why it didn't take off.
- wootland 4y ago
  Does this mean it would be easy to move off Python all together? It seems like the problem stems from everyone using pytorch at the base layer. How realistic is it recreate those apis in another, more modern language. Coding in Rust, Go... then distributing a single binary vs. pip hell seems like it would be worth it.
  - sroussey 4y ago
    Go would be interesting for the reason you could send an executable. I’d love for JS/TS to dominate as well. Use ‘bun bun’ to send an executable if need be, but also use in in web backends.
  - rfoo 4y ago
    Check https://pytorch.org/tutorials/advanced/cpp_frontend.html https://pytorch.org/tutorials/advanced/cpp_frontend.html You can easily build a standalone binary (well, it would be GiB+ if you use CUDA... but that's the cost of statically linking cu*), had you coded your model and training loop in C++. It then happily runs everywhere as long as a NVIDIA GPU driver is available (don't need to install CUDA). Protip: Your AI research team REALLY DON'T WANT TO DO THIS BECAUSE THEY LOVE PYTHON. Having Python, even with the dependency management shit, is a feature, not a bug. (if you want Rust / Go and don't want to wrapping libtorch/tf then you have a lot of work to do but yeah it's possible. also there are model compiler guys [1] where the promise is model.py in model.o out you just link it with your code) [1] https://mlc.ai https://mlc.ai
CurrentB 4y ago
Yeah, I've been using python for the first time in a while to try out some of the llm stuff and I can't believe how bad the dependency hell is. It's probably particularly bad due to the pace of change in this field. But I spend an hour getting dependencies fixed every time I touch anything. 80% of the Google Collabs I find are just outright broken. I wish there were other viable non python options to try out these things.
- bboygravity 4y ago
  No idea what a Google Collab is, but does the code come with an environment or at least a specifications of which packages and versions to use (requirements.txt)? It sounds unnecessarily weird to me that people would share Python code that simply doesn't work out at all out of the box.
  - version_five 4y ago
    Its rarely as easy as sharing a requirements.txt. There are lots of things that can still break - for examples you get weird situations where different modules require different versions of a third module. Or all the Cuda toolkit version issues thsy seem to come up with gpu stuff. When we share python, we tend to share a docker image, and even this isn't foolproof. A big problem I think is that it doesn't incentivize building something portable. And it's very hard to test across different machines. Add to that all the different practices re virtual environments, venv, conda, etc, everyone tries to install the dependencies differently or is starting from some nonstandard state. It's a mess.
    - pablo1107 4y ago
      Maybe using Nix it's a better experience for creating such an environment where you depending also on system utilities.
      - superkuh 4y ago
        Everyone is using llama.cpp because we reject the idea of giving up on system libraries like nix does. That kind of tomfoolery (at least in the desktop context) is only required when you use software projects that use libraries/languages which break forwards compatibility every 3 years. If you just write straight c++ (without c++xx, or anything like it) you can compile the code on machines from decades ago if you want.

miraculixx 4y ago

What a bad take! Python is not the cause of dependency hell. Deep dependency trees are. The only way to deal with this is to use seperate environments and to carefully specify the exact requirements. Those who claim some language would be a magical fix clearly lack experience in multiple languages.

wnoise 4y ago
It's true nothing forces or forbids this, but some languages/toolings/communities/ecosystems encourage that more than others though.