5 ms·

Python's dominance is holding us back. We need a stack with a more principled approach to environments and native dependencies.

llm_trw 2y ago
Here's what getting PyTorch built reproducibly looks like: https://hpc.guix.info/blog/2021/09/whats-in-a-package/ https://hpc.guix.info/blog/2021/09/whats-in-a-package/ Since then the whole python ecosystem has gotten worse. We are building towers on quicksand. It's not about python, it's about people who don't care about dependencies.
- hkt 2y ago
  Dependency management is just.. hard. It is one of the things where everything relies upon it but nobody thinks "hey, this is my responsibility to improve" so it is left to people who have the most motivation, academic posts, or grant funding. This is roughly the same problem that led to heartbleed for OpenSSL.
manusachi 2y ago
Do you know what other ecosystem comes closest to the existing in Python? I've heard good things about Julia. 13 years ago when I was trying to explore the field R seemed to be the most popular, but looks like not anymore. (I didn't get into the field, and do just a regular SWE, so I'm not aware of the trends). There is also a lot of development in Elixir ecosystem around the subject [1]. [1](https://dashbit.co/blog/elixir-ml-s1-2024-mlir-arrow-instructor https://dashbit.co/blog/elixir-ml-s1-2024-mlir-arrow-instruc...)
- posix_monad 2y ago
  I don't think the ML community has an appetite for learning a different language. There are Microsoft-backed F# bindings for Spark and Torch, but no one seems interested. And this is despite a pretty compelling proposition of lightweight syntax, strong typing, script-ability and great performance. The answer will probably be JavaScript. Everyone already knows the language - all that's missing is operator overloading and a few key bindings.
  - soulbadguy 2y ago
    > There are Microsoft-backed F# bindings for Spark and Torch > this is despite a pretty compelling proposition of lightweight syntax, strong typing, script-ability and great performance. For exactly the reasons your mentioned, i feel like F# would have been the perfect match for both MLE/ETL(spark pipeline) work and some of the deep learning/graph modelisation such as pytorch. Saddly, even from MSFT, the investment in F# as dried up
    - posix_monad 2y ago
      Not sure what you mean here - Microsoft are actively working on F# and the .NET Spark and Torch bindings.

llm_trw 2y ago

Here's what getting PyTorch built reproducibly looks like: https://hpc.guix.info/blog/2021/09/whats-in-a-package/ https://hpc.guix.info/blog/2021/09/whats-in-a-package/ Since then the whole python ecosystem has gotten worse. We are building towers on quicksand. It's not about python, it's about people who don't care about dependencies.

hkt 2y ago
Dependency management is just.. hard. It is one of the things where everything relies upon it but nobody thinks "hey, this is my responsibility to improve" so it is left to people who have the most motivation, academic posts, or grant funding. This is roughly the same problem that led to heartbleed for OpenSSL.

manusachi 2y ago

Do you know what other ecosystem comes closest to the existing in Python? I've heard good things about Julia. 13 years ago when I was trying to explore the field R seemed to be the most popular, but looks like not anymore. (I didn't get into the field, and do just a regular SWE, so I'm not aware of the trends). There is also a lot of development in Elixir ecosystem around the subject [1]. [1](https://dashbit.co/blog/elixir-ml-s1-2024-mlir-arrow-instructor https://dashbit.co/blog/elixir-ml-s1-2024-mlir-arrow-instruc...)

posix_monad 2y ago
I don't think the ML community has an appetite for learning a different language. There are Microsoft-backed F# bindings for Spark and Torch, but no one seems interested. And this is despite a pretty compelling proposition of lightweight syntax, strong typing, script-ability and great performance. The answer will probably be JavaScript. Everyone already knows the language - all that's missing is operator overloading and a few key bindings.
- soulbadguy 2y ago
  > There are Microsoft-backed F# bindings for Spark and Torch > this is despite a pretty compelling proposition of lightweight syntax, strong typing, script-ability and great performance. For exactly the reasons your mentioned, i feel like F# would have been the perfect match for both MLE/ETL(spark pipeline) work and some of the deep learning/graph modelisation such as pytorch. Saddly, even from MSFT, the investment in F# as dried up
  - posix_monad 2y ago
    Not sure what you mean here - Microsoft are actively working on F# and the .NET Spark and Torch bindings.