3 ms·

Ruby is nice, but if you were to actually solve this problem for Python you'd print money. The deluge of knowledge required to actually navigate Python dependen

Ruby is nice, but if you were to actually solve this problem for Python you'd print money. The deluge of knowledge required to actually navigate Python dependency management across all major OS's for the most complex use cases (typically ML stuff with deep compiler/tool chain deps) is nothing short of a nightmare. Especially on Windows, the rage I feel towards MSVC knows no bounds. venv, pip, (mini)conda, mamba, poetry, apt, yum... Obviously the last two are good to know anyways, but ugh the hours I've lost and will never get back to the rest. Anyways, god speed my dudes.

stevepike 3y ago
(founder here). The interaction between high level language package managers and system binaries or compiled extensions is definitely on our roadmap. For example the ruby net-ssh gem relies on specific versions of openssl being compiled on the machine. Do you think this is particularly difficult in ML? Is your frustration mostly from needing compile all this non-python stuff across environments?
- ZeroCool2u 3y ago
  Yeah, you're right on the money. This is basically insane in ML, but _especially_ in corporate environments, think F500, where sudo is disabled in containers and experimentation (in terms of trying different package versions) is quite challenging due to security restrictions. Frankly, it's one of the reasons that we're seeing a lot of excitement around stuff written in Rust, but with Python bindings[1][2]. Cargo makes it so easy to build across various platforms and then just produce statically compiled platform specific python wheels. If PyTorch and TF had been written using Rust, any other other ultra high perf language with a tool-chain as reliable as cargo, a lot of folks lives would be easier. [1]: https://www.getdaft.io/ https://www.getdaft.io/ [2]: https://www.pola.rs/ https://www.pola.rs/
- pabs3 3y ago
  Hmm, the Debian package of ruby-net-ssh doesn't even depend on OpenSSL at all.
  - stevepike 3y ago
    Interesting. Should it? I don't have much experience installing ruby gems with the OS package managers, I've always done it through the language-specific ones like bundler / gem. Here's a github issue showing the kind of things that come up when the versions are mismatched: https://github.com/net-ssh/net-ssh/issues/843 https://github.com/net-ssh/net-ssh/issues/843
    - pabs3 3y ago
      I was probably looking at an older or newer version to you, likely it switched from OpenSSL to pure-Ruby implementations of the crypto stuff.
jchonphoenix 3y ago
I don't think you'd print money just by telling people if a dependency is hard or easy to install though. You'd need to actually solve the problem of getting the dependency installed or suggesting some alternative solution that gets the code written. E.g. if I'm banging my head against a wall trying to upgrade to the newest version of PyTorch for some graph embeddings feature and everything breaks, knowing it's high risk doesn't help me. I still need to get the thing installed so I can use the feature.
- ZeroCool2u 3y ago
  Okay yeah, totally agree. I already know it's hard to install. If they gave me a path they were confident would work in terms of deps/versions to upgrade to, that I would pay for if it worked well.
esafak 3y ago
Do you work at a company forcing you to develop on Windows? You couldn't pay to do to that.

stevepike 3y ago

(founder here). The interaction between high level language package managers and system binaries or compiled extensions is definitely on our roadmap. For example the ruby net-ssh gem relies on specific versions of openssl being compiled on the machine. Do you think this is particularly difficult in ML? Is your frustration mostly from needing compile all this non-python stuff across environments?

ZeroCool2u 3y ago
Yeah, you're right on the money. This is basically insane in ML, but _especially_ in corporate environments, think F500, where sudo is disabled in containers and experimentation (in terms of trying different package versions) is quite challenging due to security restrictions. Frankly, it's one of the reasons that we're seeing a lot of excitement around stuff written in Rust, but with Python bindings[1][2]. Cargo makes it so easy to build across various platforms and then just produce statically compiled platform specific python wheels. If PyTorch and TF had been written using Rust, any other other ultra high perf language with a tool-chain as reliable as cargo, a lot of folks lives would be easier. [1]: https://www.getdaft.io/ https://www.getdaft.io/ [2]: https://www.pola.rs/ https://www.pola.rs/
pabs3 3y ago
Hmm, the Debian package of ruby-net-ssh doesn't even depend on OpenSSL at all.
- stevepike 3y ago
  Interesting. Should it? I don't have much experience installing ruby gems with the OS package managers, I've always done it through the language-specific ones like bundler / gem. Here's a github issue showing the kind of things that come up when the versions are mismatched: https://github.com/net-ssh/net-ssh/issues/843 https://github.com/net-ssh/net-ssh/issues/843
  - pabs3 3y ago
    I was probably looking at an older or newer version to you, likely it switched from OpenSSL to pure-Ruby implementations of the crypto stuff.

jchonphoenix 3y ago

I don't think you'd print money just by telling people if a dependency is hard or easy to install though. You'd need to actually solve the problem of getting the dependency installed or suggesting some alternative solution that gets the code written. E.g. if I'm banging my head against a wall trying to upgrade to the newest version of PyTorch for some graph embeddings feature and everything breaks, knowing it's high risk doesn't help me. I still need to get the thing installed so I can use the feature.

ZeroCool2u 3y ago
Okay yeah, totally agree. I already know it's hard to install. If they gave me a path they were confident would work in terms of deps/versions to upgrade to, that I would pay for if it worked well.

esafak 3y ago

Do you work at a company forcing you to develop on Windows? You couldn't pay to do to that.