4 ms·

What is the solution? Is there a reason why things are not better? Can all of the deps be loaded in a Docker image and be done with it?

Kuinox 4y ago
> What is the solution? There are languages ecosystem that solved these issues and never have a problem with it. Look at it, and copy their tooling.
- dismantlethesun 4y ago
  What are some of those ecosystems?
  - mattkrause 4y ago
    Julia's package manager has basically never failed on me.
    - dismantlethesun 4y ago
      Thanks. That’s something to look into then.
      - adgjlsfhk1 4y ago
        Note that the Julia package manager (while very good) is very similar to (and largely inspired by) the rust package manager.
    - obert 4y ago
      basically never, or never? ;-)
  - Kuinox 4y ago
    .NET with nuget works flawlessly, compared to bad package managers.
    - wizofaus 4y ago
      It's pretty good but certainly not flawless. Admittedly I probably don't have high standards. Cocoapods is one of the few I'd consider objectively "bad", though apparently among iOS developers it's considered one of the better ones (!).
      - Kuinox 4y ago
        That's why I said "compared to bad package managers." NuGet works, have flaws, sometimes doesn't works, but is usually reliable. On the other hands, I lost a lot of time with npm, and also a lot with pip.
matsemann 4y ago
One thing is python's dependency management is just insane in itself. When you then additionally have to have it install basically drivers for the gpu, compile loads of native code etc it becomes a hot mess. And half the ML things you want to try are academic experiments not really made for distribution, so they were made to work and that's it. So if you have some different computer setup, minor version mismatch of a dependency etc it will just break. Datasets or models excist on some url that only works for a month. It's a shame, I think the field is one where reproduction of results should be really welcome and feasible. Yeah, a docker image would be nice. Even a Dockerfile, even though that in itself may not guarantee reproducibility if you try to build it later. (And may have issues with gpu drivers etc). But at least it documents all assumptions about the setup.
- dicknuckle 4y ago
  That's the cool thing about publishing the Dockerfile and an image, one is an example that may or may not break, and the other is a functional snapshot of a working config at that point in time.
cellis 4y ago
No. This would take a very long post to explain why, but in short, it depends heavily on which NVIDIA Cards / CUDA drivers / versions of Linux you're using. Or you're running on AWS GPUs or TPUs and paying a lot more either in the form of straight up dollars or optionality.
jollybean 4y ago
Cloud services. Someone sets it up, leases out instances etc.. Or someone works to ensure that everything gels together and provides support etc.. Or in other words: 'products'

Kuinox 4y ago

> What is the solution? There are languages ecosystem that solved these issues and never have a problem with it. Look at it, and copy their tooling.

dismantlethesun 4y ago
What are some of those ecosystems?
- mattkrause 4y ago
  Julia's package manager has basically never failed on me.
  - dismantlethesun 4y ago
    Thanks. That’s something to look into then.
    - adgjlsfhk1 4y ago
      Note that the Julia package manager (while very good) is very similar to (and largely inspired by) the rust package manager.
  - obert 4y ago
    basically never, or never? ;-)
- Kuinox 4y ago
  .NET with nuget works flawlessly, compared to bad package managers.
  - wizofaus 4y ago
    It's pretty good but certainly not flawless. Admittedly I probably don't have high standards. Cocoapods is one of the few I'd consider objectively "bad", though apparently among iOS developers it's considered one of the better ones (!).
    - Kuinox 4y ago
      That's why I said "compared to bad package managers." NuGet works, have flaws, sometimes doesn't works, but is usually reliable. On the other hands, I lost a lot of time with npm, and also a lot with pip.

matsemann 4y ago

One thing is python's dependency management is just insane in itself. When you then additionally have to have it install basically drivers for the gpu, compile loads of native code etc it becomes a hot mess. And half the ML things you want to try are academic experiments not really made for distribution, so they were made to work and that's it. So if you have some different computer setup, minor version mismatch of a dependency etc it will just break. Datasets or models excist on some url that only works for a month. It's a shame, I think the field is one where reproduction of results should be really welcome and feasible. Yeah, a docker image would be nice. Even a Dockerfile, even though that in itself may not guarantee reproducibility if you try to build it later. (And may have issues with gpu drivers etc). But at least it documents all assumptions about the setup.

dicknuckle 4y ago
That's the cool thing about publishing the Dockerfile and an image, one is an example that may or may not break, and the other is a functional snapshot of a working config at that point in time.

cellis 4y ago

No. This would take a very long post to explain why, but in short, it depends heavily on which NVIDIA Cards / CUDA drivers / versions of Linux you're using. Or you're running on AWS GPUs or TPUs and paying a lot more either in the form of straight up dollars or optionality.

jollybean 4y ago

Cloud services. Someone sets it up, leases out instances etc.. Or someone works to ensure that everything gels together and provides support etc.. Or in other words: 'products'