4 ms·

I'm in a devops role where we actually reroll the Tensorflow whl in-house (to get a few tweaks like specific AVX flags turned on), but because the rest of our d

I'm in a devops role where we actually reroll the Tensorflow whl in-house (to get a few tweaks like specific AVX flags turned on), but because the rest of our deployment is apt/debs, we then turn around and wrap that whl in a deb using Spotify's excellent dh-virtualenv: https://github.com/spotify/dh-virtualenv https://github.com/spotify/dh-virtualenv There's no expertise for Bazel in-house; when we run the build, it seems to fail all its cache hits and then spend 12-13h in total compiling, much of which appears to be recompiling a specific version of LLVM. Every dependency is either vendored or pinned, including some critical things that have no ABI guarantees like Eigen, which is literally pinned to some random commit, so that causes chaos when other binaries try to link up with the underlying Tensorflow shared objects: https://github.com/tensorflow/tensorflow/blob/master/third_party/eigen3/workspace.bzl#L9 https://github.com/tensorflow/tensorflow/blob/master/third_p... And when you go down a layer into CUDA, there are even more support matrices listing exact known sets of versions of things that work together: https://docs.nvidia.com/deeplearning/tensorrt/support-matrix/index.html https://docs.nvidia.com/deeplearning/tensorrt/support-matrix... Anyway, I'm mostly just venting here. But the whole thing is an absurd nightmare. I have no idea how a normal distro would even begin to approach the task of unvendoring this stuff and shipping a set of reasonable packages for it all. The Tensorflow maintainers themselves even kind of admit the futility of it all when they propose that the easiest thing to do is just install your app on top of their pre-cooked Tensorflow GPU docker container: https://www.tensorflow.org/install https://www.tensorflow.org/install

stabbles 5y ago
You should really try out the spack package manager, it will manage all these constraints between packages, variants, dependencies for you by feeding it a very simple description of what you want to install: https://github.com/spack/spack https://github.com/spack/spack
- mikepurvis 5y ago
  Looks a bit nixy, but without the whole input-addressed hash thing.
  - tgamblin 5y ago
    And with dependency resolution (spack calls it “concretization”), which nix and guix do not have.
  - kristjansson 5y ago
    You might be interested in this talk from FOSDEM a few years ago comparing nix, spack, conda, and guix https://archive.fosdem.org/2018/schedule/event/installing_software_for_scientists/attachments/slides/2437/export/events/attachments/installing_software_for_scientists/slides/2437/20180204_installing_software_for_scientists.pdf https://archive.fosdem.org/2018/schedule/event/installing_so...
physicsguy 5y ago
+1 for Spack and also EasyBuild.
- fock 5y ago
  although spack had a lot of problems with tensorflow for years... But yes, in general it works best nowadays and with clingo coming up, I guess it will eventually beat NixOS (because hopefully browser-like dependency trees become fast then). (and I still don't get the advantages of an advanced build system like bazel, where I first have to build the right version of the build system (including all patches which might crop up with new libcs in vendored dependencies), and then it takes ages to compile (compared to any other similar software...) and I still have to fix the libc bugs...)
  - krapht 5y ago
    Bazel is for the Google-scale companies of the world who must ship petabytes around the internet. Mortals do not use Bazel - a cluster of 80 core CPUs does, and the user just downloads the cached build results. It is extremely powerful but also extremely tedious, like programming in MPI.
geofft 5y ago
FYI, I also rebuild TensorFlow in-house, hate it, and hate it a bit less now that I've realized that you can get 200+ vCPU machines in the cloud without difficulty these days. It takes about 12 hours on my workstation too, but it takes just under half an hour on an n2d-highcpu-224, for an effective cost of under $4 per build. I have a VM I power on whenever I need to rebuild TensorFlow and power off once I upload the resulting wheels. Getting my edit-compile-test-scream cycle down to under a day is definitely worth a couple of dollars to the business. (The irony of paying Google to compile their software is not lost on me.)
kortex 5y ago
> Every dependency is either vendored or pinned, including some critical things that have no ABI guarantees like Eigen, which is literally pinned to some random commit, starts sweating from GDAL PTSD I hadn't heard about dh-virtualenv. That looks super convenient, I'll definitely give that a shot. Honestly I've been kinda leaning into Conda-in-Docker images (so you can isolate while you isolate, dawg), which actually isn't so bad with a custom SHELL directive so you can RUN in a conda env. So many of my customers use it and provide algos developed in conda that it's often just easier to grab their environment.yml and stick it all in a container for deployment. The pre-cooked TF image are a godsend but also the instant you need two different versions of TF in the same pipeline, you're hosed. I've been playing with Argo Workflows to get around this. ML software packaging is the worst.
- mikepurvis 5y ago
  > two different versions of TF in the same pipeline Or if you just need to integrate with anything else whose deployment model is "lol here's a container."
tgamblin 5y ago
Check out the tensorflow package in Spack. We patch Bazel slightly to allow external dependencies/library paths, and the dependencies are factored out into their own packages: https://spack.readthedocs.io/en/latest/package_list.html#py-tensorflow https://spack.readthedocs.io/en/latest/package_list.html#py-... Spack can build with two levels of parallelism (within a build and among builds) on a single node or multiple nodes with a shared filesystem: https://spack.readthedocs.io/en/latest/packaging_guide.html#install-level-build-parallelism https://spack.readthedocs.io/en/latest/packaging_guide.html#... You can also generate GitLab pipelines of your build, which will break the jobs out as well: https://spack.readthedocs.io/en/latest/pipelines.html https://spack.readthedocs.io/en/latest/pipelines.html

stabbles 5y ago

You should really try out the spack package manager, it will manage all these constraints between packages, variants, dependencies for you by feeding it a very simple description of what you want to install: https://github.com/spack/spack https://github.com/spack/spack

mikepurvis 5y ago
Looks a bit nixy, but without the whole input-addressed hash thing.
- tgamblin 5y ago
  And with dependency resolution (spack calls it “concretization”), which nix and guix do not have.
- kristjansson 5y ago
  You might be interested in this talk from FOSDEM a few years ago comparing nix, spack, conda, and guix https://archive.fosdem.org/2018/schedule/event/installing_software_for_scientists/attachments/slides/2437/export/events/attachments/installing_software_for_scientists/slides/2437/20180204_installing_software_for_scientists.pdf https://archive.fosdem.org/2018/schedule/event/installing_so...

physicsguy 5y ago

+1 for Spack and also EasyBuild.

fock 5y ago
although spack had a lot of problems with tensorflow for years... But yes, in general it works best nowadays and with clingo coming up, I guess it will eventually beat NixOS (because hopefully browser-like dependency trees become fast then). (and I still don't get the advantages of an advanced build system like bazel, where I first have to build the right version of the build system (including all patches which might crop up with new libcs in vendored dependencies), and then it takes ages to compile (compared to any other similar software...) and I still have to fix the libc bugs...)
- krapht 5y ago
  Bazel is for the Google-scale companies of the world who must ship petabytes around the internet. Mortals do not use Bazel - a cluster of 80 core CPUs does, and the user just downloads the cached build results. It is extremely powerful but also extremely tedious, like programming in MPI.

geofft 5y ago

FYI, I also rebuild TensorFlow in-house, hate it, and hate it a bit less now that I've realized that you can get 200+ vCPU machines in the cloud without difficulty these days. It takes about 12 hours on my workstation too, but it takes just under half an hour on an n2d-highcpu-224, for an effective cost of under $4 per build. I have a VM I power on whenever I need to rebuild TensorFlow and power off once I upload the resulting wheels. Getting my edit-compile-test-scream cycle down to under a day is definitely worth a couple of dollars to the business. (The irony of paying Google to compile their software is not lost on me.)

kortex 5y ago

> Every dependency is either vendored or pinned, including some critical things that have no ABI guarantees like Eigen, which is literally pinned to some random commit, starts sweating from GDAL PTSD I hadn't heard about dh-virtualenv. That looks super convenient, I'll definitely give that a shot. Honestly I've been kinda leaning into Conda-in-Docker images (so you can isolate while you isolate, dawg), which actually isn't so bad with a custom SHELL directive so you can RUN in a conda env. So many of my customers use it and provide algos developed in conda that it's often just easier to grab their environment.yml and stick it all in a container for deployment. The pre-cooked TF image are a godsend but also the instant you need two different versions of TF in the same pipeline, you're hosed. I've been playing with Argo Workflows to get around this. ML software packaging is the worst.

mikepurvis 5y ago
> two different versions of TF in the same pipeline Or if you just need to integrate with anything else whose deployment model is "lol here's a container."

tgamblin 5y ago

Check out the tensorflow package in Spack. We patch Bazel slightly to allow external dependencies/library paths, and the dependencies are factored out into their own packages: https://spack.readthedocs.io/en/latest/package_list.html#py-tensorflow https://spack.readthedocs.io/en/latest/package_list.html#py-... Spack can build with two levels of parallelism (within a build and among builds) on a single node or multiple nodes with a shared filesystem: https://spack.readthedocs.io/en/latest/packaging_guide.html#install-level-build-parallelism https://spack.readthedocs.io/en/latest/packaging_guide.html#... You can also generate GitLab pipelines of your build, which will break the jobs out as well: https://spack.readthedocs.io/en/latest/pipelines.html https://spack.readthedocs.io/en/latest/pipelines.html