6 ms·
Incremental Builds for Haskell with Bazel
- pharmakom 4y agoBazel ergonomics / DX sucks but it does deliver on correct and fast.
- tantalor 4y agoCompared to what? Do you mean writing build rules, or the behind-the-scenes stuff?
- asadawadia 4y agoeverything
- marcyb5st 4y agoI agree on adding non native build rules, but once you get that over it is pretty epic. Also, I don't think it's more complex than SBT/Gradle (what I was using before, but admittedly was a long time ago so I don't know now).
- sp33der89 4y agoSBT is a lot less scary these days IMHO.
- vips7L 4y agoSBT is the bane of my existence.
- rtpg 4y agoFor a lot of people the alternatives are things like fabric/invoke. If you don’t care about correctness there a loads of options that work well, and don’t have problems supporting esoteric use cases such as …“let a command generate a folder of data”
- radvani13 4y agoHi, pharmakom I am the PM at bazel and would love to learn more about your DX pain points. Looking forward to connecting Regards, Radhika
- pharmakom 4y agoSure! I have a few annoyances with Bazel, but the biggest is quite fundamental. I really dislike how all rules share the same namespace for outputs. You can’t define two rules that output to the same path, and there is really little reason for this. Buck does not have this issue.
- corrral 4y agoAnyone got any stories about migrating big, crufty, multiplatform cmake C++ projects to Bazel? "It sucked but was worth it" or "it was surprisingly easy" or "LOL don't even try?"
- rufius 4y agoI’ve not got specifically what you’re describing but I’ve encountered Bazel twice in my career - both in very large Java/Scala/Go code bases (distinct repo for Go but a lot of code). Bazel is extremely underwhelming. I’ve worked with crusty ancient systems that built huge systems and Bazel is just the most clown shoes build tool in comparison. Something didn’t work? Try typing the same command repeatedly and hope that this time it sticks. Multiple commands to achieve a seemingly straightforward task? Why isn’t there a single one that will get us there. FWIW - I suspect like most tools that Google open sources, the tool makes far more sense in the context of Google’s systems and architecture. If you’re adhering to that, it’s probably coherent. I’d never choose it willingly though.
- rfoo 4y ago> Multiple commands to achieve a seemingly straightforward task? Why isn’t there a single one that will get us there. Could you elaborate on this? I seldomly need commands beside `bazel build`, `bazel test` and `bazel run` and am curious to your story.
- rufius 4y agoFull admission - I don’t own or have responsibility for how the build system I work with is setup. I’m on parental leave so details are fuzzy but roughly: - run Bazel build - expect dependencies to be built - they weren’t built - run Bazel build again - something else decides to be built - repeat ad nauseum This is within the same project directory, etc. It’s entirely possible that the project is setup in some pathological way but I’ve encountered this enough times in two companies that it’s stuck in my head.
- 4y ago
- aschleck 4y agoI really can't emphasize enough how much I love using Bazel. The ability to tell a less technical user "just run `bazel run //amazing/server`" regardless of language and know that everything will magically work (toolchain installation, future toolchain upgrades, incremental rebuilds) is really freeing. The actions graph with rules and aspects is quite powerful, so you can do things like add Java nullability checks or Python type checking remarkably easily. Recently I put together a simple build rule that strips external dependencies, archives the rest, and uploads it to a cache. Then we can easily run that archive against a pre-built container (which contains the external dependencies) in our cluster, enabling a very fast ML iteration loop on beefy cluster machines. I've also done a lot of work to enable middle-ground environments, so my users can run Python scripts like they're used to (`python script.py`) while inside of a Bazel environment, which makes it easy for them to develop quickly and then create a BUILD file when they're ready. The major downside I've experienced is that any time you're trying to do something in a less-than-Bazel way (for example relying on binaries built outside of Bazel) things can get really hairy. My containers often need various things from apt repositories, so I had to give up on rules_docker and made my own rules for Podman. I think you need someone who understands aspects and rules before adopting it, or else the sharp edges of Bazel will keep cutting you until you drop it.
- jfim 4y ago> Recently I put together a simple build rule that strips external dependencies, archives the rest, and uploads it to a cache. Then we can easily run that archive against a pre-built container (which contains the external dependencies) in our cluster, enabling a very fast ML iteration loop on beefy cluster machines. Can you tell us more about this? We're using bazel at $day_job and it's about as pleasant as gouging my eyes out with a rusty spoon. Building docker images using bazel takes forever.
- aschleck 4y agoSure! So the overall goal is to prebuild a runfiles tree containing all the external dependencies into a Docker container, and then when the user wants to run something we build a runfiles tree with all the non-external code. Then in the cluster we want to extract the user's runfiles tree on top of the prebuilt runfiles tree, and then execute the user's code. * I have an archive Starlark function that I use for both this and containers. It sets up a folder structure similar to <target>.runfiles with everything symlinked to the actual location, then it tars the whole thing following symlinks. It has a parameter to include files that start with external/ or not. * This archive function is used by my Bazel container rules, so I simply made a runner.py target that depends on every possible external Python dependency and made a Docker image with it. * I then made a Bazel rule that uses the archive function to archive a given executable without external/ and uploads it to a shared location. * At runtime runner.py is given the location as an argument, downloads it, extracts it, and then execv's it.
- xvilka 4y agoBringing Java into a Haskell project just for building doesn't sound perfect.
- mark_l_watson 4y agoThis looks really cool. If I remember correctly, when I was a consultant at Google for a while about ten years ago, they used Bazel. I have my own hack for long Haskell build times: I live in the mountains so I start a long running build process and then go on a walk.
- Kototama 4y ago> I have my own hack for long Haskell build times: I live in the mountains so I start a long running build process and then go on a walk. This is a great unexpected side-effect :-).
- Arcuru 4y ago> If I remember correctly, when I was a consultant at Google for a while about ten years ago, they used Bazel. Bazel is the open source version of Google's internal build tool, so you're correct. > I live in the mountains so I start a long running build process and then go on a walk. That's an awesome way to cope with long compiles! Maybe I should buy a slower computer...
- spockz 4y agoCabal and GHC have had incremental compilation since (almost?) the beginning/forever. How is special?
- the_duke 4y agoReading the article would answer your question...
- ParetoOptimal 4y agoA little better when TH is involved but incremental isn't as good as ghc/cabal yet: > When doing incremental builds, though, both stack and cabal-install can use the recompilation checker, and for changes deep in the dependency graph with little propagation, haskell_module is not able to beat them yet. For changes near the build targets, or which force more recompilation, haskell_module would be more competitive.
- mikepurvis 4y agoI've mostly associated Tweag with Nix-related content, so this is an interesting change. That said, it's interesting to me that there hasn't been a better Nix-Bazel bridge. Right now the story for packaging Bazel projects in Nix is really awful. You basically have Bazel run its "deps" phase, and all that stuff gets stored as a singular, gargantuan, fixed output derivation, and if you ever try to change the deps, you have to just know that you have to twiddle this magic hash [1] or it'll happily go on with the cached deps. But then, of course, every build is from scratch. A "real" Bazel story for Nix would integrate the two binary caches together, so that Nix would be aware of what Bazel was building, and the individual cached elements of it could go into the Nix store as separate entries— then you'd be able to actually get incremental builds. But I assume this would require Bazel's binary cache implementation to be pluggable. [1]: https://github.com/NixOS/nixpkgs/blob/21321a6381fd8d3660fe1cdc0485fbb5978112ce/pkgs/development/python-modules/tensorflow/default.nix#L362-L371= https://github.com/NixOS/nixpkgs/blob/21321a6381fd8d3660fe1c...