5 ms·

I used shake for some simple build scripts and to work on a predecessor of the Haskell Language Server. Due to it's lack of hermetic builds I don't think it's r

I used shake for some simple build scripts and to work on a predecessor of the Haskell Language Server. Due to it's lack of hermetic builds I don't think it's really appropriate for any project you're going to share with more than one person. That being said I think it's an absolutely amazing Haskell library for building complex systems with scheduling jobs and caching of intermediate results. Once you use it you get the sense that every large codebase has an ad hoc poorly specified build system nestled inside of it. I described the experience of using it briefly here: https://4ta.uk/p/shaking-up-the-ide https://4ta.uk/p/shaking-up-the-ide

ur-whale 5y ago
> Due to it's lack of hermetic builds Care to explain what that means or post a ref?
- gravypod 5y ago
  For a build system to be hermetic compilations (or other things) should only see declared dependencies. The least hermetic build system is Make. If you have a target that depends on `a.h` nothing is stoping you from reading `b.h`. More hermetic build systems, like Bazel, will make it so `b.h` isn't visible at all during the build process of it isn't declared as an input. This has nice properties that allow you to be sure your build rules are correct, caching your outputs based on inputs, and because you can cache things you can now remotely execute things safer.
  - ur-whale 5y ago
    Understood, and in other words: hermetic == sandbox which only contains declared deps I assume bazel implements this via containers? Does that include compilers and OS? Also, IMO, this only provides a false sense of safety unless you archive absolutely everything that's inside the sandbox.
    - gravypod 5y ago
      > hermetic == sandbox which only contains declared deps Yes, exactly. It also can include other deps like networking. A perfect hermetic build will (for most of the build process) not have access to the network. > I assume bazel implements this via containers? Sort of. Bazel's code predates "containers" by ~6 years. Bazel is the build system used at Google (sprinkle on a large amount of asterisks here). > Also, IMO, this only provides a false sense of safety unless you archive absolutely everything that's inside the sandbox. Yes, that is what a Bazel monorepo does. You can also include your language's runtime and standard library as part of the build process. See Dropbox's repo for more details on how they do this with Python: https://github.com/dropbox/dbx_build_tools https://github.com/dropbox/dbx_build_tools Some more details about what Google does are available here: https://opensource.google/docs/thirdparty/# https://opensource.google/docs/thirdparty/# Essentially, everything is built from source (more asterisks here). Since everything is built from source using Bazel you can guarantee that at a certain view of the source repository you can build a reproducible copy of all software in the repo (more asterisks). One other nice property is you can guarantee that all software in the repo is using a compatible version of each dep: https://opensource.google/docs/thirdparty/oneversion/ https://opensource.google/docs/thirdparty/oneversion/ The downside is that if you want to upgrade a library that has a breaking change this requires you update a lot of code. The upside is that doing that isn't actually that hard with the tooling available and you can guarantee that you can link together any two libraries in the repo (more asterisks) and have the build succeed without any dep version conflicts. Since you can do this some langauges with runtimes (like JVM language) can see massive speed boosts because they only need to compile changed libraries and can run a `java -cp ... your.jar` to load up deps. You can check out rules_docker for some examples of how this can be used.
      - nikita2206 5y ago
        This is so great, I wish Bazel had a section in its README with this info. Thank you
        gravypod 5y ago
        If you have further questions ask them and I can see about writing a full blog post which touches on everything I know.

ur-whale 5y ago

> Due to it's lack of hermetic builds Care to explain what that means or post a ref?

gravypod 5y ago
For a build system to be hermetic compilations (or other things) should only see declared dependencies. The least hermetic build system is Make. If you have a target that depends on `a.h` nothing is stoping you from reading `b.h`. More hermetic build systems, like Bazel, will make it so `b.h` isn't visible at all during the build process of it isn't declared as an input. This has nice properties that allow you to be sure your build rules are correct, caching your outputs based on inputs, and because you can cache things you can now remotely execute things safer.
- ur-whale 5y ago
  Understood, and in other words: hermetic == sandbox which only contains declared deps I assume bazel implements this via containers? Does that include compilers and OS? Also, IMO, this only provides a false sense of safety unless you archive absolutely everything that's inside the sandbox.
  - gravypod 5y ago
    > hermetic == sandbox which only contains declared deps Yes, exactly. It also can include other deps like networking. A perfect hermetic build will (for most of the build process) not have access to the network. > I assume bazel implements this via containers? Sort of. Bazel's code predates "containers" by ~6 years. Bazel is the build system used at Google (sprinkle on a large amount of asterisks here). > Also, IMO, this only provides a false sense of safety unless you archive absolutely everything that's inside the sandbox. Yes, that is what a Bazel monorepo does. You can also include your language's runtime and standard library as part of the build process. See Dropbox's repo for more details on how they do this with Python: https://github.com/dropbox/dbx_build_tools https://github.com/dropbox/dbx_build_tools Some more details about what Google does are available here: https://opensource.google/docs/thirdparty/# https://opensource.google/docs/thirdparty/# Essentially, everything is built from source (more asterisks here). Since everything is built from source using Bazel you can guarantee that at a certain view of the source repository you can build a reproducible copy of all software in the repo (more asterisks). One other nice property is you can guarantee that all software in the repo is using a compatible version of each dep: https://opensource.google/docs/thirdparty/oneversion/ https://opensource.google/docs/thirdparty/oneversion/ The downside is that if you want to upgrade a library that has a breaking change this requires you update a lot of code. The upside is that doing that isn't actually that hard with the tooling available and you can guarantee that you can link together any two libraries in the repo (more asterisks) and have the build succeed without any dep version conflicts. Since you can do this some langauges with runtimes (like JVM language) can see massive speed boosts because they only need to compile changed libraries and can run a `java -cp ... your.jar` to load up deps. You can check out rules_docker for some examples of how this can be used.
    - nikita2206 5y ago
      This is so great, I wish Bazel had a section in its README with this info. Thank you
      - gravypod 5y ago
        If you have further questions ask them and I can see about writing a full blog post which touches on everything I know.