5 ms·

You’re making egregious assumptions that you are only using languages that compile to executable binaries and that the way the dependencies interact with ambien

by mlthoughts2018 8y ago

You’re making egregious assumptions that you are only using languages that compile to executable binaries and that the way the dependencies interact with ambient details of the runtime environment aren’t problematic. This is exactly why the bazel-style build model fails so badly. In the companies where I’ve worked with this pattern, you end up with behemoth binaries, like a giant jar for all possible things you may do that interact with HDFS, for example. Somebody comes along and has a business case where they need a few sub-slices of that jar to function inside a special environment, say like a Jupyter-Scala notebook running inside a custom container that something like Databricks can’t currently support, and suddenly you’re doing insane dependency surgery, mangling build commands to build different versions of the jar for the different use cases. So drastically much better to publish the jar / binary / package / container / whatever to an internal registry, then express the dependency on the specific artifact that’s needed. This is genuinely a line of questioning I ask in order to reject job candidates or to reject potential employers. Your suggested way is just far too dysfunctional. > “Android apps, webservers, and language runtimes live in the same repo without issue” I don’t buy it. “Without issue” is entirely subjective here, and it’s clear from comments you make about Google and your own preferences regarding monorepos that you are at best unlikely to be sympathetic to the kinds of issues people may find to be unacceptably frustrating and choose to view them as non-issues, to falsely believe they would manifest with at least as much headache in a polyrepo situation, or to believe that even if they are sincere downsides of a monorepo that still they don’t rise to a level of importance to see them as distinguishing failure cases. Companies like Google exhibit deep-seated dogmatism about such things, and given this, veteran employees in such places are almost by definition produced by a selection bias effect that reinforces the dogma: anyone with valid dissatisfaction with the monorepo steamroller most likely would have left or declined the job in the first place, just reinforcing the monoculture.

joshuamorton 8y ago
I work mainly in Python. So no I'm not at all making that assumption. If you need slices of that jar, you never depend on the jar artifact. You depend on the source files/ targets you need. It's not hard or tricky. You should only ever produce executable binaries. Library targets or jars that are there as deps should not ever be collected as artifacts. Trying to store intermediate artifacts externally is bad. Don't do it. If you try to use a system in exactly the way it says not to, you'll have problems. That's your fault though.
- mlthoughts2018 8y ago
  You may need different versions of the same source files or targets, due to incompatibilities of the underlying runtime environment. For example, custom compiled versions of TensorFlow with different sets of optimization flags enabled for different use cases, or different build variants of the same conda package for different runtime compiled versions of e.g. numpy for different services. Trying to solve this without providing versioned artifacts in an external storage repository is bad, don’t do it.
  - joshuamorton 8y ago
    You can define compiler flags in BUILD files. My_binary and my_binary_opt can live side by side. You can even select dependencies depending on the runtime[0]. Like I said, not hard. I can think of at least 3 situations where we swap flags or dependencies based on runtime that I've touched this week. [0]: https://docs.bazel.build/versions/master/be/functions.html#select https://docs.bazel.build/versions/master/be/functions.html#s...
    - mlthoughts2018 8y ago
      Managing something like this with the select functionality in bazel is a terrible way to solve it. The conditions necessary to make the decision of which build variant to choose can be arbitrarily complicated (for example, the logical test could be the execution of some arbitrary script like embodied in a .PHONY target in Make) and also rely on caching logic or details of whether the ambient environment performing the build is a CI environment or not, and cannot be limited to flag arguments. The identically same compiled version may be required by multiple consumers, even if they are distributed from each other. Forcing both consumers to rebuild some dependencies in such a case (as opposed to relying on pre-built artifacts from an artifact repository) is often an unacceptable performance overhead, and very wasteful even in cases when someone can afford to sit around and wait. It can also be affected by portability.. for example differences in Docker for Mac vs say Docker for Ubuntu when the underlying artifact is a container. Much better to ensure the container that is built passes all required portability tests as a publication criteria on a container repo, so that the different consumers on different platforms can be sure if it is part of their build workflow, they are getting the correct artifact and it cannot be subject to idiosyncrasies of the platform or ambient environment at local build time. The necessary build target for a given select condition may change over time, while still requiring backward compatibility with a former build of that same target from those same build conditions, effectively making a timestamp into an implicit version identifier for a backwards compatible version, meaning that re-running an identical build command with identical parameters at a later time will produce a build that succeeds for some consumers (who don’t need the backward compatibility or may even require a fundamental change that renders backward compatibility impossible) while failing for other consumers who require backward compatibility. I’m sure you have plenty of “not hard” rebuttals for how to mangle bazel to address these types of things, which is meaningless because solving it with versioned, permanent artifacts is simply better. It requires you to be extremely noseblind with diehard dogmatic insistence on using your “hammer for every nail” favorite to not see this. It’s also telling to me that you focus on what you perceive to be “not hard” in your preferred tooling and have said this in several comments as if you believe it means something. I believe you are missing the point.

joshuamorton 8y ago

I work mainly in Python. So no I'm not at all making that assumption. If you need slices of that jar, you never depend on the jar artifact. You depend on the source files/ targets you need. It's not hard or tricky. You should only ever produce executable binaries. Library targets or jars that are there as deps should not ever be collected as artifacts. Trying to store intermediate artifacts externally is bad. Don't do it. If you try to use a system in exactly the way it says not to, you'll have problems. That's your fault though.

mlthoughts2018 8y ago
You may need different versions of the same source files or targets, due to incompatibilities of the underlying runtime environment. For example, custom compiled versions of TensorFlow with different sets of optimization flags enabled for different use cases, or different build variants of the same conda package for different runtime compiled versions of e.g. numpy for different services. Trying to solve this without providing versioned artifacts in an external storage repository is bad, don’t do it.
- joshuamorton 8y ago
  You can define compiler flags in BUILD files. My_binary and my_binary_opt can live side by side. You can even select dependencies depending on the runtime[0]. Like I said, not hard. I can think of at least 3 situations where we swap flags or dependencies based on runtime that I've touched this week. [0]: https://docs.bazel.build/versions/master/be/functions.html#select https://docs.bazel.build/versions/master/be/functions.html#s...
  - mlthoughts2018 8y ago
    Managing something like this with the select functionality in bazel is a terrible way to solve it. The conditions necessary to make the decision of which build variant to choose can be arbitrarily complicated (for example, the logical test could be the execution of some arbitrary script like embodied in a .PHONY target in Make) and also rely on caching logic or details of whether the ambient environment performing the build is a CI environment or not, and cannot be limited to flag arguments. The identically same compiled version may be required by multiple consumers, even if they are distributed from each other. Forcing both consumers to rebuild some dependencies in such a case (as opposed to relying on pre-built artifacts from an artifact repository) is often an unacceptable performance overhead, and very wasteful even in cases when someone can afford to sit around and wait. It can also be affected by portability.. for example differences in Docker for Mac vs say Docker for Ubuntu when the underlying artifact is a container. Much better to ensure the container that is built passes all required portability tests as a publication criteria on a container repo, so that the different consumers on different platforms can be sure if it is part of their build workflow, they are getting the correct artifact and it cannot be subject to idiosyncrasies of the platform or ambient environment at local build time. The necessary build target for a given select condition may change over time, while still requiring backward compatibility with a former build of that same target from those same build conditions, effectively making a timestamp into an implicit version identifier for a backwards compatible version, meaning that re-running an identical build command with identical parameters at a later time will produce a build that succeeds for some consumers (who don’t need the backward compatibility or may even require a fundamental change that renders backward compatibility impossible) while failing for other consumers who require backward compatibility. I’m sure you have plenty of “not hard” rebuttals for how to mangle bazel to address these types of things, which is meaningless because solving it with versioned, permanent artifacts is simply better. It requires you to be extremely noseblind with diehard dogmatic insistence on using your “hammer for every nail” favorite to not see this. It’s also telling to me that you focus on what you perceive to be “not hard” in your preferred tooling and have said this in several comments as if you believe it means something. I believe you are missing the point.