4 ms·

>The identically same compiled version may be required by multiple consumers, even if they are distributed from each other. Forcing both consumers to rebuild so

>The identically same compiled version may be required by multiple consumers, even if they are distributed from each other. Forcing both consumers to rebuild some dependencies in such a case (as opposed to relying on pre-built artifacts from an artifact repository) is often an unacceptable performance overhead, and very wasteful even in cases when someone can afford to sit around and wait. Bazel handles all of this caching internally (yes even in the distributed case), so you don't need a secondary artifact repository. The build system will make sure you don't rebuild things unless they've changed. This is like the first selling point of bazel[0]. From bazel.build: > With advanced local and distributed caching, optimized dependency analysis and parallel execution, you get fast and incremental builds. Literally the first callout. >The conditions necessary to make the decision of which build variant to choose can be arbitrarily complicated If you cannot express the conditions of your builds statically, something in your CI process is horribly broken. Your build system should not be trying to guess whether its invoked on CI or locally or whatnot. I should get the same artifact, no matter what, no matter where I build it. The configuration of your build artifacts should be explicit. > Much better to ensure the container that is built passes all required portability tests as a publication criteria on a container repo, so that the different consumers on different platforms can be sure if it is part of their build workflow, they are getting the correct artifact and it cannot be subject to idiosyncrasies of the platform or ambient environment at local build time. This has nothing to do with the build system though. Yes, running tests under all the versions is a good idea. You can do that. You can set up a :mac and :ubuntu environment and make sure tests pass on both (by running your CI on both mac and ubuntu). >The necessary build target for a given select condition may change over time, while still requiring backward compatibility with a former build of that same target from those same build conditions, effectively making a timestamp into an implicit version identifier for a backwards compatible version, meaning that re-running an identical build command with identical parameters at a later time will produce a build that succeeds for some consumers (who don’t need the backward compatibility or may even require a fundamental change that renders backward compatibility impossible) while failing for other consumers who require backward compatibility. This doesn't make sense. Like, I can't even comprehend what you're doing without an example. Timestamp should never be relevant, revision will be. Again, you're only ever building from HEAD and a cache based on artifact hashes. Bazel doesn't use timestamps anywhere. Timestamp based builds are an antipattern. Make is bad. You shouldn't use it. You should use anything else. So in bazel land, when you check out a previous revision, you'll build a working version. When you check out the current version, you'll build a working version. Those builds will both be incremental, only rebuilding the modified dependencies, but will be byte for byte equivalent to a clean build made at any time. >It’s also telling to me that you focus on what you perceive to be “not hard” in your preferred tooling and have said this in several comments as if you believe it means something. I believe you are missing the point. Pretty much every complaint comes from you misusing bazel. That doesn't make bazel a bad tool. It means you don't appear to understand how it works or best practices. You misapplying a tool doesn't make the tool bad, it makes you a bad user. And you're going to read that as me saying bazel doesn't work for the things you're doing. Which is incorrect. I'm almost positive it does. You just are making (wrongheaded) assumptions about how dependency management is supposed to work and trying to make bazel fit your worldview instead of the opposite. Again, user error, not tooling issues. Stop trying to shoehorn an additional artifact repository in around bazel. Stop trying to hack all these make-style isms into a bazel workflow. Use the tool as it is meant to be used and perhaps your complaints about it being problematic will go away. As I'm reading your comments, all I see is you repeatedly trying to misuse the tool and then complain that its bad when your misuse turns out to be annoying. [0]: https://docs.bazel.build/versions/master/remote-caching.html https://docs.bazel.build/versions/master/remote-caching.html

mlthoughts2018 8y ago
You _really_ are missing the point man. I’m not talking about basic caching local to the build environment nor am I talking about caching of code or intermediate artifacts themselves to e.g. reduce compile times. For example, I worked previously a system that automatically trained several dozen different logistic regression models every day, each one tailored to a specific customer or customer vertical. For ourbuild system we had to check s3 to see if the most updated trained model checkpoint had completed yet, and download it locally to use as part of several builds if so. If not, the fallback logic was to check if the newest vertical-specific model file was completed for the day, and use that, and if not, the fall back to a previous day’s model. Based on which of these checkpoints was selected, several different follow-up selections would be made in terms of environment variables and build flags, and the result would be cached locally for that developer so that the remote check of s3 model builds only happened at most once per hour (configurable by the dev as well). It is very frustrating speaking with you because you seem to believe that a very shallow way of thinking about literally just the build tool (e.g. pants or bazel style caching) is somehow a concise and objectively scoped response to what I’m saying, but you just keep digging yourself into deeper graves by demonstrating how dogmatically and narrow-mindedly you think about these problems. Your responses do not come across as rooted in objective analysis of build capabilities, but it seems like you think you’re presenting yourself that way, when really you are being extremely subjective and lacking imagination about all the things people may need from build systems. > “This doesn't make sense. Like, I can't even comprehend what you're doing without an example.“ It is clear you are not trying to understand. I am NOT saying someone should ever use a timestamp as an actual part of versioning. I am saying that the select feature in bazel can de facto make that happen, and it has nothing at all to do with misusing bazel (which looks a lot like a No True Scotsman fallacy that you’re now throwing in the mix). If I have some bazel select criteria C that resolves to a build action A, and then later in time I need for the exact same criteria C to instead build A2 (an upgraded version of A that is not backward compatible), this is a problem. Sure, I can muck around with revision history of the BUILD file, but what if I need A from the old revision but I need Foo and Bar from the newest revision? Nobody is saying you _cannot_ find a way to hack through revision history and make bazel do it... Whether bazel can do it is not the point. The point is that one super easy and effective way is to have a permanent versioned artifact of A that never needs to be rebuilt, and express the dependency on the artifact. Nothing about my points indicates any misuse of bazel. There are just use cases that bazel is bad at. The fact that you cannot admit that bazel might not solve every problem and that if bazel presents an annoyance it must be down to someone misusing it is seriously alarming. You have got to be more self-aware of how dogmatic you are being about this. It’s just sincerely not reasonable, yet this is the whole way Google operates regarding monorepo, bazel, and a bunch of other stuff.

mlthoughts2018 8y ago

You _really_ are missing the point man. I’m not talking about basic caching local to the build environment nor am I talking about caching of code or intermediate artifacts themselves to e.g. reduce compile times. For example, I worked previously a system that automatically trained several dozen different logistic regression models every day, each one tailored to a specific customer or customer vertical. For ourbuild system we had to check s3 to see if the most updated trained model checkpoint had completed yet, and download it locally to use as part of several builds if so. If not, the fallback logic was to check if the newest vertical-specific model file was completed for the day, and use that, and if not, the fall back to a previous day’s model. Based on which of these checkpoints was selected, several different follow-up selections would be made in terms of environment variables and build flags, and the result would be cached locally for that developer so that the remote check of s3 model builds only happened at most once per hour (configurable by the dev as well). It is very frustrating speaking with you because you seem to believe that a very shallow way of thinking about literally just the build tool (e.g. pants or bazel style caching) is somehow a concise and objectively scoped response to what I’m saying, but you just keep digging yourself into deeper graves by demonstrating how dogmatically and narrow-mindedly you think about these problems. Your responses do not come across as rooted in objective analysis of build capabilities, but it seems like you think you’re presenting yourself that way, when really you are being extremely subjective and lacking imagination about all the things people may need from build systems. > “This doesn't make sense. Like, I can't even comprehend what you're doing without an example.“ It is clear you are not trying to understand. I am NOT saying someone should ever use a timestamp as an actual part of versioning. I am saying that the select feature in bazel can de facto make that happen, and it has nothing at all to do with misusing bazel (which looks a lot like a No True Scotsman fallacy that you’re now throwing in the mix). If I have some bazel select criteria C that resolves to a build action A, and then later in time I need for the exact same criteria C to instead build A2 (an upgraded version of A that is not backward compatible), this is a problem. Sure, I can muck around with revision history of the BUILD file, but what if I need A from the old revision but I need Foo and Bar from the newest revision? Nobody is saying you _cannot_ find a way to hack through revision history and make bazel do it... Whether bazel can do it is not the point. The point is that one super easy and effective way is to have a permanent versioned artifact of A that never needs to be rebuilt, and express the dependency on the artifact. Nothing about my points indicates any misuse of bazel. There are just use cases that bazel is bad at. The fact that you cannot admit that bazel might not solve every problem and that if bazel presents an annoyance it must be down to someone misusing it is seriously alarming. You have got to be more self-aware of how dogmatic you are being about this. It’s just sincerely not reasonable, yet this is the whole way Google operates regarding monorepo, bazel, and a bunch of other stuff.