5 ms·
You _really_ are missing the point man. I’m not talking about basic caching local to the build environment nor am I talking about caching of code or intermediat
by mlthoughts2018 8y ago
You _really_ are missing the point man. I’m not talking about basic caching local to the build environment nor am I talking about caching of code or intermediate artifacts themselves to e.g. reduce compile times.
For example, I worked previously a system that automatically trained several dozen different logistic regression models every day, each one tailored to a specific customer or customer vertical.
For ourbuild system we had to check s3 to see if the most updated trained model checkpoint had completed yet, and download it locally to use as part of several builds if so. If not, the fallback logic was to check if the newest vertical-specific model file was completed for the day, and use that, and if not, the fall back to a previous day’s model.
Based on which of these checkpoints was selected, several different follow-up selections would be made in terms of environment variables and build flags, and the result would be cached locally for that developer so that the remote check of s3 model builds only happened at most once per hour (configurable by the dev as well).
It is very frustrating speaking with you because you seem to believe that a very shallow way of thinking about literally just the build tool (e.g. pants or bazel style caching) is somehow a concise and objectively scoped response to what I’m saying, but you just keep digging yourself into deeper graves by demonstrating how dogmatically and narrow-mindedly you think about these problems.
Your responses do not come across as rooted in objective analysis of build capabilities, but it seems like you think you’re presenting yourself that way, when really you are being extremely subjective and lacking imagination about all the things people may need from build systems.
> “This doesn't make sense. Like, I can't even comprehend what you're doing without an example.“
It is clear you are not trying to understand. I am NOT saying someone should ever use a timestamp as an actual part of versioning.
I am saying that the select feature in bazel can de facto make that happen, and it has nothing at all to do with misusing bazel (which looks a lot like a No True Scotsman fallacy that you’re now throwing in the mix).
If I have some bazel select criteria C that resolves to a build action A, and then later in time I need for the exact same criteria C to instead build A2 (an upgraded version of A that is not backward compatible), this is a problem.
Sure, I can muck around with revision history of the BUILD file, but what if I need A from the old revision but I need Foo and Bar from the newest revision?
Nobody is saying you _cannot_ find a way to hack through revision history and make bazel do it... Whether bazel can do it is not the point.
The point is that one super easy and effective way is to have a permanent versioned artifact of A that never needs to be rebuilt, and express the dependency on the artifact.
Nothing about my points indicates any misuse of bazel. There are just use cases that bazel is bad at.
The fact that you cannot admit that bazel might not solve every problem and that if bazel presents an annoyance it must be down to someone misusing it is seriously alarming. You have got to be more self-aware of how dogmatic you are being about this. It’s just sincerely not reasonable, yet this is the whole way Google operates regarding monorepo, bazel, and a bunch of other stuff.
- joshuamorton 8y ago> For ourbuild system we had to check s3 This isn't a build system. What you're doing isn't building. Trying to use a build system to do what you're doing is a bad idea. Using make to do what you're doing is a bad idea too, but it works because its just a shell script alias thing and not a build system. If its not something you can statically generate based on your code, and only your checked in code, at a certain revision, its not something you're building. >If I have some bazel select criteria C that resolves to a build action A, and then later in time I need for the exact same criteria C to instead build A2 (an upgraded version of A that is not backward compatible), this is a problem. You update the select rule. The select rule is version controlled. There's no timestamping anywhere, only what is at HEAD at revision X vs. revision Y. If you check out the older revision, you'll use the old criteria. The new one will use the new criteria, no matter the time. >I am NOT saying someone should ever use a timestamp as an actual part of versioning. Yet you're advocating make, which does exactly that ;) >but what if I need A from the old revision but I need Foo and Bar from the newest revision? You don't. You should never need to mix different revisions because everything is always buildable at head. If you mean you have some data file that's incompatible with your current schema or something then that's a different question, but again has 0 to do with your build system. To make it crystalline: versioning your data is not the same thing as versioning your builds. Bazel solves the first problem, because bazel is a build system. You seem to be trying to use bazel for something its bad at and then complaining that its a terrible tool because you're trying to version cron-generated, non-hermetic data files as part of your build process, which like yes bazel is bad at because that's not a build artifact, its an external non-hermetic data artifact. Pass your non-hermetic model data into your hermetic production binary via a flag. Separate your concerns. Problem solved. >The fact that you cannot admit that bazel might not solve every problem and that if bazel presents an annoyance it must be down to someone misusing it is seriously alarming. Except that I was right. Bazel is indeed not good for what you're doing, but that's because you're not managing builds, but ML model data. You're attempting to use a build system for managing not-builds. You're experiencing pain. This is not a surprise. Its a classic example of the XY problem. You: "Bazel is a terrible build system, it causes pain and shouldn't be used" Us: Some discussion Conclusion: "What you're doing isn't managing build artifacts, so of course a tool designed to do that isn't solving your problems. Dogmatism helps here. Use a build system to manage your builds and a different tool to manage your ML models.