4 ms·

I get that, but again, there's no reason Waf can't be used as a base for building that. I actually use Waf for cross compilation extensively, and have built som

by bogwog 4y ago

I get that, but again, there's no reason Waf can't be used as a base for building that. I actually use Waf for cross compilation extensively, and have built some tools around it with Conan for my own projects. Waf can handle cross compilation just fine, but it's up to you to build what that looks like for your project (a common pattern I see is custom Context subclasses for each target) Memory management, broken build scripts, etc. can all be handled with Waf as well. In the simplest case, you can just wrap a `recurse` call in a try catch block, or you can build something much more sophisticated around how your projects are structured. Note, I'm not trying to argue that Google/Facebook "should have used X". There are a million reasons to pick X over Y, even if Y is the objectively better choice. Sometimes, molding X to be good enough is more efficient than spending months just researching options hoping you'll find Y. I'm just curious to know if they did evaluate Waf, why did they decide against it.

davnn 4y ago
> the core is <10k lines of Python with zero dependencies Isn‘t that already a no-go, to write a performance critical system in a slow programming language?
- taeric 4y ago
  I am no python fan, but find it laughably hard to believe it could be what makes a build coordination system slow.
  - Too 4y ago
    On clean builds the python tax will be dwarfed be the thousands of calls to clang yes. That’s not the scenario you need to optimize for. What’s more important is that incremental builds are snappy, since that is what developers do 100 times per day. I’ve seen some projects with 100MB+ ninja-files that even ninja itself, proud for being written in optimized c++, takes a second or two to parse on each build invocation. Convert that to python and you likely land in 5-20 sec range instead. Enough to alt-tab and get distracted by something else. Google code base is likely even larger than this. A background daemon that holds the graph in memory would probably handle it. In the big scheme such a design is likely better anyway. But needs a big upfront design and is a lot more complex than just reparsing a file each time. Side note: For some, even the interpreter startup is annoying. Personally I find it negligible, especially after 3.11 you can almost claim it’s snappy.
    - taeric 4y ago
      Code bases that big are strawmen for most companies. Yes, they happen; but as often they should be segmented into smaller things. That don't require monolithic build setups.
      - joshuamorton 4y ago
        The context for this thread was weather Facebook considered waf in participation, so it is very relevant.
        taeric 4y ago
        Certainly fair. I had meandered on to "in general" way too quickly.
klodolph 4y ago
I don’t see how using Waf as a base would help in any way. It seems like a massive mismatch for the problems that Facebook and Google are solving. You seem to be fond of Waf, maybe if you elaborated why you think that Waf would be a good base for compiling massive, multi-language code-bases, I could understand where you are coming from. Where I am coming from—it feels like Waf is kind of a better version of autotools, or something like that, and it’s just not in the same league. It’s like comparing a bicycle to a cargo ship. Like, “Why didn’t the people designing the cargo ship use the bicycle as a starting point?” I don’t want to abuse analogies here, but that’s what the question sounds like to me. This is based on my relatively limited experience using Waf (and SCons, which I know is different), and my experience using Bazel and Buck. Having spent a lot of time with Buck and Bazel, there are just so many little things you run into where you go, “Oh, that explains why Buck or Bazel is designed that way.” These design decisions permeate Buck and Bazel (Pants, Please, etc.) I just don’t see how Waf can be used as a base. I really do see this as a new “generation” of build systems, with Buck, Bazel, Please, and Pants, and everything else seems so much more primitive by comparison.
- bogwog 4y ago
  I’m coming from the perspective of someone who has been working with it for a while, and coincidentally very intensely hacking away at it recently. The thing about waf is that it’s more designed like a framework than a typical build tool. If you look at the code, it’s split into a core library (thats the <10k loc I estimated), and additional tools that do things like add C++ or Java build support. That’s one of the reasons I like Waf, since it becomes a powerful toolkit for creating a custom build system once you strip away the thin outer layer. There is no one-size-fits-all build system, so a tool that can be molded like waf is very powerful imo. I guess it’s hard to get that point across without experiencing it. There are just so many good design decisions everywhere. For example, extensibility comes easily because task generator methods are “flat”, and ordering is implemented via constraints. This means you can easily slip your own functions between any built in generator method to manipulate their inputs or outputs. It’s like a sub-build system just for creating Task objects. Also, I don’t want to give the impression that I think waf would have been a better choice for these companies. I’ve kind of been defending it a lot in this thread, but my original point/question was just to know if they evaluated waf/what they thought about it. After so many comments I feel like I might be coming off as hostile… which isn’t my intention.

by bogwog 4y ago

davnn 4y ago

> the core is <10k lines of Python with zero dependencies Isn‘t that already a no-go, to write a performance critical system in a slow programming language?

taeric 4y ago
I am no python fan, but find it laughably hard to believe it could be what makes a build coordination system slow.
- Too 4y ago
  On clean builds the python tax will be dwarfed be the thousands of calls to clang yes. That’s not the scenario you need to optimize for. What’s more important is that incremental builds are snappy, since that is what developers do 100 times per day. I’ve seen some projects with 100MB+ ninja-files that even ninja itself, proud for being written in optimized c++, takes a second or two to parse on each build invocation. Convert that to python and you likely land in 5-20 sec range instead. Enough to alt-tab and get distracted by something else. Google code base is likely even larger than this. A background daemon that holds the graph in memory would probably handle it. In the big scheme such a design is likely better anyway. But needs a big upfront design and is a lot more complex than just reparsing a file each time. Side note: For some, even the interpreter startup is annoying. Personally I find it negligible, especially after 3.11 you can almost claim it’s snappy.
  - taeric 4y ago
    Code bases that big are strawmen for most companies. Yes, they happen; but as often they should be segmented into smaller things. That don't require monolithic build setups.
    - joshuamorton 4y ago
      The context for this thread was weather Facebook considered waf in participation, so it is very relevant.
      - taeric 4y ago
        Certainly fair. I had meandered on to "in general" way too quickly.

klodolph 4y ago

I don’t see how using Waf as a base would help in any way. It seems like a massive mismatch for the problems that Facebook and Google are solving. You seem to be fond of Waf, maybe if you elaborated why you think that Waf would be a good base for compiling massive, multi-language code-bases, I could understand where you are coming from. Where I am coming from—it feels like Waf is kind of a better version of autotools, or something like that, and it’s just not in the same league. It’s like comparing a bicycle to a cargo ship. Like, “Why didn’t the people designing the cargo ship use the bicycle as a starting point?” I don’t want to abuse analogies here, but that’s what the question sounds like to me. This is based on my relatively limited experience using Waf (and SCons, which I know is different), and my experience using Bazel and Buck. Having spent a lot of time with Buck and Bazel, there are just so many little things you run into where you go, “Oh, that explains why Buck or Bazel is designed that way.” These design decisions permeate Buck and Bazel (Pants, Please, etc.) I just don’t see how Waf can be used as a base. I really do see this as a new “generation” of build systems, with Buck, Bazel, Please, and Pants, and everything else seems so much more primitive by comparison.

bogwog 4y ago
I’m coming from the perspective of someone who has been working with it for a while, and coincidentally very intensely hacking away at it recently. The thing about waf is that it’s more designed like a framework than a typical build tool. If you look at the code, it’s split into a core library (thats the <10k loc I estimated), and additional tools that do things like add C++ or Java build support. That’s one of the reasons I like Waf, since it becomes a powerful toolkit for creating a custom build system once you strip away the thin outer layer. There is no one-size-fits-all build system, so a tool that can be molded like waf is very powerful imo. I guess it’s hard to get that point across without experiencing it. There are just so many good design decisions everywhere. For example, extensibility comes easily because task generator methods are “flat”, and ordering is implemented via constraints. This means you can easily slip your own functions between any built in generator method to manipulate their inputs or outputs. It’s like a sub-build system just for creating Task objects. Also, I don’t want to give the impression that I think waf would have been a better choice for these companies. I’ve kind of been defending it a lot in this thread, but my original point/question was just to know if they evaluated waf/what they thought about it. After so many comments I feel like I might be coming off as hostile… which isn’t my intention.