3 ms·

I could be wrong as I haven't dug into the waf docs too too much, but I think the major difference between waf and Buck is the ability to handle dependency mana

I could be wrong as I haven't dug into the waf docs too too much, but I think the major difference between waf and Buck is the ability to handle dependency management between various projects in a large org. The documentation and examples for waf seem to be around building one project, in one language, with an output of statistics and test results. I am sure this is a simplification for education and documentation purposes, but it does leave a vague area around "what if I have more than 1 or 2 build targets + 5 libs + 2 apps + 3 interdependent helper libraries?" Buck seems to be different in that it does everything waf does but also has clear `dep` files to map dependencies between various libraries within a large repository with many, many different languages and build environments. The key thing here being, I suspect that within Meta's giant repositories of various projects, they have a tight inter-linking between all these libraries and wanted build tooling that could not only build everything, but be able to map the dependency trees between everything as well. Pair that with a bunch of consolidated release mapping between the disparate projects and their various links and you have a reason why someone would likely choose Buck over waf purely from a requirements side. As for another reason they likely chose Buck over waf. It would appear that waf is a capable, but lesser known project in the wider dev community. I say this because when I look into waf, I mostly see it compared against CMake. Its mental state resides mostly in the minds of C++ devs. Either because of NIHS (not invented here syndrome) or fear that the project wouldn't be maintained over time, Meta may have decided to just roll their own tooling. They seem to be really big on the whole "being the SDK of the internet" as of late. I could see them not wanting to support an independent BSD licensed library they don't have complete control over. These are just my thoughts, I could be completely wrong about everything I've said, but they're my best insights into why they likely didn't consider waf for this.

bogwog 4y ago
It’s true that Waf doesn’t come with dependency management out of the box (EDIT: unless you count pkg-config), so maybe that’s why (besides NIHS). The way I handle it is with another excellent project called Conan (https://conan.io/ https://conan.io/) However, if you’re going to build a custom package management system anyways, there’s no reason you couldn’t build it on top of waf. Again, the core is tiny enough that one engineer could realistically hold the entire thing in their head. But I don’t think we’re going to get it right speculating here lol. I’m sure there was more to it than NIHS, or being unaware of waf.
- joshuamorton 4y ago
  A number of things like being written in python start to matter at big scale. I love python, but cli startup time in python is actually a concern for apps used many times daily by many engineers. Fixing that or moving to a daemon or whatever starts to take more time than just redoing it from scratch, and if the whole thing is 10k lines of python, it's something a domain expert can mostly reimplement in a week to better serve the fb specific needs.
  - bogwog 4y ago
    I've been using Waf for a couple of years, including on retro thinkpads from ~08. I've never run into issues with the startup time for waf and/or Python. Even if the interpreter were 100x slower to start and execute than it currently is, that time would be negligible next to the time spent waiting for a compiler or other build task to complete. And if it is too slow, there's profiling support for tracking down bottlenecks, and many different ways to optimize them. This includes simply optimizing your own code, or changing waf internal behavior to optimize specific scenarios. There's even a tool called "fast_partial" which implements a lot more caching than usual project-wide to reduce time spent executing Python during partial rebuilds in projects with an obscene number of tasks. > Fixing that or moving to a daemon or whatever starts to take more time than just redoing it from scratch, and if the whole thing is 10k lines of python, it's something a domain expert can mostly reimplement in a week to better serve the fb specific needs. Well, considering Buck just went through a from-scratch rewrite, I would argue otherwise. Although, to be fair, that 10k count is just for the core waflib. There are extra modules to support compiling C/C++/Java/etc for real projects. (also, waf does have a daemon tool, but it has external dependencies so it's not included by default)
    - joshuamorton 4y ago
      > Well, considering Buck just went through a from-scratch rewrite, I would argue otherwise Based on what, the idea that waf fits their needs better than the tool they wrote and somehow wouldn't need to be rewritten or abandoned? > Even if the interpreter were 100x slower to start and execute than it currently is, that time would be negligible next to the time spent waiting for a compiler or other build task to complete. This wrongly assumes that clean builds are the only use case. Keep in mind that in many cases when using buck or bazel, a successful build can complete without actually compiling anything, because all of the artifacts are cached externally. > There's even a tool called "fast_partial" which implements a lot more caching than usual project-wide to reduce time spent executing Python during partial rebuilds in projects with an obscene number of tasks Right, the point that this is a concern to some people, and that there's clearly some tradeoff here such that it isn't the default immediately rings alarm bells.

bogwog 4y ago

It’s true that Waf doesn’t come with dependency management out of the box (EDIT: unless you count pkg-config), so maybe that’s why (besides NIHS). The way I handle it is with another excellent project called Conan (https://conan.io/ https://conan.io/) However, if you’re going to build a custom package management system anyways, there’s no reason you couldn’t build it on top of waf. Again, the core is tiny enough that one engineer could realistically hold the entire thing in their head. But I don’t think we’re going to get it right speculating here lol. I’m sure there was more to it than NIHS, or being unaware of waf.

joshuamorton 4y ago
A number of things like being written in python start to matter at big scale. I love python, but cli startup time in python is actually a concern for apps used many times daily by many engineers. Fixing that or moving to a daemon or whatever starts to take more time than just redoing it from scratch, and if the whole thing is 10k lines of python, it's something a domain expert can mostly reimplement in a week to better serve the fb specific needs.
- bogwog 4y ago
  I've been using Waf for a couple of years, including on retro thinkpads from ~08. I've never run into issues with the startup time for waf and/or Python. Even if the interpreter were 100x slower to start and execute than it currently is, that time would be negligible next to the time spent waiting for a compiler or other build task to complete. And if it is too slow, there's profiling support for tracking down bottlenecks, and many different ways to optimize them. This includes simply optimizing your own code, or changing waf internal behavior to optimize specific scenarios. There's even a tool called "fast_partial" which implements a lot more caching than usual project-wide to reduce time spent executing Python during partial rebuilds in projects with an obscene number of tasks. > Fixing that or moving to a daemon or whatever starts to take more time than just redoing it from scratch, and if the whole thing is 10k lines of python, it's something a domain expert can mostly reimplement in a week to better serve the fb specific needs. Well, considering Buck just went through a from-scratch rewrite, I would argue otherwise. Although, to be fair, that 10k count is just for the core waflib. There are extra modules to support compiling C/C++/Java/etc for real projects. (also, waf does have a daemon tool, but it has external dependencies so it's not included by default)
  - joshuamorton 4y ago
    > Well, considering Buck just went through a from-scratch rewrite, I would argue otherwise Based on what, the idea that waf fits their needs better than the tool they wrote and somehow wouldn't need to be rewritten or abandoned? > Even if the interpreter were 100x slower to start and execute than it currently is, that time would be negligible next to the time spent waiting for a compiler or other build task to complete. This wrongly assumes that clean builds are the only use case. Keep in mind that in many cases when using buck or bazel, a successful build can complete without actually compiling anything, because all of the artifacts are cached externally. > There's even a tool called "fast_partial" which implements a lot more caching than usual project-wide to reduce time spent executing Python during partial rebuilds in projects with an obscene number of tasks Right, the point that this is a concern to some people, and that there's clearly some tradeoff here such that it isn't the default immediately rings alarm bells.