3 ms·

A number of things like being written in python start to matter at big scale. I love python, but cli startup time in python is actually a concern for apps used

A number of things like being written in python start to matter at big scale. I love python, but cli startup time in python is actually a concern for apps used many times daily by many engineers. Fixing that or moving to a daemon or whatever starts to take more time than just redoing it from scratch, and if the whole thing is 10k lines of python, it's something a domain expert can mostly reimplement in a week to better serve the fb specific needs.

bogwog 4y ago
I've been using Waf for a couple of years, including on retro thinkpads from ~08. I've never run into issues with the startup time for waf and/or Python. Even if the interpreter were 100x slower to start and execute than it currently is, that time would be negligible next to the time spent waiting for a compiler or other build task to complete. And if it is too slow, there's profiling support for tracking down bottlenecks, and many different ways to optimize them. This includes simply optimizing your own code, or changing waf internal behavior to optimize specific scenarios. There's even a tool called "fast_partial" which implements a lot more caching than usual project-wide to reduce time spent executing Python during partial rebuilds in projects with an obscene number of tasks. > Fixing that or moving to a daemon or whatever starts to take more time than just redoing it from scratch, and if the whole thing is 10k lines of python, it's something a domain expert can mostly reimplement in a week to better serve the fb specific needs. Well, considering Buck just went through a from-scratch rewrite, I would argue otherwise. Although, to be fair, that 10k count is just for the core waflib. There are extra modules to support compiling C/C++/Java/etc for real projects. (also, waf does have a daemon tool, but it has external dependencies so it's not included by default)
- joshuamorton 4y ago
  > Well, considering Buck just went through a from-scratch rewrite, I would argue otherwise Based on what, the idea that waf fits their needs better than the tool they wrote and somehow wouldn't need to be rewritten or abandoned? > Even if the interpreter were 100x slower to start and execute than it currently is, that time would be negligible next to the time spent waiting for a compiler or other build task to complete. This wrongly assumes that clean builds are the only use case. Keep in mind that in many cases when using buck or bazel, a successful build can complete without actually compiling anything, because all of the artifacts are cached externally. > There's even a tool called "fast_partial" which implements a lot more caching than usual project-wide to reduce time spent executing Python during partial rebuilds in projects with an obscene number of tasks Right, the point that this is a concern to some people, and that there's clearly some tradeoff here such that it isn't the default immediately rings alarm bells.
  - bogwog 4y ago
    No offense, but I think you're reading too much into my casual comments here to guide your understanding of waf, rather than the actual waf docs. Waf isn't optimized for clean builds (quite the contrary), and neither you nor I know whether the waf defaults are insufficient for whatever Buck is being used for. I just pointed out the existence of that "fast_partial" thing to show how deep into waf internals a project-specific optimization effort could go. But discussions about optimization are pointless without real world measurements and data.
    - lmm 4y ago
      The fact that it's implemented and not on by default is a red flag any way you slice it. Either it's implemented but unreliable, or it's reliable but the maintainers don't think it's worth turning on for some reason (why?).
  - dikei 4y ago
    Exactly, one of the key selling point Bazel/Buck is their caching systems: very high cache hit rate, with no inconsistency, which allows very fast incremental build: 0-change build takes close to 0 seconds.
cozzyd 4y ago
Just imagine how much memory a large dependency graph would take in Python... Especially considering how poor Python's support for shared memory concurrency is...

bogwog 4y ago

I've been using Waf for a couple of years, including on retro thinkpads from ~08. I've never run into issues with the startup time for waf and/or Python. Even if the interpreter were 100x slower to start and execute than it currently is, that time would be negligible next to the time spent waiting for a compiler or other build task to complete. And if it is too slow, there's profiling support for tracking down bottlenecks, and many different ways to optimize them. This includes simply optimizing your own code, or changing waf internal behavior to optimize specific scenarios. There's even a tool called "fast_partial" which implements a lot more caching than usual project-wide to reduce time spent executing Python during partial rebuilds in projects with an obscene number of tasks. > Fixing that or moving to a daemon or whatever starts to take more time than just redoing it from scratch, and if the whole thing is 10k lines of python, it's something a domain expert can mostly reimplement in a week to better serve the fb specific needs. Well, considering Buck just went through a from-scratch rewrite, I would argue otherwise. Although, to be fair, that 10k count is just for the core waflib. There are extra modules to support compiling C/C++/Java/etc for real projects. (also, waf does have a daemon tool, but it has external dependencies so it's not included by default)

joshuamorton 4y ago
> Well, considering Buck just went through a from-scratch rewrite, I would argue otherwise Based on what, the idea that waf fits their needs better than the tool they wrote and somehow wouldn't need to be rewritten or abandoned? > Even if the interpreter were 100x slower to start and execute than it currently is, that time would be negligible next to the time spent waiting for a compiler or other build task to complete. This wrongly assumes that clean builds are the only use case. Keep in mind that in many cases when using buck or bazel, a successful build can complete without actually compiling anything, because all of the artifacts are cached externally. > There's even a tool called "fast_partial" which implements a lot more caching than usual project-wide to reduce time spent executing Python during partial rebuilds in projects with an obscene number of tasks Right, the point that this is a concern to some people, and that there's clearly some tradeoff here such that it isn't the default immediately rings alarm bells.
- bogwog 4y ago
  No offense, but I think you're reading too much into my casual comments here to guide your understanding of waf, rather than the actual waf docs. Waf isn't optimized for clean builds (quite the contrary), and neither you nor I know whether the waf defaults are insufficient for whatever Buck is being used for. I just pointed out the existence of that "fast_partial" thing to show how deep into waf internals a project-specific optimization effort could go. But discussions about optimization are pointless without real world measurements and data.
  - lmm 4y ago
    The fact that it's implemented and not on by default is a red flag any way you slice it. Either it's implemented but unreliable, or it's reliable but the maintainers don't think it's worth turning on for some reason (why?).
- dikei 4y ago
  Exactly, one of the key selling point Bazel/Buck is their caching systems: very high cache hit rate, with no inconsistency, which allows very fast incremental build: 0-change build takes close to 0 seconds.

cozzyd 4y ago

Just imagine how much memory a large dependency graph would take in Python... Especially considering how poor Python's support for shared memory concurrency is...