5 ms·

> If there are any people in the Buck team here, I would be curious to hear if you all happened to evaluate waf before choosing to build Buck? There’s no way W

> If there are any people in the Buck team here, I would be curious to hear if you all happened to evaluate waf before choosing to build Buck? There’s no way Waf can handle code bases as large as the ones inside Facebook (Buck) or Google (Bazel). Waf also has some problems with cross-compilation, IIRC. Waf would simply choke. If you think about the problems you run into with extremely large code bases, then the design decisions behind Buck/Bazel/etc. start to make a lot of sense. Things like how targets are labeled as //package:target, rather than paths like package/target. Package build files are only loaded as needed, so your build files can be extremely broken in one part of the tree, and you can still build anything that doesn’t depend on the broken parts. In large code bases, it is simply not feasible to expect all of your build scripts to work all of the time. The Python -> Starlark change was made because the build scripts need to be completely hermetic and deterministic. Starlark is reusable outside Bazel/Buck precisely because other projects want that same hermeticity and determinism. Waf is nice but I really want to emphasize just how damn large the codebases are that Bazel and Buck handle. They are large enough that you cannot load the entire build graph into memory on a single machine—neither Facebook nor Google have the will to load that much RAM into a single server just to run builds or build queries. Some of these design decisions are basically there so that you can load subsets of the build graph and cache parts of the build graph. You want to hit cache as much as possible. I’ve used Waf and its predecessor SCons, and I’ve also used Buck and Bazel.

nextaccountic 4y ago
> They are large enough that you cannot load the entire build graph into memory on a single machine You mean, multiple gigabytes for build metadata, that just says things like that X depends on Y and to build Y you run command Z?
- klodolph 4y ago
  Yes. By “multiple gigabytes” I am talking about >100 GB. Maybe >1 TB.
  - nextaccountic 4y ago
    How is this even possible? I take that this data is highly compressible, right?
    - phyrex 4y ago
      It wouldn’t be compressed in ram though, would it?
      - nextaccountic 4y ago
        There are in-memory https://en.wikipedia.org/wiki/Succinct_data_structure https://en.wikipedia.org/wiki/Succinct_data_structure but I actually don't mean that specifically: I mean that, for example, there must be tons of strings with common prefixes, like file paths (which can be stored in a trie to have faster access and compress data in ram) or very similar strings (like compiler invocations that mostly have the same flags), and other highly redundant data that can usually be used to cut down on memory requirements. I highly doubt that, after doing all those tricks, you still end up with 100GB - 1TB of build data.
        Shish2k 4y ago
        You could do those tricks and cut down memory, perhaps even 10x, but they come at the cost of increased CPU time. Designing the system in such a way that you only ever need to load a tiny subset of the graph at one time gives you a 1000x saving for memory and CPU.
        nextaccountic 4y ago
        Some of those tricks may actually decrease CPU time (by fetching less data from RAM and using the CPU cache more effectively). And you can also apply any optimizations for partial loading on top of that. I guess the downside is that the system would be more complex overall, but you can probably get 80% of the result with not so large changes
- esprehn 4y ago
  Yes, the codebases at Google and FB contain billions of files. Article from 2016 about the scale, and of course it's only grown dramatically since then: https://m-cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://m-cacm.acm.org/magazines/2016/7/204032-why-google-st...
bogwog 4y ago
I get that, but again, there's no reason Waf can't be used as a base for building that. I actually use Waf for cross compilation extensively, and have built some tools around it with Conan for my own projects. Waf can handle cross compilation just fine, but it's up to you to build what that looks like for your project (a common pattern I see is custom Context subclasses for each target) Memory management, broken build scripts, etc. can all be handled with Waf as well. In the simplest case, you can just wrap a `recurse` call in a try catch block, or you can build something much more sophisticated around how your projects are structured. Note, I'm not trying to argue that Google/Facebook "should have used X". There are a million reasons to pick X over Y, even if Y is the objectively better choice. Sometimes, molding X to be good enough is more efficient than spending months just researching options hoping you'll find Y. I'm just curious to know if they did evaluate Waf, why did they decide against it.
- davnn 4y ago
  > the core is <10k lines of Python with zero dependencies Isn‘t that already a no-go, to write a performance critical system in a slow programming language?
  - taeric 4y ago
    I am no python fan, but find it laughably hard to believe it could be what makes a build coordination system slow.
    - Too 4y ago
      On clean builds the python tax will be dwarfed be the thousands of calls to clang yes. That’s not the scenario you need to optimize for. What’s more important is that incremental builds are snappy, since that is what developers do 100 times per day. I’ve seen some projects with 100MB+ ninja-files that even ninja itself, proud for being written in optimized c++, takes a second or two to parse on each build invocation. Convert that to python and you likely land in 5-20 sec range instead. Enough to alt-tab and get distracted by something else. Google code base is likely even larger than this. A background daemon that holds the graph in memory would probably handle it. In the big scheme such a design is likely better anyway. But needs a big upfront design and is a lot more complex than just reparsing a file each time. Side note: For some, even the interpreter startup is annoying. Personally I find it negligible, especially after 3.11 you can almost claim it’s snappy.
jsgf 4y ago
With Buck2, memory taken for the graph is a concern, but it fits into a single host's RAM.
- klodolph 4y ago
  Interesting. I know that for Buck 1, some workloads didn’t fit entirely in RAM.

nextaccountic 4y ago

> They are large enough that you cannot load the entire build graph into memory on a single machine You mean, multiple gigabytes for build metadata, that just says things like that X depends on Y and to build Y you run command Z?

klodolph 4y ago
Yes. By “multiple gigabytes” I am talking about >100 GB. Maybe >1 TB.
- nextaccountic 4y ago
  How is this even possible? I take that this data is highly compressible, right?
  - phyrex 4y ago
    It wouldn’t be compressed in ram though, would it?
    - nextaccountic 4y ago
      There are in-memory https://en.wikipedia.org/wiki/Succinct_data_structure https://en.wikipedia.org/wiki/Succinct_data_structure but I actually don't mean that specifically: I mean that, for example, there must be tons of strings with common prefixes, like file paths (which can be stored in a trie to have faster access and compress data in ram) or very similar strings (like compiler invocations that mostly have the same flags), and other highly redundant data that can usually be used to cut down on memory requirements. I highly doubt that, after doing all those tricks, you still end up with 100GB - 1TB of build data.
      - Shish2k 4y ago
        You could do those tricks and cut down memory, perhaps even 10x, but they come at the cost of increased CPU time. Designing the system in such a way that you only ever need to load a tiny subset of the graph at one time gives you a 1000x saving for memory and CPU.
        nextaccountic 4y ago
        Some of those tricks may actually decrease CPU time (by fetching less data from RAM and using the CPU cache more effectively). And you can also apply any optimizations for partial loading on top of that. I guess the downside is that the system would be more complex overall, but you can probably get 80% of the result with not so large changes
esprehn 4y ago
Yes, the codebases at Google and FB contain billions of files. Article from 2016 about the scale, and of course it's only grown dramatically since then: https://m-cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://m-cacm.acm.org/magazines/2016/7/204032-why-google-st...

bogwog 4y ago

I get that, but again, there's no reason Waf can't be used as a base for building that. I actually use Waf for cross compilation extensively, and have built some tools around it with Conan for my own projects. Waf can handle cross compilation just fine, but it's up to you to build what that looks like for your project (a common pattern I see is custom Context subclasses for each target) Memory management, broken build scripts, etc. can all be handled with Waf as well. In the simplest case, you can just wrap a `recurse` call in a try catch block, or you can build something much more sophisticated around how your projects are structured. Note, I'm not trying to argue that Google/Facebook "should have used X". There are a million reasons to pick X over Y, even if Y is the objectively better choice. Sometimes, molding X to be good enough is more efficient than spending months just researching options hoping you'll find Y. I'm just curious to know if they did evaluate Waf, why did they decide against it.

davnn 4y ago
> the core is <10k lines of Python with zero dependencies Isn‘t that already a no-go, to write a performance critical system in a slow programming language?
- taeric 4y ago
  I am no python fan, but find it laughably hard to believe it could be what makes a build coordination system slow.
  - Too 4y ago
    On clean builds the python tax will be dwarfed be the thousands of calls to clang yes. That’s not the scenario you need to optimize for. What’s more important is that incremental builds are snappy, since that is what developers do 100 times per day. I’ve seen some projects with 100MB+ ninja-files that even ninja itself, proud for being written in optimized c++, takes a second or two to parse on each build invocation. Convert that to python and you likely land in 5-20 sec range instead. Enough to alt-tab and get distracted by something else. Google code base is likely even larger than this. A background daemon that holds the graph in memory would probably handle it. In the big scheme such a design is likely better anyway. But needs a big upfront design and is a lot more complex than just reparsing a file each time. Side note: For some, even the interpreter startup is annoying. Personally I find it negligible, especially after 3.11 you can almost claim it’s snappy.

jsgf 4y ago

With Buck2, memory taken for the graph is a concern, but it fits into a single host's RAM.

klodolph 4y ago
Interesting. I know that for Buck 1, some workloads didn’t fit entirely in RAM.