6 ms·

>since the bottleneck in large codebases is not a compute, e.g. headers preprocessing, but it's a memory bandwidth. SSD bandwidth: 4-10GB/s RAM bandwidth: 5-10

>since the bottleneck in large codebases is not a compute, e.g. headers preprocessing, but it's a memory bandwidth. SSD bandwidth: 4-10GB/s RAM bandwidth: 5-10x that, say 40GB/s. If compute was not a bottleneck, the entire linux kernel should compile in less than 1 second.

menaerus 1y ago
On 40-core or 64-core machine there's more compute than you will ever need for a compilation process. Compilation is a heavy I/O workload not a heavy compute workload, in most cases, where it actually matters.
- lingolango 1y ago
  Linux is ~1.5GB of source text and the output is typically a binary less than 100MB. That should take a few hundred milliseconds to read in from an SSD or be basically instant from RAM cache, and then a few hundred ms to write out the binary. So why does it take minutes to compile? Compilation is entirely compute bound, the inputs and outputs are minuscule data sizes, in the order of megabytes for typical projects - maybe gigabytes for multi million line projects, but that is still only a second or two from an SSD.
  - menaerus 1y ago
    Output as a result is 100mb. Process of compilation accumulates magnitudes more data. Evidence is the constant memory pressure you have in 32G or 64G or even 128G systems. Now given that the process of compilation on even such high end systems take non trivial amount of time, tens of minutes, what do you think how much data bounces from and in memory? It accumulates to a lot more than what you suggest.
  - bluGill 1y ago
    I don't build linux from source, but in my tests with large machines (and my C++ work project with more than 10 million lines of code) somewhere between 40 and 50 cores compile speed starts decreasing as you add more cores. When I moved my source files to a ramdisk the speed got even worse so I know disk IO isn't the issue (there was a lot of RAM on this machine so I don't expect to run low on RAM even with that many cores in use). I don't know how to find the truth, but all signs point to memory bandwidth being the issue. Of course the above is specific to the machines I did my testing on. A different machine may have other differences from my setup. Still my experience matches the claim: at 40 cores memory bandwidth is the bottleneck not CPU speed. Most people don't have 40+ core machines to play with, and so will not see those results. The machines I tested on cost > $10,000 so most would argue that is not affordable.
    - menaerus 1y ago
      One of the biggest reasons why people see so much compilation improvement speed on Apple M chips - massive bandwidth improvement in contrast to other machines, even some older servers. 100G/s single core main memory. It starts to drop, e.g. it doesn't scale linearly, when you add more and more cores to the workload, due to L3 contention I'd say, but it goes up to 200G/s IIRC.
  - Someone 1y ago
    > So why does it take minutes to compile? I’m not claiming anything about it being I/O or compute bound, but you are missing some sources of I/O: - the compiler reads many source files (e.g. headers) multiple times - the compiler writes and then reads lots of intermediate data - the OS may have to swap out memory Also, there may be resource contention that makes the system do neither I/O nor compute for part of the build.
    - lingolango 1y ago
      Tried building sqlite amalgamation just now. Input: single .c file 8.5MB. Output: 1.8MB object file. Debug build took 1.5s. Release build (O2) took about 6s. That is about 3 orders of magntiude slower than what this machine is capable of in terms of IO from disk.
    - sgerenser 1y ago
      The fact that something doesn’t scale past X cores doesn’t mean that it is I/O bound! For most C++ toolchains, any given translation unit can only be compiled on a single core. So if you have a big project, but there’s a few files that alone take 1+ minute to compile, the entire compilation can’t possibly take any less than 1 minute even if you had infinite cores. That’s not even getting into linking, which is also usually at least partially if not totally a serial process. See also https://en.m.wikipedia.org/wiki/Amdahl%27s_law https://en.m.wikipedia.org/wiki/Amdahl%27s_law
- anarazel 1y ago
  This is just wildly wrong. On an older 2 socket workstation, with relatively poor memory bandwidth, I ran a linux kernel compile. perf stat --topdown --td-level 2 indicates that memory bandwidth is not a bottleneck. Fetch latency, branch mispredicts and the frontend are. I also analyzed the memory bandwidth using perf stat --per-socket -M memory_bandwidth_read,memory_bandwidth_write -a -r0 sleep 1 and it never gets anywhere close to the memory bandwidth the system can trivially utilize (it barely reaches the bandwidth a single core can utilize). iostat indicates there are pretty much no reads/writes happening on the relevant disks. Every core is 100% busy.
  - bluGill 1y ago
    How many cores on that workstation? The claim is you need 40 cores to observe that - very few people have access to such a thing - they exist, but they are expensive.
    - anarazel 1y ago
      That workstation has 2x10 cores / 20 threads. I also executed the test on a newer workstation with 2x24 cores with similar results, but I thought the older workstation is more interesting, as the older workstation has a much worse memory bandwidth. Sorry, but compilation is simply not memory bandwidth bound. There are significant memory latency effects, but bandwidth != latency.
      - menaerus 1y ago
        I doubt you can saturate the bandwidth with dual-socket configuration with each having 10 cores. Perhaps if you have very recent cores, which I believe you don't, but Intel design hasn't been that good. What you're also measuring in your experiment, and needs to be taken into account, is the latency across the NUMA nodes which is ridiculously high, 1.5x to 2x to the local node, amounting to usually ~130ns. Because of this, in NUMA configurations, you usually need more (Intel) cores to saturate the bw. I know because I have one sitting at my desk. Memory bandwidth saturation usually begins at ~20 cores with the Intel design that is roughly ~5 year old. I might be off with that number but it's roughly something like that. Other cores if you have them burning the cycles are just sitting there and waiting in the line for the bus to become free.
jayd16 1y ago
This is making the assumption that source is read once and that there is no intermediate data to write and read. Unless the working set fits in cache, you'll have I/O and can be I/O bound.

menaerus 1y ago

On 40-core or 64-core machine there's more compute than you will ever need for a compilation process. Compilation is a heavy I/O workload not a heavy compute workload, in most cases, where it actually matters.

lingolango 1y ago
Linux is ~1.5GB of source text and the output is typically a binary less than 100MB. That should take a few hundred milliseconds to read in from an SSD or be basically instant from RAM cache, and then a few hundred ms to write out the binary. So why does it take minutes to compile? Compilation is entirely compute bound, the inputs and outputs are minuscule data sizes, in the order of megabytes for typical projects - maybe gigabytes for multi million line projects, but that is still only a second or two from an SSD.
- menaerus 1y ago
  Output as a result is 100mb. Process of compilation accumulates magnitudes more data. Evidence is the constant memory pressure you have in 32G or 64G or even 128G systems. Now given that the process of compilation on even such high end systems take non trivial amount of time, tens of minutes, what do you think how much data bounces from and in memory? It accumulates to a lot more than what you suggest.
- bluGill 1y ago
  I don't build linux from source, but in my tests with large machines (and my C++ work project with more than 10 million lines of code) somewhere between 40 and 50 cores compile speed starts decreasing as you add more cores. When I moved my source files to a ramdisk the speed got even worse so I know disk IO isn't the issue (there was a lot of RAM on this machine so I don't expect to run low on RAM even with that many cores in use). I don't know how to find the truth, but all signs point to memory bandwidth being the issue. Of course the above is specific to the machines I did my testing on. A different machine may have other differences from my setup. Still my experience matches the claim: at 40 cores memory bandwidth is the bottleneck not CPU speed. Most people don't have 40+ core machines to play with, and so will not see those results. The machines I tested on cost > $10,000 so most would argue that is not affordable.
  - menaerus 1y ago
    One of the biggest reasons why people see so much compilation improvement speed on Apple M chips - massive bandwidth improvement in contrast to other machines, even some older servers. 100G/s single core main memory. It starts to drop, e.g. it doesn't scale linearly, when you add more and more cores to the workload, due to L3 contention I'd say, but it goes up to 200G/s IIRC.
- Someone 1y ago
  > So why does it take minutes to compile? I’m not claiming anything about it being I/O or compute bound, but you are missing some sources of I/O: - the compiler reads many source files (e.g. headers) multiple times - the compiler writes and then reads lots of intermediate data - the OS may have to swap out memory Also, there may be resource contention that makes the system do neither I/O nor compute for part of the build.
  - lingolango 1y ago
    Tried building sqlite amalgamation just now. Input: single .c file 8.5MB. Output: 1.8MB object file. Debug build took 1.5s. Release build (O2) took about 6s. That is about 3 orders of magntiude slower than what this machine is capable of in terms of IO from disk.
  - sgerenser 1y ago
    The fact that something doesn’t scale past X cores doesn’t mean that it is I/O bound! For most C++ toolchains, any given translation unit can only be compiled on a single core. So if you have a big project, but there’s a few files that alone take 1+ minute to compile, the entire compilation can’t possibly take any less than 1 minute even if you had infinite cores. That’s not even getting into linking, which is also usually at least partially if not totally a serial process. See also https://en.m.wikipedia.org/wiki/Amdahl%27s_law https://en.m.wikipedia.org/wiki/Amdahl%27s_law
anarazel 1y ago
This is just wildly wrong. On an older 2 socket workstation, with relatively poor memory bandwidth, I ran a linux kernel compile. perf stat --topdown --td-level 2 indicates that memory bandwidth is not a bottleneck. Fetch latency, branch mispredicts and the frontend are. I also analyzed the memory bandwidth using perf stat --per-socket -M memory_bandwidth_read,memory_bandwidth_write -a -r0 sleep 1 and it never gets anywhere close to the memory bandwidth the system can trivially utilize (it barely reaches the bandwidth a single core can utilize). iostat indicates there are pretty much no reads/writes happening on the relevant disks. Every core is 100% busy.
- bluGill 1y ago
  How many cores on that workstation? The claim is you need 40 cores to observe that - very few people have access to such a thing - they exist, but they are expensive.
  - anarazel 1y ago
    That workstation has 2x10 cores / 20 threads. I also executed the test on a newer workstation with 2x24 cores with similar results, but I thought the older workstation is more interesting, as the older workstation has a much worse memory bandwidth. Sorry, but compilation is simply not memory bandwidth bound. There are significant memory latency effects, but bandwidth != latency.
    - menaerus 1y ago
      I doubt you can saturate the bandwidth with dual-socket configuration with each having 10 cores. Perhaps if you have very recent cores, which I believe you don't, but Intel design hasn't been that good. What you're also measuring in your experiment, and needs to be taken into account, is the latency across the NUMA nodes which is ridiculously high, 1.5x to 2x to the local node, amounting to usually ~130ns. Because of this, in NUMA configurations, you usually need more (Intel) cores to saturate the bw. I know because I have one sitting at my desk. Memory bandwidth saturation usually begins at ~20 cores with the Intel design that is roughly ~5 year old. I might be off with that number but it's roughly something like that. Other cores if you have them burning the cycles are just sitting there and waiting in the line for the bus to become free.

jayd16 1y ago

This is making the assumption that source is read once and that there is no intermediate data to write and read. Unless the working set fits in cache, you'll have I/O and can be I/O bound.