5 ms·

Something I've wondered about is why CPUs have separate scalar and vector ALUs. Would it be possible to simply have 16x scalar ALUs that can be used either indi

Something I've wondered about is why CPUs have separate scalar and vector ALUs. Would it be possible to simply have 16x scalar ALUs that can be used either individually or ganged together in groups of 4, 8, or 16 to execute the various vector instructions?

dzaima 2y ago
Intel has shared ports across scalar and SIMD for a long time (only recently they've split them apart). I don't know whether they share silicon though, quite likely that they don't. The complexity of using parts of vector ALUs might not be worth the saved silicon (even for something as expensive as float division, which has both SIMD and scalar versions (both operate in the same register file, just differing in the number of elements processed), both Intel and AMD have the same throughput for the scalar version and 128-bit SIMD version[0]). It additionally allows separating bypass networks, e.g. the output of a SIMD add will never be directly used as a memory address (or, in the case of a gather, an extra cycle or two of latency is acceptable considering the amount of loads that will follow and in general the throughput-centric nature of SIMD), whereas for a scalar op that's an extremely important path. Or, in general, it allows making different tradeoffs between scalar & SIMD. [0]: https://uops.info/table.html?search=div%20xmm)&cb_lat=on&cb_tp=on&cb_HSW=on&cb_SKL=on&cb_ADLP=on&cb_ZEN2=on&cb_ZEN4=on&cb_measurements=on&cb_sse=on https://uops.info/table.html?search=div%20xmm)&cb_lat=on&cb_...
magicalhippo 2y ago
Good question. I imagine that for instruction encoding density one could for example encode instructions so that "add4 ra0, rb4" would do a 4-wide addition ra0+rb4 up to ra3+rb7. Possibly with restrictions like only "aligned" register access, ie an 8-way op can only use regs 0-7 or 8-15. Swizzle instructions can then handle reordering ala SSE. Don't forget about 2-way ops as well.
- jiggawatts 2y ago
  I was just thinking of the back end ALUs but you make a good point: it might also make sense to unify scalar and vector instructions. If the right encoding is used it might allow scalar code to retain its efficiencies.
  - magicalhippo 2y ago
    For "just" the back-end I was thinking the instruction count overhead would outweigh the gain. That is you'd get less effective instruction cache and require far more decoding units from all the instructions required to fill all those ALUs. So yea, that's why I was imagining an ISA where you'd have instructions that were 1, 2, 4 etc wide, encoded in the instruction. You'd only need two bits for up to 2^4=16 wide instructions. Of course could be I am terribly mistaken, I'm by no means an expert in this.
  - dzaima 2y ago
    For unified scalar and vector instructions, MRISC32 might be interesting: https://mrisc32.bitsnbites.eu/ https://mrisc32.bitsnbites.eu/. Though it still has separate vector & scalar register files, and I don't think it's intended for the type of thing discussed above.
Remnant44 2y ago
Setting aside the complexity of "detachable" ALU clusters, the bigger problem is that essentially, we can currently drive all those ALUs _because_ they're SIMD and executing the same instruction. For example, if you could un-gang even a single Zen5 ALU and do 16 independent scalar FMAs at once, you now need the front-end of the processor to decode and issue 16 instructions/cycle for each 1 instruction that it currently does! That's hopelessly wider than the front-end of the CPU can process. It needs to decode the instructions and schedule them, and both of those are complex operations that can be power hungry / slow (not to mention the instruction fetch bandwidth is now through the roof!). SIMD bypasses that by doing lots of operations with a single instruction. It would be extremely difficult to achieve with just scalar instructions.
adrian_b 2y ago
When a team at IBM has coined the word "superscalar" in 1987, they have written a widely cited research paper in which they have argued that a "superscalar" CPU is better than a "vector CPU" ("vector" CPUs were well known at that time and had been used for more than a decade, especially in supercomputers), therefore all vector CPUs should be replaced with superior "superscalar" CPUs. Their theory was in essence that when you have 16x scalar ALUs it is always better to be able to use them individually, instead of having them used by a single instruction. If you have the hardware that allows using the 16 scalar ALUs individually, by 16 independent instructions, that is obviously much more powerful. It can handle all the cases that can be handled by SIMD, but also many other cases. The IBM research paper has been extremely influential. Before it, the target for most CPU design teams was to make a pipelined CPU able to execute 1 instruction per clock cycle, after it everybody has switched to attempting to design CPUs with an IPC as high as possible. Even Intel, who prioritized the CPU production cost over the CPU performance, has evolved through 80486 (1989), Pentium (1993) and Pentium Pro (1995) eventually reaching the stage of having a high-performance superscalar CPU. Nevertheless, around 1995 the initial hype about superscalar CPUs has begun to dissipate. It became understood that even if a superscalar CPU would always be faster than a vector CPU with the same number of ALUs, the cost of the superscalar CPU increases very quickly and super-linearly with the number of ALUs. For high enough values, doubling the number of ALUs in a superscalar CPU increases both the area and the power consumption by a factor much greater than 2. The result of understanding the limitations of the superscalar CPUs was a smaller step back, resurrecting the vector CPUs, but in combination with the superscalar CPUs. Now all modern CPUs use this combination, instead of using only one of the two design variants. For instance, Zen 5 contains 32 FP64 adders. However, it can do only 4 independent FP64 operations simultaneously. Therefore it can do 4 scalar FP64 additions per cycle. Or it can gang pairs of ALUs and it can do 4 additions of length-2 FP64 vectors per cycle. Or it can gang groups of 4 ALUs and it can do 4 additions of length-4 FP64 vectors per cycle. Or it can gang groups of 8 ALUs and it can do 4 additions of length-8 FP64 vectors per cycle. Only in the last case all the 32 existing FP64 adders are used. The same is in all modern CPUs. All have 2 limits. One limit is the number of ALUs and the second limit is the number of simultaneous independent operations a.k.a. the number of execution ports. The number of execution ports limits the number of independent scalar operations. The total number of ALUs limits the number of equivalent scalar operations when they are not independent, but the ALUs are ganged for vector operations.

dzaima 2y ago

Intel has shared ports across scalar and SIMD for a long time (only recently they've split them apart). I don't know whether they share silicon though, quite likely that they don't. The complexity of using parts of vector ALUs might not be worth the saved silicon (even for something as expensive as float division, which has both SIMD and scalar versions (both operate in the same register file, just differing in the number of elements processed), both Intel and AMD have the same throughput for the scalar version and 128-bit SIMD version[0]). It additionally allows separating bypass networks, e.g. the output of a SIMD add will never be directly used as a memory address (or, in the case of a gather, an extra cycle or two of latency is acceptable considering the amount of loads that will follow and in general the throughput-centric nature of SIMD), whereas for a scalar op that's an extremely important path. Or, in general, it allows making different tradeoffs between scalar & SIMD. [0]: https://uops.info/table.html?search=div%20xmm)&cb_lat=on&cb_tp=on&cb_HSW=on&cb_SKL=on&cb_ADLP=on&cb_ZEN2=on&cb_ZEN4=on&cb_measurements=on&cb_sse=on https://uops.info/table.html?search=div%20xmm)&cb_lat=on&cb_...

magicalhippo 2y ago

Good question. I imagine that for instruction encoding density one could for example encode instructions so that "add4 ra0, rb4" would do a 4-wide addition ra0+rb4 up to ra3+rb7. Possibly with restrictions like only "aligned" register access, ie an 8-way op can only use regs 0-7 or 8-15. Swizzle instructions can then handle reordering ala SSE. Don't forget about 2-way ops as well.

jiggawatts 2y ago
I was just thinking of the back end ALUs but you make a good point: it might also make sense to unify scalar and vector instructions. If the right encoding is used it might allow scalar code to retain its efficiencies.
- magicalhippo 2y ago
  For "just" the back-end I was thinking the instruction count overhead would outweigh the gain. That is you'd get less effective instruction cache and require far more decoding units from all the instructions required to fill all those ALUs. So yea, that's why I was imagining an ISA where you'd have instructions that were 1, 2, 4 etc wide, encoded in the instruction. You'd only need two bits for up to 2^4=16 wide instructions. Of course could be I am terribly mistaken, I'm by no means an expert in this.
- dzaima 2y ago
  For unified scalar and vector instructions, MRISC32 might be interesting: https://mrisc32.bitsnbites.eu/ https://mrisc32.bitsnbites.eu/. Though it still has separate vector & scalar register files, and I don't think it's intended for the type of thing discussed above.

Remnant44 2y ago

Setting aside the complexity of "detachable" ALU clusters, the bigger problem is that essentially, we can currently drive all those ALUs _because_ they're SIMD and executing the same instruction. For example, if you could un-gang even a single Zen5 ALU and do 16 independent scalar FMAs at once, you now need the front-end of the processor to decode and issue 16 instructions/cycle for each 1 instruction that it currently does! That's hopelessly wider than the front-end of the CPU can process. It needs to decode the instructions and schedule them, and both of those are complex operations that can be power hungry / slow (not to mention the instruction fetch bandwidth is now through the roof!). SIMD bypasses that by doing lots of operations with a single instruction. It would be extremely difficult to achieve with just scalar instructions.

adrian_b 2y ago

When a team at IBM has coined the word "superscalar" in 1987, they have written a widely cited research paper in which they have argued that a "superscalar" CPU is better than a "vector CPU" ("vector" CPUs were well known at that time and had been used for more than a decade, especially in supercomputers), therefore all vector CPUs should be replaced with superior "superscalar" CPUs. Their theory was in essence that when you have 16x scalar ALUs it is always better to be able to use them individually, instead of having them used by a single instruction. If you have the hardware that allows using the 16 scalar ALUs individually, by 16 independent instructions, that is obviously much more powerful. It can handle all the cases that can be handled by SIMD, but also many other cases. The IBM research paper has been extremely influential. Before it, the target for most CPU design teams was to make a pipelined CPU able to execute 1 instruction per clock cycle, after it everybody has switched to attempting to design CPUs with an IPC as high as possible. Even Intel, who prioritized the CPU production cost over the CPU performance, has evolved through 80486 (1989), Pentium (1993) and Pentium Pro (1995) eventually reaching the stage of having a high-performance superscalar CPU. Nevertheless, around 1995 the initial hype about superscalar CPUs has begun to dissipate. It became understood that even if a superscalar CPU would always be faster than a vector CPU with the same number of ALUs, the cost of the superscalar CPU increases very quickly and super-linearly with the number of ALUs. For high enough values, doubling the number of ALUs in a superscalar CPU increases both the area and the power consumption by a factor much greater than 2. The result of understanding the limitations of the superscalar CPUs was a smaller step back, resurrecting the vector CPUs, but in combination with the superscalar CPUs. Now all modern CPUs use this combination, instead of using only one of the two design variants. For instance, Zen 5 contains 32 FP64 adders. However, it can do only 4 independent FP64 operations simultaneously. Therefore it can do 4 scalar FP64 additions per cycle. Or it can gang pairs of ALUs and it can do 4 additions of length-2 FP64 vectors per cycle. Or it can gang groups of 4 ALUs and it can do 4 additions of length-4 FP64 vectors per cycle. Or it can gang groups of 8 ALUs and it can do 4 additions of length-8 FP64 vectors per cycle. Only in the last case all the 32 existing FP64 adders are used. The same is in all modern CPUs. All have 2 limits. One limit is the number of ALUs and the second limit is the number of simultaneous independent operations a.k.a. the number of execution ports. The number of execution ports limits the number of independent scalar operations. The total number of ALUs limits the number of equivalent scalar operations when they are not independent, but the ALUs are ganged for vector operations.