8 ms·

Fujitsu A64FX Microarchitecture Manual [pdf]

73 points by rrss 6y ago

aoki 6y ago
For anyone else who was confused: this is an ARM design aimed at supercomputers, and it is used in a recent system that topped TOP500.
- gnufx 6y ago
  And specifically designed (along with the other system components) with suitable trades-off across the sort of work envisaged for it.
whatever1 6y ago
It is so weird that Intel and AMD blew their headstart so badly. I see so many players who come up with ARM implementations that come close or even exceed the performance / energy envelope of Intel/Amd.
- tyingq 6y ago
  This one counts as exceeding, at least on some measures. The chip has 1TB/s of memory bandwidth from the 48 cores to its on-package 32GB of memory. Looks like the smallest thing you can buy, though, is a 2U server with 8 sockets, 2 of them populated, for $40k.
- dragontamer 6y ago
  Fujitsu has been making TOP500 supercomputers for years, maybe decades. They're not a small player. They used to make UltraSPARCs, but since Sun / Oracle is a toxic wasteland now, its natural for them to move to ARM instead. The previous Fujitsu supercomputer, the K, was also #1 when it came out in 2011. (https://en.wikipedia.org/wiki/K_computer https://en.wikipedia.org/wiki/K_computer) ----------- The main thing that been proven is that a company like Fujitsu can transition between ISAs: the SPARC ISA into the ARM ISA, and not miss a beat.
  - agumonkey 6y ago
    What surprises me, slightly, is how Fujitsu never tried to expand into more markets.
    - archi42 6y ago
      Which markets exactly? For generic x86, they have server, workstation and blade ranges: https://www.fujitsu.com/global/products/computing/servers/primergy/ https://www.fujitsu.com/global/products/computing/servers/pr... I'm not in that business/role, so I don't know how they compare to e.g. Dell or HP; especially regarding support. But I've got one of their older 1U rackmounts in my basement and it's a pretty nice machine.
      - agumonkey 6y ago
        consumer / prosumer / workstations is what I had in mind
        jzwinck 6y ago
        Like this? https://www.fujitsu.com/global/products/computing/pc/workstations/celsius-w5010/index.html https://www.fujitsu.com/global/products/computing/pc/worksta... And this? https://www.fujitsu.com/global/products/computing/pc/desktops/esprimo-d9010/index.html https://www.fujitsu.com/global/products/computing/pc/desktop...
        agumonkey 6y ago
        then my radar is way too narrow it's been 15 years since I've seen a fujitsu desktop in EU. I mean I regularly see way more panasonic toughbooks around and it's actually rare.
        _trampeltier 6y ago
        I use at home since many years just Fujitsu Notebooks and also have an W550 workstation. Also for my mom, I bought for her many years ago a Fujitsu Notebook. They come without any bloatware and just run and run forever. The design is nice, I like hiw they have mostly a red line over the device. But they are not cheap to buy. I bought lately an U939X notebook and he is really light, but still everithing (touch and pen) works also with Linux. https://m.youtube.com/watch?v=hS2uogVMK8U https://m.youtube.com/watch?v=hS2uogVMK8U
- djmips 6y ago
  To a certain extent it's always been about bandwidth. Everyone is using the same memory technology.
  - dragontamer 6y ago
    Fujitsu's previous "K" supercomputer was #1 on TOP500 and GREEN500 when it came out. Heck, it was #1 on Green500 for many, many years after its release. K held #1 on HPCG from 2011 through 2018. Fujitsu just has good tech. Its kind of mysterious since its aimed at the Japanese market instead of the US market, so we don't hear much about it. But its clearly a world-class processor company.
    - linksnapzz 6y ago
      My impression is that the Fujitsu K-machine && NEC SX-ACE equipment is designed, built, and sold almost exclusively for the Japanese market; courtesy of massive subsidies from the Japanese gov’t for the purpose of maintaining that expertise in-country.
      - gnufx 6y ago
        K was basically a one-off, but you could buy systems with similar technology, though they weren't common outside Japan. A64FX machines are appearing in the West now. There was a recent substantial award for one to a UK academic/Met Office consortium, in which there's some interest.
        linksnapzz 6y ago
        Yeah, by K-machine I mean any of the PrimeHPC-FX class systems using SPARC64-V derivatives; I'm sure they sold a few outside Japan, but I don't think that not having the wider global market would've deterred Fujitsu from making them. I know some of Fujitsu's A64 modules were making it into Cray systems as well; isn't that what U. Bristol has?
        6y ago
- ChuckMcM 6y ago
  As I see it, Intel and AMD see the super computer market as "niche" and while it gives good publicity, there are perhaps 1 or 2 such machines sold each year. As a result they spend their time and "innovation beans" on things that make desktop and laptop computers more desirable. IBM's Power architecture is another computer ecosystem that is really innovative and cool but you hardly hear about it. Different target markets. The second part of the issue is "head start." For literally decades the "cost" of a computer architecture was perhaps 3% chips and 97% software support for those chips. The "Windows" lock (by having the #1 OS run only on x86 architecture machines) meant that Intel really didn't have to do anything for years but shrink their transistor size and they could "win." Linux took away the Windows "moat", networking took away the "enterprise server" moat, and gcc took away the "compiler/tools" moat. Between the three of them it gave everyone a chance to play if they wanted to stake out a niche. ARM took "low power" and captured phones for which battery power is key. By licensing the architecture widely and inexpensively they enabled parallel investment that well exceeded anything a single manufacturer could apply to a particular architecture. By some estimates you have over $50 billion dollars / year being spent on R&D that accelerates ARM compared to Intel's $15 billion R&D budget for 2020? Not so much "blowing their head start" as is it getting mowed down by not being able to invest as much as the other guys.
  - bhouston 6y ago
    It used to be a big deal when intel machines started to show up and then rule the supercomputer benchmarks. It used to be all specialized chips like IBM Power. It was when Intel was ascendant. Now ARM is doing the same thing. Arm is attacking Intel at the top and bottom and middle leaving less and less space for them to operate. Intel lost embedded to arm first. Then the lost mobile. Then they lost the Apple laptops and later this year the Apple desktops. They are starting to lose the super computers. The nail in the coffin will come with the cloud servers if those vendors could get something close to Apple’s custom Arm cpu performance.
    - ori_b 6y ago
      This worries me. Intel was never very clean, but it's so much more open than every other option. There's even complete GPU documentation.
mhh__ 6y ago
It would be nice if apple would bother to publish one for their chips.
- protomyth 6y ago
  Apple doesn't do normal programming documentation these days, so I don't expect something that in-depth.
  - mhh__ 6y ago
    That's true, but microarchitecture docs and optimization manuals aren't like regular documentation. Apart from the intellectual dick-waving, it's just plain interesting.
    - jabl 6y ago
      Apple hasn't even documented their matrix extension, apparently thinking that people shouldn't compile against them directly but instead use their accelerate framework. Further, they use their private llvm branch and haven't pushed the pipeline description tables for AFAIK any of their arm chips to upstream llvm. So while I agree it would be interesting, I wouldn't hold my breath.
CalChris 6y ago
This is a lot like the manuals that ARM publishes for its IP. For example, for the A76 [1] they publish the Arm Cortex-A76 Software Optimization Guide. But I haven't seen this for any non-ARM ARMv8 implementation such as Apple, NVidia, ... [1] https://developer.arm.com/documentation/swog307215/ https://developer.arm.com/documentation/swog307215/
- my123 6y ago
  For Nvidia, the CPU manual is part of the SoC Manual, downloadable at https://developer.nvidia.com/embedded/downloads#?cl=date,-1 https://developer.nvidia.com/embedded/downloads#?cl=date,-1. (search for TRM) Xavier has 8 Carmel cores and Parker/X2 has 2 Denver2 cores and 4 Cortex-A57s. The SoC manual includes instruction timings for the NV cores.
nullc 6y ago
This is much more detailed than what I've seen available for other high performance modern microprocessors.

73 points by rrss 6y ago

aoki 6y ago

For anyone else who was confused: this is an ARM design aimed at supercomputers, and it is used in a recent system that topped TOP500.

gnufx 6y ago
And specifically designed (along with the other system components) with suitable trades-off across the sort of work envisaged for it.

whatever1 6y ago

It is so weird that Intel and AMD blew their headstart so badly. I see so many players who come up with ARM implementations that come close or even exceed the performance / energy envelope of Intel/Amd.

tyingq 6y ago
This one counts as exceeding, at least on some measures. The chip has 1TB/s of memory bandwidth from the 48 cores to its on-package 32GB of memory. Looks like the smallest thing you can buy, though, is a 2U server with 8 sockets, 2 of them populated, for $40k.
dragontamer 6y ago
Fujitsu has been making TOP500 supercomputers for years, maybe decades. They're not a small player. They used to make UltraSPARCs, but since Sun / Oracle is a toxic wasteland now, its natural for them to move to ARM instead. The previous Fujitsu supercomputer, the K, was also #1 when it came out in 2011. (https://en.wikipedia.org/wiki/K_computer https://en.wikipedia.org/wiki/K_computer) ----------- The main thing that been proven is that a company like Fujitsu can transition between ISAs: the SPARC ISA into the ARM ISA, and not miss a beat.
- agumonkey 6y ago
  What surprises me, slightly, is how Fujitsu never tried to expand into more markets.
  - archi42 6y ago
    Which markets exactly? For generic x86, they have server, workstation and blade ranges: https://www.fujitsu.com/global/products/computing/servers/primergy/ https://www.fujitsu.com/global/products/computing/servers/pr... I'm not in that business/role, so I don't know how they compare to e.g. Dell or HP; especially regarding support. But I've got one of their older 1U rackmounts in my basement and it's a pretty nice machine.
    - agumonkey 6y ago
      consumer / prosumer / workstations is what I had in mind
      - jzwinck 6y ago
        Like this? https://www.fujitsu.com/global/products/computing/pc/workstations/celsius-w5010/index.html https://www.fujitsu.com/global/products/computing/pc/worksta... And this? https://www.fujitsu.com/global/products/computing/pc/desktops/esprimo-d9010/index.html https://www.fujitsu.com/global/products/computing/pc/desktop...
        agumonkey 6y ago
        then my radar is way too narrow it's been 15 years since I've seen a fujitsu desktop in EU. I mean I regularly see way more panasonic toughbooks around and it's actually rare.
      - _trampeltier 6y ago
        I use at home since many years just Fujitsu Notebooks and also have an W550 workstation. Also for my mom, I bought for her many years ago a Fujitsu Notebook. They come without any bloatware and just run and run forever. The design is nice, I like hiw they have mostly a red line over the device. But they are not cheap to buy. I bought lately an U939X notebook and he is really light, but still everithing (touch and pen) works also with Linux. https://m.youtube.com/watch?v=hS2uogVMK8U https://m.youtube.com/watch?v=hS2uogVMK8U
djmips 6y ago
To a certain extent it's always been about bandwidth. Everyone is using the same memory technology.
- dragontamer 6y ago
  Fujitsu's previous "K" supercomputer was #1 on TOP500 and GREEN500 when it came out. Heck, it was #1 on Green500 for many, many years after its release. K held #1 on HPCG from 2011 through 2018. Fujitsu just has good tech. Its kind of mysterious since its aimed at the Japanese market instead of the US market, so we don't hear much about it. But its clearly a world-class processor company.
  - linksnapzz 6y ago
    My impression is that the Fujitsu K-machine && NEC SX-ACE equipment is designed, built, and sold almost exclusively for the Japanese market; courtesy of massive subsidies from the Japanese gov’t for the purpose of maintaining that expertise in-country.
    - gnufx 6y ago
      K was basically a one-off, but you could buy systems with similar technology, though they weren't common outside Japan. A64FX machines are appearing in the West now. There was a recent substantial award for one to a UK academic/Met Office consortium, in which there's some interest.
      - linksnapzz 6y ago
        Yeah, by K-machine I mean any of the PrimeHPC-FX class systems using SPARC64-V derivatives; I'm sure they sold a few outside Japan, but I don't think that not having the wider global market would've deterred Fujitsu from making them. I know some of Fujitsu's A64 modules were making it into Cray systems as well; isn't that what U. Bristol has?
        6y ago
ChuckMcM 6y ago
As I see it, Intel and AMD see the super computer market as "niche" and while it gives good publicity, there are perhaps 1 or 2 such machines sold each year. As a result they spend their time and "innovation beans" on things that make desktop and laptop computers more desirable. IBM's Power architecture is another computer ecosystem that is really innovative and cool but you hardly hear about it. Different target markets. The second part of the issue is "head start." For literally decades the "cost" of a computer architecture was perhaps 3% chips and 97% software support for those chips. The "Windows" lock (by having the #1 OS run only on x86 architecture machines) meant that Intel really didn't have to do anything for years but shrink their transistor size and they could "win." Linux took away the Windows "moat", networking took away the "enterprise server" moat, and gcc took away the "compiler/tools" moat. Between the three of them it gave everyone a chance to play if they wanted to stake out a niche. ARM took "low power" and captured phones for which battery power is key. By licensing the architecture widely and inexpensively they enabled parallel investment that well exceeded anything a single manufacturer could apply to a particular architecture. By some estimates you have over $50 billion dollars / year being spent on R&D that accelerates ARM compared to Intel's $15 billion R&D budget for 2020? Not so much "blowing their head start" as is it getting mowed down by not being able to invest as much as the other guys.
- bhouston 6y ago
  It used to be a big deal when intel machines started to show up and then rule the supercomputer benchmarks. It used to be all specialized chips like IBM Power. It was when Intel was ascendant. Now ARM is doing the same thing. Arm is attacking Intel at the top and bottom and middle leaving less and less space for them to operate. Intel lost embedded to arm first. Then the lost mobile. Then they lost the Apple laptops and later this year the Apple desktops. They are starting to lose the super computers. The nail in the coffin will come with the cloud servers if those vendors could get something close to Apple’s custom Arm cpu performance.
  - ori_b 6y ago
    This worries me. Intel was never very clean, but it's so much more open than every other option. There's even complete GPU documentation.

mhh__ 6y ago

It would be nice if apple would bother to publish one for their chips.

protomyth 6y ago
Apple doesn't do normal programming documentation these days, so I don't expect something that in-depth.
- mhh__ 6y ago
  That's true, but microarchitecture docs and optimization manuals aren't like regular documentation. Apart from the intellectual dick-waving, it's just plain interesting.
  - jabl 6y ago
    Apple hasn't even documented their matrix extension, apparently thinking that people shouldn't compile against them directly but instead use their accelerate framework. Further, they use their private llvm branch and haven't pushed the pipeline description tables for AFAIK any of their arm chips to upstream llvm. So while I agree it would be interesting, I wouldn't hold my breath.

CalChris 6y ago

This is a lot like the manuals that ARM publishes for its IP. For example, for the A76 [1] they publish the Arm Cortex-A76 Software Optimization Guide. But I haven't seen this for any non-ARM ARMv8 implementation such as Apple, NVidia, ... [1] https://developer.arm.com/documentation/swog307215/ https://developer.arm.com/documentation/swog307215/

my123 6y ago
For Nvidia, the CPU manual is part of the SoC Manual, downloadable at https://developer.nvidia.com/embedded/downloads#?cl=date,-1 https://developer.nvidia.com/embedded/downloads#?cl=date,-1. (search for TRM) Xavier has 8 Carmel cores and Parker/X2 has 2 Denver2 cores and 4 Cortex-A57s. The SoC manual includes instruction timings for the NV cores.

nullc 6y ago

This is much more detailed than what I've seen available for other high performance modern microprocessors.