8 ms·

Assessing Cavium’s ThunderX2: The Arm Server Dream Realized

99 points by drewg123 8y ago

jsgo 8y ago
sounds great, didn't I read that ARM was backing off on server though (hoping it was a bad article). edit: thanks, you all nailed it. I remember now and it was Qualcomm. Whew, I'm glad as I was bummed that this meant there was a good step for a piece of tech that would be DoA. Really wish Qualcomm wasn't abandoning based on how it appeared they were making progress as well to be a good competitor (and competition has been great where applied).
- pertymcpert 8y ago
  On the contrary. That was Qualcomm perhaps you read about.
- mtgx 8y ago
  Not Arm, but Qualcomm. It was also a strange and sudden decision. I don't think they really wanted to do it, but might have been forced to do it by the circumstances: anti-trust lawsuits, Apple dumping them, Broadcom attempting a hostile takeover, U.S. government pressuring them not to sell to Broadcom no matter what, etc. One or all of these may have forced Qualcomm to cut its losses. Either way, I hope they get to sell the Centriq line to some other company that can provide competition to both Cavium and Intel/AMD/IBM in the server space. We need more than one Arm chip to compete in the space, otherwise Cavium's chip could also be seen as some kind of outlier and be ignored by the industry.
  - wyldfire 8y ago
    > One or all of these may have forced Qualcomm to cut its losses. Yeah, I think the LBO really scared Qualcomm. Their royalties fund a lot of the other work in the company and I think they've learned that it also makes them a target. A domestic LBO could happen and then there'd be no escape hatch. They pledged to cut costs when Broadcom announced their intentions, this may be a part of that. I don't think of them selling the server chip business as "cutting its losses" -- instead I would call it narrowing its focus back to businesses closer to its core competency. The Centriq launch with Amberwing seemed to be relatively positive for a first design release. They didn't turn the industry upside-down but they probably achieved most of what they set out to do. It will take some time to convince server customers to switch, recompile their code and get their vendors to recompile their code. > We need more than one Arm chip to compete in the space, otherwise Cavium's chip could also be seen as some kind of outlier and be ignored by the industry. Agreed. There's momentum that's been built [1,2] and I'd hate to see it flub now. [1] https://www.redhat.com/en/blog/red-hat-introduces-arm-server-support-red-hat-enterprise-linux https://www.redhat.com/en/blog/red-hat-introduces-arm-server... [2] https://www.suse.com/c/news/suse-steps-up-to-support-innovative-arm-solutions-for-customers/ https://www.suse.com/c/news/suse-steps-up-to-support-innovat...
    - jsgo 8y ago
      I hope they do something with it though. Whether sell it to someone who can be a good caretaker to it, share lessons and designs with other companies that could then build upon it, spinning it off, whatever they choose. Just abandoning it outright seems like an unfortunate move. I won't say bad, because I'm sure they have their reasons, but unfortunate in that it seemed like it was on track to be viable. Microsoft's Surface Pro line, to me, was kind of an "eh, that's nice I guess" up until the Pro 3 at which point it seemed like a platform that had hit its stride. The stuff I was reading on Amberwing seemed like it wasn't a silver bullet, but what it did work for, it worked very well when comparing price vs performance so one could hope that with time, they could make it even better.
  - auslander 8y ago
    Or NSA was pushing for closed source system firmware a.k.a blobs
newprint 8y ago
I would curious, where can I buy this system ? Anyone one know resellers ? At $$$ listed, I would buy cheapest system in a heartbeat.
- ajross 8y ago
  This is a press release detailing the what appears to be the system under test: https://www.cavium.com/news/gigabyte-announces-thunderxstation-industry-s-first-armv8-workstation-based-on-cavium-s-thunderx2-processor https://www.cavium.com/news/gigabyte-announces-thunderxstati... I can't find a mention of it on Gigabyte's site, but there's an email there you could try. Honestly: this is a brand new device that clearly isn't actually in the channel yet. They shipped out test systems for press hits like this, it's not really a "product" quite yet.
- rwmj 8y ago
  Cavium are actually pretty good out of all the ARM server vendors of, you know, selling the hardware. (For some reason other vendors found that really hard.) The ThunderX2 has not been generally released yet, but I'm expecting it'll turn up on the same sites selling the old ThunderX, eg. https://www.avantek.co.uk/store/arm-servers.html https://www.avantek.co.uk/store/arm-servers.html
  - aseipp 8y ago
    Out of curiosity I looked, and they actually already have pricing for the ThunderX2 Workstation from Gigabyte -- the very one mentioned in the article: about 10,000 GBP out of the box. https://www.avantek.co.uk/store/avantek-thunderx2-arm-workstation-thunderx2station.html https://www.avantek.co.uk/store/avantek-thunderx2-arm-workst... (They have ThunderX2 blades on the server section of their store with no prices yet, however.)
- shaklee3 8y ago
  You can try the ThunderX on packet.net.
alberth 8y ago
I thought TDP was a selling point of ARM for servers. Cavium has a higher TDP (180W) than either of the Intel Xeon's listed in the article (150W, 165W). Is this more about "performance per watt" then? EDIT: Also makes me wonder just how much more energy efficient and/or performant Cavium would be if they moved away from TSMC 16nm fab over to Samsung 10nm fab (like Qualcomm). There's a sizeable difference between 16nm vs 10nm (assuming the fabs are using the same metrics). Qualcomm using Samsung 10nm fab is only 120W.
- Symmetry 8y ago
  Small microservers with cellphone class cores in them were something a few companies pursued but they never really took off. And there was one company with Atom based servers in the same space, it wasn't just all ARM A9s. This one is about trading off minimum latency for more throughput with lots of more medium sized cores in a way that's probably better suited to most server workloads.
- notacoward 8y ago
  Yes, it's about performance per watt, and also performance per dollar. The Intel chips lose badly by that measure. Also, TDP isn't necessarily the right figure of merit here. It's defined and measured differently by different vendors, so it's not always directly comparable. The more important and more directly comparable measure is actual system-wide consumption when running a defined workload.
  - ajross 8y ago
    > The Intel chips lose badly by that measure. That's true, but then the Intel chips shown in that chart are Xeon Gold/Platinum parts that heretofore have had essentially no competition in the market and thus command huge premiums. Intel is making crazy bank on its server parts, basically. So in the presence of real competition (which this looks likely to be, though to be fair it's not actually in the market yet and is likely priced at a point intended to grab eyeballs), it's likely that Xeon parts will drop in price. You won't be looking at that delta for long if Intel starts losing market share. So... if they were merely cost-equivalent, is this part worth it? Because the pure silicon metrics here (performance per watt and per die area), as mentioned upthread, are sort of a wash. And maturity/stability issues (c.f. the bit in the linked article where they point out that the Cavium board draws 300W at idle!) kill products in the datacenter space all the time. Consumers are OK with occasional glitches, but no one is willing to put anything on their rack that has any kind of failure risk at all.
notacoward 8y ago
Between this and Zen, Intel is definitely starting to feel some heat. The next year or so is going to be interesting. Maybe Intel will recover; maybe we'll look back and see this as the point where they lost their grip.
- axaxs 8y ago
  I used to think this, but really don't see it happening. Intel has a lot of weight, money, and is pretty vertically integrated. They've been rather happy to stick where they're at, where the money is good. And no company is really threatening them there, and may not in the foreseeable future. But make no mistake, once that market begins to erode, they'll branch out more. Their mobile efforts have been mostly half assed, but I have no doubt they could cream both Apple and Qualcomm if they threw their resources at it.
alberth 8y ago
Can anyone speak to the techical merits (or disadvantages) of arranging your Cores by Mesh or Ring? I noticed all the ARM chips use a Ring bus whereas Intel uses a Mesh.
- dweekly 8y ago
  Fewer hops: https://software.intel.com/en-us/articles/intel-xeon-processor-scalable-family-technical-overview https://software.intel.com/en-us/articles/intel-xeon-process...
- Symmetry 8y ago
  Intel was using a ring too until their most recent architecture. Sometimes multiple intersecting rings when the number of cores grew too large.
- brandmeyer 8y ago
  Cost versus variation in memory access latency.
thisisit 8y ago
An important side note here is that Cavium has been bought by Marvell Semiconductors: https://in.reuters.com/article/cavium-m-a-marvell-technlgy/marvell-technology-to-buy-rival-chipmaker-cavium-for-6-billion-idINKBN1DK1TL https://in.reuters.com/article/cavium-m-a-marvell-technlgy/m... So future of ThunderX2 remains to be seen.
- wtallis 8y ago
  I don't think Marvell has anything that overlaps with ThunderX2, but they definitely have a lot of complementary products. I don't think they have any reason to cancel ThunderX2 if it looks like it can be successful.
auslander 8y ago
If it could Libreboot, that would be like first free of firmware blobs practical server in decades. Good Marvell bought it. Their ARM boards are already fw blob free.
- spacenick88 8y ago
  Have a look at Talos II, it's using POWER9 with completely open firmware
  - auslander 8y ago
    Checks out, I stand corrected, thanks. libreboot.org/news/talos.html
pmontra 8y ago
I know ARM processors shouldn't be subject to the recent vulnerabilities, or not as much as Intel and AMD, but I'm surprised that the article doesn't address that point. A decent CPU not vulnerable to Spectre, Meltdown and friends should be able to sell well only because of that.
- floatboth 8y ago
  The ISA doesn't matter very much. Meltdown is specifically Intel's fault. Spectre is, AFAIK, inherent to out-of-order execution. So the original ThunderX is not vulnerable, ThunderX 2 is.
yazr 8y ago
Can i get the SPECjbb benchmark mentioned in the article ? The web site claims that it is pay-only $1500 https://www.spec.org/order.html https://www.spec.org/order.html
monocasa 8y ago
> branch intensive code (databases, AI...) AI is branch intensive? Like, video game style state machines are, but that's not what people mean these days when the say AI in connection with servers, right?

99 points by drewg123 8y ago

jsgo 8y ago

sounds great, didn't I read that ARM was backing off on server though (hoping it was a bad article). edit: thanks, you all nailed it. I remember now and it was Qualcomm. Whew, I'm glad as I was bummed that this meant there was a good step for a piece of tech that would be DoA. Really wish Qualcomm wasn't abandoning based on how it appeared they were making progress as well to be a good competitor (and competition has been great where applied).

pertymcpert 8y ago
On the contrary. That was Qualcomm perhaps you read about.
mtgx 8y ago
Not Arm, but Qualcomm. It was also a strange and sudden decision. I don't think they really wanted to do it, but might have been forced to do it by the circumstances: anti-trust lawsuits, Apple dumping them, Broadcom attempting a hostile takeover, U.S. government pressuring them not to sell to Broadcom no matter what, etc. One or all of these may have forced Qualcomm to cut its losses. Either way, I hope they get to sell the Centriq line to some other company that can provide competition to both Cavium and Intel/AMD/IBM in the server space. We need more than one Arm chip to compete in the space, otherwise Cavium's chip could also be seen as some kind of outlier and be ignored by the industry.
- wyldfire 8y ago
  > One or all of these may have forced Qualcomm to cut its losses. Yeah, I think the LBO really scared Qualcomm. Their royalties fund a lot of the other work in the company and I think they've learned that it also makes them a target. A domestic LBO could happen and then there'd be no escape hatch. They pledged to cut costs when Broadcom announced their intentions, this may be a part of that. I don't think of them selling the server chip business as "cutting its losses" -- instead I would call it narrowing its focus back to businesses closer to its core competency. The Centriq launch with Amberwing seemed to be relatively positive for a first design release. They didn't turn the industry upside-down but they probably achieved most of what they set out to do. It will take some time to convince server customers to switch, recompile their code and get their vendors to recompile their code. > We need more than one Arm chip to compete in the space, otherwise Cavium's chip could also be seen as some kind of outlier and be ignored by the industry. Agreed. There's momentum that's been built [1,2] and I'd hate to see it flub now. [1] https://www.redhat.com/en/blog/red-hat-introduces-arm-server-support-red-hat-enterprise-linux https://www.redhat.com/en/blog/red-hat-introduces-arm-server... [2] https://www.suse.com/c/news/suse-steps-up-to-support-innovative-arm-solutions-for-customers/ https://www.suse.com/c/news/suse-steps-up-to-support-innovat...
  - jsgo 8y ago
    I hope they do something with it though. Whether sell it to someone who can be a good caretaker to it, share lessons and designs with other companies that could then build upon it, spinning it off, whatever they choose. Just abandoning it outright seems like an unfortunate move. I won't say bad, because I'm sure they have their reasons, but unfortunate in that it seemed like it was on track to be viable. Microsoft's Surface Pro line, to me, was kind of an "eh, that's nice I guess" up until the Pro 3 at which point it seemed like a platform that had hit its stride. The stuff I was reading on Amberwing seemed like it wasn't a silver bullet, but what it did work for, it worked very well when comparing price vs performance so one could hope that with time, they could make it even better.
- auslander 8y ago
  Or NSA was pushing for closed source system firmware a.k.a blobs

newprint 8y ago

I would curious, where can I buy this system ? Anyone one know resellers ? At $$$ listed, I would buy cheapest system in a heartbeat.

ajross 8y ago
This is a press release detailing the what appears to be the system under test: https://www.cavium.com/news/gigabyte-announces-thunderxstation-industry-s-first-armv8-workstation-based-on-cavium-s-thunderx2-processor https://www.cavium.com/news/gigabyte-announces-thunderxstati... I can't find a mention of it on Gigabyte's site, but there's an email there you could try. Honestly: this is a brand new device that clearly isn't actually in the channel yet. They shipped out test systems for press hits like this, it's not really a "product" quite yet.
rwmj 8y ago
Cavium are actually pretty good out of all the ARM server vendors of, you know, selling the hardware. (For some reason other vendors found that really hard.) The ThunderX2 has not been generally released yet, but I'm expecting it'll turn up on the same sites selling the old ThunderX, eg. https://www.avantek.co.uk/store/arm-servers.html https://www.avantek.co.uk/store/arm-servers.html
- aseipp 8y ago
  Out of curiosity I looked, and they actually already have pricing for the ThunderX2 Workstation from Gigabyte -- the very one mentioned in the article: about 10,000 GBP out of the box. https://www.avantek.co.uk/store/avantek-thunderx2-arm-workstation-thunderx2station.html https://www.avantek.co.uk/store/avantek-thunderx2-arm-workst... (They have ThunderX2 blades on the server section of their store with no prices yet, however.)
shaklee3 8y ago
You can try the ThunderX on packet.net.

alberth 8y ago

I thought TDP was a selling point of ARM for servers. Cavium has a higher TDP (180W) than either of the Intel Xeon's listed in the article (150W, 165W). Is this more about "performance per watt" then? EDIT: Also makes me wonder just how much more energy efficient and/or performant Cavium would be if they moved away from TSMC 16nm fab over to Samsung 10nm fab (like Qualcomm). There's a sizeable difference between 16nm vs 10nm (assuming the fabs are using the same metrics). Qualcomm using Samsung 10nm fab is only 120W.

Symmetry 8y ago
Small microservers with cellphone class cores in them were something a few companies pursued but they never really took off. And there was one company with Atom based servers in the same space, it wasn't just all ARM A9s. This one is about trading off minimum latency for more throughput with lots of more medium sized cores in a way that's probably better suited to most server workloads.
notacoward 8y ago
Yes, it's about performance per watt, and also performance per dollar. The Intel chips lose badly by that measure. Also, TDP isn't necessarily the right figure of merit here. It's defined and measured differently by different vendors, so it's not always directly comparable. The more important and more directly comparable measure is actual system-wide consumption when running a defined workload.
- ajross 8y ago
  > The Intel chips lose badly by that measure. That's true, but then the Intel chips shown in that chart are Xeon Gold/Platinum parts that heretofore have had essentially no competition in the market and thus command huge premiums. Intel is making crazy bank on its server parts, basically. So in the presence of real competition (which this looks likely to be, though to be fair it's not actually in the market yet and is likely priced at a point intended to grab eyeballs), it's likely that Xeon parts will drop in price. You won't be looking at that delta for long if Intel starts losing market share. So... if they were merely cost-equivalent, is this part worth it? Because the pure silicon metrics here (performance per watt and per die area), as mentioned upthread, are sort of a wash. And maturity/stability issues (c.f. the bit in the linked article where they point out that the Cavium board draws 300W at idle!) kill products in the datacenter space all the time. Consumers are OK with occasional glitches, but no one is willing to put anything on their rack that has any kind of failure risk at all.

notacoward 8y ago

Between this and Zen, Intel is definitely starting to feel some heat. The next year or so is going to be interesting. Maybe Intel will recover; maybe we'll look back and see this as the point where they lost their grip.

axaxs 8y ago
I used to think this, but really don't see it happening. Intel has a lot of weight, money, and is pretty vertically integrated. They've been rather happy to stick where they're at, where the money is good. And no company is really threatening them there, and may not in the foreseeable future. But make no mistake, once that market begins to erode, they'll branch out more. Their mobile efforts have been mostly half assed, but I have no doubt they could cream both Apple and Qualcomm if they threw their resources at it.

alberth 8y ago

Can anyone speak to the techical merits (or disadvantages) of arranging your Cores by Mesh or Ring? I noticed all the ARM chips use a Ring bus whereas Intel uses a Mesh.

dweekly 8y ago
Fewer hops: https://software.intel.com/en-us/articles/intel-xeon-processor-scalable-family-technical-overview https://software.intel.com/en-us/articles/intel-xeon-process...
Symmetry 8y ago
Intel was using a ring too until their most recent architecture. Sometimes multiple intersecting rings when the number of cores grew too large.
brandmeyer 8y ago
Cost versus variation in memory access latency.

thisisit 8y ago

An important side note here is that Cavium has been bought by Marvell Semiconductors: https://in.reuters.com/article/cavium-m-a-marvell-technlgy/marvell-technology-to-buy-rival-chipmaker-cavium-for-6-billion-idINKBN1DK1TL https://in.reuters.com/article/cavium-m-a-marvell-technlgy/m... So future of ThunderX2 remains to be seen.

wtallis 8y ago
I don't think Marvell has anything that overlaps with ThunderX2, but they definitely have a lot of complementary products. I don't think they have any reason to cancel ThunderX2 if it looks like it can be successful.

auslander 8y ago

If it could Libreboot, that would be like first free of firmware blobs practical server in decades. Good Marvell bought it. Their ARM boards are already fw blob free.

spacenick88 8y ago
Have a look at Talos II, it's using POWER9 with completely open firmware
- auslander 8y ago
  Checks out, I stand corrected, thanks. libreboot.org/news/talos.html

pmontra 8y ago

I know ARM processors shouldn't be subject to the recent vulnerabilities, or not as much as Intel and AMD, but I'm surprised that the article doesn't address that point. A decent CPU not vulnerable to Spectre, Meltdown and friends should be able to sell well only because of that.

floatboth 8y ago
The ISA doesn't matter very much. Meltdown is specifically Intel's fault. Spectre is, AFAIK, inherent to out-of-order execution. So the original ThunderX is not vulnerable, ThunderX 2 is.

yazr 8y ago

Can i get the SPECjbb benchmark mentioned in the article ? The web site claims that it is pay-only $1500 https://www.spec.org/order.html https://www.spec.org/order.html

monocasa 8y ago

> branch intensive code (databases, AI...) AI is branch intensive? Like, video game style state machines are, but that's not what people mean these days when the say AI in connection with servers, right?