4 ms·

> AMD is attempting to vertically integrate next year with their upcoming Pollara 400G NIC, which supports Ultra Ethernet, hopefully making AMD competitive with

by ryao 2y ago

> AMD is attempting to vertically integrate next year with their upcoming Pollara 400G NIC, which supports Ultra Ethernet, hopefully making AMD competitive with Nvidia. Infiniband is an industry standard. It is weird to see the industry invent yet another standard to do effectively the same thing just because Nvidia is using it. This “Nvidia does things this way so let’s do it differently” mentality is hurting AMD: * Nvidia has a unified architecture so let’s split ours into RDNA and CDNA. * Nvidia has a unified driver, so let’s make a different driver for every platform. * Nvidia made a virtual ISA (PTX) for backward compatibility. Let’s avoid that. * Nvidia is implementing tensor cores. Let’s avoid those on RDNA. Then implement them on CDNA and call them matrix cores. * Nvidia is using Infiniband like the rest of the HPC community. Let’s use Ethernet. I am sure people can find more examples. Also, they seem to have realized their mistake in splitting their architecture into RDNA and CDNA, since they are introducing UDNA in the future to unify them like Nvidia does.

dralley 2y ago
You're painting this like AMD is off to play in their own sandbox when it's more like the entire industry is trying to develop an alternative to Infiniband. Ultra Ethernet is a joint project between dozens of companies organized under the Linux Foundation. https://www.phoronix.com/news/Ultra-Ethernet-Consortium https://www.phoronix.com/news/Ultra-Ethernet-Consortium >> The Linux Foundation has established the Ultra Ethernet Consortium "UED" as an industry-wide effort founded by AMD, Arista, Broadcom, Cisco, Eviden, HPE, Intel, Meta, and Microsoft for designing a new Ethernet-based communication stack architecture for high performance networking. You probably can't call it "industry standard" yet but the goal is obviously for it to become one.
- ryao 2y ago
  I wrote: > It is weird to see the industry invent yet another standard to do effectively the same thing just because Nvidia is using it. This is a misstep for all involved, AMD included. Even if AMD is following everyone else by jumping off a bridge, AMD is still jumping too.
croes 2y ago
You hardly beat someone by copying him. They have way more experience in the field you try to catch up.
- imtringued 2y ago
  AMD doesn't need to beat Nvidia, they just need to match them at a lower price point.
  - croes 2y ago
    That‘s beating them at the price point.
  - sokoloff 2y ago
    In business, that combination is nearly impossible to distinguish from beating them.
- ryao 2y ago
  You don’t beat someone by doing everything worse either.
mixblast 2y ago
Infiniband is a monopoly from NVidia (Mellanox). Everyone else would much rather use Ethernet which is the actual industry standard.
- iszomer 2y ago
  Depends on deployment: Ethernet has more drawbacks encapsulating PCIe packet traffic than Infiniband does, or lack thereof with RDMA.
- ryao 2y ago
  Others can build infiniband hardware according to the standard. There used to be at least two companies building infiniband hardware until Intel killed QLogic’s infiniband division in a misguided attempt to make its own monopoly. :/
  - mrlongroots 2y ago
    Or they can develop a RoCEv2 that works (which is basically what UltraEthernet is). Infiniband is not fun - it's a special snowflake of an interconnect that sits parallel to the rest of your datacenter network, and can not really run a standard TCP/IP codebase (yeah IPoIB is a thing but still). Do the Nvidia boxes really need a scale-out IP network as well as an Infiniband network? Plus the spec is old. Packet spraying and trimming, better ordering guarantees, queue pair scalability... a whole bunch of enhancements have been incorporated into UE all the while being compatible with regular Ethernet. Qlogic was never really an Infiniband vendor --- their qib driver is still in the Linux codebase and essentially emulates verbs on top of a messaging-based design.
    - ryao 2y ago
      We must agree to disagree then. Infiniband is an awesome technology. It was originally meant to to serve as a fabric to connect all components in a computer. That plan died with the dotcom bubble as it was too ambitious and therefore risky, so it was relegated to networking where it was an incredibly good job in delivering messages reliably and fast. It still am hopeful someone will do that someone will do it.
Flenkno 2y ago
Nvidia has a unique problem, wants to move fast and has a shit load of money. No need for Nvidia to go first to an industry standard and neither for AMD. Personally would be great its getting backported but its so far away from an normal use case.
- ryao 2y ago
  Nvidia is the one who went to an industry standard before their competitors in this space. It was created in 1999 and is called infiniband: https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/InfiniBand Infiniband is extremely popular in the HPC space, which is why Nvidia adopted it. Everyone else saw Nvidia adopt it and said "Let us make a new network standard to be incompatible". This is mind boggling. Even more mind boggling is that many of the companies in the Ultra Ethernet Consortium are members of the Infiniband Trade Association, AMD included: https://www.infinibandta.org/member-listing/ https://www.infinibandta.org/member-listing/ This would be like the automotive industry forming a consortium to invent new incompatible wheels to exclude a successful upstart that adopted their existing standard wheel designs. With trillions of dollars in revenue on the line, you would think that companies would use existing networking standards to focus on building competitive hardware with reduced time to market, yet they are instead reinventing networking standards just because they can. This is a huge gift to Nvidia, since it means that everyone else is wasting time and money instead of being competitive.
  - mrlongroots 2y ago
    Sorry to hound you for the third time, but this is wrong: > Infiniband is extremely popular in the HPC space Not anymore. There used to be Cray Aries/GNI, psm/psm2, and now there's Slingshot, the new Cornelis stuff etc. There's almost no Infiniband now.
    - ryao 2y ago
      The top500 says otherwise: https://www.infinibandta.org/infiniband-and-roce-advances-further-in-the-top500-november-2024-list/ https://www.infinibandta.org/infiniband-and-roce-advances-fu... Where are you getting your information?
mrlongroots 2y ago
> Infiniband is an industry standard Infiniband is not an industry standard lol. Maybe it used to be, but it definitely is not anymore. Most Infiniband vendors are dead. The only product from those days that endures is Cornelis' Omnipath, and even that only emulated the Infiniband API back with its first gen, and then evolved to be its own thing. At this point, Infiniband is as good as a proprietary interconnect only sold by Nvidia/Mellanox.
- ryao 2y ago
  Infiniband is very much an industry standard: https://www.infinibandta.org/member-listing/ https://www.infinibandta.org/member-listing/ As far as I know, everyone is free to sign up with the infiniband trade association and implement the specification. Furthermore, if you use RDMA over Ethernet, you are using infiniband at a low level. RoCE which enables it was originally called Ethernet over infiniband. It is maintained by the infiniband trade association. Omnipath was Intel’s failed effort to try to kill an open standard. It purchased QLogic’s infiniband business, killed it in favor of omnipath and sold it when it failed.
  - mrlongroots 2y ago
    > Furthermore, if you use RDMA over Ethernet, you are using infiniband at a low level. RoCE which enables it was originally called Ethernet over infiniband. This is wrong. RDMA over Ethernet is... RDMA over Ethernet. There is no Infiniband involved. RoCE was motivated by supporting RDMA, which was then an IB-only feature, over regular Ethernet. The user-level APIs are the same (verbs), but the underlying architecture is all different --- it is traditional Ethernet with link-level flow control to make it lossless (pause frames).
    - ryao 2y ago
      This is not what I have been told by others.
      - mrlongroots 2y ago

by ryao 2y ago

dralley 2y ago

You're painting this like AMD is off to play in their own sandbox when it's more like the entire industry is trying to develop an alternative to Infiniband. Ultra Ethernet is a joint project between dozens of companies organized under the Linux Foundation. https://www.phoronix.com/news/Ultra-Ethernet-Consortium https://www.phoronix.com/news/Ultra-Ethernet-Consortium >> The Linux Foundation has established the Ultra Ethernet Consortium "UED" as an industry-wide effort founded by AMD, Arista, Broadcom, Cisco, Eviden, HPE, Intel, Meta, and Microsoft for designing a new Ethernet-based communication stack architecture for high performance networking. You probably can't call it "industry standard" yet but the goal is obviously for it to become one.

ryao 2y ago
I wrote: > It is weird to see the industry invent yet another standard to do effectively the same thing just because Nvidia is using it. This is a misstep for all involved, AMD included. Even if AMD is following everyone else by jumping off a bridge, AMD is still jumping too.

croes 2y ago

You hardly beat someone by copying him. They have way more experience in the field you try to catch up.

imtringued 2y ago
AMD doesn't need to beat Nvidia, they just need to match them at a lower price point.
- croes 2y ago
  That‘s beating them at the price point.
- sokoloff 2y ago
  In business, that combination is nearly impossible to distinguish from beating them.
ryao 2y ago
You don’t beat someone by doing everything worse either.

mixblast 2y ago

Infiniband is a monopoly from NVidia (Mellanox). Everyone else would much rather use Ethernet which is the actual industry standard.

iszomer 2y ago
Depends on deployment: Ethernet has more drawbacks encapsulating PCIe packet traffic than Infiniband does, or lack thereof with RDMA.
ryao 2y ago
Others can build infiniband hardware according to the standard. There used to be at least two companies building infiniband hardware until Intel killed QLogic’s infiniband division in a misguided attempt to make its own monopoly. :/
- mrlongroots 2y ago
  Or they can develop a RoCEv2 that works (which is basically what UltraEthernet is). Infiniband is not fun - it's a special snowflake of an interconnect that sits parallel to the rest of your datacenter network, and can not really run a standard TCP/IP codebase (yeah IPoIB is a thing but still). Do the Nvidia boxes really need a scale-out IP network as well as an Infiniband network? Plus the spec is old. Packet spraying and trimming, better ordering guarantees, queue pair scalability... a whole bunch of enhancements have been incorporated into UE all the while being compatible with regular Ethernet. Qlogic was never really an Infiniband vendor --- their qib driver is still in the Linux codebase and essentially emulates verbs on top of a messaging-based design.
  - ryao 2y ago
    We must agree to disagree then. Infiniband is an awesome technology. It was originally meant to to serve as a fabric to connect all components in a computer. That plan died with the dotcom bubble as it was too ambitious and therefore risky, so it was relegated to networking where it was an incredibly good job in delivering messages reliably and fast. It still am hopeful someone will do that someone will do it.

Flenkno 2y ago

Nvidia has a unique problem, wants to move fast and has a shit load of money. No need for Nvidia to go first to an industry standard and neither for AMD. Personally would be great its getting backported but its so far away from an normal use case.

ryao 2y ago
Nvidia is the one who went to an industry standard before their competitors in this space. It was created in 1999 and is called infiniband: https://en.wikipedia.org/wiki/InfiniBand https://en.wikipedia.org/wiki/InfiniBand Infiniband is extremely popular in the HPC space, which is why Nvidia adopted it. Everyone else saw Nvidia adopt it and said "Let us make a new network standard to be incompatible". This is mind boggling. Even more mind boggling is that many of the companies in the Ultra Ethernet Consortium are members of the Infiniband Trade Association, AMD included: https://www.infinibandta.org/member-listing/ https://www.infinibandta.org/member-listing/ This would be like the automotive industry forming a consortium to invent new incompatible wheels to exclude a successful upstart that adopted their existing standard wheel designs. With trillions of dollars in revenue on the line, you would think that companies would use existing networking standards to focus on building competitive hardware with reduced time to market, yet they are instead reinventing networking standards just because they can. This is a huge gift to Nvidia, since it means that everyone else is wasting time and money instead of being competitive.
- mrlongroots 2y ago
  Sorry to hound you for the third time, but this is wrong: > Infiniband is extremely popular in the HPC space Not anymore. There used to be Cray Aries/GNI, psm/psm2, and now there's Slingshot, the new Cornelis stuff etc. There's almost no Infiniband now.
  - ryao 2y ago
    The top500 says otherwise: https://www.infinibandta.org/infiniband-and-roce-advances-further-in-the-top500-november-2024-list/ https://www.infinibandta.org/infiniband-and-roce-advances-fu... Where are you getting your information?

mrlongroots 2y ago

> Infiniband is an industry standard Infiniband is not an industry standard lol. Maybe it used to be, but it definitely is not anymore. Most Infiniband vendors are dead. The only product from those days that endures is Cornelis' Omnipath, and even that only emulated the Infiniband API back with its first gen, and then evolved to be its own thing. At this point, Infiniband is as good as a proprietary interconnect only sold by Nvidia/Mellanox.

ryao 2y ago
Infiniband is very much an industry standard: https://www.infinibandta.org/member-listing/ https://www.infinibandta.org/member-listing/ As far as I know, everyone is free to sign up with the infiniband trade association and implement the specification. Furthermore, if you use RDMA over Ethernet, you are using infiniband at a low level. RoCE which enables it was originally called Ethernet over infiniband. It is maintained by the infiniband trade association. Omnipath was Intel’s failed effort to try to kill an open standard. It purchased QLogic’s infiniband business, killed it in favor of omnipath and sold it when it failed.
- mrlongroots 2y ago
  > Furthermore, if you use RDMA over Ethernet, you are using infiniband at a low level. RoCE which enables it was originally called Ethernet over infiniband. This is wrong. RDMA over Ethernet is... RDMA over Ethernet. There is no Infiniband involved. RoCE was motivated by supporting RDMA, which was then an IB-only feature, over regular Ethernet. The user-level APIs are the same (verbs), but the underlying architecture is all different --- it is traditional Ethernet with link-level flow control to make it lossless (pause frames).
  - ryao 2y ago
    This is not what I have been told by others.
    - mrlongroots 2y ago