4 ms·
In terms of terraflops, AWS likely has the largest GPU compute capacity out there, and they could use idle GPU capacity to train their models, and Anthorpic has
by carlsborg 4y ago
In terms of terraflops, AWS likely has the largest GPU compute capacity out there, and they could use idle GPU capacity to train their models, and Anthorpic has some top tier researchers building it. Give it time.
- solomatov 4y agoIt's not that simple. You couldn't just use any random GPU in the cloud, you need to connect them using the right kind of interconnect (Mellanox bough by NVidia) which is very expensive.
- lightbendover 4y agoAs companies reduce their tech expenditure, there will be plenty of excess capacity to collocate and properly connect by high-speed switches. There is also a lot of research (not sure it by Amazon as well) going into better distributes model training.
- deleted 4y ago[deleted]
- carlsborg 4y agoIndeed. The AWS p4 instances are wired together by the next generation NVLink/NLink internconnect (600 GB/s bidirectional bandwidth) and also the Elastic Fabric Adapters (400 GB/s) which is likely Mellanox under the covers.
- Moissanite 4y agoEFA is not InfiniBand. It has similar goals (low latency via kernel bypass), but is implemented using AWS Nitro NICs and regular Ethernet networking. Other providers (Azure and Oracle) do offer InfiniBand, but that creates a problem with "islands" of compute which can't be easily expanded. There are also more security considerations when using IB in a multi-tenant environment which makes it not such a slam-dunk advantage as you might first assume.
- carlsborg 3y agoYou're right, EFA is custom nitro NICs. Although Mellanox (now Nvidia ConnectX) is not solely Inifiband either - Nvidia offers them with Ethernet upto 400GbE as well, same speed as EFA says on the label.