3 ms·

When you look at the moat for these companies, is it really justified? I just think about ARM and Intel/AMD. I find it hard to believe that existing GPUs with

When you look at the moat for these companies, is it really justified? I just think about ARM and Intel/AMD. I find it hard to believe that existing GPUs with their memory bandwidth considerations are optimal architecture and that in 20y we will have the same compute model But the last time I did any work with cuda was a decade ago, so very rough ideas here

tomoyoirl 3y ago
I’m not sure of all the details of the architecture, but my understanding is that there’s all sorts of “GPU Direct Storage” and massive on-chip CPU/GPU shared memory spaces on the new GH100s, plus all the Mellanox stuff for interconnects. At least in the datacenter space, anyway.
- nostrebored 3y ago
  Thanks! The last time I cared about this I was doing SGD on GPU clusters and memory bandwidth was the limiting factor. Almost the entirety of the work was focused on chunking the problem so that a maximal number of subproblems was on the GPU. If storage really is becoming integrated at similar speed (EG not on an order of magnitude difference like L2 vs RAM) then I think I'm wrong
jillesvangurp 3y ago
Exactly. Nvidia's current dominance is based on the notion that leading AI related libraries and tools depend on Cuda and run well on that. But that's not something that will last. Many parties are working on making these libraries work against alternative architectures and untangling them from Nvidia specifics. Apple's laptops for example are pretty popular with people into LLMs. Why? They are good laptops with decent enough hardware and lots of people have them. The key point with that is that there's no nvidia hardware on Apple's platform and that a lot of work seems to be going into making sure Apple silicon is supported well in all relevant libraries and tools. And they are of course not the only ones. A lot of the bigger AI companies are using their own chips and platforms. Tesla, Google, Microsoft, Amazon, etc. And then there are Intel and AMD that are equally eager to grab more of the AI market. Add to that the emerging open source ecosystem around LLMs and you have a nice recipe for vendor neutral libraries used by world+dog and users simply cherry picking the market for the best hardware that can run those libraries. Nvidia has enjoyed a nice software moat for the last years but it won't last. Everyone is trying to engineer around them currently.
alecco 3y ago
They are way ahead and gaining speed: https://news.ycombinator.com/item?id=39355625 https://news.ycombinator.com/item?id=39355625 And also the integration. The Nvidia datacenter clusters are ready as soon as you turn them on, unlike every other vendor until now requiring very difficult fine tuning for months.

tomoyoirl 3y ago

I’m not sure of all the details of the architecture, but my understanding is that there’s all sorts of “GPU Direct Storage” and massive on-chip CPU/GPU shared memory spaces on the new GH100s, plus all the Mellanox stuff for interconnects. At least in the datacenter space, anyway.

nostrebored 3y ago
Thanks! The last time I cared about this I was doing SGD on GPU clusters and memory bandwidth was the limiting factor. Almost the entirety of the work was focused on chunking the problem so that a maximal number of subproblems was on the GPU. If storage really is becoming integrated at similar speed (EG not on an order of magnitude difference like L2 vs RAM) then I think I'm wrong

jillesvangurp 3y ago

Exactly. Nvidia's current dominance is based on the notion that leading AI related libraries and tools depend on Cuda and run well on that. But that's not something that will last. Many parties are working on making these libraries work against alternative architectures and untangling them from Nvidia specifics. Apple's laptops for example are pretty popular with people into LLMs. Why? They are good laptops with decent enough hardware and lots of people have them. The key point with that is that there's no nvidia hardware on Apple's platform and that a lot of work seems to be going into making sure Apple silicon is supported well in all relevant libraries and tools. And they are of course not the only ones. A lot of the bigger AI companies are using their own chips and platforms. Tesla, Google, Microsoft, Amazon, etc. And then there are Intel and AMD that are equally eager to grab more of the AI market. Add to that the emerging open source ecosystem around LLMs and you have a nice recipe for vendor neutral libraries used by world+dog and users simply cherry picking the market for the best hardware that can run those libraries. Nvidia has enjoyed a nice software moat for the last years but it won't last. Everyone is trying to engineer around them currently.

alecco 3y ago

They are way ahead and gaining speed: https://news.ycombinator.com/item?id=39355625 https://news.ycombinator.com/item?id=39355625 And also the integration. The Nvidia datacenter clusters are ready as soon as you turn them on, unlike every other vendor until now requiring very difficult fine tuning for months.