4 ms·
This seemed like a big topic when I was interviewing with Meta and nVidia some months ago. Meta had a few good YouTube videos about the problems of dealing wit
by CoastalCoder 3y ago
This seemed like a big topic when I was interviewing with Meta and nVidia some months ago.
Meta had a few good YouTube videos about the problems of dealing with this many GPUs at scale.
- keefle 3y agoCould you link me the YouTube videos/articles in question? It happens to be my research area and I'm interested in knowing how big companies such as meta deal with multi-GPU systems
- CoastalCoder 3y agoI don't have them bookmarked anymore, but they may have been from this playlist: [0] [0] https://www.youtube.com/playlist?list=PLBnLThDtSXOw_kePWy3CSG7LLf3KdGqQz https://www.youtube.com/playlist?list=PLBnLThDtSXOw_kePWy3CS...
- keefle 3y agoThank you for sharing! I'll hunt it down
- mackid 3y agoMark did a good video on ChatGPT infra. [1]. https://techcommunity.microsoft.com/t5/microsoft-mechanics-blog/what-runs-chatgpt-inside-microsoft-s-ai-supercomputer-featuring/ba-p/3830281 https://techcommunity.microsoft.com/t5/microsoft-mechanics-b...
- rlupi 3y agoThank you all for the replies. I picked this one, but the answer is for every other reply. I didn't know about some of these links. Solving these problems has basically been by job for the past 6 years, for Google's TPU systems and some of the GPU systems (the non-Cloud ones). It is a pity that, after the pandemic, it has been impossible to give similar conference talks at my company (at least if you are employed in Europe, due to restrictions on travel outside one's country). The Meta sessions are very interesting. I wasn't aware of their work on Arcadia; Nvidia has also their own systems (Nvidia Air / Omniverse) in this area.