4 ms·
So this is the secret sauce for speeding up AI stuff, right? Memory bandwidth is the thing that for training large language models or diffusion models, you run
by wincy 3y ago
So this is the secret sauce for speeding up AI stuff, right? Memory bandwidth is the thing that for training large language models or diffusion models, you run up against? Or is there other stuff? What would a perfect machine look like for generating the next Stable Diffusion or ChatGPT5? They train on H100X8 machines with HBM3 right now, is that correct?
- xnx 3y agoThe linked part of this presentation goes into detail on best available machine learning hardware: https://www.youtube.com/watch?v=EFe7-WZMMhc&t=1217s https://www.youtube.com/watch?v=EFe7-WZMMhc&t=1217s
- bobim 3y agoComputational Fluid Dynamics and implicit Finite Elements Analysis are still severely memory bandwidh limited on our 128 cores zen3 dual sockets servers. More bandwidth is always welcome.
- kookamamie 3y agoZen 3, as in Milan? Upgrade to Genoa and you'll see a very nice bump in performance due to DDR5 support with 12 memory channels, plus due to the added AVX512 instruction set.
- bayindirh 3y agoThe problem is, unless you reorder and optimize your matrices to make the prefetcher happy, you end up throwing a lot of data, because you don't access the memory in the linear fashion where the prefetchers assume. So, having more channels is always nice, but this "more cowbells" approach doesn't bring in the performance you expect in some cases.
- markhahn 3y agomore channels (and the fact that each ddr5 dimm is basically two channels) is a major win that doesn't depend on prefetching. it is perhaps underappreciated that CPU cores have gotten a lot more "thready" even in the absence of SMT, since each core has hundreds of instructions in flight. you know, kind of like how GPU cores (CUs, SMs) do...
- bayindirh 3y agoIt depends on prefetching very much. I have a similar simulation code, which can saturate the cores pretty efficiently (It was running with >1.7M integrations/sec/core and scaling linearly on last decade's hardware). But when you access to square submatrices inside a 3000x3000 matrix, and CPU prefetcher brings the whole n rows, and you discard most of the rows to get another part of the matrix, that bandwidth is wasted. Instead, you can rearrange your matrices to make the prefetcher happy, and do not throw what prefetcher brings, because you make it bring which is going to be used next. When I was doing last analysis runs, it was running with ~20% cache trash meaning I'm wasting 20% of the data I'm pulling in regardless. This is a huge loss, considering I was saturating memory bandwidth before the cores.
- mastax 3y agoWhich is an interesting aside, but unless I'm missing something the amount of wastage due to the prefetcher isn't at all proportional to the number of available memory channels so it's non sequitur.
- bayindirh 3y agoNo, the wastage is due to because I'm not accessing data in a "linear" fashion if I lay out the matrices naively (actually pulling n by n square submatrices from the big matrix), and prefetcher brings in data with its baked in locality rules, but its assumptions doesn't fit my access pattern. So 20 percent of the data is thrown away. If I change how I fill and lay my matrices out, I can align my data with how prefetcher assumes, and way less data will be trashed during the process. I didn't do that because it needed a codebase-wide refactorization, and my code was already 30x faster than reference implementation with better accuracy, so we didn't pursue that avenue.
- bobim 3y agoI guess money is the rate determining step here. Secondary issue is that cores are also getting faster and then need more bandwidth. Solution is a low core count, maybe 2-3 per channel, more servers and serious interconnect, but cost…
- Scene_Cast2 3y agoI've heard that running CFD on thr CPU is still quite popular and prevalent, despite GPU implementations being widely available. What is it about the algorithm that makes GPUs less suitable? Model sizes? Branching? Data (non-)locality?
- bobim 3y agoMemory is one limiting factor, needing over 512 Gb is not uncommon. But complex physics like chemical reactions, particle tracking, any user field functions, is probably what is tricky to implement on a GPU.
- brucethemoose2 3y ago> What would a perfect machine look like for generating the next Stable Diffusion or ChatGPT5? A recent Microsoft paper proposed that the "perfect" inference machine is a ton of large-ish, modestly clocked, SRAM heavy chips piped together on a motherboard. There is no RAM. The model weights are split up between the chips, and operations are pipelined like they already are in llms that are split into "layers". And there is no expensive interposer, or crazy interconnect like Cerebras. The chips are just wired together on a cheap motherboard, as the bandwidth between layers is relatively modest. ...It seems like this would work for training too. But what do I know? If it doesn't, the next "ideal" architecture is probably Cerebras's wafer-scale machines. And I am thinking they make use of TSMC's TSV tech (like AMD's X3D chips) to cram more SRAM onto a single wafer. Theres also some fuss asserting that doing a bunch of matrix multiplication is an inefficient way to emulate physical neural networks, so maybe future architectures will be radically different?
- nimish 3y agoCerebras' strategy is the winner then if hardware was all that mattered
- brucethemoose2 3y agoYeah, I guess in the reality the perfect hardware is "an Nvidia GPU with sufficient VRAM" because its everywhere and already works by default.