7 ms·
Disclosure: HPC admin who works with NIVIDA cards here. Because, no. It's not as simple as that. NVIDIA has a complete ecosystem now. They have cards. They ha
by bayindirh 2y ago
Disclosure: HPC admin who works with NIVIDA cards here.
Because, no. It's not as simple as that.
NVIDIA has a complete ecosystem now. They have cards. They have cards of cards (platforms), which they produce, validate and sell. They have NVLink crossbars and switches which connects these cards on their card of cards with very high speeds and low latency.
For inter-server communication they have libraries which coordinate cards, workloads and computations.
They bought Mellanox, but that can be used by anyone, so there's no lock-in for now.
As a tangent, NVIDIA has a whole set of standards for pumping tremendous amount of data in and out of these mesh of cards. Let it be GPU-Direct storage or specialized daemons which handle data transfers on and off cards.
If you think that you can connect n cards on PCIe bus and just send workloads to them and solve problems magically, you'll hurt yourself a lot, both performance and psychology wise.
You have to build a stack which can perform these things with maximum possible performance to be able to compute with NVIDIA. It's not just emulating CUDA, now. Esp., on the high end of the AI spectrum (GenAI, MultiCard, MultiSystem, etc.).
For other lower end, multi-tenant scenarios, they have card virtualization, MIG, etc. for card sharing. You have to complete on that, too, for cloud and smaller applications.
- dgfitz 2y agoHow does any of this make money?
- arcticbull 2y agoHaving the complete ecosystem affords them significant margins.
- dgfitz 2y agoAgainst what?
- arcticbull 2y agoAs of today they have SaaS company margins as a hardware company which is practically unheard of.
- dgfitz 2y agoI mean, do they?
- deleted 2y ago[deleted]
- lyime 2y agoWhat? It's like the most profitable set of products in tech. You have companies like Meta, MSFT, Amazon, Google etc spending $5B every few years buying this hardware.
- dgfitz 2y agoStale money is moving around. Nothing changed .
- HeatrayEnjoyer 2y agoWhat is stale money?
- bayindirh 2y agoWhen this walled garden is the only way to use GPUs with high efficiency and everybody is using this stack, and NVIDIA controlling the supply of these "platform boards" to OEMs, they don't make money, but they literally print it. However, AMD is coming for them because a couple of high profile supercomputer centers (LUMI, Livermore, etc.) are using Instinct cards and pouring money to AMD to improve their cards and stack. I have not used their (Instinct) cards, yet, but their Linux driver architecture is way better than NVIDIA.
- ryao 2y agoI think he was asking how HPC makes money. The answer as far as I know is that it often does not for anyone other than the vendors. I wonder where you got your information on AMD’s “Linux driver architecture”. It is reportedly a mess: https://news.ycombinator.com/item?id=34832660 https://news.ycombinator.com/item?id=34832660 So far, I have been very happy with Nvidia’s Linux drivers.
- ryao 2y agoHPC is used in research, which is often not expected to make money. The hope is that the research will result in something that makes money. One example would be drug discovery. Another would be weather prediction, which is not so much a way to make money, but to minimize losses.
- throwaway48476 2y agoAll of that is highly relevant for training but what the poster was asking for is a desktop inference card.
- bayindirh 2y agoYou use at least half of this stack for desktop setups. You need copying daemons, the ecosystem support (docker-nvidia, etc.), some of the libraries, etc. even when you're on a single system. If you're doing inference on a server; MIG comes into play. If you're doing inference on a larger cloud, GPU-direct storage comes into play. It's all modular.
- WanderPanda 2y agoNo you don‘t need much bandwidth between cards for inference
- bayindirh 2y agoCopying daemons (gdrcopy) is about pumping data in and out of a single card. docker-nvidia and rest of the stack is enablement for using cards. GPU-Direct is about pumping data from storage devices to cards, esp. from high speed storage systems across networks. MIG actually shares a single card to multiple instances, so many processes or VMs can use a single card for smaller tasks. Nothing I have written in my previous comment is related to inter-card, inter-server communication, but all are related to disk-GPU, CPU-GPU or RAM-CPU communication. Edit: I mean, it's not OK to talk about downvoting, and downvote as you like but, I install and enable these cards for researchers. I know what I'm installing and what it does. C'mon now. :D
- mikhael 2y agoMostly, I think, we don’t really understand your argument that Intel couldn’t easily replicate the parts needed only for inference.
- 2y ago
- postalrat 2y agoLets see how quickly that changes if intel releases cards with massive amounts of ram for a fraction of the cost.
- jmward01 2y agoMost of the above infra is predicated on limiting RAM so that you need so much communication between cards. Bump the RAM up and you could do single card inference and all those connections become overhead that could have gone to more ram. For training there is an argument still, but even there the more RAM you have the less all that connectivity gains you. RAM has been used to sell cards and servers for a long time now, it is time to open the floodgates.
- foobiekr 2y agoCorrect for inference - the main use of the interconnect is RDMA requests between GPUs to fit models that wouldn't otherwise fit. Not really correct for training - training has a lot of all-to-all problems, so hierarchical reduction is useful but doesn't really solve the incast problem - Nvlink _bandwidth_ is less of an issue than perhaps the SHARP functions in the NVLink switch ASICs.
- epistasis 2y agoRather than tackling the entire market at once, they could start with one section and build from there. NVIDIA didn't get to where it was in a year, it took many strategic acquisitions. (All the networking and other HPC-specialized stuff I was buying a decade ago has seemingly been bought by NVIDIA). Start by being a "second vendor" for huge customers of NVIDIA that want to foster competition, as well as a few others willing to take risks, and build from there.
- tw04 2y agoIntel has already bought and killed everything they need to compete here. They seem incapable of sticking to any market that isn’t x86. Likely because when they were making those acquisitions they were drunk on margin and didn’t want to focus anywhere else.
- ryao 2y agoKilling QLogic’s infiniband business after buying it was a major loss for the industry.
- RobotToaster 2y ago> They seem incapable of sticking to any market that isn’t x86 Even within that they seem to have difficulty expanding to new areas, remember edison?
- teekert 2y agoI have a question for you, since I’m somewhat entering the HPC world. In the EU the EuroHPC-JU is building what they call AI factories, afaict these are just batch processing (Slurm I think) clusters with GPUs in the nodes. So I wonder where you’d place those cards of cards. Are you saying there is another, perhaps better ways to use massive amounts of these cards? Or is that still in the “super powerful workstation” domain? Thanx in advance.
- treprinum 2y agoView it as Raspberry Pi for AI workloads. Initial stage is for enthusiasts that would develop the infra, figure out what is possible and spread the word. Then the next phase will be SME industry adoption, making it commercially interesting, while bypassing Nvidia completely. At some point it would live its own life and big players jump in. Classical disrupt strategy via low cost unique offerings.
- renecito 2y ago> ... local generative AI lunch I agree, just for my PC, something that'd enable small devs to create interesting foundation model apps that'd deploy to users using this local AI cards to run these new Apps.
- davrosthedalek 2y agoThere might be an hen-egg problem if the apps end up requiring a 128Gb AI accelerator card. You only get the card if there are apps to run, and you only develop the apps if the cards are wide-spread. With so much RAM, the cards will not be let's-through-them-into-a-cheap-build cheap. I think there have to be a couple of killer apps that run "OK" with CPU or GPU, but would run tremendously better with such a card.
- tgtweak 2y agoI think he's mostly referring to inference and not training, which I entirely agree with - a 4x version of this card for workstations would do really well - even some basic interconnect between the cards a la nvlink would really drive this home. The training can come after, with some inference and runtime optimizations on the software stack.
- thelastparadise 2y agoSo what you're saying is Intel, or any other would-be NVIDIA competitor, needs to put out fast interconnects, not just compute cards. This is true. I'm not sure your argument stands when it comes to OP's idea of a single card with 128GB VRAM. This would be enough to run ~180B models with reasonable quantization --we're not near maxing out the capability of 180B yet (see the latest 32B models performing near public SOTA). This indeed would push rapid and wide adoption and be quite disruptive. But sure, it wouldn't instantly enable competitive training of 405B models.
- knowitnone 2y agoyes but if Intel provided the base, people will flock to it build software for it. Intel doesn't even need be involved but they should.
- 3abiton 2y agoBut all the tricks of developing a solid software stack to support the HW are already out no? The basic principles are there, to my understanding the main challenge is not doing this development in tandem with the HW people and the requirments to support older legacy device which makes it harder for example for Amd to compete. The only challenges,which intel are prepped to face is logistics and fabs. On a separate note, project like JAX are aiming to circumvent that abstraction layer cuda adds to nvidia, so having decent hardware competition is definitely an option. Just some time ago, vllm fully supported amd gpus! We need more competition.
- wordofx 2y ago> Disclosure: HPC admin who works with NIVIDA cards here. > Because, no. It's not as simple as that. Wow what he said is way above your head! Please reread what he wrote.
- ryao 2y agoI have been hacking on local llama 3 inference software (for the CPU, but I have been thinking about how I would port it to a GPU) and would like to do a rebuttal: https://github.com/ryao/llama3.c https://github.com/ryao/llama3.c Inference workloads are easy to parallelize to N cards with minimal connectivity between them. The Nvlink crossbars and switches just are not needed. In particular, inference can be divided into two distinct phases, which are input processing (prompt processing) and output generation (token generation). They are remarkably different in their requirements. Input processing is compute bound via GEMM operations while output generation is memory bandwidth bound via GEMV operations. Technically, you can do the input processing via GEMV too by processing 1 token at a time, but that is slow, so you do not want to do that. Anyway, these phases can be further subdivided into the model’s layers. You can have 1 GPU per layer with the logits passing from GPU to GPU in a pipeline. The GPUs just need the layer’s weights and the key-value cache for all of the tokens in that layer in memory to be able to work effectively. For llama 3.1 405B, there are 126 layers, so that is up to 126 GPUs. That is of course slightly slower than if you just had 1 GPU with an incredible amount of VRAM, but you can always have more than one query in flight to get better than 1 GPU’s worth of performance from this pipeline approach. There are other ways of doing parallelization too, such as having output processing use GEMM to do multiple queries in parallel. This would be what others call batching, although I am only interested in doing 1 query at a time right now, so I have not touched it. In essence, you can connect n cards on PCIe and have them solve inferencing problems magically, with the right software. Training is a different matter and I cannot comment on it as I have not studied it yet.
- dogcomplex 2y agowhat kind of bandwidth/latency between GPUs would one need in that setup to not be bottlenecking? What you're describing sounds quite forgiving. Is it forgiving enough that we could potentially connect those GPUs over a LAN, or even a remote decentralized cloud of host computers? From my understanding that's certainly possible to do without the latency hurting much with large batching between inference layers
- ryao 2y agoThis depends on: * the model dimension * how many bits per variable are used by your quantization * how many tokens are being processed per step (input processing can do all N input tokens simultaneously and output processing can only do 1 at a time when doing a single query) * how many times you split the model layers across multiple GPUs The model dimensions are: * 4096 for llama 3/3.1 8B. * 8192 for llama 3/3.1 70B. * 16384 for llama 3.1 405B. The model layers are: * 32 for llama 3/3.1 8B * 80 for llama 3/3.1 70B * 126 for llama 3.1 405B The amount of data that needs to be transferred for each split is surprisingly small. Each time you move the calculation of a subsequent layer to a different GPU, you need to transfer an array that is of size model_dimension * num_tokens * bits_per_variable. Then this reduces to a classic network transfer time problem, where you consider both time until the first byte arrives and the transfer time until the last byte arrives. Reality will likely be longer than that idealized scenario, especially since you need to send a signal saying to begin computing. Input processing can tackle so many tokens simultaneously that it probably is not worth thinking too much about this penalty there. Output processing is where the penalty is more significant, since you will incur these costs for every token. Let’s say we are doing fp16 or bf16 on llama 3 8B. Then we need to transfer 8KB every time we move the calculation for another layer to another GPU. If you use RDMA and do this over 10GbE, the transfer time would be 6.4 microseconds. If we assume the time to first byte and time to do a signal to begin processing is 3.6 microseconds combined (chosen to round things up), then we get a penalty of 10 microseconds per split, per token. If you are doing 60 tokens per second and split things across 4 GPUs over the network, you have a penalty of 30 microseconds per token. It will run about 0.003% slower and you are not going to notice this at all. Assuming 10GbE with RDMA is somewhat idealized, although I needed to pick something to give some numbers. In any case, the equation for figuring what factor slower it would be is 1 / (1 + time to do transfers and trigger processing per each token in seconds). That would mean under a less ideal situation where the penalty is 5 milliseconds per token, the calculation will be ~0.99502487562 times what it would have been had it been done in a hypothetical single GPU that has all of the VRAM needed, but otherwise the same specifications. This penalty is also not very noticeable. In conclusion, you are right. I actually recall seeing a random YouTube video talking about a software project that does clustered interferencing, so people are already doing this. Unfortunately, I do not remember the project name, channel name or video name.
- vicentwu 2y agoOff the topoc. I think, in the long-term , inference should be done along with some kind of training.
- stephen_cagle 2y agoPretty sure they are talking about inference in the post you are responding to. Training the model obviously needs far more compute, but running them locally is what most devs are interested in.
- nine_k 2y agoYou are of course right WRT datacenter use of GPUs. The OP spoke about local generation though. It is of course a smaller market, but a market nevertheless, and the more amateurs and students are using your product, the more of them would consider applying it in more professional settings. Sun (Sparc) and HP (PA-RISC) used to own most of the server market in 1990, but lost most of it to x86 by 2000. Few people had a Sun box with Solaris, but tons of people had access to a PC with Linux, which was inferior in many ways, but well-known and much less locked-up.
- weatherlite 2y ago> NVIDIA has a complete ecosystem now. They have cards. They have cards of cards Nvidia is starting to sound like a house of cards to me.
- reissbaker 2y agoOn servers you're right: but for local LLM inference, I think you're wrong. For local LLMs most people are bottlenecked by not having enough VRAM: pretty much no one is running a 70b model on Nvidia GPUs locally, just due to the expense. You don't need maximum performance: you need it to run at all, which most people can't do for the good models — at least, not without heavy quantization that pretty badly lobotomizes them. Apple is the king right now of local LLM inference, just because of their unified memory architecture meaning that people can get large amounts of "VRAM" (since all RAM is VRAM). They're not as fast as Nvidia — not even close to an H100, for example. But they don't need to be. No consumer can afford an H100, but they can afford a Mac.