5 ms·
Nvidia Hopper Architecture In-Depth
- musha68k 4y agoDoes anyone know where a mere mortal can try these with least markup asked by the corresponding cloud provider?
- sbierwagen 4y agoThis is a preview, H100 hasn't started shipping yet. There's been some speculation that if the Ethereum merge puts a lot of 3000-series cards on the used market then Nvidia might delay releasing this generation of GPU.
- wmf 4y agoThe 100 series aren't used for mining and used desktop GPUs don't compete with them. Hopper will be released when it's ready.
- musha68k 4y agoI’m asking as I would like to try parsec with it. Playing modern games in 4K+ with maxed out settings (and any ML “image reconstruction” turned off). It would be great to try this if possible!
- learndeeply 4y agoYou're confusing the H100, a high-end data-center card with the RTX 4000 series.
- hughw 4y agoWill AWS please rent me just one of these? I can't use 8 at a time. I need the fine granularity they offer with p3 instances.
- jiggawatts 4y agoThe mention of "Confidential Computing support" and the word "secure" over and over makes me strangely nervous. I have a sneaking suspicion that somewhere in an NSA datacenter there will be racks upon racks of these things running transformers over every text message and voice call coming from certain regions...
- coolspot 4y agoIt is perhaps less about NSA and more about DRM that should scare us in NVidia doing “confidential computing”.
- wfrew 4y agoWhile I can totally imagine that being true, I imagine this is probably more a response to cloud providers wanting to subdivide each physical GPU into virtual/multi-tenant GPUs. They'll want to be able to provide strong security guarantees to customers who're renting compute on a multi-tenant GPU and are pushing their sensitive data onto it
- amluto 4y ago“Confidential Computing” is an up-and-coming model in which a user (e.g. cloud customer) trusts a hardware vendor (e.g. Intel, AMD, nVidia), and that vendor has a mechanism by which systems with their hardware can run user code and attest, remotely, to the user that the code has not been tampered with and that the system’s owner (e.g. Amazon) is not exfiltrating user data. This could be used for DRM or nefarious purposes, but that’s not the point. Obviously this is no more secure than the underlying hardware. All the attacks against SGX, for example, will still apply unless they are fixed.
- fulafel 4y agoThis kind of thing has a history, eg this was on top of the TPM platform: https://web.archive.org/web/20060822043633/http://www.hpl.hp.com/personal/Wenbo_Mao/research/tcgridsec.pdf https://web.archive.org/web/20060822043633/http://www.hpl.hp... I wonder what's different this time.
- wfrew 4y agoAdding network addressing to the GPU interconnect is kind of fascinating Am I right in thinking the GPU-to-GPU communication is just shuttling chunks of data around for sharing inputs/outputs of computations? Or is there some other coordination going on between the GPUs directly with regards to the actual computations each is running? (Or is that still being managed wholly by the CPUs they're attached to?)
- chessgecko 4y agoMaybe it’s to speed up multi gpu matrix multiplies. They’re useful for serving/training gpt3 size models
- learndeeply 4y agoBoth - the first is for sharding tensors across GPUs, the second is to do an all reduce (e.g. for distributed data parallel to synchronize gradients)
- modeless 4y agoFP8 is interesting. I wonder if training could ever be made to work in this format? I see they have two options for mantissa size which is interesting. But no unsigned option? Losing a whole bit to the sign seems unfortunate when you only have eight.
- sbierwagen 4y agoIf you can train with 8 bit ints, then FP8 should be fine: https://arxiv.org/abs/2009.13108 https://arxiv.org/abs/2009.13108 A group at IBM has been claiming for years that you can train with as little as FP4: https://papers.nips.cc/paper/2020/file/13b919438259814cd5be8cb45877d577-Paper.pdf https://papers.nips.cc/paper/2020/file/13b919438259814cd5be8...
- modeless 4y agoThere have been a lot of papers on techniques for training in low precision, even as low as binary. But the largest models that would benefit most aren't using those techniques, which indicates to me that they don't work as well as their paper abstracts suggest.
- sdenton4 4y agoMy experience with quantization is that a little is easy, and a lot takes work. Quantization aware training helps, but it's finicky and adds extra steps to the training process, which in turn slows down iteration speed.