cavisne

10 ms·

151.
by cavisne 2y ago
Large clouds tend to be not very good for GPU clusters. The security & management of multi-tenant GPU's is very complex. They have to buy from Nvidia so there is no negotiating leverage. And they know you will be a high maintenance
152.
by cavisne 2y ago
ML training is a tightly interconnected HPC style workload (network bound), which is a market Infiniband has been targeting for a long time. Nvidia saw that and bought Mellanox, and made NCCL/GPU's work really well with Infiniband
153.
by cavisne 2y ago
They are focusing in the press release and the report on DC energy needs though. I see that as foreshadowing that things will be worse going forward.
154.
by cavisne 2y ago
I ran into this in France, where the ticket station used QR codes & NFC. The QR code readers are scratched up meaning printed copies without backlight didn’t work. And iPhone opens Apple wallet when you bring it near a NFC reader, hidin
155.
by cavisne 3y ago
This awesome talk [1] from OpenAI covers this topic quite a bit, one useful takeaway is how GPU compute is basically static, gone are the days of autoscaling as there is nothing to autoscale to. I think that beyond optimizing batch size, ma
156.
by cavisne 3y ago
Good video on the issues with extension cords in the US https://www.youtube.com/watch?v=K_q-xnYRugQ&t=1131s
157.
by cavisne 3y ago
The Programming Massively Parallel Processors textbook has a chapter on histograms. You have basically mentioned the main trick though (privatization). For this problem I think coursening would also help (so everything isn’t queued up on th
158.
by cavisne 3y ago
How does this compare in bandwidth and latency to nvlink? (I’m aware it’s not available on the consumer cards)
159.
by cavisne 3y ago
Nothing compared to Nvidia though. The FLOPS and memory bandwidth is simply not there.
160.
by cavisne 3y ago
Is there an equivalent to this reference for Intel Gaudi? https://docs.nvidia.com/cuda/parallel-thread-execution/index...
161.
by cavisne 3y ago
Yeah, I think AMD will really struggle with the cloud providers. Even Nvidia GPU's are tricky to sandbox, and it sounds like the AMD cards are really easy for the tenant to break (or at least force a restart of the underlying host). AW
162.
by cavisne 3y ago
Very unlikely, Bezos & Gates basically did that and they are still massive targets of criticism. If Elon had decided to just roll over a couple years ago and do everything the media asked him to do (shutdown the Fremont factory during c
163.
by cavisne 3y ago
They operate on outdated fab's (roughly state of the art - 1) https://en.wikipedia.org/wiki/Tensor_Processing_Unit#Product... They also do have a serious presence/spend on things like HBM, semianalysis has so
164.
by cavisne 3y ago
Considering private repos used to require a paid subscription misremembering seems likely. It used to be public by default, and enough people got confused by this that AWS & Github used to specifically scan repo's for accidentally
165.
by cavisne 3y ago
Comes from here [1]. Basically 100 racks of H vs 8 racks of B. I think there may be a typo though, I assume this also includes liquid-cooled vs air-cooled. [1] https://nvdam.widen.net/s/xqt56dflgh/nvidia-blackwell-
166.
by cavisne 3y ago
Could you share the paper? Found this but not sure if it is the same concept https://arxiv.org/pdf/2307.02666.pdf
167.
by cavisne 3y ago
They already have.
168.
by cavisne 3y ago
LLM training is arcane and expensive to experiment with. So OpenAI had to waste a lot of time and GPU-hours on things that didn't work to learn the tricks that did work. Most of the competitors have lineage straight back to OpenAI, eg
169.
by cavisne 3y ago
One of the interesting things when weights are open sourced is the community can often improve the results. See all the bugs fixed in Gemma for an example.
170.
by cavisne 3y ago
Probably https://www.youtube.com/watch?v=nOxKexn3iBo (or just skimming the attached colab).
171.
by cavisne 3y ago
Causes every thread in the block to wait until they have reached this point. Worth reading a cuda primer for more details on blocks/warps. Since the threads are relying on each other to fill the SRAM with all needed data if you didn’t
172.
by cavisne 3y ago
Depending on how they treat oversubscription, Wireless ISP's cant really offer this speed reliably, so they would lose their subsidy. So building housing in a remote area only served by a WISP gets more expensive. In practice I don&#x2
173.
by cavisne 3y ago
Spanners interleaved tables seem like a similar solution, ie you interleave customer data so it all ends up on the same set of hosts for performance, while still having the ability to create transactions across customers.
174.
by cavisne 3y ago
I suspect thats basically it (operations one by one and then pipelining to saturate). Thats basically what Groq does also AFAIK. From their website it seems the chips are designed to be connected together into one big topology, the "Ga
175.
by cavisne 3y ago
My priors were that Anthropic is the most left leaning & most secretive of the big labs. So this is quite surprising!
176.
by cavisne 3y ago
A recent presentation on the architecture https://youtu.be/WQDMKTEgQnY?si=W0E9Kq6P280l3Wcl IMO we still need an MLPerf submission or similar to really understand if this is more efficient or more efficient only if you also
177.
by cavisne 3y ago
Reading this was awesome, but also makes me so glad I bought a Tesla. What a nightmare vs a simple fixed price.
178.
by cavisne 3y ago
Doesn’t meta famously not do much testing at all? Ie they use experiments to “test in prod”.
179.
by cavisne 3y ago
One additional factor is you need to beg for GPU capacity from cloud providers, so they are investing in them, and also giving them preferential capacity over companies they don’t invest in.
180.
by cavisne 3y ago
My impression is they still have a fairly efficient culture, the main problem with joining a bubble company would be if they are overhiring/losing focus. Just sell shares when they vest to reduce your risk.

More ›