gdiamos

15 ms·

211.
by gdiamos 2y ago
The top of the nasdaq is full of companies that build computers (from phones to data centers), not companies that only do algorithms. A clever AI algorithm run on rented compute is not a moat.
212.
by gdiamos 2y ago
If your goal is to lift the limit, why put it in? We periodically raise flop limits in export control law. The intention is still to limit China and Iran. Would any computer industry accept a government mandated limit on perf? Should NVIDI
213.
by gdiamos 2y ago
“But the big guys are struggling getting past 100KB, so ‘think of the small guys’ doesn’t make sense when the limit is 640KB.” How do people on a computer technology forum ignore the 10,000x improvement in computers over 30 years due to adv
214.
by gdiamos 2y ago
Cost as a perf metric is meaningless and the history of computer benchmarks has repeatedly proven this point. There is a reason why we report time (speedup) in spec instead of $$ The price you pay depends on who you are and who is giving it
215.
by gdiamos 2y ago
Tell that to me when we get to llama 15
216.
by gdiamos 2y ago
Advanced technology will drop the cost of training. The flop targets in that bill would be like saying “640KB of memory is all we will ever need” and outlawing anything more. Imagine what other countries would have done to us if we allowed
217.
by gdiamos 2y ago
If that bill had passed I would have seriously considered moving my AI company out of the state.
218.
by gdiamos 2y ago
Criticism is an important part of the scientific process. Whichever approach ends up winning is improved by careful evaluation and replication of results
219.
by gdiamos 2y ago
Go NVIDIA
220.
by gdiamos 2y ago
Do tech giants really want to give Sam that much money and power - it looks like the answer is yes. What will he spend it on?
221.
by gdiamos 2y ago
Do inference frameworks like vllm support vision?
222.
by gdiamos 2y ago
Llama 3.2 includes a 1B parameter model. This should be 8x higher throughput for data pipelines. In our experience, smaller models are just fine for simple tasks like reading paragraphs from PDF documents.
223.
by gdiamos 2y ago
Says the hotchips keynote - https://www.youtube.com/watch?v=rsxCZAE8QNA&t=2480s - data that is moved is about 100x higher power than data that is local You have a good point about intermediate data. This would not apply
224.
by gdiamos 2y ago
It takes a lot of energy to move data
225.
by gdiamos 2y ago
SSDs are 3D with high stacking. They retain data without burning power, unlike SRAM. What if you put some of these 3D transistors tightly integrated with a 3D SSD and power gated them? It seems like you would beat the memory wall. Obviousl
226.
by gdiamos 2y ago
Is this why there is more attention on PIM - processing in memory? You don’t burn active power?
227.
by gdiamos 2y ago
backward activations does but typically not backwards weight gradients. Why compute gradients with regards to weights that aren't going to be updated?
228.
by gdiamos 2y ago
I tried with Llama 3.1 and it hallucinated MFU. It also completely misinterpreted multi-node scaling. ========================================================== Let's break down the results described in the post. Context: The post is
229.
by gdiamos 2y ago
That’s approximately 0.8% MFU - h100 would get more like 30% or 40% MFU if well tuned 405e9 parameters 2 flops per matrix multiply per parameter 3 matrix multiplies for (forward, backward param, and backward activation) passes batch size 16
230.
Notes on Building LLMs in 2024
4 points by gdiamos 2y ago|0 comments
231.
by gdiamos 2y ago
It did indeed get merged into the CUDA group but I think the internal CUDA project predated it, or at least, several of the engineers working on it did
232.
by gdiamos 2y ago
Disagree - https://arxiv.org/abs/2406.17642 We cover halting problem and intractable problems in the related work. Of course LLMs cannot give answers to intractable problems. I also don’t see why you should call an ans
233.
LLM Inference at the Memory Wall(lamini.ai)
1 points by gdiamos 2y ago|0 comments
234.
by gdiamos 2y ago
If it isn’t already obvious - hardware to run these models is the moat
235.
by gdiamos 2y ago
We know that LLMs hallucinate, but we can also remove them. I’d love to see a future generation of a model that doesn’t hallucinate on key facts that are peer and expert reviewed. Like the Wikipedia of LLMs https://arxiv.org&#x2F
236.
by gdiamos 2y ago
Mostly talking to people irl Good communities are tough to find and you have to constantly build them
237.
by gdiamos 2y ago
I unplugged from Reddit when they did this and my life is much better. Reddit had a good community and content in the early days, but as it grew in popularity and squeezed profit the value dropped. I think it’s funny that early LLM projects
238.
by gdiamos 2y ago
At $1.127/lb of aluminum that only comes out to a bit less than $30,000 for the raw materials. That's astounding.
239.
by gdiamos 2y ago
How much copper would you need for a 4,300km 2GW subsea cable?
240.
by gdiamos 2y ago
Glad to see the use of SLURM The number of times I see people trying to reinvent the HPC wheel astounds me

More ›