3 ms·

I would expect specific hardware acceleration for LLM if it becomes very relevant. That would improve latency and costs. I use ChatGPT throughout the day as a

I would expect specific hardware acceleration for LLM if it becomes very relevant. That would improve latency and costs. I use ChatGPT throughout the day as a widely knowledgeable coworker that periodically eats a bag of mushrooms, and have paid the $20 / month. I’d note this description matches all the most useful coworkers I’ve had, except that they cost a ton more per month.

fnbr 4y ago
What specific hardware acceleration is possible? I think TPUs are already as good as you can get for accelerating large MLPs (half of what a LLM is). Is there that much overhead for the attention blocks?
- dylan522p 4y ago
  No TPUs are getting better architecturally
- sinenomine 4y ago
  TPU is a pretty conservative design, there are 2x...5x speedups available with full-stack optimization, and then some.
pavelstoev 4y ago
Before spending $$$ on the hardware accelerators, there is still a lot to achieve using software optimizations for inference and trainings workloads. We optimized GPT-J, and others, for real. https://centml.ai/benchmarks/ https://centml.ai/benchmarks/
- Const-me 4y ago
  You could also port that code from nVidia CUDA somewhere else. If you wanna run on servers, Vulkan Compute will probably do the trick. I have ported ML inference to Windows desktops with DirectCompute https://github.com/Const-me/Whisper https://github.com/Const-me/Whisper These NVIDIA A40 GPUs mentioned on the page you’ve linked cost $4000 (Amazon) to $10000 (Dell), deliver similar performance to Intel A770 which cost $350. That’s 1 order of magnitude difference in cost efficiency.
omnicognate 4y ago
Better to treat it as a devious and eloquent coworker commiting corporate sabotage for a competing company, who is paid by the factoid to get you to believe incorrect things. Google failed at this by being not nearly paranoid enough when putting their Bard demo together. Everyone laughed at them but ChatGPT does exactly the same thing. (Someone will probably claim ChatGPT would be less likely than Bard to commit the sort of fabrication spotted in the demo, but it's not. I gave ChatGPT the exact same question but replaced JWST with CERN and it told me antimatter was discovered at CERN when according to CERN's own website [1] it was both predicted and observed decades before CERN existed. I didn't have to fish for this. It was the first result for basically the exact same question and was a highly believable fabrication of the exact same kind, attributing a discovery that had been made earlier. I switched JWST for CERN just in case there has been any special training regarding JWST in light of the publicity - no idea if there has been, of course.) [1] https://timeline.web.cern.ch/timeline-header/86 https://timeline.web.cern.ch/timeline-header/86
- fnordpiglet 4y ago
  So, I use chatgpt as a wingman while I program to give me explanations and snippets. I don’t think it’s malicious, and I’d say it’s not a whole lot worse than my human coworkers in terms of coming up with malarkey. Humans say and do all sorts of wacky shit with full confidence. I stay credulous and I’m not asking it to do my work, I’m asking it to explain edges I’m unfamiliar with. If I find it hallucinated an answer it usually means what I want to do isn’t actually possible but it contrived a plausible way to answer that could be true. If I push around the edges and correct it but it keeps coming up with garbage, I can be pretty certain my line of reasoning is at a dead end and I need to backtrack or accept I need a different solution to a different question. I’ll verify this using Kagi and it’s usually pretty trivial to verify I’m right. This feels exactly like my experience over the last 30 years developing software with human coworkers.
- fnordpiglet 4y ago
  Did you read the response carefully? I know there’s a randomness factor in the response but the response I got to your prompt (helpful to provide the exact prompt) said that CERN studies antimatter, not discover it. I also asked it how was antimatter discovered and it accurately explained the timeline. What new discoveries from the CERN can I tell my 9 year old about? Here are a few recent discoveries from CERN that you can explain to your 9-year-old: Higgs Boson: CERN scientists discovered the Higgs Boson particle in 2012, which helps explain how other particles have mass. You can explain it to your child as a particle that helps give everything in the universe its size and weight. Antimatter: CERN scientists study antimatter, which is a type of matter that has the opposite charge of normal matter. This helps us understand more about the basic building blocks of the universe. Dark Matter: CERN scientists are searching for evidence of dark matter, which is a mysterious substance that makes up most of the universe but cannot be seen directly. Scientists believe that dark matter could help explain the behavior of galaxies and other celestial objects. Neutrinos: CERN scientists study neutrinos, which are tiny particles that are produced in the sun and other celestial objects. These particles can travel through the universe and can help us understand more about how the universe works. Remember, it's important to keep the explanations simple and use examples that your child can relate to, like using a ball as a representation of a particle.

fnbr 4y ago

What specific hardware acceleration is possible? I think TPUs are already as good as you can get for accelerating large MLPs (half of what a LLM is). Is there that much overhead for the attention blocks?

dylan522p 4y ago
No TPUs are getting better architecturally
sinenomine 4y ago
TPU is a pretty conservative design, there are 2x...5x speedups available with full-stack optimization, and then some.

pavelstoev 4y ago

Before spending $$$ on the hardware accelerators, there is still a lot to achieve using software optimizations for inference and trainings workloads. We optimized GPT-J, and others, for real. https://centml.ai/benchmarks/ https://centml.ai/benchmarks/

Const-me 4y ago
You could also port that code from nVidia CUDA somewhere else. If you wanna run on servers, Vulkan Compute will probably do the trick. I have ported ML inference to Windows desktops with DirectCompute https://github.com/Const-me/Whisper https://github.com/Const-me/Whisper These NVIDIA A40 GPUs mentioned on the page you’ve linked cost $4000 (Amazon) to $10000 (Dell), deliver similar performance to Intel A770 which cost $350. That’s 1 order of magnitude difference in cost efficiency.

omnicognate 4y ago

Better to treat it as a devious and eloquent coworker commiting corporate sabotage for a competing company, who is paid by the factoid to get you to believe incorrect things. Google failed at this by being not nearly paranoid enough when putting their Bard demo together. Everyone laughed at them but ChatGPT does exactly the same thing. (Someone will probably claim ChatGPT would be less likely than Bard to commit the sort of fabrication spotted in the demo, but it's not. I gave ChatGPT the exact same question but replaced JWST with CERN and it told me antimatter was discovered at CERN when according to CERN's own website [1] it was both predicted and observed decades before CERN existed. I didn't have to fish for this. It was the first result for basically the exact same question and was a highly believable fabrication of the exact same kind, attributing a discovery that had been made earlier. I switched JWST for CERN just in case there has been any special training regarding JWST in light of the publicity - no idea if there has been, of course.) [1] https://timeline.web.cern.ch/timeline-header/86 https://timeline.web.cern.ch/timeline-header/86

fnordpiglet 4y ago
So, I use chatgpt as a wingman while I program to give me explanations and snippets. I don’t think it’s malicious, and I’d say it’s not a whole lot worse than my human coworkers in terms of coming up with malarkey. Humans say and do all sorts of wacky shit with full confidence. I stay credulous and I’m not asking it to do my work, I’m asking it to explain edges I’m unfamiliar with. If I find it hallucinated an answer it usually means what I want to do isn’t actually possible but it contrived a plausible way to answer that could be true. If I push around the edges and correct it but it keeps coming up with garbage, I can be pretty certain my line of reasoning is at a dead end and I need to backtrack or accept I need a different solution to a different question. I’ll verify this using Kagi and it’s usually pretty trivial to verify I’m right. This feels exactly like my experience over the last 30 years developing software with human coworkers.
fnordpiglet 4y ago
Did you read the response carefully? I know there’s a randomness factor in the response but the response I got to your prompt (helpful to provide the exact prompt) said that CERN studies antimatter, not discover it. I also asked it how was antimatter discovered and it accurately explained the timeline. What new discoveries from the CERN can I tell my 9 year old about? Here are a few recent discoveries from CERN that you can explain to your 9-year-old: Higgs Boson: CERN scientists discovered the Higgs Boson particle in 2012, which helps explain how other particles have mass. You can explain it to your child as a particle that helps give everything in the universe its size and weight. Antimatter: CERN scientists study antimatter, which is a type of matter that has the opposite charge of normal matter. This helps us understand more about the basic building blocks of the universe. Dark Matter: CERN scientists are searching for evidence of dark matter, which is a mysterious substance that makes up most of the universe but cannot be seen directly. Scientists believe that dark matter could help explain the behavior of galaxies and other celestial objects. Neutrinos: CERN scientists study neutrinos, which are tiny particles that are produced in the sun and other celestial objects. These particles can travel through the universe and can help us understand more about how the universe works. Remember, it's important to keep the explanations simple and use examples that your child can relate to, like using a ball as a representation of a particle.