4 ms·

This isn't a hardware feat, this is a software triumph. They didn't make special purpose hardware to run a model. They crafted a large model so that it could

This isn't a hardware feat, this is a software triumph. They didn't make special purpose hardware to run a model. They crafted a large model so that it could run on consumer hardware (a phone).

pdpi 8mo ago
It's both. We haven't had phones running laptop-grade CPUs/GPUs for that long, and that is a very real hardware feat. Likewise, nobody would've said running a 400b LLM on a low-end laptop was feasible, and that is very much a software triumph.
- bigyabai 8mo ago
  > We haven't had phones running laptop-grade CPUs/GPUs for that long Agree to disagree, we've had laptop-grade smartphone hardware for longer than we've had LLMs.
  - pdpi 8mo ago
    Kind of. We've had solid CPUs for a while, but GPUs have lagged behind (and they're the ones that matter for this particular application). iPhones still lead by a comfortable margin on this front, but have historically been pretty limited on the IO front (only supported USB2 speeds until recently).
    - bigyabai 7mo ago
      The GPUs are perfectly solid. Cheap Android handsets have shipped with Vulkan compliance for almost a decade now; the GPUs are equally-featured to consoles and PCs. The same goes for Apple handsets that run byte-identical Metal Compute Shaders to the Mac. For desktop use they are perfectly amenable. The hardware lacks nothing required for inference or gaming that dGPUs ordinarily support. And even if you raise the requirements, we still have to contend with cheap CUDA-capable GPUs like the one in the ($300!!!) Nintendo Switch, or the Jetson SOCs. The mobile market has had tons of high-speed/low-power options for a very long time now.
  - mnkyprskbd 7mo ago
    We had LLMs for about 5 minutes or so. Hardly a measure of time for an industry that goes back half a century and then some.
smallerize 8mo ago
The iPhone 17 Pro launched 8 months ago with 50% more RAM and about double the inference performance of the previous iPhone Pro (also 10x prompt processing speed).
- deleted 8mo ago
  [deleted]
SV_BubbleTime 8mo ago
>triumph It’s been a lot of years, but all I can hear after reading that is … I’m making a note here, huge success
- breggles 8mo ago
  It's hard to overstate my satisfaction!
- GorbachevyChase 8mo ago
  There’s no use crying over every mistake. You just keep on trying until you run out of cake.
anemll 8mo ago
both, tbh

This isn't a hardware feat, this is a software triumph. They didn't make special purpose hardware to run a model. They crafted a large model so that it could run on consumer hardware (a phone).

pdpi 8mo ago

It's both. We haven't had phones running laptop-grade CPUs/GPUs for that long, and that is a very real hardware feat. Likewise, nobody would've said running a 400b LLM on a low-end laptop was feasible, and that is very much a software triumph.

bigyabai 8mo ago
> We haven't had phones running laptop-grade CPUs/GPUs for that long Agree to disagree, we've had laptop-grade smartphone hardware for longer than we've had LLMs.
- pdpi 8mo ago
  Kind of. We've had solid CPUs for a while, but GPUs have lagged behind (and they're the ones that matter for this particular application). iPhones still lead by a comfortable margin on this front, but have historically been pretty limited on the IO front (only supported USB2 speeds until recently).
  - bigyabai 7mo ago
    The GPUs are perfectly solid. Cheap Android handsets have shipped with Vulkan compliance for almost a decade now; the GPUs are equally-featured to consoles and PCs. The same goes for Apple handsets that run byte-identical Metal Compute Shaders to the Mac. For desktop use they are perfectly amenable. The hardware lacks nothing required for inference or gaming that dGPUs ordinarily support. And even if you raise the requirements, we still have to contend with cheap CUDA-capable GPUs like the one in the ($300!!!) Nintendo Switch, or the Jetson SOCs. The mobile market has had tons of high-speed/low-power options for a very long time now.
- mnkyprskbd 7mo ago
  We had LLMs for about 5 minutes or so. Hardly a measure of time for an industry that goes back half a century and then some.

smallerize 8mo ago

The iPhone 17 Pro launched 8 months ago with 50% more RAM and about double the inference performance of the previous iPhone Pro (also 10x prompt processing speed).

deleted 8mo ago
[deleted]

SV_BubbleTime 8mo ago

>triumph It’s been a lot of years, but all I can hear after reading that is … I’m making a note here, huge success

breggles 8mo ago
It's hard to overstate my satisfaction!
GorbachevyChase 8mo ago
There’s no use crying over every mistake. You just keep on trying until you run out of cake.

anemll 8mo ago

both, tbh