3 ms·
What's insane? The claim that recall is spyware isn't that bad of an exaggeration, and the claim that the NPU is hype is a reasonable opinion to have. If that
by Dylan16807 2y ago
What's insane? The claim that recall is spyware isn't that bad of an exaggeration, and the claim that the NPU is hype is a reasonable opinion to have.
If that 7% of the die goes unused it's not a huge waste but it's not very good either. If you're not using it enough to affect your battery then you might not get much benefit, because the GPU can do the same tasks about half as fast, it's just less efficient (and it would be a lot more than half if the NPU die space was converted into GPU compute blocks). And for serious tasks it's not very powerful. There's a range where it's good, but not as big a range as people might expect.
- imtringued 2y agoIf my speculation about block FP16 is correct, it might be possible to hit 24 to 48 teraflops on the NPU. This means it would be entirely memory bottlenecked even in prompt processing. There would basically be no application where you would run into the NPU being a limitation. What I fear though is that the NPU will be gimped with a single infinity fabric port, which would limit it to a relatively weak 62GB/s out of 120GB/s. For crazy people who want to use 128k token context, that might turn out to make it or break it, in the end.