4 ms·
Despite my ignorance on what would be involved, I like to fantasize that companies like Apple (who own their own chip design and hardware) can put these ternary
by JKCalhoun 3mo ago
Despite my ignorance on what would be involved, I like to fantasize that companies like Apple (who own their own chip design and hardware) can put these ternary pack/unpack operations in hardware with a single instruction for each (and also instructions for performing the various matrix convolutions using ternary numbers) so that we can get on-device LLMs.
I suppose with the recent quantized Bonsai models we're already seeing on-device LLMs for phones and the like… But I (again, forgive my ignorance) assume that there is an order of magnitude or more in performance sitting out there if we get custom hardware instructions.
- WorldMaker 3mo agoThere is a possibility space there. Some transistors are "naturally" ternary (which we use today for things like binary error correction). Also the Soviet Union went down the rabbit hole of ternary computers and programming languages, which is fascinating history to explore. The interesting questions are if enough LLMs use ternary enough and for long enough into the future to make investments in ternary hardware worth it. Ternary hardware is probably less general purpose than the modern GPU design that is being used for LLMs today and the crossover of "general purpose floating point matrix math engine" for graphics as well as LLMs a useful one, particularly because we know the graphics revenue models pretty well (and gamers are pretty upset at GPU pricing right now because of LLM demand).