4 ms·

There was a paper about LLM running on same power as a light bulb. https://arxiv.org/abs/2406.02528 https://arxiv.org/abs/2406.02528 https://news.ucsc.edu/202

• transpute 2y ago
  Claims 90% memory reduction with OSS code for replication on standard GPUs, https://github.com/ridgerchu/matmulfreellm https://github.com/ridgerchu/matmulfreellm > ..avoid using matrix multiplication using two main techniques. The first is a method to force all the numbers within the matrices to be ternary, meaning they can take one of three values: negative one, zero, or positive one. This allows the computation to be reduced to summing numbers rather than multiplying.. Instead of multiplying every single number in one matrix with every single number in the other matrix.. the matrices are overlaid and only the most important operations are performed.. researchers were able to maintain the performance of the neural network by introducing time-based computation in the training of the model. This enables the network to have a “memory” of the important information it processes, enhancing performance. > ... On standard GPUs.. network achieved about 10 times less memory consumption and operated about 25 percent faster.. could provide a path forward to enabling the algorithms to run at full capacity on devices with smaller memory like smartphones.. Over three weeks, the team created a [FPGA] prototype of their hardware.. surpasses human-readable throughput.. on just 13 watts of power. Using GPUs would require about 700 watts of power, meaning that the custom hardware achieved more than 50 times the efficiency of GPUs.