4 ms·
When it needs to do more precise numerical calculations, perhaps, like a human, it could just use a calculator?
by aussieguy1234 2y ago
When it needs to do more precise numerical calculations, perhaps, like a human, it could just use a calculator?
- kromem 2y agoThat's not really a concern. If you have a trillion parameter 8-bit fp network or a trillion parameter 1.5-bit ternary network, based on the scaling in Microsoft's paper the latter will actually perform better. A lot of the current thinking is that the nodes themselves act as superpositions for a virtualized network in a multidimensional vector space, so precision is fairly arbitrary for the base nodes and it may be that constraining the individual node values actually allows for a less fuzzy virtualized network by the end of the training. You could still have a very precise 'calculator' feature in the virtual space no matter the underlying parameter precision, and because each parameter is being informed by overlapping virtual features, may even have less unexpected errors and issues with lower precision nodes.
- xwolfi 2y agoYup, your response makes me think they should just use a calculator, like everyone.
- imtringued 2y agoI don't know what you mean. They already use the GPU as a calculator.
- ben_w 2y agoI believe you four are talking about different things; the models are executed on very good "calculators" (if you want to call the GPUs that), but themselves are not very good at being used as calculators. LLMs are sufficiently good hammers that people see everything as a nail, then talk about how bad they are at driving screws.
- CuriouslyC 2y agoPeople are trying to make these monolithic god models right now because everyone's chasing OpenAI's example with ChatGPT, but that approach is going to run out of steam for a lot of reasons. We're eventually going to end up with an agent model that does just what you say, recognizes specialized problem types such as math and calls the appropriate tool, such as a calculator or symbolic computation engine.
- zozbot234 2y agoMany models support function calling/RAG already, these are very similar features from a structural POV. But of course it's harder to train a model for such tasks, compared to just fitting some existing training set.
- CuriouslyC 2y agoI'm not sure having the LLM as the top level piece is the right approach though. Async is the direction we want to go, and LLMS are inherently synchronous. Additionally, LLMs are cumbersome to train and tune, having an agent that calls smaller models would unlock the power of the community to build customized models for specific needs.
- szundi 2y agoWe would like to have something like us. My daughter is just 3.5yrs old, she is inherently hard to train too, decades left. However, we find ourselves quite good.
- CuriouslyC 2y agoYou're assuming you're a monolith, but in fact you have many subnetworks in your brain. The neocortex acts to process information from other parts of the brain, and it itself comprised of a network of interacting modules.
- szundi 2y agoChatGPT routinely does this already
- darkerside 2y agoIt should just be able to program. A programming language is the ultimate verbal-to-math translation layer. Tangent but... programming languages aren't designed for computers. Computers are perfectly happy with assembly or even binary. Programming languages are designed for humans, not just so we can see what others have done, but so that we can understand what we ourselves have done. We give the variable a name because we can't remember 0x0010101; but the computer remembers them both just fine.