4 ms·
That's false. It has been shown that LLMs can perform e.g. gradient descent internally [1], which can explain why they are so good at few shot prompting. The un
by sigmoid10 2y ago
That's false. It has been shown that LLMs can perform e.g. gradient descent internally [1], which can explain why they are so good at few shot prompting. The universal approximation theorem already tells us that a single layer is sufficient to approximate any function, so it should come as no surprise that modern deep networks with many layers should be able to perform iterative optimisations.
[1] https://arxiv.org/abs/2212.10559 https://arxiv.org/abs/2212.10559
- ChicagoDave 2y agoWhat’s the success rate once you hit 20 prompts? What’s the success rate if you hit 30 prompts 40 prompts? I’m pretty sure that as you increase the complexity of your questioning, the LLM is just gonna flat out fail and no change to the vector database is going to improve that.
- sigmoid10 2y agoWhat are you talking about? This has nothing to do with vector databases. This is about the basic capabilities of attention networks and dense layers.
- slashdave 2y agoSo you want to encapsulate reason into a simple function?
- sigmoid10 2y agoWhy not? At the most simple level, the human brain also just takes a set of inputs and produces a set of outputs. There is a huge complicated function behind that, but complexity is no longer an issue thanks to modern compute capabilities.
- ChicagoDave 2y agoYou clearly don't understand short-term and long-term memory and using multiple disciplinary information in problem solving. LLMs can only provide answers to what already exists. They cannot invent new answers. Therefore, they cannot handle complexity.
- sigmoid10 2y agoThat's so wrong, I don't even know where to begin. You should really look up the fundamentals of how these models work instead of listening to the "statistical parrot" nonsense that is constantly spewed around HN.
- slashdave 2y agoIt doesn't sound like you understand what a function is. There is a mathematical definition you need to look up.
- sigmoid10 2y agoTo be honest it doesn't sound like you understand it. Or maybe look up the universal approximation theorem that says this is very much possible. Many people just have this dangerous tendency to put their own mind on a pedestal. That's how they justified their superiority over slaves or minorities in the past and it's how they will justify it over machines in the future.
- Iwan-Zotow 2y agoIs it related to KAN?