3 ms·

>guessing at best Yes, but at the same time no. It's guessing based on math, so it is guessing the answer out of a filtered set of "likely correct" answers. Th

>guessing at best Yes, but at the same time no. It's guessing based on math, so it is guessing the answer out of a filtered set of "likely correct" answers. The reason it's novel is because it can go so fast now that it seems conversational, but as nothing is free the trade off is to get better math, better probabilities, and better guesses, you have to keep expanding the size of the model which costs immense amounts of electricity for diminishing returns. There is also a limit to the amount of times you can "just make it bigger" before it actually performs more poorly, as I understand it. LLMs feel like a shortcut because it is a shortcut compared to developing something that actually understands human language. We just train them on data so it knows what words usually comes next in a series. So if the training data is good, the output is more likely to be good. But it doesn't understand what the "correct" answer is and is entirely unaware of what the content it was trained on actually means. So they will likely never be perfect because they were designed to be "good enough". I can foresee them being a part of a greater AI in the future though.

MuffinFlavored 3y ago
> you have to keep expanding the size of the model Can you be more specific here? Expand the size in possible parameters? Expand the size in context? I get what you are saying in theory: what if every GPT-4 instance had 100TB of RAM it could use, what could it achieve then, but I can't think of how/what would fill that RAM that we are limited on today? I thought the more parameters you introduce, the more risk you have for hallucination because it's harder to pick which token should be next given the "more possible outcomes" total.
- I_Am_Nous 3y ago
  I believe you are correct, I mentioned "There is also a limit to the amount of times you can "just make it bigger" before it actually performs more poorly, as I understand it." But there may also be a certain sweet spot we can slowly increase as we do more research. We lose accuracy when we quantize from a larger model to a smaller one (such as 16b quantized to 4b) and it's usually done for speed, but a larger model quantized down tends to be more accurate than a model trained to be that particular size. So it would be interesting to see how it would work if researchers build up to much larger parameter sizes, but quantized them down so they were easier to run but still kept some of the benefits of the increased parameter count. I also imagine parameter count is currently more limited by how much we can process at once, so we could be looking at the NES and trying to imagine PS5 games based on what we know about the NES :)

MuffinFlavored 3y ago

> you have to keep expanding the size of the model Can you be more specific here? Expand the size in possible parameters? Expand the size in context? I get what you are saying in theory: what if every GPT-4 instance had 100TB of RAM it could use, what could it achieve then, but I can't think of how/what would fill that RAM that we are limited on today? I thought the more parameters you introduce, the more risk you have for hallucination because it's harder to pick which token should be next given the "more possible outcomes" total.

I_Am_Nous 3y ago
I believe you are correct, I mentioned "There is also a limit to the amount of times you can "just make it bigger" before it actually performs more poorly, as I understand it." But there may also be a certain sweet spot we can slowly increase as we do more research. We lose accuracy when we quantize from a larger model to a smaller one (such as 16b quantized to 4b) and it's usually done for speed, but a larger model quantized down tends to be more accurate than a model trained to be that particular size. So it would be interesting to see how it would work if researchers build up to much larger parameter sizes, but quantized them down so they were easier to run but still kept some of the benefits of the increased parameter count. I also imagine parameter count is currently more limited by how much we can process at once, so we could be looking at the NES and trying to imagine PS5 games based on what we know about the NES :)