4 ms·
> you have to keep expanding the size of the model Can you be more specific here? Expand the size in possible parameters? Expand the size in context? I get wha
by MuffinFlavored 3y ago
> you have to keep expanding the size of the model
Can you be more specific here? Expand the size in possible parameters? Expand the size in context? I get what you are saying in theory: what if every GPT-4 instance had 100TB of RAM it could use, what could it achieve then, but I can't think of how/what would fill that RAM that we are limited on today? I thought the more parameters you introduce, the more risk you have for hallucination because it's harder to pick which token should be next given the "more possible outcomes" total.
- I_Am_Nous 3y agoI believe you are correct, I mentioned "There is also a limit to the amount of times you can "just make it bigger" before it actually performs more poorly, as I understand it." But there may also be a certain sweet spot we can slowly increase as we do more research. We lose accuracy when we quantize from a larger model to a smaller one (such as 16b quantized to 4b) and it's usually done for speed, but a larger model quantized down tends to be more accurate than a model trained to be that particular size. So it would be interesting to see how it would work if researchers build up to much larger parameter sizes, but quantized them down so they were easier to run but still kept some of the benefits of the increased parameter count. I also imagine parameter count is currently more limited by how much we can process at once, so we could be looking at the NES and trying to imagine PS5 games based on what we know about the NES :)