4 ms·
Better on which axis? Do you want an AI that takes one hour to respond to? Some would for certain fields, but getting something fast and cheap is going to be ha
by mensetmanusman 4y ago
Better on which axis? Do you want an AI that takes one hour to respond to? Some would for certain fields, but getting something fast and cheap is going to be hard now that Moore’s law is over.
- AlecSchueler 4y agoDon't we all agree that GPT4 is "better" than GPT3? How are we evaluating that if the axis is such a mystery. Yeah maybe we can't quantify it like I can't tell you one writer is better than another in a quantitative but we can both still read their work and come to an understanding.
- nwlieb 4y agoThe runtime is quadratic for a given context size, although it seems like there is some progress on this front https://gwern.net/note/attention https://gwern.net/note/attention
- MichaelZuo 4y agoExponential scaling for a presumable GPT-5 suggests it's response time will be unusably long for the vast majority of use cases, and probably cost multiple dollars USD per query. Not to mention there doesn't actually exist enough English text data in the world to even double GPT-4's training set.
- epups 4y agoCompute will also scale exponentially in coming years. The data source limitation seems to be a harder barrier, I think many companies are experimenting with AI generated content for training at this point.
- MichaelZuo 4y ago> Compute will also scale exponentially in coming years. Cost per transistor scaling has already plateaued or perhaps even inverted with TSMC's latest and greatest. And the new chips, even after 25 layers of EUV lithography, more than doubling the previous record, and an extra year of fine tuning, has total SRAM size scaling of -5% and logic scaling of -42%. These are numbers verified by experienced semi people.
- darkerside 4y agoINSUFFICIENT DATA FOR MEANINGFUL ANSWER https://users.ece.cmu.edu/~gamvrosi/thelastq.html https://users.ece.cmu.edu/~gamvrosi/thelastq.html
- prenoob 4y agoI wasnt aware that it was his favorite story, thanks for posting it
- zirgs 4y agoYes - image generation AIs also take longer to generate higher resolution pictures. I'd rather have a LLM that thinks a bit longer than a LLM that spits out wrong answers immediately.
- fauxpause_ 4y agoIdk. I’m not the one making the claim we’ve already nailed the optimal point though.