5 ms·

Maybe a weird question, by why does everyone assume that we will continue to have improvements and GPT-5,6,7,8 will surely "get there"? Isn't it possible that

Maybe a weird question, by why does everyone assume that we will continue to have improvements and GPT-5,6,7,8 will surely "get there"? Isn't it possible that we've almost reached the limit of this technology and it won't improve much for a very long time? Like self driving cars.

crop_rotation 4y ago
Self driving cars have a very different problem. Regardless of the improvements and sufficiency they reach, their errors can be fatal. One news headline about a self driving car crash will ensure that the technology doesn't get any approvals. ChatGPT for all it's flaws can't jump out of the computer and harm the user. It can harm users in others ways, but those are less known or understood by the general public, and thus it is much easier to take to general availability. I agree on the limit part that it might be possible the technology has reached a limit, however, it is still better to think about what happens if it hasn't and the rate of improvement continues.
dpkirchner 4y ago
I doubt it. I would expect that if GPT's "knowledge" reaches some limit, OpenAI could just hire subject matter experts to write books for the model to read, and then use those to (effectively) down-rank material that differs substantially. If we're talking about the modeling itself, I think we're a long long ways from any limits. There's room for improvements such as cheap personalized results, faster hardware with more memory for lower costs, bot-to-bot communication allowing my instance to interface with yours to come up with mutual understanding, etc. Pretty much all of the components for this exist already in some form, they just need to be optimized and implemented on a large scale and made available to us humans.
- JieJie 4y ago
  I found this Veritasium video interesting, about a company that is paring down models so they fit on analog chips, so the chips can be used in self-contained devices. It also appears that quantum computers may be even better for matrix math that transformers rely on. If all these hardware advances pan out... https://www.youtube.com/watch?v=GVsUOuSjvcg https://www.youtube.com/watch?v=GVsUOuSjvcg https://phys.org/news/2023-01-analog-quantum-previously-unsolvable-problems.html https://phys.org/news/2023-01-analog-quantum-previously-unso...
- ThrowawayR2 4y ago
  It takes months to years to write a well researched and in-depth book and subject matter experts are not necessarily good authors. Beyond that, each book only moves the needle a little bit as well because of the size of the training corpus. If there's going to be improvement from that path, it will be at best slow and incremental.
  - dpkirchner 4y ago
    You're right, maybe a whole book is unrealistic. I guess it's more likely that they'd have someone make small corrections, like if enough people flag a common result as wrong or poor.

crop_rotation 4y ago

Self driving cars have a very different problem. Regardless of the improvements and sufficiency they reach, their errors can be fatal. One news headline about a self driving car crash will ensure that the technology doesn't get any approvals. ChatGPT for all it's flaws can't jump out of the computer and harm the user. It can harm users in others ways, but those are less known or understood by the general public, and thus it is much easier to take to general availability. I agree on the limit part that it might be possible the technology has reached a limit, however, it is still better to think about what happens if it hasn't and the rate of improvement continues.

dpkirchner 4y ago

I doubt it. I would expect that if GPT's "knowledge" reaches some limit, OpenAI could just hire subject matter experts to write books for the model to read, and then use those to (effectively) down-rank material that differs substantially. If we're talking about the modeling itself, I think we're a long long ways from any limits. There's room for improvements such as cheap personalized results, faster hardware with more memory for lower costs, bot-to-bot communication allowing my instance to interface with yours to come up with mutual understanding, etc. Pretty much all of the components for this exist already in some form, they just need to be optimized and implemented on a large scale and made available to us humans.

JieJie 4y ago
I found this Veritasium video interesting, about a company that is paring down models so they fit on analog chips, so the chips can be used in self-contained devices. It also appears that quantum computers may be even better for matrix math that transformers rely on. If all these hardware advances pan out... https://www.youtube.com/watch?v=GVsUOuSjvcg https://www.youtube.com/watch?v=GVsUOuSjvcg https://phys.org/news/2023-01-analog-quantum-previously-unsolvable-problems.html https://phys.org/news/2023-01-analog-quantum-previously-unso...
ThrowawayR2 4y ago
It takes months to years to write a well researched and in-depth book and subject matter experts are not necessarily good authors. Beyond that, each book only moves the needle a little bit as well because of the size of the training corpus. If there's going to be improvement from that path, it will be at best slow and incremental.
- dpkirchner 4y ago
  You're right, maybe a whole book is unrealistic. I guess it's more likely that they'd have someone make small corrections, like if enough people flag a common result as wrong or poor.