4 ms·

Sorta predictions, sorta hopes... GPT-4 or another huge language model like it will be released, and we will see diminishing returns continue to plateau. Grap

Sorta predictions, sorta hopes... GPT-4 or another huge language model like it will be released, and we will see diminishing returns continue to plateau. Graph-based approaches in NLP will continue to grow in significance, with KGs, ontologies, and semantic graphs seeing growing adoption for their interpretability. Model robustness in ML will probably remain a significant obstacle to productionization, holding back use-cases that need high reliability e.g. self-driving and front-facing chatbots. Reinforcement Learning may see increased research focus for applications in NLP and user interaction.

oezi 5y ago
> GPT-4 or another huge language model like it will be released, and we will see diminishing returns continue to plateau. It was my understanding that GPT-3 showed pretty well that the returns are not dimishing but rather surprisingly still scaling along with data rather nicely.
- KhoomeiK 5y ago
  Figure 3.8 in the GPT-3 paper (SuperGLUE Performance) pretty clearly shows a plateau in performance, with the increase in performance from 13B to 175B parameters (13.5x params, +6 score) not being significantly different from the increase between 6.7B to 13B (1.9x params, +4 score).