3 ms·
One thing to consider is that while the large transformers used in LLMs might have these diminishing returns, we don't know what discrete jump in model architec
by samrus 3y ago
One thing to consider is that while the large transformers used in LLMs might have these diminishing returns, we don't know what discrete jump in model architecture might come next. That model might gain a lot from even more training data. And might gain more from the semi structured data on reddit than the slightly less structured data on Wikipedia and Twitter
Maybe
- danudey 3y agoI would argue that Wikipedia is much more structured than Reddit. > we don't know what discrete jump in model architecture might come next This is true, but as long as these models are based on creating sentences based on what sentences that it's seen have looked like, and not on fetching and understanding verifiable facts, these services will do more harm than good overall. If anything, these models need two sources of training data: 1. The standard language model (as it is now) to be able to generate and process queries and provide understandable answers 2. A database of verifiable factual information that it can query in order to prevent it from completely hallucinating information and then asserting that it is verifiable factual information when asked [0]. Until we can solve the AI hallucination problem, these systems are going to require users to be much more careful with information they're given than most people can manage right now. [0] https://www.cbsnews.com/news/lawyer-chatgpt-court-filing-avianca/ https://www.cbsnews.com/news/lawyer-chatgpt-court-filing-avi...