3 ms·

What is the reason LLM training requires so much data? It appears they require the entire output of human civilization in order to stay competitive, but why? Is

• kalbadia 2y ago
  According to Ars Technica in the linked article : "In the longer term, the deal also means that OpenAI can openly and officially utilize Condé Nast articles to train future AI language models, which includes successors to GPT-4o. In this case, "training" means feeding content into an AI model's neural network so the AI model can better process conceptual relationships. AI training is an expensive and computationally intense process that happens rarely, usually prior to the launch of a major new AI model, although a secondary process called "fine-tuning" can continue over time. Having access to high-quality training data, such as vetted journalism, improves AI language models' ability to provide accurate answers to user questions."
• Ukv 2y ago
  > [...] are they just very inefficient in how they learn? I'd challenge: inefficient compared to what? For many tasks, the only more capable systems we know of are biological ones that evolved on 4 billion years' worth of data followed by years of constant input from all senses. Gradient descent is incredibly fast and data-efficient by comparison (hence use of mutation + natural selection for machine learning is mostly limtied to tiny toy examples).