5 ms·

> “Pre-training as we know it will unquestionably end,” Sutskever said onstage. > “We’ve achieved peak data and there’ll be no more.” > During his NeurIPS tal

• _aavaa_ 2y ago
  > just as oil is a finite resource, the internet contains a finite amount of human-generated content. I guess now they’re being explicit about the blatantly extractive nature of these businesses and their models.
• kibae 2y ago
  I always suspected that bots on Reddit were used to gain karma and then eventually sell the account, but maybe they're also being used for some kind of RLHF.
• vitorgrs 2y ago
  Not sure if this was a good example. Getty already license their images to Nvidia. And they already have a generative image service... I believe it's power by Nvidia model.
• popularonion 2y ago
  > What will replace Internet data for training? Curated synthetic datasets? Enter Neuralink
  • zxexz 2y ago
    Really not sure what you mean by this, could you explain?
    • Gigachad 2y ago
      AI can just suck up the content of peoples brains for training data.
      • zxexz 2y ago
        Yeah, people will go crazy for GPT-o2 trained on the readings of sensors "barely embedded" in the brains tortured monkeys, for sure. EDIT: This comment may have been a bit too sassy. I get the thought behind the original comment, but I personally question the direction and premise of the Neuralink project, and know I am not alone in that regard. That being said, taking a step back, there for sure are plenty of rich data sources for non-text multimodal data.
      • phillipcarter 2y ago
        You need to go back to Twitter with low-quality posts like this.
• YetAnotherNick 2y ago
  Humans doesn't need trillions of tokens to reason or ability to know what they know. While a certain part of it comes from evolution, I think we have already matched the part that came from evolution using internet data, like basic language skills, basic world modelling. Current pretraining takes lot more data than a human would, and you don't need to look into all Getty images to draw a picture and so would a self aware/improving model(whatever that means). To reach expert level in any field, just training next tokens for internet data or any data is not the solution.
  • exe34 2y ago
    > Humans doesn't need trillions of tokens I wonder about that. we can fine tune on calculus with much fewer tokens, but I'd be interested in some calculations of how many tokens evolution provides us (it's not about the DNA itself, but all the other things that were explored and discarded and are now out of reach) - but also the sheer amount of physics learnt by a baby by crawling around and putting everything in its mouth.
    • YetAnotherNick 2y ago
      Yes, as I said in the last comment. With current training techniques, one internet data is enough to give models what is given by evolution. For further training, I believe we would need different techniques to make the model self aware about its knowledge. Also, I believe a person who is blind and paralyzed for life could still attain knowledge if educated well enough.(can't find any study here tbh)
      • exe34 2y ago
        yeah blind and paralysed from birth - I'm doubtful that hearing along would give you the physics training. although if it can be done, then it means the evolutionary pre-training is even more impressive.
  • bronco21016 2y ago
    > Humans doesn't need trillions of tokens to reason or ability to know what they know. It seems to me by the time we’re 5-6 we’ve likely already been exposed to trillions of tokens. Just think of how many hours of video and audio tokens have already come to your brain by that point. We also have constant input from senses like touch and proprioception that help shape our understanding of the world around us. I think there are plenty more tokens freely available out in the world. We just haven’t figured out how to have machines capture them yet.
• menaerus 2y ago
  > What will replace Internet data for training? Curated synthetic datasets? Perhaps a different take at this could be: if I wanted to train "state law" LLM that is exceedingly good in interpreting state law, what are the obstacles to download all the law and regulations material for given state that will allow me to train LLM such that it becomes 95th percentile of all law trainees and lawyers. In that case, and my point is, that we already don't need an "Internet". We just need a sufficiently sized and curated domain-specific dataset and the result we can get is already scary. "State law" LLM was just an example but the same logic applies to basically any other domain - want a domain-specific (LLM) expert? Train it.
  • pas 2y ago
    you need context for the dry statutes sure, you download all the legal arguments, and hope that putting all this on top of a general LLM which has enough context to deal with usual human, American, contemporary stuff the argument is that it's not really enough for the next jump (as it would need "exponentially" more data) as far as a I understand
    • menaerus 2y ago
      I don't understand the limitation, e.g. how much data do you need to train the "law state" specific LLM that doesn't know anything else but that? Such LLM does not need to have 400B of parameters since it's not a general knowledge LLM but perhaps I'm wrong on this (?). So my point rather is that it may very well be, let's take for example, a 30B parameters LLM which in turn means that we might have just enough data to train it. Larger contexts in smaller models are a solved problem.
      • petesergeant 2y ago
        > how much data do you need to train the "law state" specific LLM that doesn't know anything else but that? Law doesn’t exist in a vacuum. You can’t have a useful LLM for state law that doesn’t have an exceptional grounding in real world objects of mechanics. You could force a bright young child to memorize a large text, but without a strong general model of the world, they’re just regurgitating words rather than able to reason about it.
• deleted 2y ago
  [deleted]
• fidotron 2y ago
  > What will replace Internet data for training? Curated synthetic datasets? My take is that the access Meta, Google etc. have to extra data has reduced the amount of research into using synthetic data because they have had such a surplus of it relative to everyone else. For example, when I've done training of object detectors (quite out of date now) I used Blender 3D models, scripts to adjust parameters, and existing ML models to infer camera calibration and overlay orientation. This works amazingly well for subsequently identifying the real analogue of the object, and I know of people doing vehicle training in similar ways using game engines. There were several surprising tactical details to all this which push the accuracy up dramatically and you don't see too widely discussed, like ensuring that things which are not relevant are properly randomized in the training set, such as the surface texture of the 3D models. (i.e. putting random fractal patterns on the object for training improves how robust the object detector is to disturbance in reality).
• robg 2y ago
  The ones that stand out to me are industries like pharmaceuticals and energy exploration, where the data silos are the point of their (assumed) competitive advantages. Why even the playing by opening up those datasets when keeping them closed locks in potential discoveries? Open data is the basis of the Internet. But whole industries are based on keeping discoveries closely guarded for decades.
• seydor 2y ago
  Robots can acquire data on their own (hopefully not via human dissection)
  • rapjr9 2y ago
    This is indeed what I thought he was saying, AI needs to dynamically learn, just training on static data sets is no longer enough to advance. So continuous learning is the future, and the best source of data for continuous learning is people themselves. Don't know what form that might take, instrumenting lots of people with sensors? Robots interacting with people? Self driving cars learning from human drivers? (already happening) Ingesting all video from surveillance cameras? Whatever form the input data takes, continuous learning would be an advance in high level AI. There's been work on it over the decades, not sure how that work might relate to recent LLM work.
• parkaboy 2y ago
  I wonder if we will see (or already are/have been seeing) the XR/smart glasses space heat up. Seems eventually like a great way to generate and hoover up massive amounts of fresh training data.
• RicoElectrico 2y ago
  I think we're not close to running out of training data. It's just that we would like knowledge, but not necessary behavior of said texts. LLMs are very bad at recalling popular memes (known by any seasoned netizen) if they had no press coverage. Maybe training with 4chan isn't as pointless if you could make it memorize it, but not imitate it. Also, what about movie scripts and song lyrics? Transcripts of well known YouTube videos? Hell, television programs even.
  • stavros 2y ago
    We've run out of training data that definitely did not contain LLM outputs.
    • DAGdug 2y ago
      What about non-text modalities - image and video, specifically?
      • riffraff 2y ago
        video is probably still fine, but images sourced from the internet now contain a massive amount of AI slop. It seems, for example, that many newsletters, blogs etc resort to using AI-generated images to give some color to their writings (which is something I too intended to do, before realizing how annoyed I am by it)
  • gcollard- 2y ago
    All the publicly accessible sources you mentioned have already been scraped or licensed to avoid legal issues. This is why it’s often said, “there’s no public data left to train on.” For evidence of this, consider observing non-English-speaking young children (ages 2–6) using ChatGPT’s voice mode. The multimodal model frequently interprets a significant portion of their speech as “thank you for watching my video,” reflecting child-like patterns learned from YouTube videos.
• zozbot234 2y ago
  Synthetic datasets are useless (other than for very specific purposes, such as enforcing known strong priors, and even then it's way better to do it directly by changing the architecture). You're better off spending that compute by making multiple passes over the data you do have.
  • HeatrayEnjoyer 2y ago
    This is contrary to what the big AI labs have found. Synthetic data is the new game in town.
    • kranke155 2y ago
      Ilya is saying it doesn’t work in this talk apparently.
      • seanhunter 2y ago
        Ilya could be wrong. I don’t think the question is decided yet in general. We already know that in lots of fields fake data can be used in ways that are as useful as or even more useful than the real thing[1], but in my understanding that tends to be situations where we have an objective function that is unambiguous and known beforehand. Meta has some very impressive work on synthetic data for training and my (uninformed) read was that is the state of the art in eg voice recognition at the moment.[2] [1] Eg Sobel sequences in a monte carlo simulation instead of real random numbers. They allow better coverage of the space of a simulation from fewer paths. https://www.sciencedirect.com/science/article/abs/pii/0041555367901449 https://www.sciencedirect.com/science/article/abs/pii/004155... [2] Seems a good overview is https://arxiv.org/html/2404.07503 https://arxiv.org/html/2404.07503
  • toxik 2y ago
    Most priors are not encodable as architecture though, or only partially.
• oldgradstudent 2y ago
  > There are massive proprietary datasets out there which people avoid using for training due to copyright concerns. The main legal concern is their unwillingness to pay to access these datasets.
  • zozbot234 2y ago
    Yup, there's also a huge amount of copyright-free, public domain content on the Internet which just has to be transcribed, and would provide plenty of valuable training to a LLM on all sorts of varied language use. (Then you could use RAG over some trusted set of data to provide the bare "facts" that the LLM is supposed to be talking about.) But guess what, writing down that content accurately from scans costs money (and no, existing OCR is nowhere near good enough), so the job is left to purely volunteer efforts.
• numpad0 2y ago
  > What will replace Internet data for training? It means unlimited scaling with Transformer LLM is over. They need a new architecture that scales better. Internet data respawns when they click [New Game...], oil analogy is an analogy and not a fact, but anyways total amount available in a single game is finite so combustion efficiency matters.
• sureglymop 2y ago
  Perhaps these models will be more personalized and there will be more personal data collection. I am currently building a platform for heavy personal data collection including a keylogger, logging mouse positions and window focus, screenshots à la recall, open browser tabs and much more. My goal is to gather data now that may become useful later. It's mainly for personal use but I'd he surprised if e.g. iphones weren't headed in the same direction.
• lo_fye 2y ago
  I think this will be the one thing that causes Google to revive its plan to scan all books in existence. They had started it, and built the machines to do it, and were making good progress... until Copyright hit them. BUT if they're not making the full text publicly accessible, and are "only" training AI on it, who knows if that would still be a problem. It's definitely a vast treasure trove of information, often with citations, and (presumably) hyper-linkable sources.