3 ms·

Does it look so different though? What happens when the LLMs are good enough to propose practical code improvements that will result in more-useful LLMs? And wh

Does it look so different though? What happens when the LLMs are good enough to propose practical code improvements that will result in more-useful LLMs? And when we start generating LLM-driven agents that actually train and deploy these more-useful LLMs? Are we taking on faith that we'll never get there? This is what a lot of skeptics fail to appreciate imo: That yes the current generation of models are not threatening, and on their own they probably never will be. But there will likely come a point where they can be used to recursively self-improve, and that can potentially lead to very dangerous scenarios.

c_crank 3y ago
Currently, one of the problems AI developers are dealing with is that LLMs training on data that is 'polluted' by AI generated content get dumber at predicting their desired outcomes. Even if there was a way to gain efficiency through self modification rather than losing it, it would likely hit a plateau of diminishing returns, like most things in the world do.
- yanderekko 3y ago
  >Even if there was a way to gain efficiency through self modification rather than losing it, it would likely hit a plateau of diminishing returns, like most things in the world do. And maybe you're fine with betting your life on that, but the people who are more-hesitant on this are not crazy.
  - c_crank 3y ago
    Most of them I would not call crazy, but relying on poor assumptions.
    - simonh 3y ago
      The fact that AI alignment is not yet a solved problem, and according to AI safety researchers we don’t have a clear path to solving it, is not an assumption. Further I don’t think that worrying about that in the context of possible super intelligent general AI is poor reasoning.
      - c_crank 3y ago
        AI safety researchers are prone to making wild and spurious claims about what a superintelligent AI is capable of, often relying on scenarios from science fiction or very out there fields of physics to try and get people to pay heed to their ideas. If I had a dime for every time 'possible human extinction' was brought up...
      - sverona 3y ago
        AI alignment is the closest thing to an unsolvable problem I've ever seen.
- throwaway4aday 3y ago
  That paper[0] mentions two defects, catastrophic forgetting which occurs during continual learning. GPT type models do not use this type of training and use supervised training instead which utilizes a curated corpus of text. The second is model collapse which is essentially just bad data making its way into the training set, once again a sufficiently curated corpus eliminates this problem. Microsoft has published a few papers about their research in using LLMs to train new LLM models. They have used training data consisting of essentially recorded chatGPT interactions which consist of the system prompt, user request and model response[1] to successfully train a smaller model that outperforms Vicuna-13B. The other paper[2] demonstrates using a combination of curated web data and machine generated textbooks and exercises (using GPT-3.5) and they produced a small model (1.3B) that outperforms several larger models on programming tasks. The point here is that the simple presence of machine generated content in the dataset is not an immediate detriment to the model being trained on it. It all hinges on the quality of the data regardless of whether it is of human or machine origin. Think about it, if the information is correct then there is no difference whether it came from a human or a machine. Machine generated data is not secretly cursed or other such nonsense. It's either correct or incorrect, nothing more. [0] https://arxiv.org/pdf/2305.17493v2.pdf https://arxiv.org/pdf/2305.17493v2.pdf [1] https://arxiv.org/pdf/2306.02707.pdf https://arxiv.org/pdf/2306.02707.pdf [2] https://arxiv.org/pdf/2306.11644.pdf https://arxiv.org/pdf/2306.11644.pdf

c_crank 3y ago

Currently, one of the problems AI developers are dealing with is that LLMs training on data that is 'polluted' by AI generated content get dumber at predicting their desired outcomes. Even if there was a way to gain efficiency through self modification rather than losing it, it would likely hit a plateau of diminishing returns, like most things in the world do.

yanderekko 3y ago
>Even if there was a way to gain efficiency through self modification rather than losing it, it would likely hit a plateau of diminishing returns, like most things in the world do. And maybe you're fine with betting your life on that, but the people who are more-hesitant on this are not crazy.
- c_crank 3y ago
  Most of them I would not call crazy, but relying on poor assumptions.
  - simonh 3y ago
    The fact that AI alignment is not yet a solved problem, and according to AI safety researchers we don’t have a clear path to solving it, is not an assumption. Further I don’t think that worrying about that in the context of possible super intelligent general AI is poor reasoning.
    - c_crank 3y ago
      AI safety researchers are prone to making wild and spurious claims about what a superintelligent AI is capable of, often relying on scenarios from science fiction or very out there fields of physics to try and get people to pay heed to their ideas. If I had a dime for every time 'possible human extinction' was brought up...
    - sverona 3y ago
      AI alignment is the closest thing to an unsolvable problem I've ever seen.
throwaway4aday 3y ago
That paper[0] mentions two defects, catastrophic forgetting which occurs during continual learning. GPT type models do not use this type of training and use supervised training instead which utilizes a curated corpus of text. The second is model collapse which is essentially just bad data making its way into the training set, once again a sufficiently curated corpus eliminates this problem. Microsoft has published a few papers about their research in using LLMs to train new LLM models. They have used training data consisting of essentially recorded chatGPT interactions which consist of the system prompt, user request and model response[1] to successfully train a smaller model that outperforms Vicuna-13B. The other paper[2] demonstrates using a combination of curated web data and machine generated textbooks and exercises (using GPT-3.5) and they produced a small model (1.3B) that outperforms several larger models on programming tasks. The point here is that the simple presence of machine generated content in the dataset is not an immediate detriment to the model being trained on it. It all hinges on the quality of the data regardless of whether it is of human or machine origin. Think about it, if the information is correct then there is no difference whether it came from a human or a machine. Machine generated data is not secretly cursed or other such nonsense. It's either correct or incorrect, nothing more. [0] https://arxiv.org/pdf/2305.17493v2.pdf https://arxiv.org/pdf/2305.17493v2.pdf [1] https://arxiv.org/pdf/2306.02707.pdf https://arxiv.org/pdf/2306.02707.pdf [2] https://arxiv.org/pdf/2306.11644.pdf https://arxiv.org/pdf/2306.11644.pdf