42 ms·

> which is just a belief that magic is real Is there a law of thermodynamics which prevents AI from writing code which would train a better AI? Never learned t

• advael 3y ago
  To be honest, I think a lot of smart people are willing to believe in magic when they've demonstrated some strong capability and the people funding their company want magic to happen.
  • killerstorm 3y ago
    It's not magic, though. If AI can do work of a human, it can do work of a human. It's a trivial statement, and inability to see it is a hard cope. Are you gonna to take a bet "AI won't be able to do X in 10 years" for some X which people can learn to do now? If you're unwilling to bet then you believe that AI would plausibly be able to perform any human job, including job of AI researcher.
    • advael 3y ago
      I don't claim it's impossible, just that there isn't a clear path from what exists now to that reality, and that the explanation presented by the above commenter (and I suppose OpenAI's website) does not clarify what they think the path is
      • hackerlight 3y ago
        What will AutoGPT look like if we have 100x more compute and another 10 years of research breakthroughs? It will be pretty damn good. If it can do the cognitive work of an AI researcher, well, there's your recursive self-improvement, at least on the research front (not so much on the hardware/energy front, physical constraints are trickier and will slow down progress in practice). I don't know the exact path there, because if I did I'd publish and win the Turing Award. But it seems to be a plausible outcome in the medium-term future, at least if you go with Hinton's view that current methods are capable of understanding and reasoning, and not LeCun's view that it's all a dead end.
        • advael 3y ago
          I won't comment on whether I believe those researchers hold those views as you describe them, but as you describe them, I think both those descriptions of the state of AI research are untrue. The capabilities demonstrated by transformer models seem necessary but not sufficient to understand and reason, meaning that while they're not necessarily a "dead end", it is far from guaranteed that adding more compute will get them there Of course if we allow for any arbitrary "research breakthrough" to happen then any outcome that's physically possible could happen, and I agree with you that superhuman artificial intelligence is possible. Nonetheless it remains unclear what research breakthroughs need to happen, how difficult they will be, and whether handing a company like OpenAI lots of money and chips will get that done, and it remains even more unclear whether that is a desirable outcome, given that the priorities of that company seem to shift considerably each time their budget is increased (As is the norm in this economic environment, to be clear, that is not a unique problem of OpenAI) Obviously OpenAI has every reason to claim that it can do this and to claim that it will use the results in a way designed to benefit humanity as a whole. The people writing this promotional copy and the people working there may even believe both of these things. However, based on the information available, I don't think the first claim is credible. The second claim becomes less credible the more of the company's original mission gets jettisoned as its priorities align more to its benefactors, which we have seen happen rather rapidly
      • killerstorm 3y ago
        We can reason about it without knowing the path. E.g. somebody in 1950s could say "If you have enough compute you can do photorealistic quality computer graphics". If you ask them how to build a GPU they won't know. Their statement is about principal possibility.
        • advael 3y ago
          Yes, and there are lots of predictions about when that would happen that turned out to be very wrong. Even if there is a clear path and specific people assigned to do a thing, it is famously always more difficult than expected for those people to correctly estimate how long it will take. Forgive me for being skeptical of random laypeople giving me timelines for an unknown unknown based on an objective that is ill-specified for work being done on something for which there is currently a lot of marketing hype
    • rdedev 3y ago
      At the end of the day it can only get as far as the data it has. Let's say you want to make a drug that inhibits a protein. The AI can generate plausible drugs but to see if it actually works you need to test it in the lab and then on an animal etc. now you can have an AI that has a perfect understanding of how a drug interacts with a protein but wesuch data is not available in the first place. Without that you can't just simply scale gpt type models
    • xanderlewis 3y ago
      ‘Doing the work of a human’ is something that is very hard to define or quantify in many cases. You sound very confident, but you don’t address this at all; you simply assume it’s a given. Relevant: https://www.jaakkoj.com/concepts/doorman-fallacy https://www.jaakkoj.com/concepts/doorman-fallacy
      • advael 3y ago
        Yea, I think we're in kind of a vicious recursive cycle of imperfect-metric-reinforcement (reward-hacking I suppose, though often implemented in economics as well as code) rather than one of recursive self-improvement in a more holistic sense. Optimization is really good at turning small problems of this nature into big ones more quickly
• woopsn 3y ago
  They do. Altman is saying their tech may be poised to capture the sum of all value in Earth's future light cone. Saying "well that is not physically impermissible" doesn't make it real. In any case nobody has ever shown that recursive self-improvement "takes off", and nor is that what we should expect a priori.
• rdedev 3y ago
  Even if recursive self improvement does work out my hunch is that is going to be logarithmic instead of exponential mostly down to just availability of data. It might go beyond human intelligence but I don't think it will reach singularity
  • lucubratory 3y ago
    This is why the big bet for AI-assisted AI-development long term is synthetic data. A big part of the reason so much money and resources is going into synthetic data right now is not just out of economic necessity, but because there have been extremely encouraging results with synthetic data (e.g. 'Textbooks Are All You Need', AlphaZero).
    • rdedev 3y ago
      I wouldn't count aplha zero since it's reinforcement learning. That technique you can generate high quality data all the time since the rules are fixed. Not everything can be trained using that way
      • lucubratory 3y ago
        The chess knowledge and skills of LLMs comes from them ingesting a sufficient number of chess games in text format (the amount will be proportional to both other data you have and the compute you have), same with the ability of LLMs to play other games or solve other fixed rule/perfect knowledge puzzles. AlphaZero and its cousins showed that you can generate an effectively infinite quantity of extremely high-quality data in those domains. There is a possibility that the benefit to an LLM's general intelligence from giving it e.g. one billion ~4600 ELO level games is only in improving its ability to play chess. Given the results many studies have reported in cross-learning with LLMs, I doubt that though. The potential is that generating a lot of extremely high level logic and puzzle solving and providing it as extremely high quality synthetic data to an LLM can improve its general reasoning and logic capabilities - that would be huge, and is one of the promises of synthetic data.
• astrange 3y ago
  > Is there a law of thermodynamics which prevents AI from writing code which would train a better AI? You need to apply Wittgenstein here. This appears to be true because you haven't defined "better". If you define it, it'll become obvious that this is either false or true, but if it is true it'll be obvious in a way that doesn't make it sound interesting anymore. (For one thing our current "AI" don't come from "writing code", they just come from training bigger models on the same data. For another, making changes to code doesn't make it exponentially better, and instead breaks it if you're not careful.) > I guess people working there believe in magic. Yes, OpenAI was literally founded by a computer worshipping religious cult. > People believe that self-improvement might happen when AI is around human-level. Humans don't have a "recursive self-improvement" ability. Also not obvious that an AI that was both "aligned" and "capable of recursive self-improvement" would choose to do it; if you're an AI and you're making a new improved AI, how do you know it's aligned? It sounds unsafe.
  • Tenobrus 3y ago
    it is very clear to me that humans do in fact have a recursive self-improvement ability, and i'm confused why you think otherwise
    • astrange 3y ago
      I think people can read books (self improvement) and have children (recursive), but neither of those are both.
      • lucubratory 3y ago
        Why do you think that the human population is more intelligent, knowledgeable, and achieves greater technological feats as time goes on? It's because of recursive self-improvement, we are raised and educated into being better in a quite general sense, which includes being better at raising and educating; nearly every generation this cycle repeats and has for all of human history, at least since we acquired language. We also build machines that help us to make better machines, and then we use those better machines to make even better machines, another example of recursive self-improvement.
• koe123 3y ago
  > I guess people working there believe in magic. I've been thinking about this recently. Personally, I've yet to see any compelling evidence that an LLM, let alone any AI, can operate really well "out of distribution". It's capabilities (in my experience) seem to be spanned by the data it's trained on. Hence, this supposed property that it can "train itself", generating new knowledge in the process, is yet to be proven in my mind. That raises the question for me: why do OpenAI staff believe what they believe? If I'm being optimistic, I suppose they may have seen unreleased tech, motivating their beliefs that seemingly AGI is on the horizon. If I'm being cynical, the promise of AGI probably draws in much more investment. Thus, anyone with a stake in OpenAI has an incentive to promote this narrative of imminent AGI, regardless of how realistic it is technically. This is of course just based on what I've seen and read, I'd love to see evidence that counter my claims.
  • killerstorm 3y ago
    The question is not whether it can work right now, but whether it is possible in the future (i.e. whether it's possible in principle). I think the concern about out-of-distribution is overstated. If we train it on predicting machine learning papers, writing machine learning papers is not out-of-distribution. You might say "but writing NOVEL papers" would be OOD; but there's no sharp boundary between old and new. Model's behavior is usually smooth, so it's not like it will output random bs if you try to predict 2025 papers. And predicting 2025 papers in 2024 all we need to do "recursive self-improvement". (There are also many ways to shift distribution towards where you want it to be, e.g. aesthetics tuning, guidance in diffusion models, etc. Midjourney does not faithfully replicate distribution in the input training set, it's specifically tuned to create more pleasing outputs. So I don't see "oh but we don't have 2025 papers in the training set yet!" being an insurmountable problem.) But more generally, seeing models as interpolators is useful only to some extent. We use statistical language when training the models, but that doesn't mean that all output should be interpreted as statistics. E.g. suppose I trained a model which generates a plausible proofs. I can combine it with proof-checker (which is much easier than generating a proof), and wrap it into a single function `generate_proof` which is guaranteed to generate a correct proof (it will loop until a plausible proof checks out). Now the statistics do not matter much. It's just a function. If there's such a thing as a general reasoning step, then all we need is a function which perform that. Then we just add an outer loop to explore a tree of possibilities using these steps. And further improvements might be in making these steps faster and better. Does reasoning generalize? I'd say everything points to "yes". Math is used in variety of fields. We are yet to find something where math doesn't work. If you get somebody educated in mathematical modeling and give them a new field to model, they won't complain about math being out-of-distribution. If you look at LLMs today, they struggle with outputting JSON. It's clearly not an out-of-distribution problem, it's a problem with training - the dataset was too noisy, it had too many examples where somebody requests a JSON but gets a JSON-wrapped-in-Markdown. It's just an annoying data cleanup problem, nothing fundamental. I think it's reasonable to assume that within 5 years OpenAI, Google, etc, will manage to clean up their datasets and train more capable, reliable models which demonstrate good reasoning capabilities. FWIW I believe that if we hit a wall on a road towards AGI that might actually be good to buy more time to research what we actually want out of AGI. But I doubt that any wall will last more than 5 years, as it already seems almost within the reach...