8 ms·

The Myth of AI Omniscience: AI's Epistemological Limits

83 points by cpwalker 3y ago

mtkhaos 3y ago
Reading through there is a heavy fault in logic. Statically reasonable limitations to llm output based on training material, ignores hallucinations. And hallucinations ignore the sheer chance of new emergent information by odds.
- K0balt 3y ago
  Hallucinations are not a different process from normal output. They are merely called hallucinations rather than insights because they are wrong. There is no magic in LLMs , nor in human thought. It is all a matter of symbol vectorisation and inferring relationships within the multidimensional memetic matrix. AI absolutely can achieve original insight- but not one that humans could not also make if looking at the same data. This is because the “intelligence” in generative AI is human bounded in the training data, not in the engine that processes it. I strongly suspect the same is true of humans. After all, we modelled neural networks after our own brains, why would we not expect them to achieve similar results using similar means? Wasn’t that the whole point? It never ceases to amaze me how people go all pikachu face when I say that our thought process is probably a lot like generative models. Step one, make a facsimile of thing. Step 2: be surprised when thing can also be viewed as being similar to the facsimile. Lol. Hubris is our primary characteristic.
  - mtkhaos 3y ago
    By odds and in the context of RLHF the human could very well thumbs up the output without recognition of said hallucination. As hallucination is a general term giving to said phenomenon. Otherwise the question Emerges why 2023? And why was that not known colloquially before hand. When the basis of these Algorithms come the the 80s?
    - K0balt 3y ago
      Because models got sophisticated enough that wrong outputs started looking plausible answers rather than random garbage. Sometimes they still spew random garbage though.
      - mtkhaos 3y ago
        But isn't a binary absolute in this case and cannot be used in the basis of first principle's assumption of truth.
deleted 3y ago
[deleted]
ozy 3y ago
Why would AGI be bound to human made models? Why can it no develop its own?
- K0balt 3y ago
  Recursive insight is possible with a model that self trains, but right now that would result in a detour into unreality. Perhaps with the right systems of vetting prior to incorporating new data into the retraining set. Right now they just get stupider if you train them on their own output, which suggests that the quality of the data available in the training set is higher than the quality of output produced by the model as a general rule. The fidelity is < 1.0 . Apparently, it is possible to achieve fidelity >1 (the growth of human knowledge) but our algorithms are not so great at this point, it seems.
  - visarga 3y ago
    Not necessarily. For example Anthropic's ConstitutionalAI (CAI) leverages the model to substitute human judgments in RLHF, effectuating essentially RLAIF. CAI information is used to fine-tune the Claude model. Broadly speaking, you require statistics at echelon N+1 when you are at rung N. We can amplify models by providing them additional time, self-reflexion, demand step by step planning, allow external tools, tune it on human preferences, or give it feedback from executing a code, or from a robot.
    - K0balt 3y ago
      Yeah, it makes some sense that you could use a more intense introspection to train weaker ones… I wonder what the human analogue for that looks like. Maybe working up a proof and then quizzing yourself on it? As long as we get >N supervision and the difference is more than the model retrograde, it seems that could work. But it seems like there is a definite limit to that. The N-n1 difference will only stay above the improvement delta up to a point.
      - visarga 3y ago
        The model would learn from feedback, not just regurgitate the training set, as long as the model is part of a system that can generate this feedback. AlphaGo Zero had self play for feedback. Robots can check task execution success. Even chatting with us generates feedback to the model.
joak 3y ago
«xAI, a venture whose ambitious aim is to “understand the true nature of the universe.”» C'mon, we already have the answer it's 42
tudorw 3y ago
I'm interested in seeing how brains change as AI usage increases, in the same way many of us don't attempt to memorize phone numbers, what skills will we offload to 'AI', will we be aware that it's happening? Perhaps more interesting, what new adaptations or adjustments may occur as a result of this augmentation. I hope that our brains ability to synthesize new thoughts both extrapolated and 'from thin air' will remain a useful skill regardless.
- zarzavat 3y ago
  The problem with AI is that it doesn’t really free you to do anything else, because as soon as it does, that thing becomes the next open research problem. For example, you could say that ChatGPT frees writers to focus on creativity. But someone somewhere is working on making LLMs more creative and less formulaic.
K0balt 3y ago
Rather than viewing generative AI as a form of artificial intelligence, I posit that it should be seen as an automated tool for tapping into human cultural, linguistic, and empirical knowledge. Data and computation are two sides of the same coin. The 'intelligence' in AI is embedded within the data, with the computational model serving as a tool to access and express this inherent intelligence. I would argue for a change in perspective towards AI, one that recognizes LLMs as powerful tools for accessing the vast wealth of human cultural knowledge rather than viewing them as a separate form of intelligence. We must carefully consider critical ethical considerations about control, access, and trust that will become increasingly relevant as these tools become more integrated into our everyday lives. This paradigm shift carries some implications: LLMs will not achieve superintelligence: Although these models can process information quickly and access a wide range of knowledge, they lack the superior reasoning or inference abilities that would classify them as superintelligence. LLMs as an extension of human thought: These models can automate and amplify human capabilities but do not introduce new abilities beyond what is already present in human thought processes. LLMs as mirrors of human culture and knowledge: These models reflect the recorded artifacts of human language, art, and culture. They can make the inherent intelligence within these artifacts accessible, providing a vast information resource. Implications for the future: Access to this "memetic matrix" of human knowledge will become a fundamental part of being human as these tools become more integrated into our lives, bringing up issues of ownership, access, and the potential for misuse. Thought consolidation and control of inference engines: There's a potential risk that control of inference engines by a small number of companies could lead to a consolidation of thought that threatens democratic governance. I propose a diversity of federated or self-hosted inference tools as solutions to mitigate this risk. The necessity for trust and individuality: As these tools become more influential in our lives, maintaining trust in our individual thoughts and avoiding the uncritical acceptance of synthesized ideas from sources with opaque motives will become increasingly important. Synthetic Inference relies on a vast cultural commons: We cannot allow these commons to be closed off and owned by a few big companies. This resource is to approximate totality of all human knowledge, language, and culture. It belongs to all of humanity. Training data must be open, free, and available for examination.
- Davidzheng 3y ago
optimalsolver 3y ago
>OpenAI, the company behind ChatGPT, recently announced a massive investment into research on “superalignment,” The head of the “superalignment” project at OpenAI recently appeared on the AXRP podcast talking about the Automated Alignment Researcher they’re developing. The host asked him how they were going to align the AAR itself, and if they can do that, what would be left for the AAR to do, but he didn’t seem to understand the question. Not encouraging signs.
famouswaffles 3y ago
LLMs don't learn to simulate or mimic, that's just a byproduct. They learn to predict the training corpus. There is absolutely nothing about the act of prediction that necessitates an upper bound of intelligence on the corpus itself. https://www.pnas.org/doi/full/10.1073/pnas.2016239118 https://www.pnas.org/doi/full/10.1073/pnas.2016239118 They found representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity in an LLM only trained to predict protein structure sequences. No folding, nothing explicit in the corpus itself. Yet those truths manifest as a necessity of accurate predictions. Prediction is not bound by what the data explicitly shows. >A language model’s vocabulary is limited to the words that exist within the model’s training texts, which means a LLM can only refer to objects and relations that we humans have already discerned, named, and written about. This is not true and is easy enough to test.
- dvt 3y ago
  > This is not true and is easy enough to test. How exactly is this not true? Embeddings are literally a mapping of (English) words to numbers.
  - Davidzheng 3y ago
    But because the tokens are not word level they are capable of making up words that humans have never used before.
    - dvt 3y ago
      Most tokens are word or sub-word level. A token can be "find" or two tokens can be "find" and "ing." To argue that this isn't inherently using the underlying grammar rules is plainly wrong. To argue that an LLM trained on English text is capable of "making up words" (in any meaningful sense of that sentence) is even more wrong.
      - _ea1k 3y ago
        You might think it is wrong, but I think it is a perfectly cromulent case of AI "making up words".
simonh 3y ago
The article is claiming that advanced AIs cannot become more intelligent than humans, essentially because LLMs cannot become more intelligent than humans. LLMs aren’t the be all and end all of AI though. They’re an impressive but inherently limited stepping stone, with a very constrained scope of applicability and capabilities. There is no reason to suppose that future, much more advanced architectures can’t surpass humans in reasoning ability. Advanced AI doesn’t have to be omniscient fir us not yon be able to predict it’s capabilities, it just needs to be smarter than us. By definition if something is smarter than you, it’s not possible for you to anticipate and predict its behaviour. Therefore in principle it is not possible to predict whether that behaviour will be to your liking. That’s the problem.
- rst 3y ago
  More specifically, the article assumes that the only way "AI"s will ever be produced is the way that LLMs are -- by being trained off pre-existing human generated media, and existing as static objects after that. In particular, this assumes that the system doesn't subsequently modify itself based on further interactions with the real world (e.g., doing experiments, and taking note of the results). That's assuming an awful lot, to put it mildly.
  - simonh 3y ago
    It's assuming something we know to be false. AlphaZero wasn't even trained against humans at all, but still beats them at Go and Chess handily.
- wrsh07 3y ago
  I think the paper would similarly claim something like "a chess ai trained only on human games can't play better than the humans who played those games" I think that's false - you could imagine it playing at GM level without making mistakes that real GMs make But it's missing an important fact: chess ais aren't limited to learning only from human games. They can learn from self play. You can extend this to physics (if we can simulate physics precisely enough, there's no reason ai can't make fundamental discoveries in physics)
obiefernandez 3y ago
he lost me at "a language model’s vocabulary is limited to the words that exist within the model’s training texts, which means a LLM can only refer to objects and relations that we humans have already discerned, named, and written about." This is trivially demonstrated to be a false statement, as GPT is capable of synthesizing entirely novel words based on very little input guidance.
- epups 3y ago
  I find it exhausting when articles are written about ChatGPT which can be trivially demonstrated false by simply testing their assumptions. Noam Chomsky pulled a similar one a while ago.
  - mahathu 3y ago
    Whats up with the number of mind-numbingly stupid takes in this thread? Are you actually this dumb or is there some elaborate in-joke i missed? Just because LLMs can (poorly) come up with new words when you instruct them to doesn't mean they're actually "thinking" in novel concepts, its just another indirectly grounded label for existing concepts. https://dai.fmph.uniba.sk/~retova/CSCTR/materials/CSCTR_07sem_Barsalou_perc_sym_sys_1999.pdf https://dai.fmph.uniba.sk/~retova/CSCTR/materials/CSCTR_07se...
Xcelerate 3y ago
I found this article a bit fluffy and lacking on technical substance. If we want to entertain the limits of AGI, we need to look at concepts like Solomonoff induction, which were conceived of and researched in the 1960s (and apparently mostly forgotten about since then). Given an input sequence of bits, what is the best (i.e., “oracle”) posterior distribution possible given a domain prior and the universal prior? How would a resource-limited approximation widen this posterior distribution? What are the practical limits to achieving the shortest Turing machines that reproduce the input sequence? These are interesting questions that should be looked into if we want to discuss AGI in the epistemological limit. Also, I tend to get frustrated with predictions by “AI experts” about what a future with AGI will look like. We don’t have any world experts on AGI currently, for the simple reason that it doesn’t exist yet. We have experts (or I’d prefer to say “first movers”) on particular types of machine learning based upon neural networks, but that’s about it. A true AGI would almost be unpredictable by definition, so human attempts to forecast the future in this regard are mostly sensationalism or a projection of some kind of personal agenda (cough cough Marc Andreessen).
johndough 3y ago
> To sum up: because LLMs are fundamentally limited to i) using our vocabulary, ii) “understanding” concepts in the ways we do, and iii) “talking” in the ways we do, then, at best, a LLM can only mirror back to us the order we have carved, the truth we have honed. That conclusion is not correct. Generative models can combine existing concepts in novel ways which have never been considered before. This is most easily seen with text-to-image models. For example the teddy bear swimming in an Olympic pool [1] or many other surreal examples [2] were not part of the training set. So as long as an answer can be expressed as a combination of primitive concepts, LLMs can generate them. [1] https://imagen.research.google/main_gallery_images/teddy-bear-swimming-butterfly.jpg https://imagen.research.google/main_gallery_images/teddy-bea... [2] https://imagen.research.google/ https://imagen.research.google/
- Capricorn2481 3y ago
  That's not an LLM, though. We haven't seen evidence yet that an LLM can combine existing language in novel ways. In fact, we've seen over and over that LLMs are quite generic. Compared to text-to-image, which is seemingly impossible to use without getting something weird
  - glenstein 3y ago
    Is there something about the nature of language and linguistic meaning that would make it difficult to combine language in novel ways?
    - TuringTourist 3y ago
      I would posit the opposite, language exists to be combined in any number of ways easily and still communicate well. A language that cannot do this does not I think exist.
      - glenstein 3y ago
        Same. I find it hard to makes sense of why anyone would claim that computers can innovate visually but not with language. It would make more sense if someone at least felt that they both could, or both couldn't.
gibsonf1 3y ago
It seems silly to even mention epistemology when current AI has no Intelligence at all yet.
Procrastes 3y ago
This focuses on the argument that some hyper-sophisticated future LLM will not discover the secrets of the universe on its own. That's plausible, but beside the point when it comes to the potential dangers of A.I. Not all future A.I.s will be LLMs. Understanding the universe isn't the only way to pose an existential threat. This is an old discussion with much deeper insights already available than I can parrot here, but one obvious counter just to this argument is that even an LLM can be an existential threat, depending what it can convince you it knows.
lastingwin 3y ago
I think it's safe to say that we should disregard any substack/medium posts, especially when it comes to AI.
progrus 3y ago
Yay, we need more of this before OpenAI finishes hypnotizing the global south (lol).
ta8645 3y ago
People are still desperate to maintain humankind at the top of the intellectual pyramid. It's tiresome and a waste of energy. There are definitely limits to intelligence, nothing will ever be smart enough to predict the future perfectly. But to believe that humans represent the peak of what's possible, is just hubris and nothing else.
visarga 3y ago
> Future LLMs, regardless of model architecture, are fundamentally constrained in this way, by virtue of the fact that they are trained on human-written texts. Therefore, the outputs of a LLM, at best, reflect our current understanding of the universe and nothing more. That constraint holds only for pure LLMs. There is a secondary source of learning - external feedback to the model. When a LLM is part of a larger system, it can integrate feedback to improve itself, like AlphaGo Zero. A LLM+compiler could self-optimize by observing compilation errors and program outputs. A LLM+game could self-improve by conditioning actions on the score. Humans could create a preference dataset and model for RLHF to refine the LLM based on human feedback. A LLM+robot could learn from past experiences to improve its planning. Similarly, humans need tools to conduct research. We can't do pure research without labs and experiments. LLMs need external confirmation the same way we do. The outputs of a pure LLM merely reflect our current understanding of the universe. But coupled with real-world feedback, LLMs have the potential to learn and discover new knowledge.
CuriouslyC 3y ago
This is plainly false. With reinforcement learning, an objective function and the ability to generate random solutions it's possible for AI to stumble upon sequences/strategies that greatly outperform human state of the art.
RoboTeddy 3y ago
> But the ways in which a LLM can “talk about” the universe (and everything it contains) are limited to the ways in which humans have previously talked about the universe. This is often said, but it isn't so. The task of predicting the next token in human speech really well requires immense intelligence — potentially far more intelligence than possessed by the original speaker! Imagine yourself engaging in the task of listening to someone who isn't that smart speak and then trying to figure out what they'll say next — in doing so, you might make all sorts of extrapolations about the person, their motivations, their manner, their dialect, etc — calling on all sorts of internal models that you've built up about people over time. This is what models are being trained to do when we train them on predicting tokens. There are concrete examples of models inventing new ways of thinking that are not described in their training set. For example, when training a transformer from scratch to perform addition mod P (and having no training data other than examples of addition mod P), the transformer was able to discover the use of discrete fourier transforms and trigonometric identities [1]. As we can see, neural nets can build all sorts of internal mental models that no one explained to them beforehand. These internal mental models can then be elicited and used for other purposes by e.g. fine-tuning. I think a good mental model for transformers/neural nets is that they're automatic scientists. They figure out ways of modeling things in order to predict the output from the input — which is what scientists do! As part of this, they can de-facto discover new theories, and come to rely on the theories that prove useful in their prediction task. Also, not all tokens in the training set are from human speech, so models are being trained to model all manner of data-generating processes. [1] https://arxiv.org/pdf/2301.05217.pdf https://arxiv.org/pdf/2301.05217.pdf
- gremlinunderway 3y ago
  This is a great example when it comes to trying to understand the epistemological limits of AI. People inherently fall back on an argument that presumes the human mind works like a neural network. There's an interesting theory that I've never been able to identify the origins of, that humans like to think that we created technology from needs based on our models of the world, whereas we ignore the effects of daily technology on our thinking frameworks. The argument essentially states that we never "discovered" the circulatory system of the heart or "discovered" it works like a pump and valves, rather instead right around the same time this theoretical work was being investigated on the heart is when the industrial revolution was in full swing. Thus, we modelled the heart as pumps and valves because that's the technology we were surrounded by. The heart isn't somehow inherently a "pump" and we "discovered" that, we just started using the pump metaphor because it seemed to help do other things. I'm sure though that the metaphor has it limits. Typically, the narrative around the invention of machine learning models is that we started coding computers to be more like recently "discovered" models of the brain. Under this theory its the opposite. Right as cognitive sciences started developing as a novel field of research is when we developed computers. So, in classic form we decomposed the brain in atomic fashion, and the 'atoms' of measurement we chose to use ended up being bits and bytes. Distinguishing between inventing "novel" things and gobbledygook is completely subjective and based on the viewer's own models. It's proving these abilities after the fact, not before it. Thousand monkeys on typewriters etc. This measurement of "accuracy" is completely forgetting everything that Kuhn discovered about scientific knowledge. If you've got a community of like 3 people who research some incredibly esoteric scientific field, only those 3 people could ever accurately judge the full extent of their domain. A model could generate a series of tokens that for the rest of the world is gobbledygook, but to these 3 scientists it makes perfect sense. This doesn't really endeavour me to believe that there's anything "novel" about what AI "predicts". It just throws out enough combinations and we conveniently ignore the huge gaps when its wrong, but then jump up and down excitedly when it's "right" (as if its discovered some universal material objective truth).
waffletower 3y ago
"Therefore, the outputs of a LLM, at best, reflect our current understanding of the universe and nothing more." Others have reflected on the problems with this statement, and I agree, LLMs can be prompted to navigate the training corpus and create novel output, countering this claim. However, these criticisms neglect to credit the external intelligence inherent to the prompts themselves. AutoGPT and its ilk have yet to create convincing agency: for the time being the current state of AI is a mere extension of human intelligence and will not be making lofty discoveries without direct human interaction. LLMs are a form of human co-intelligence.
entropyneur 3y ago
LLMs are trained on human-written texts, therefore their output can only reflect our current understanding - am I reading this right? Hopefully not as that would be insultingly wrong.
arisAlexis 3y ago
Anything with started with "the myth" arguing about something that is at the limits of knowledge without a clear answer is unavoidably biased by a person with big ego.
mannykannot 3y ago
Putting aside the question of what LLMs, specifically, are capable of, I feel the author's central argument has already been shown to be false, through events outside of AI. In a nutshell, the argument is that, as AI will have just the knowledge that is available to us, expressed in the languages we use, we will be able to anticipate and put a stop to it doing anything catastrophic. Exactly the same argument, however, could be made over computer and communications security. If anything, those trying to keep things secure have the advantage in knowledge, at least initially, yet zero-day exploits keep surfacing, some of them at fundamental levels such as processor architecture. Nothing truly catastrophic has happened yet, but it cannot be said to be impossible. The flaw in the argument is that one can easily know all the bare facts without seeing all the implications but someone - or something - else could do so.
ilaksh 3y ago
LLMs have obvious limitations and a big contributor is the training data and the way they are trained in a supervised way. Although that is not as much of a limitation as with most other AIs because of the "in-context learning" ability. But that doesn't mean that future LLMs couldn't conceivably learn abstractions than are at a higher level or even perhaps inaccessible to humans in number of layers or complexity. I think the obvious type of superintelligence that is only a few years away is what I call hyperspeed AI. LLMs are a very specific application which we will be able to accelerate greatly. Within a decade the output speed will be dozens of times faster than human thought. Because they will have robust reasoning and extreme "thinking speed", these AIs will probably be connected to industry and military applications. Humans will need to be removed from the loop because waiting for a human decision means the competitors' AIs race ahead the equivalent of days or weeks. If this is fine then it will create a precarious situation for humans where they are only nominally in control and things like advanced AI "viruses" might become very dangerous. It's also certain that researchers will continue to develop new types of AIs that incorporate more animal (such as human)-like capabilities.
creer 3y ago
Why do we need anything close to superintelligence or "omniscience" (wtf) before we get profound changes to how humanity operates. It's really an obsession! Instead compare to the average human, at most. Compare to refrigeration? Compare to fire? Compare to computers? And that's even if the bar is placed at the singularity. Human progress as we have known it so far is not founded on superhumans but on a lot of work by a mix of merely visionary, hard working, and just plain average people. Just speeding that an order of magnitude changes everything.

83 points by cpwalker 3y ago

Reading through there is a heavy fault in logic. Statically reasonable limitations to llm output based on training material, ignores hallucinations. And hallucinations ignore the sheer chance of new emergent information by odds.

K0balt 3y ago
Hallucinations are not a different process from normal output. They are merely called hallucinations rather than insights because they are wrong. There is no magic in LLMs , nor in human thought. It is all a matter of symbol vectorisation and inferring relationships within the multidimensional memetic matrix. AI absolutely can achieve original insight- but not one that humans could not also make if looking at the same data. This is because the “intelligence” in generative AI is human bounded in the training data, not in the engine that processes it. I strongly suspect the same is true of humans. After all, we modelled neural networks after our own brains, why would we not expect them to achieve similar results using similar means? Wasn’t that the whole point? It never ceases to amaze me how people go all pikachu face when I say that our thought process is probably a lot like generative models. Step one, make a facsimile of thing. Step 2: be surprised when thing can also be viewed as being similar to the facsimile. Lol. Hubris is our primary characteristic.
- mtkhaos 3y ago
  By odds and in the context of RLHF the human could very well thumbs up the output without recognition of said hallucination. As hallucination is a general term giving to said phenomenon. Otherwise the question Emerges why 2023? And why was that not known colloquially before hand. When the basis of these Algorithms come the the 80s?
  - K0balt 3y ago
    Because models got sophisticated enough that wrong outputs started looking plausible answers rather than random garbage. Sometimes they still spew random garbage though.
    - mtkhaos 3y ago
      But isn't a binary absolute in this case and cannot be used in the basis of first principle's assumption of truth.

deleted 3y ago

[deleted]

ozy 3y ago

Why would AGI be bound to human made models? Why can it no develop its own?

K0balt 3y ago
Recursive insight is possible with a model that self trains, but right now that would result in a detour into unreality. Perhaps with the right systems of vetting prior to incorporating new data into the retraining set. Right now they just get stupider if you train them on their own output, which suggests that the quality of the data available in the training set is higher than the quality of output produced by the model as a general rule. The fidelity is < 1.0 . Apparently, it is possible to achieve fidelity >1 (the growth of human knowledge) but our algorithms are not so great at this point, it seems.
- visarga 3y ago
  Not necessarily. For example Anthropic's ConstitutionalAI (CAI) leverages the model to substitute human judgments in RLHF, effectuating essentially RLAIF. CAI information is used to fine-tune the Claude model. Broadly speaking, you require statistics at echelon N+1 when you are at rung N. We can amplify models by providing them additional time, self-reflexion, demand step by step planning, allow external tools, tune it on human preferences, or give it feedback from executing a code, or from a robot.
  - K0balt 3y ago
    Yeah, it makes some sense that you could use a more intense introspection to train weaker ones… I wonder what the human analogue for that looks like. Maybe working up a proof and then quizzing yourself on it? As long as we get >N supervision and the difference is more than the model retrograde, it seems that could work. But it seems like there is a definite limit to that. The N-n1 difference will only stay above the improvement delta up to a point.
    - visarga 3y ago
      The model would learn from feedback, not just regurgitate the training set, as long as the model is part of a system that can generate this feedback. AlphaGo Zero had self play for feedback. Robots can check task execution success. Even chatting with us generates feedback to the model.

joak 3y ago

«xAI, a venture whose ambitious aim is to “understand the true nature of the universe.”» C'mon, we already have the answer it's 42

tudorw 3y ago

I'm interested in seeing how brains change as AI usage increases, in the same way many of us don't attempt to memorize phone numbers, what skills will we offload to 'AI', will we be aware that it's happening? Perhaps more interesting, what new adaptations or adjustments may occur as a result of this augmentation. I hope that our brains ability to synthesize new thoughts both extrapolated and 'from thin air' will remain a useful skill regardless.

zarzavat 3y ago
The problem with AI is that it doesn’t really free you to do anything else, because as soon as it does, that thing becomes the next open research problem. For example, you could say that ChatGPT frees writers to focus on creativity. But someone somewhere is working on making LLMs more creative and less formulaic.

K0balt 3y ago

Rather than viewing generative AI as a form of artificial intelligence, I posit that it should be seen as an automated tool for tapping into human cultural, linguistic, and empirical knowledge. Data and computation are two sides of the same coin. The 'intelligence' in AI is embedded within the data, with the computational model serving as a tool to access and express this inherent intelligence. I would argue for a change in perspective towards AI, one that recognizes LLMs as powerful tools for accessing the vast wealth of human cultural knowledge rather than viewing them as a separate form of intelligence. We must carefully consider critical ethical considerations about control, access, and trust that will become increasingly relevant as these tools become more integrated into our everyday lives. This paradigm shift carries some implications: LLMs will not achieve superintelligence: Although these models can process information quickly and access a wide range of knowledge, they lack the superior reasoning or inference abilities that would classify them as superintelligence. LLMs as an extension of human thought: These models can automate and amplify human capabilities but do not introduce new abilities beyond what is already present in human thought processes. LLMs as mirrors of human culture and knowledge: These models reflect the recorded artifacts of human language, art, and culture. They can make the inherent intelligence within these artifacts accessible, providing a vast information resource. Implications for the future: Access to this "memetic matrix" of human knowledge will become a fundamental part of being human as these tools become more integrated into our lives, bringing up issues of ownership, access, and the potential for misuse. Thought consolidation and control of inference engines: There's a potential risk that control of inference engines by a small number of companies could lead to a consolidation of thought that threatens democratic governance. I propose a diversity of federated or self-hosted inference tools as solutions to mitigate this risk. The necessity for trust and individuality: As these tools become more influential in our lives, maintaining trust in our individual thoughts and avoiding the uncritical acceptance of synthesized ideas from sources with opaque motives will become increasingly important. Synthetic Inference relies on a vast cultural commons: We cannot allow these commons to be closed off and owned by a few big companies. This resource is to approximate totality of all human knowledge, language, and culture. It belongs to all of humanity. Training data must be open, free, and available for examination.

Davidzheng 3y ago

optimalsolver 3y ago

>OpenAI, the company behind ChatGPT, recently announced a massive investment into research on “superalignment,” The head of the “superalignment” project at OpenAI recently appeared on the AXRP podcast talking about the Automated Alignment Researcher they’re developing. The host asked him how they were going to align the AAR itself, and if they can do that, what would be left for the AAR to do, but he didn’t seem to understand the question. Not encouraging signs.

famouswaffles 3y ago

LLMs don't learn to simulate or mimic, that's just a byproduct. They learn to predict the training corpus. There is absolutely nothing about the act of prediction that necessitates an upper bound of intelligence on the corpus itself. https://www.pnas.org/doi/full/10.1073/pnas.2016239118 https://www.pnas.org/doi/full/10.1073/pnas.2016239118 They found representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity in an LLM only trained to predict protein structure sequences. No folding, nothing explicit in the corpus itself. Yet those truths manifest as a necessity of accurate predictions. Prediction is not bound by what the data explicitly shows. >A language model’s vocabulary is limited to the words that exist within the model’s training texts, which means a LLM can only refer to objects and relations that we humans have already discerned, named, and written about. This is not true and is easy enough to test.

dvt 3y ago
> This is not true and is easy enough to test. How exactly is this not true? Embeddings are literally a mapping of (English) words to numbers.
- Davidzheng 3y ago
  But because the tokens are not word level they are capable of making up words that humans have never used before.
  - dvt 3y ago
    Most tokens are word or sub-word level. A token can be "find" or two tokens can be "find" and "ing." To argue that this isn't inherently using the underlying grammar rules is plainly wrong. To argue that an LLM trained on English text is capable of "making up words" (in any meaningful sense of that sentence) is even more wrong.
    - _ea1k 3y ago
      You might think it is wrong, but I think it is a perfectly cromulent case of AI "making up words".

simonh 3y ago

The article is claiming that advanced AIs cannot become more intelligent than humans, essentially because LLMs cannot become more intelligent than humans. LLMs aren’t the be all and end all of AI though. They’re an impressive but inherently limited stepping stone, with a very constrained scope of applicability and capabilities. There is no reason to suppose that future, much more advanced architectures can’t surpass humans in reasoning ability. Advanced AI doesn’t have to be omniscient fir us not yon be able to predict it’s capabilities, it just needs to be smarter than us. By definition if something is smarter than you, it’s not possible for you to anticipate and predict its behaviour. Therefore in principle it is not possible to predict whether that behaviour will be to your liking. That’s the problem.

rst 3y ago
More specifically, the article assumes that the only way "AI"s will ever be produced is the way that LLMs are -- by being trained off pre-existing human generated media, and existing as static objects after that. In particular, this assumes that the system doesn't subsequently modify itself based on further interactions with the real world (e.g., doing experiments, and taking note of the results). That's assuming an awful lot, to put it mildly.
- simonh 3y ago
  It's assuming something we know to be false. AlphaZero wasn't even trained against humans at all, but still beats them at Go and Chess handily.
wrsh07 3y ago
I think the paper would similarly claim something like "a chess ai trained only on human games can't play better than the humans who played those games" I think that's false - you could imagine it playing at GM level without making mistakes that real GMs make But it's missing an important fact: chess ais aren't limited to learning only from human games. They can learn from self play. You can extend this to physics (if we can simulate physics precisely enough, there's no reason ai can't make fundamental discoveries in physics)

obiefernandez 3y ago

he lost me at "a language model’s vocabulary is limited to the words that exist within the model’s training texts, which means a LLM can only refer to objects and relations that we humans have already discerned, named, and written about." This is trivially demonstrated to be a false statement, as GPT is capable of synthesizing entirely novel words based on very little input guidance.

epups 3y ago
I find it exhausting when articles are written about ChatGPT which can be trivially demonstrated false by simply testing their assumptions. Noam Chomsky pulled a similar one a while ago.
- mahathu 3y ago
  Whats up with the number of mind-numbingly stupid takes in this thread? Are you actually this dumb or is there some elaborate in-joke i missed? Just because LLMs can (poorly) come up with new words when you instruct them to doesn't mean they're actually "thinking" in novel concepts, its just another indirectly grounded label for existing concepts. https://dai.fmph.uniba.sk/~retova/CSCTR/materials/CSCTR_07sem_Barsalou_perc_sym_sys_1999.pdf https://dai.fmph.uniba.sk/~retova/CSCTR/materials/CSCTR_07se...

Xcelerate 3y ago

I found this article a bit fluffy and lacking on technical substance. If we want to entertain the limits of AGI, we need to look at concepts like Solomonoff induction, which were conceived of and researched in the 1960s (and apparently mostly forgotten about since then). Given an input sequence of bits, what is the best (i.e., “oracle”) posterior distribution possible given a domain prior and the universal prior? How would a resource-limited approximation widen this posterior distribution? What are the practical limits to achieving the shortest Turing machines that reproduce the input sequence? These are interesting questions that should be looked into if we want to discuss AGI in the epistemological limit. Also, I tend to get frustrated with predictions by “AI experts” about what a future with AGI will look like. We don’t have any world experts on AGI currently, for the simple reason that it doesn’t exist yet. We have experts (or I’d prefer to say “first movers”) on particular types of machine learning based upon neural networks, but that’s about it. A true AGI would almost be unpredictable by definition, so human attempts to forecast the future in this regard are mostly sensationalism or a projection of some kind of personal agenda (cough cough Marc Andreessen).

johndough 3y ago

> To sum up: because LLMs are fundamentally limited to i) using our vocabulary, ii) “understanding” concepts in the ways we do, and iii) “talking” in the ways we do, then, at best, a LLM can only mirror back to us the order we have carved, the truth we have honed. That conclusion is not correct. Generative models can combine existing concepts in novel ways which have never been considered before. This is most easily seen with text-to-image models. For example the teddy bear swimming in an Olympic pool [1] or many other surreal examples [2] were not part of the training set. So as long as an answer can be expressed as a combination of primitive concepts, LLMs can generate them. [1] https://imagen.research.google/main_gallery_images/teddy-bear-swimming-butterfly.jpg https://imagen.research.google/main_gallery_images/teddy-bea... [2] https://imagen.research.google/ https://imagen.research.google/

Capricorn2481 3y ago
That's not an LLM, though. We haven't seen evidence yet that an LLM can combine existing language in novel ways. In fact, we've seen over and over that LLMs are quite generic. Compared to text-to-image, which is seemingly impossible to use without getting something weird
- glenstein 3y ago
  Is there something about the nature of language and linguistic meaning that would make it difficult to combine language in novel ways?
  - TuringTourist 3y ago
    I would posit the opposite, language exists to be combined in any number of ways easily and still communicate well. A language that cannot do this does not I think exist.
    - glenstein 3y ago
      Same. I find it hard to makes sense of why anyone would claim that computers can innovate visually but not with language. It would make more sense if someone at least felt that they both could, or both couldn't.

gibsonf1 3y ago

It seems silly to even mention epistemology when current AI has no Intelligence at all yet.

Procrastes 3y ago

This focuses on the argument that some hyper-sophisticated future LLM will not discover the secrets of the universe on its own. That's plausible, but beside the point when it comes to the potential dangers of A.I. Not all future A.I.s will be LLMs. Understanding the universe isn't the only way to pose an existential threat. This is an old discussion with much deeper insights already available than I can parrot here, but one obvious counter just to this argument is that even an LLM can be an existential threat, depending what it can convince you it knows.

lastingwin 3y ago

I think it's safe to say that we should disregard any substack/medium posts, especially when it comes to AI.

progrus 3y ago

Yay, we need more of this before OpenAI finishes hypnotizing the global south (lol).

ta8645 3y ago

People are still desperate to maintain humankind at the top of the intellectual pyramid. It's tiresome and a waste of energy. There are definitely limits to intelligence, nothing will ever be smart enough to predict the future perfectly. But to believe that humans represent the peak of what's possible, is just hubris and nothing else.

visarga 3y ago

> Future LLMs, regardless of model architecture, are fundamentally constrained in this way, by virtue of the fact that they are trained on human-written texts. Therefore, the outputs of a LLM, at best, reflect our current understanding of the universe and nothing more. That constraint holds only for pure LLMs. There is a secondary source of learning - external feedback to the model. When a LLM is part of a larger system, it can integrate feedback to improve itself, like AlphaGo Zero. A LLM+compiler could self-optimize by observing compilation errors and program outputs. A LLM+game could self-improve by conditioning actions on the score. Humans could create a preference dataset and model for RLHF to refine the LLM based on human feedback. A LLM+robot could learn from past experiences to improve its planning. Similarly, humans need tools to conduct research. We can't do pure research without labs and experiments. LLMs need external confirmation the same way we do. The outputs of a pure LLM merely reflect our current understanding of the universe. But coupled with real-world feedback, LLMs have the potential to learn and discover new knowledge.

CuriouslyC 3y ago

This is plainly false. With reinforcement learning, an objective function and the ability to generate random solutions it's possible for AI to stumble upon sequences/strategies that greatly outperform human state of the art.

RoboTeddy 3y ago

> But the ways in which a LLM can “talk about” the universe (and everything it contains) are limited to the ways in which humans have previously talked about the universe. This is often said, but it isn't so. The task of predicting the next token in human speech really well requires immense intelligence — potentially far more intelligence than possessed by the original speaker! Imagine yourself engaging in the task of listening to someone who isn't that smart speak and then trying to figure out what they'll say next — in doing so, you might make all sorts of extrapolations about the person, their motivations, their manner, their dialect, etc — calling on all sorts of internal models that you've built up about people over time. This is what models are being trained to do when we train them on predicting tokens. There are concrete examples of models inventing new ways of thinking that are not described in their training set. For example, when training a transformer from scratch to perform addition mod P (and having no training data other than examples of addition mod P), the transformer was able to discover the use of discrete fourier transforms and trigonometric identities [1]. As we can see, neural nets can build all sorts of internal mental models that no one explained to them beforehand. These internal mental models can then be elicited and used for other purposes by e.g. fine-tuning. I think a good mental model for transformers/neural nets is that they're automatic scientists. They figure out ways of modeling things in order to predict the output from the input — which is what scientists do! As part of this, they can de-facto discover new theories, and come to rely on the theories that prove useful in their prediction task. Also, not all tokens in the training set are from human speech, so models are being trained to model all manner of data-generating processes. [1] https://arxiv.org/pdf/2301.05217.pdf https://arxiv.org/pdf/2301.05217.pdf

gremlinunderway 3y ago
This is a great example when it comes to trying to understand the epistemological limits of AI. People inherently fall back on an argument that presumes the human mind works like a neural network. There's an interesting theory that I've never been able to identify the origins of, that humans like to think that we created technology from needs based on our models of the world, whereas we ignore the effects of daily technology on our thinking frameworks. The argument essentially states that we never "discovered" the circulatory system of the heart or "discovered" it works like a pump and valves, rather instead right around the same time this theoretical work was being investigated on the heart is when the industrial revolution was in full swing. Thus, we modelled the heart as pumps and valves because that's the technology we were surrounded by. The heart isn't somehow inherently a "pump" and we "discovered" that, we just started using the pump metaphor because it seemed to help do other things. I'm sure though that the metaphor has it limits. Typically, the narrative around the invention of machine learning models is that we started coding computers to be more like recently "discovered" models of the brain. Under this theory its the opposite. Right as cognitive sciences started developing as a novel field of research is when we developed computers. So, in classic form we decomposed the brain in atomic fashion, and the 'atoms' of measurement we chose to use ended up being bits and bytes. Distinguishing between inventing "novel" things and gobbledygook is completely subjective and based on the viewer's own models. It's proving these abilities after the fact, not before it. Thousand monkeys on typewriters etc. This measurement of "accuracy" is completely forgetting everything that Kuhn discovered about scientific knowledge. If you've got a community of like 3 people who research some incredibly esoteric scientific field, only those 3 people could ever accurately judge the full extent of their domain. A model could generate a series of tokens that for the rest of the world is gobbledygook, but to these 3 scientists it makes perfect sense. This doesn't really endeavour me to believe that there's anything "novel" about what AI "predicts". It just throws out enough combinations and we conveniently ignore the huge gaps when its wrong, but then jump up and down excitedly when it's "right" (as if its discovered some universal material objective truth).

waffletower 3y ago

"Therefore, the outputs of a LLM, at best, reflect our current understanding of the universe and nothing more." Others have reflected on the problems with this statement, and I agree, LLMs can be prompted to navigate the training corpus and create novel output, countering this claim. However, these criticisms neglect to credit the external intelligence inherent to the prompts themselves. AutoGPT and its ilk have yet to create convincing agency: for the time being the current state of AI is a mere extension of human intelligence and will not be making lofty discoveries without direct human interaction. LLMs are a form of human co-intelligence.

entropyneur 3y ago

LLMs are trained on human-written texts, therefore their output can only reflect our current understanding - am I reading this right? Hopefully not as that would be insultingly wrong.

arisAlexis 3y ago

Anything with started with "the myth" arguing about something that is at the limits of knowledge without a clear answer is unavoidably biased by a person with big ego.

mannykannot 3y ago

Putting aside the question of what LLMs, specifically, are capable of, I feel the author's central argument has already been shown to be false, through events outside of AI. In a nutshell, the argument is that, as AI will have just the knowledge that is available to us, expressed in the languages we use, we will be able to anticipate and put a stop to it doing anything catastrophic. Exactly the same argument, however, could be made over computer and communications security. If anything, those trying to keep things secure have the advantage in knowledge, at least initially, yet zero-day exploits keep surfacing, some of them at fundamental levels such as processor architecture. Nothing truly catastrophic has happened yet, but it cannot be said to be impossible. The flaw in the argument is that one can easily know all the bare facts without seeing all the implications but someone - or something - else could do so.

ilaksh 3y ago

LLMs have obvious limitations and a big contributor is the training data and the way they are trained in a supervised way. Although that is not as much of a limitation as with most other AIs because of the "in-context learning" ability. But that doesn't mean that future LLMs couldn't conceivably learn abstractions than are at a higher level or even perhaps inaccessible to humans in number of layers or complexity. I think the obvious type of superintelligence that is only a few years away is what I call hyperspeed AI. LLMs are a very specific application which we will be able to accelerate greatly. Within a decade the output speed will be dozens of times faster than human thought. Because they will have robust reasoning and extreme "thinking speed", these AIs will probably be connected to industry and military applications. Humans will need to be removed from the loop because waiting for a human decision means the competitors' AIs race ahead the equivalent of days or weeks. If this is fine then it will create a precarious situation for humans where they are only nominally in control and things like advanced AI "viruses" might become very dangerous. It's also certain that researchers will continue to develop new types of AIs that incorporate more animal (such as human)-like capabilities.

creer 3y ago

Why do we need anything close to superintelligence or "omniscience" (wtf) before we get profound changes to how humanity operates. It's really an obsession! Instead compare to the average human, at most. Compare to refrigeration? Compare to fire? Compare to computers? And that's even if the bar is placed at the singularity. Human progress as we have known it so far is not founded on superhumans but on a lot of work by a mix of merely visionary, hard working, and just plain average people. Just speeding that an order of magnitude changes everything.