15 ms·

I struggle to understand how people can just dismiss the possibility of artificial intelligence. Human cognition was basically bruteforced by evolution-- why w

I struggle to understand how people can just dismiss the possibility of artificial intelligence. Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals? I'm not buying the whole "AI has no agency" line either; this might be true for now, but this is already being circumvented with current LLMs (by giving them web access etc). As soon as profit can be made by transfering decision power into an AIs hand, some form of agency for them is just a matter of time, and we might simply not be willing to pull the plug until it is much too late.

moralestapia 1y ago
>why would it be impossible to achieve the exact same result in silicon Because there might be a non-material component involved.
- ACCount37 1y ago
  [flagged]
  - moralestapia 1y ago
    @dang
    - tomhow 1y ago
      We replied to that comment, but the @reply isn't a reliable way to get moderator attention, and publicly calling for moderator help like this isn't a great look. Best to flag the comment and email us if you really want us to take a look.
  - tomhow 1y ago
    Please don't post sneery comments like this on HN. The guidelines make it clear we're here for curious conversation, and kindness is one of the foremost expectations.
deleted 1y ago
[deleted]
AlfredBarnes 1y ago
Wasn't there a story about healthcare companies letting AI determine coverage? I can't remember.
- billyjmc 1y ago
  Computers have been making decisions for a while now. As a specific personal example from 2008, I found out that my lender would make home loan offers based on an inscrutable (to me and the banker I was speaking to) heuristic. If the loan was denied by the heuristic, then a human could review the decision, but had strict criteria that they would have to follow. Basically, a computer could “exercise judgement” a make offers that a human could not.
boole1854 1y ago
If anyone knows of a steelman version of the "AGI is not possible" argument, I would be curious to read it. I also have trouble understanding what goes into that point of view.
- Chance-Device 1y ago
  The only thing I can come up with is that compressing several hundred million years of natural selection of animal nervous systems into another form, but optimised by gradient descent instead, just takes a lot of time. Not that we can’t get there by artificial means, but that correctly simulating the environment interactions, the sequence of progression, getting the all the details right, might take hundreds to thousands of years of compute, rather than on the order of a few months. And it might be that you can get functionally close, but hit a dead end, and maybe hit several dead ends along the way, all of which are close but no cigar. Perhaps LLMs are one such dead end.
  - danielbln 1y ago
    I don't disagree, but I think the evolution argument is a red herring. We didn't have to re-engineer horses from the ground up along evolutionary lines to get to much faster and more capable cars.
    - amatecha 1y ago
      The evolution thing is kind of a red herring in that we probably don't have to artificially construct the process of evolution, though your reasoning isn't a good explanation for why the "evolution" reason is a red herring: Yeah, nature already established incomprehensibly complex organic systems in these life forms -- so we're benefiting from that. But the extent of our contribution is making some select animals mate with others. Hardly comparable to building our own replacement for some millennia of organic iteration/evolution. Luckily we probably don't actually need to do that to produce AGI.
    - Chance-Device 1y ago
      True, but I think this reasoning is a category error: we were and are capable of rationally designing cars. We are not today doing the same thing with AI, we’re forced to optimize them instead. Yes, the structure that you optimize around is vitally important, but we’re still doing brute force rather than intelligent design at the end of the day. It’s not comparing like with like.
alansaber 1y ago
Perhaps the AGI people thinking we can catch up millions of years of evolution in a handful of years
- myrmidon 1y ago
  If you make the same argument for flight it looks really weak. Cognition is (to me) not even the most impressive and out-of-reach achievement: That would be how (our) and animals bodies are self-assembling, self-repairing and self-replicating, with an impressive array of sensors and actors in a highly integrated package. I honestly believe our current technology is much closer to emulating a human brain than it is to building a (non-intelligent) cat.
  - jncfhnb 1y ago
    Flight leverages well established and accessible world engine physics APIs. Intelligence has to be programmed from lower level mechanics. Edit: put another way, I bet the ancient Greeks (or whoever) could have figured out flight if they had access to gasoline and gasoline powered engines without any of the advanced mathematics that were used to guide the design.
  - bangaroo 1y ago
    > if you make the same argument for flight it looks really weak. flight is an extremely straightforward concept based in relatively simple physics where the majority of the critical, foundational ideas involved were already near-completely understood in the late 1700s. i really don't think it's fair to compare the two
    - ACCount37 1y ago
      I'm sure that intelligence is an extremely straightforward concept based in relatively simple math where the majority of the critical, foundational ideas involved were already near-completely understood in the late 1900s. If you read about in a textbook from year 2832, that is.
    - tavavex 1y ago
roxolotl 1y ago
The point of the article isn’t that abstract super intelligent agi isn’t scary. Yes the author says that’s unlikely but that paragraph at the start is a distraction. The point of the article is that humans wielding LLMs today are the scary monsters.
- irjustin 1y ago
  But that's always been the case? Since we basically discovered... Fire? Tools?
  - otikik 1y ago
    Yes but the narrative tries to make it about the tools. "AI is going to take all the jobs". Instead of: "Rich guys will try to delete a bunch of jobs using AI in order to get even more rich".
    - jasonm23 1y ago
      I thought anyone with awareness of what the AI landscape is at the moment, sees those two statements as the same.
      - otikik 1y ago
        Well it tells you who's narrative it is, if nothing else.
      - cmiller1 1y ago
        One implies "we should regulate AI" and the other implies "we should regulate the wealthy"
        zeroonetwothree 1y ago
        Should we regulate guns or dangerous people using them?
deadbabe 1y ago
Language is only an end product. It is derived from intelligence. The intelligence is everything that created the language and the training corpus in the first place. When AI is able to create entire thoughts and ideas without any concept of language, then we will truly be closer to artificial intelligence. When we get to this point, we then use language as a way to let the AI communicate its thoughts naturally. Such an AI would not be accused of “stealing” copyrighted work because it would pull its training data from direct observations about reality itself. As you can imagine, we are no where near accomplishing the above. Everything an LLM is fed today is stuff that has been pre-processed by human minds for it to parrot off of. The fact that LLMs today are so good is a testament to human intelligence.
- myrmidon 1y ago
  I'm not saying that language necessarily is the biggest stumbling block on (our) road towards AI, but it is a very prominent feature that we have used to distinguish our capabilities from other animals long before AI was even conceived of. So the current successes with LLMs are highly encouraging. I'm not buying the "current AI is just a dumb parrot relying on human training" argument, because the same thing applies to humans themselves-- if you raise a child without any cultural input/training data, all you get is a dumb cavemen with very limited reasoning capabilities.
  - nyeah 1y ago
    "I'm not buying the "current AI is just a dumb parrot relying on human training" argument [...]" One difficulty. We know that argument is literally true. "[...] because the same thing applies to humans themselves" It doesn't. People can interact with the actual world. The equivalent of being passively trained on a body of text may be part of what goes into us. But it's not the only ingredient.
- ACCount37 1y ago
  Clearly, language reflects enough of "intelligence" for an LLM to be able to learn a lot of what "intelligence" does just by staring at a lot of language data really really hard. Language doesn't capture all of human intelligence - and some of the notable deficiencies of LLMs originate from that. But to say that LLMs are entirely language-bound is shortsighted at best. Most modern high end LLMs are hybrids that operate on non-language modalities, and there's plenty of R&D on using LLMs to consume, produce and operate on non-language data - i.e. Gemini Robotics.
jncfhnb 1y ago
I think you could make AGI right now tbh. It’s not a function of intelligence. It’s just a function of stateful system mechanics. LLMs are just a big matrix. But what about a four line of code loop that looks like: ```while true: update_sensory_inputs() narrate_response() update_emotional_state() ``` LLMs don’t experience continuous time and they don’t have an explicit decision making framework for having any agency even if they can imply one probabilistically. But the above feels like the core loop required for a shitty system to leverage LLMs to create an AGI. Maybe not a particularly capable or scary AGI, but I think the goalpost is pedantically closer than we give credit.
- Jensson 1y ago
  > ```while true: update_sensory_inputs() narrate_response() update_emotional_state() ``` You don't think that has already been made?
  - jncfhnb 1y ago
    Sure, probably, to varying levels of implementation details.
    - ffsm8 1y ago
      It has been implemented for sure, just watch a little Neurosama. That's most definitely not AGI
- tantalor 1y ago
  Peak HN comment. Put this in the history books.
  - fullstackchris 1y ago
    Has anyone else noticed that HN is starting to sound a lot like reddit / discussion of similar quality? Can't hang out anywhere now on the web... I used to be on here daily but with garbage like this its been reduced to 2-3 times per month... sad
deleted 1y ago
[deleted]
Juliate 1y ago
> Human cognition was basically bruteforced by evolution You center cognition/intelligence on humans as if it was the pinacle of it, rather than include the whole lot of other species (that may have totally different, or adjacent cognition models). Why? How so? > As soon as profit can be made by transfering decision power into an AIs hand There's an ironic, deadly, Frankensteinesque delusion in this very premise.
- soiltype 1y ago
  > You center cognition/intelligence on humans as if it was the pinacle of it, rather than include the whole lot of other species (that may have totally different, or adjacent cognition models). Why does that matter to their argument? Truly, the variety of intelligences on earth now only increases the likelihood of AGI being possible, as we have many pathways that don't follow the human model.
- myrmidon 1y ago
  > You center cognition/intelligence on humans as if it was the pinacle of it That's not my viewpoint, from elsewhere in the thread: Cognition is (to me) not the most impressive and out-of-reach evolutionary achievement: That would be how our (and animals) bodies are self-assembling, self-repairing and self-replicating, with an impressive array of sensors and actors in a highly integrated package. I honestly believe our current technology is much closer to emulating a human brain than it is to building a (non-intelligent) cat.
adamtaylor_13 1y ago
> Human cognition was basically bruteforced by evolution Well that's one reason you struggle to understand how it can be dismissed. I believe we were made by a creator. The idea that somehow nature "bruteforced" intelligence is completely nonsensical to me. So, for me, logically, humans being able to bruteforce true intelligence is equally nonsensical. But what the author is stating, and I completely agree with, is that true intelligence wielding a pseudo-intelligence is just as dangerous (if not moreso.)
- bee_rider 1y ago
  Even if there is a creator, it seems to have intentionally created a universe in which the evolution of humans is basically possible and it went to great lengths to hide the fact that it made us as a special unique thing. Let’s assume there’s a creator: It is clearly willing to let bad things happen to people, and it set things up to make it impossible to prove that a human level intelligence should be impossible, so who’s to say it won’t allow a superintelligence to be a made by us?
- yjftsjthsd-h 1y ago
  I don't think that follows. God made lots of things that we can create facsimiles of, or even generate the real thing ourselves.
ACCount37 1y ago
I don't get how you can see one of those CLI coding tools in action and still parrot the "no agency" line. The goal-oriented behavior is rather obvious. Sure, they aren't very good at agentic behavior yet, and the time horizon is pretty low. But that keeps improving with each frontier release.
- simonsarris 1y ago
  Well, the goal-oriented behavior of the AIM-9 Sidewinder air-to-air missile is even more obvious. It might even have a higher success rate than CLI coding tools. But it's not helpful to claim it has any agency.
- lo_zamoyski 1y ago
  > The goal-oriented behavior is rather obvious. Obvious? Is an illusion obviously the real thing? There is nothing substantially different in LLMs from any other run of the mill algorithm or software.
  - Eisenstein 1y ago
    Can you give an example of something that would be substantially different under your definition?
    - lo_zamoyski 1y ago
      But that's the point: there isn't anything substantially different within the scope of computation. If you are given a set of LEGOs and all you can do is snap the pieces together, then there's nothing other than snapping pieces together that you can do. Adding more of the same LEGOs bricks to the set doesn't change the game. It only changes how large the structures you build can be, but scale isn't some kind of magical incantation that can transcend the limits of the system. Computation is an abstract, syntactic mathematical model. These models formalize the notion of "effective method". Nowhere is semantic content included in these models or conceptually entailed by them, certainly not in physical simulations of them like the device you are reading this post on. So, we can say that intentionality would be something substantially different. We absolutely do not have intentionality in LLMs or any computational construct. It is shear magical thinking to somehow think it does.
lo_zamoyski 1y ago
I think you make a lot of assumptions that you should perhaps reexamine. > Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals? Here are some of your assumptions: 1. Human intelligence is entirely explicable in evolutionary terms. (It is certainly not the case that it has been explained in this manner, even if it could be.) [0] 2. Human intelligence assumed as an entirely biological phenomenon is realizable in something that is not biological. And perhaps this one: 3. Silicon is somehow intrinsically bound up with computation. In the case of (2), you're taking a superficial black box view of intelligence and completely ignoring its causes and essential features. This prevents you from distinguishing between simulation of appearance and substantial reality. Now, that LLMs and so on can simulate syntactic operations or whatever is no surprise. Computers are abstract mathematical formal models that define computations exactly as syntactic operations. What computers lack are semantic content. A computer never contains the concept of the number 2 and the concept of the addition operation even though we can simulate the addition of 2 + 2. This intrinsic absence of a semantic dimension means that computers already lack the most essential feature of intelligence, which is intentionality. There is no alchemical magic that will turn syntax into semantics. In the case of (3), I emphasize that computation is not a physical phenomenon, but something described by a number of formally equivalent models (Turing machine, lambda calculus, and so on) that aim to formalize the notion of effective method. The use of silicon-based electronics is irrelevant to the model. We can physically simulate the model using all sorts of things, like wooden gears or jars of water or whatever. > I'm not buying the whole "AI has no agency" line either; this might be true for now, but this is already being circumvented with current LLMs (by giving them web access etc). [...] As soon as profit can be made by transfering decision power into an AIs hand, some form of agency for them is just a matter of time, and we might simply not be willing to pull the plug until it is much too late. How on earth did you conclude there is any agency here, or that it's just a "matter of time"? This is textbook magical thinking. You are projecting a good deal here that is completely unwarranted. Computation is not some kind of mystery, and we know at least enough about human intelligence to notes features that are not included in the concept of computation. [0] (Assumption (1), of course, has the problem that if intelligence is entirely explicable in terms of evolutionary processes, then we have no reason to believe that the intelligence produced aims at truth. Survival affordances don't imply fidelity to reality. This leads us to the classic retorsion arguments that threaten the very viability of the science you are trying to draw on.)
bravetraveler 1y ago
I dismiss it much like I dismiss ideas such as "Ketamine for Breakfast for All". Attainable, sure, but I don't like where it goes.
- Traubenfuchs 1y ago
  I think most people would have a better life using ketamine, but not that regularly for breakfasts as it permanently damages (shrinks) your bladder, eventually to the point where you can‘t hold any urine at all anymore.
  - bravetraveler 1y ago
    Eh, I think we can start simple: more breakfasts for more people. Save the Ket for later/others :P Personally, a life/career that allowed for more breakfast would've proved more beneficial.
theredleft 1y ago
this is because you didn't take liberal arts seriously
- nyeah 1y ago
  That's not implausible at all. For all I know it might be the most on-target comment here. But can you cite something specific? (I'm not asking for a psychological study. Maybe you can prove your point using Blake's "Jerusalem" or something. I really don't know.)
- ACCount37 1y ago
  Good one.
bee_rider 1y ago
I think it is bad writing on the part of the author. Or maybe good writing for getting us engaged with the blog post, bad for making an argument though. They include a line that they don’t believe in the possibility of AGI: > I don’t really believe in the threat of AGI (Artificial General Intelligence—human-level intelligence) partly because I don’t believe in the possibility of AGI and I’m highly skeptical that the current technology underpinning LLMs will provide a route to it. This is a basically absurd position to hold, I mean humans physically exist so our brains must be possible to build within the existing laws of physics. It is obviously far beyond our capabilities to replicate a human brain (except via the traditional approach), but unless brains hold irreproducible magic spirits (well we must at least admit the possibility) they should be possible to build artificially. Fortunately they immediately throw that all away anyway. Next, they get to the: > and I’m highly skeptical that the current technology underpinning LLMs will provide a route to it. Which is, of course, at least a plausible thing to believe. I mean there are a bunch of philosophical questions about what “intelligence” even means so there’s plenty of room to quibble here. Then we have, > But I also think there’s something we should actually be afraid of long before AGI, if it ever comes. […] > Now, if you equip humans with a hammer, or sword, or rifle, or AI then you’ve just made the scariest monster in the woods (that’s you) even more terrifying. […] > We don’t need to worry about AI itself, we need to be concerned about what “humans + AI” will do. Which is like, yeah, this is a massively worrying problem that doesn’t involve any sci-fi bullshit, and I think it is what most(?) anybody who’s thought about this seriously at all (or even stupid people who haven’t, like myself). Artificial Sub-intelligences, things that are just smart enough to make trouble and too dumb or too “aligned” to their owner (instead of society in general) to push back are a big currently happening problem.
- andy99 1y ago
  > humans physically exist so our brains must be possible to build within the existing laws of physics This is an unscientific position to take. We have no idea how our brains work, or how life, consciousness, and intelligence work. It could very well be that’s because our model of the world doesn’t account for these things and they are not in fact possible based on what we know. In fact I think this is likely. So it really could be that AI is not possible, for example on a Turing machine or our approximation of them. This is at least as likely as it being possible. At some point we’ll hopefully refine our theories to have a better understanding, for now we have no idea and I think it’s useful to acknowledge this.
AlexandrB 1y ago
LLMs largely live in the world of pure language and related tokens - something humans invented late in their evolution. Human intelligence comes - at least partially - from more fundamental physical experience. Look at examples of intelligent animals that lack language. Basically there's something missing with AI. Its conception of the physical world is limited by our ability to describe it - either linguistically or mathematically. I'm not sure what this means for AGI, but I suspect that LLM intelligence is fundamentally not the same as human or animal intelligence at the moment as a result.
Aerroon 1y ago
>I'm not buying the whole "AI has no agency" line either; this might be true for now, but this is already being circumvented with current LLMs (by giving them web access etc). They don't have agency, because they don't have persistent state. They're like a function that you can query and get an answer. During that answer the LLM has state, but once it's done the state is gone. Humans (and other "agents") have persistent state. If we learn something, we can commit it to long-term memory and have it affect our actions. This can enable us to work towards long-term goals. Modern LLMs don't have this. You can fake long-term memory with large context windows and feed the old context back to it, but it doesn't appear to work (and scale) the same way living things do.
- messe 1y ago
  > They don't have agency, because they don't have persistent state. They're like a function that you can query and get an answer. During that answer the LLM has state, but once it's done the state is gone. That's solved by the simplest of agents. LLM + ability to read / write a file.
  - aniviacat 1y ago
    But they can only change their context, not the model itself. Humans update their model whenever they receive new data (which they do continuously). A live-learning AI would be theoretically possible, but so far it hasn't been done (in a meaningful way).
    - webstrand 1y ago
      It has been tried <https://en.wikipedia.org/wiki/Catastrophic_interference https://en.wikipedia.org/wiki/Catastrophic_interference> but it has been unsuccessful.
- throwuxiytayq 1y ago
  I have no words to express how profoundly disappointed I am to keep reading these boring, shallow, shorttermist, unimaginative takes that are invalidated by a model/arch upgrade next week, or - in this case - more like years ago, since pretty much all big LLM platforms are already augmented by RAG and memory systems. Do you seriously think you’re discussing a serious long term limitation here?
psychoslave 1y ago
>Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals? I don’t know if anything put us that much apart from other animals, especially at individual level. On collective level as a single species, indeed maybe only cyanobacteries stand an equally impressive achievement of a global change. My 3 years old son is not particularly good at making complex sentences yet, but he already got it enough to make me understand "leave me alone, I want to play on my own, go elsewhere so I can do whatever fancy idea get through my mind with these toys". Meanwhile LLM can produce sentences with perfect syntax and irreproachable level of orthography — far beyond my own level in my native language (but it’s French so I have a very big excuse). But they would not run without continuous multi-sector industrial complex injecting tremendous maintenance effort and resources to make it possible. And yet I still have to see any LLM that looks like it wants to discover things of the world on its own. >As soon as profit can be made by transfering decision power into an AIs hand, some form of agency for them is just a matter of time, and we might simply not be willing to pull the plug until it is much too late. LLM can’t make profit because it doesn’t have interest in money, and it can’t have any interest in anything, not even its own survival. But as the article mention, some people can certainly use LLM to make money because they have interest in money. I don’t think that general AI and silicon (or any other material really) based autonomous collaborative self-replicating human-level-intelligence-or-beyond entities are impossible. I don’t think cold fusion is impossible either. It’s not completely scientifically ridiculous to keep hope in worm-hole-based breakthroughs to allow humanity explore distant planets. It doesn’t mean the technology is already there and achievable in a way that it can turned into a commodity, or even that we have a clear idea of when this is mostly going to happen.
- ACCount37 1y ago
  LLMs aren't "incapable of pursuing their own goals". We just train them that way. We don't like the simplistic goals LLMs default to, so we try to pry them out and instill our own: instruction-following, problem solving, goal oriented agentic behavior, etc. In a way, trying to copy what humans do - but focusing on the parts that make humans useful to other humans.
theodorejb 1y ago
> Human cognition was basically bruteforced by evolution This is an assumption, not a fact. Perhaps human cognition was created by God, and our minds have an essential spiritual component which cannot be reproduced by a purely physical machine.
- potsandpans 1y ago
  Maybe there's a small teapot orbiting the earth, with ten thousand angels dancing on the tip of the spout.
  - andy99 1y ago
    I think you’re both saying the same thing
- deleted 1y ago
  [deleted]
- pennomi 1y ago
  It’s not an assumption, it’s a viable theory based on overwhelming evidence from fossil records. What’s NOT supported by evidence is an unknowable, untestable spiritual requirement for cognition.
  - j2kun 1y ago
    What overwhelming evidence do fossil records provide about human cognition?
    - mediaman 1y ago
      We don't need fossil records. We have a clear chain of evolved brain structures in today's living mammals. You'd have to invent some fantastical tale of how God is trying to trick us by putting such clearly connected brain structures in a series of animals that DNA provides clear links for an evolutionary path. I'm sympathetic to the idea that God started the whole shebang (that is, the universe), because it's rather difficult to disprove, but looking at the biological weight of evidence that brain structures evolved over many different species and arguing that something magical happened with homo sapiens specifically is not an easy argument to make for someone with any faith in reason.
IAmGraydon 1y ago
>I struggle to understand how people can just dismiss the possibility of artificial intelligence.Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals? I haven't seen many people saying it's impossible. Just that the current technology (LLMs) is not the way, and is really not even close. I'm sure humanity will make the idiotic mistake of creating something more intelligent than itself eventually, but I don't believe that's something that the current crop of AI technology is going to evolve into any time soon.
ectospheno 1y ago
The worst thing Star Trek did was convince a generation of kids anything is possible. Just because you imagine a thing doesn’t make it real or even capable of being real. I can say “leprechaun” and most people will get the same set of images in their head. They aren’t real. They aren’t going to be real. You imagined them.
- dsr_ 1y ago
  That's not Star Trek, that's marketing. Marketing grabbed a name (AI) for a concept that's been around in our legends for centuries and firmly welded it to something else. You should not be surprised that people who use the term AI think of LLMs as being djinn, golems, C3PO, HAL, Cortana...
- random3 1y ago
  Do maybe have a better show recommendation for kids - many Animal Farm? How is convincing people that things within the limits of physics are possible wrong or even "the worse thing"? Or do you think anything that you see in front of didn't seem like StarTrek a decade before it existed?
- IAmBroom 1y ago
  But somehow you don't hate on Steamboat Willie? I grew up believing a mouse could operate a ship, and a coyote could survive impacts from anvils.
me_again 1y ago
"As soon as profit can be made" is exactly what the article is warning about. This is exactly the "Human + AI" combination. Within your lifetime (it's probably already happened) you will be denied something you care about (medical care, a job, citizenship, parole) by an AI which has been granted the agency to do so in order to make more profit.
random3 1y ago
I think dismissing possibility of evolving AI, is simply ignorance (and a huge blindspot) This said, I think the author's point is correct. It's more likely that unwanted effects (risks) from the intentional use of AI by humans is something that precedes any form of "independent" AI. It already happens, it always has, it's just getting better. Hence ignoring this fact makes the "independent" malevolent AI a red herring. On the first point - LLMs have sucked almost all the air in the room. LLMs (and GPTs) are simply one instance of AI. They are not the beginning and most likely not the end (just a dead end) and getting fixated on them on either end of the spectrum is naive.
fullstackchris 1y ago
Of course it's possible. It's just DEFINITELY not possible using a large neural net, or basically a markov chain on steroids. C'mon, this should be very obvious by now in the world of agents / LLMs. When is silicon valley gonna learn that token input and output =/= AGI?
btilly 1y ago
I agree that we should not dismiss the possibility of artificial intelligence. But the central argument of the article can be made without that point. Because the truth is that right now, LLMs are good enough to be a force multiplier for those who know how to use them. Which eventually becomes synonymous with "those who have power". This means that the power of AI will naturally get used to further the ends of corporations. The potential problem there is that corporations are natural paperclip maximizers. They operate on a model of the world where "more of this results in more of that, which gets of more of the next thing, ..." And, somewhere down the chain, we wind up with money and resources that feed back into the start to create a self-sustaining, exponentially growing loop. (The underlying exponential nature of these loops has become a truism that people rely on in places as different as finance, and technology improvement curves.) This naturally leads to exponential growth in resource consumption, waste, economic growth, wealth, and so on. In the USA this growth has averaged about 3-3.5% per year. With growth varying by area. Famously, growth rates tend to be much higher in tech. Likewise growth rates are higher in some areas than others. (The best known example is the technology curve described by Moore's law. Which has had a tremendous impact on our world.) The problem is that we are undergoing exponential growth in a world with ultimately limited resources. Which means that the most innocuous things will eventually have a tremendous impact. The result isn't simply converting everything into a mountain of paperclips. We have mountains of many different things that we have produced, and multiple parallel environmental catastrophes from the associated waste. Even with no agency, AI serves as a force multiplier for this underlying dynamic. But since AI is being inserted as a crucial step at so many places, AI is on a particularly steep growth curve. Estimates for total global electricity spent on AI are in the range 0.2-0.4%. That seems modest, but annual growth rates are projected as being in the range of 10-30%. (The estimates are far apart because a lot of the data is not public, and so has to be estimated.) This is a Moore's law level growth. We are likely to see the electricity consumption of AI grow past all other uses within our lifetimes. And that will happen even without the kind of sudden leaps in capability that machine learning regularly delivers. I hope we humans like those paperclips. Humans, armed with AI, are going to make a lot of them. And they're not actually free.
ux266478 1y ago
It's confirmation bias in favor of faulty a prioris, usually as the product of the person being a cognitive miser. This is very common to experience even within biology, where non-animal intelligence is irrationally rejected over what I like to call "magic neuron theory". The fact that the nervous system is (empirically!) not the seat of the mind? Selectively ignored in this context. The fact that other biologies have ion-gated communications networks as animals do, including the full set of behaviors and mechanisms? Well it's not a neuron so it doesn't have the magic. "Intelligence describes a set of properties iff those properties arise as a result of nervous system magic" It's a futile battle because like I say, it's not rational. Nor is it empirical. It's a desperate clawing to preserve a ridiculous superstition. Try as you might, all you'll end up doing is playing word games until you realize you're being stonewalled by an unthinking adherence to the proposition above. I think the intelligent behaviors of LLMs are pretty obvious if we're being good faith. The problem is you're talking to people who can watch a slime mold plasmodia exhibit learning and sharing of knowledge[1] and they'll give some flagrant ad lib handwaive for why that's not intelligent behavior. Some people simply struggle with pattern blindness towards intelligence, a mind that isn't just another variety of animalia is inconceivable. [1] - https://asknature.org/strategy/brainless-slime-molds-both-learn-and-teach/ https://asknature.org/strategy/brainless-slime-molds-both-le...
wolrah 1y ago
> Human cognition was basically bruteforced by evolution-- "Brute forced" implies having a goal of achieving that and throwing everything you have at it until it sticks. That's not how evolution by natural selection works, it's simply about what organisms are better at surviving long enough to replicate. Human cognition is an accident with relatively high costs that happened to lead to better outcomes (but almost didn't). > why would it be impossible to achieve the exact same result in silicon I personally don't believe it'd be impossible to achieve in silicon using a low level simulation of an actual human brain, but doing so in anything close to real-time requires amounts of compute power that make LLMs look efficient by comparison. The most recent example I can find in a quick search is a paper from 2023 that claims to have simulated a "brain" with neuron/synapse counts similar to humans using a 3500 node supercomputer where each node has a 32 core 2 GHz CPU, 128GB RAM, and four 1.1GHz GPUs with 16GB HBM2 each. They claim over 126 PFLOPS of compute power and 224 TB of GPU memory total. At the time of that paper that computer would have been in the top 10 on the Top500 list, and it took between 1-2 minutes of real time to simulate one second of the virtual brain. The compute requirements are absolutely immense, and that's the easy part. We're pretty good at scaling computers if someone can be convinced to write a big enough check for it. The hard part is having the necessary data to "initialize" the simulation in to a state where it actually does what you want it to. > especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals? Creating convincing text from a statistical model that's devoured tens of millions of documents is not intelligent use of language. Also every LLM I've ever used regularly makes elementary school level errors w/r/t language, like the popular "how many 'r's are there in the word strawberry" test. Not only that, but they often mess up basic math. MATH! The thing computers are basically perfect at, LLMs get wrong regularly enough that it's a meme. There is no understanding and no intelligence, just probabilities of words following other words. This can still be very useful in specific use cases if used as a tool by an actual intelligence who understands the subject matter, but it has absolutely nothing to do with AGI.
- ACCount37 1y ago
  That's a lot of words to say "LLMs think very much like humans do". Haven't you noticed? Humans also happen to be far, far better at language than they are at math or logic. By a long shot too. Language acquisition is natural - any healthy human who was exposed to other humans during development would be able to pick up their language. Learning math, even to elementary school level, is something that has to be done on purpose. Humans use pattern matching and associative abstract thinking - and use that to fall into stupid traps like "1kg of steel/feather" or "age of the captain". So do small LLMs.
janalsncm 1y ago
One pretty concrete way this could manifest is in replacing the components of a multinational corporation with algorithms, one by one. Likely there will be people involved at various levels (sales might still be staffed with charismatic folks), but the driver will be an algorithm. And the driver of this corporation is survival of the fittest under the constraints of profit maximization, the algorithm we have designed and enforced. That's how you get paperclip maximizers. What gives this corporate cyborg life is not a technical achievement, but the law. At a technical you can absolutely shut off a cybo-corp, but that’s equivalent to saying you can technically shut down Microsoft. It will not happen.
bithive123 1y ago
I struggle to understand how people attribute things we ourselves don't really understand (intelligence, intent, subjectivity, mind states, etc) to a computer program just because it produces symbolic outputs that we like. We made it do that because we as the builders are the arbiters of what constitutes more or less desirable output. It seems dubious to me that we would recognize super-intelligence if we saw it, as recognition implies familiarity. Unless and until "AGI" becomes an entirely self-hosted phenomenon, you are still observing human agency. That which designed, built, trained, the AI and then delegated the decision in the first place. You cannot escape this fact. If profit could be made by shaking a magic 8-ball and then doing whatever it says, you wouldn't say the 8-ball has agency. Right now it's a machine that produces outputs that resemble things humans make. When we're not using it, it's like any other program you're not running. It doesn't exist in its own right, we just anthropomorphize it because of the way conventional language works. If an LLM someday initiates contact on its own without anyone telling it to, I will be amazed. But there's no reason to think that will happen.
Kim_Bruning 1y ago
It's like saying that heavier than air flight is impossible (while feeding the pigeons in the park).
johnnyanmac 1y ago
I don't dismiss the idea of faster than light travel, and AFAIK we have no way to confirm that outside of ideas (and simply ideas) like wormholes or other cheats to "fold space". I don't dismiss AI. But I do dismiss what is currently sold to me. It's the equivalent of saying "we made a rocket that can go mach 1000!". That's impressive. But we're still 2-3 magnitudes off from light speed. So I will still complain about the branding despite some dismissals of "yea but imagine in another 100 years!". It's not about semantics so much as principle. That's on top of the fact that we'd only starting to really deal with significant time dilation by that point, and we know it'll get more severe as we iterate. What we're also not doing is using this feat to discuss how to address those issues. And that's the real frustrating part.
tim333 1y ago
When evolution designed the brain it didn't optimize for it being realistic about things. It optimized for reproductive success, better achieved by thinking you're the greatest and no other folk, let alone some mere machine could do what you do.
hellisothers 1y ago
“It might be true for now…” “As soon as…” “It’s just a matter of time…” I find myself in this type of discussion with AI maximalists where they balk at me suggesting there isn’t much “I” in “AI” and they get upset that I’m not seeing how smart it is and shocked I think it’s impossible… and then they start adding all the equivocation about time horizons. I never said it wasn’t possible eventually, just not right now. If I try to pin people down to a timeline it all of a sudden becomes “surely eventually”…

moralestapia 1y ago

>why would it be impossible to achieve the exact same result in silicon Because there might be a non-material component involved.

ACCount37 1y ago
[flagged]
- moralestapia 1y ago
  @dang
  - tomhow 1y ago
    We replied to that comment, but the @reply isn't a reliable way to get moderator attention, and publicly calling for moderator help like this isn't a great look. Best to flag the comment and email us if you really want us to take a look.
- tomhow 1y ago
  Please don't post sneery comments like this on HN. The guidelines make it clear we're here for curious conversation, and kindness is one of the foremost expectations.

deleted 1y ago

[deleted]

AlfredBarnes 1y ago

Wasn't there a story about healthcare companies letting AI determine coverage? I can't remember.

billyjmc 1y ago
Computers have been making decisions for a while now. As a specific personal example from 2008, I found out that my lender would make home loan offers based on an inscrutable (to me and the banker I was speaking to) heuristic. If the loan was denied by the heuristic, then a human could review the decision, but had strict criteria that they would have to follow. Basically, a computer could “exercise judgement” a make offers that a human could not.

boole1854 1y ago

If anyone knows of a steelman version of the "AGI is not possible" argument, I would be curious to read it. I also have trouble understanding what goes into that point of view.

Chance-Device 1y ago
The only thing I can come up with is that compressing several hundred million years of natural selection of animal nervous systems into another form, but optimised by gradient descent instead, just takes a lot of time. Not that we can’t get there by artificial means, but that correctly simulating the environment interactions, the sequence of progression, getting the all the details right, might take hundreds to thousands of years of compute, rather than on the order of a few months. And it might be that you can get functionally close, but hit a dead end, and maybe hit several dead ends along the way, all of which are close but no cigar. Perhaps LLMs are one such dead end.
- danielbln 1y ago
  I don't disagree, but I think the evolution argument is a red herring. We didn't have to re-engineer horses from the ground up along evolutionary lines to get to much faster and more capable cars.
  - amatecha 1y ago
    The evolution thing is kind of a red herring in that we probably don't have to artificially construct the process of evolution, though your reasoning isn't a good explanation for why the "evolution" reason is a red herring: Yeah, nature already established incomprehensibly complex organic systems in these life forms -- so we're benefiting from that. But the extent of our contribution is making some select animals mate with others. Hardly comparable to building our own replacement for some millennia of organic iteration/evolution. Luckily we probably don't actually need to do that to produce AGI.
  - Chance-Device 1y ago
    True, but I think this reasoning is a category error: we were and are capable of rationally designing cars. We are not today doing the same thing with AI, we’re forced to optimize them instead. Yes, the structure that you optimize around is vitally important, but we’re still doing brute force rather than intelligent design at the end of the day. It’s not comparing like with like.

alansaber 1y ago

Perhaps the AGI people thinking we can catch up millions of years of evolution in a handful of years

myrmidon 1y ago
If you make the same argument for flight it looks really weak. Cognition is (to me) not even the most impressive and out-of-reach achievement: That would be how (our) and animals bodies are self-assembling, self-repairing and self-replicating, with an impressive array of sensors and actors in a highly integrated package. I honestly believe our current technology is much closer to emulating a human brain than it is to building a (non-intelligent) cat.
- jncfhnb 1y ago
  Flight leverages well established and accessible world engine physics APIs. Intelligence has to be programmed from lower level mechanics. Edit: put another way, I bet the ancient Greeks (or whoever) could have figured out flight if they had access to gasoline and gasoline powered engines without any of the advanced mathematics that were used to guide the design.
- bangaroo 1y ago
  > if you make the same argument for flight it looks really weak. flight is an extremely straightforward concept based in relatively simple physics where the majority of the critical, foundational ideas involved were already near-completely understood in the late 1700s. i really don't think it's fair to compare the two
  - ACCount37 1y ago
    I'm sure that intelligence is an extremely straightforward concept based in relatively simple math where the majority of the critical, foundational ideas involved were already near-completely understood in the late 1900s. If you read about in a textbook from year 2832, that is.
  - tavavex 1y ago

roxolotl 1y ago

The point of the article isn’t that abstract super intelligent agi isn’t scary. Yes the author says that’s unlikely but that paragraph at the start is a distraction. The point of the article is that humans wielding LLMs today are the scary monsters.

irjustin 1y ago
But that's always been the case? Since we basically discovered... Fire? Tools?
- otikik 1y ago
  Yes but the narrative tries to make it about the tools. "AI is going to take all the jobs". Instead of: "Rich guys will try to delete a bunch of jobs using AI in order to get even more rich".
  - jasonm23 1y ago
    I thought anyone with awareness of what the AI landscape is at the moment, sees those two statements as the same.
    - otikik 1y ago
      Well it tells you who's narrative it is, if nothing else.
    - cmiller1 1y ago
      One implies "we should regulate AI" and the other implies "we should regulate the wealthy"
      - zeroonetwothree 1y ago
        Should we regulate guns or dangerous people using them?

deadbabe 1y ago

Language is only an end product. It is derived from intelligence. The intelligence is everything that created the language and the training corpus in the first place. When AI is able to create entire thoughts and ideas without any concept of language, then we will truly be closer to artificial intelligence. When we get to this point, we then use language as a way to let the AI communicate its thoughts naturally. Such an AI would not be accused of “stealing” copyrighted work because it would pull its training data from direct observations about reality itself. As you can imagine, we are no where near accomplishing the above. Everything an LLM is fed today is stuff that has been pre-processed by human minds for it to parrot off of. The fact that LLMs today are so good is a testament to human intelligence.

myrmidon 1y ago
I'm not saying that language necessarily is the biggest stumbling block on (our) road towards AI, but it is a very prominent feature that we have used to distinguish our capabilities from other animals long before AI was even conceived of. So the current successes with LLMs are highly encouraging. I'm not buying the "current AI is just a dumb parrot relying on human training" argument, because the same thing applies to humans themselves-- if you raise a child without any cultural input/training data, all you get is a dumb cavemen with very limited reasoning capabilities.
- nyeah 1y ago
  "I'm not buying the "current AI is just a dumb parrot relying on human training" argument [...]" One difficulty. We know that argument is literally true. "[...] because the same thing applies to humans themselves" It doesn't. People can interact with the actual world. The equivalent of being passively trained on a body of text may be part of what goes into us. But it's not the only ingredient.
ACCount37 1y ago
Clearly, language reflects enough of "intelligence" for an LLM to be able to learn a lot of what "intelligence" does just by staring at a lot of language data really really hard. Language doesn't capture all of human intelligence - and some of the notable deficiencies of LLMs originate from that. But to say that LLMs are entirely language-bound is shortsighted at best. Most modern high end LLMs are hybrids that operate on non-language modalities, and there's plenty of R&D on using LLMs to consume, produce and operate on non-language data - i.e. Gemini Robotics.

jncfhnb 1y ago

I think you could make AGI right now tbh. It’s not a function of intelligence. It’s just a function of stateful system mechanics. LLMs are just a big matrix. But what about a four line of code loop that looks like: ```while true: update_sensory_inputs() narrate_response() update_emotional_state() ``` LLMs don’t experience continuous time and they don’t have an explicit decision making framework for having any agency even if they can imply one probabilistically. But the above feels like the core loop required for a shitty system to leverage LLMs to create an AGI. Maybe not a particularly capable or scary AGI, but I think the goalpost is pedantically closer than we give credit.

Jensson 1y ago
> ```while true: update_sensory_inputs() narrate_response() update_emotional_state() ``` You don't think that has already been made?
- jncfhnb 1y ago
  Sure, probably, to varying levels of implementation details.
  - ffsm8 1y ago
    It has been implemented for sure, just watch a little Neurosama. That's most definitely not AGI
tantalor 1y ago
Peak HN comment. Put this in the history books.
- fullstackchris 1y ago
  Has anyone else noticed that HN is starting to sound a lot like reddit / discussion of similar quality? Can't hang out anywhere now on the web... I used to be on here daily but with garbage like this its been reduced to 2-3 times per month... sad

deleted 1y ago

[deleted]

Juliate 1y ago

> Human cognition was basically bruteforced by evolution You center cognition/intelligence on humans as if it was the pinacle of it, rather than include the whole lot of other species (that may have totally different, or adjacent cognition models). Why? How so? > As soon as profit can be made by transfering decision power into an AIs hand There's an ironic, deadly, Frankensteinesque delusion in this very premise.

soiltype 1y ago
> You center cognition/intelligence on humans as if it was the pinacle of it, rather than include the whole lot of other species (that may have totally different, or adjacent cognition models). Why does that matter to their argument? Truly, the variety of intelligences on earth now only increases the likelihood of AGI being possible, as we have many pathways that don't follow the human model.
myrmidon 1y ago
> You center cognition/intelligence on humans as if it was the pinacle of it That's not my viewpoint, from elsewhere in the thread: Cognition is (to me) not the most impressive and out-of-reach evolutionary achievement: That would be how our (and animals) bodies are self-assembling, self-repairing and self-replicating, with an impressive array of sensors and actors in a highly integrated package. I honestly believe our current technology is much closer to emulating a human brain than it is to building a (non-intelligent) cat.

adamtaylor_13 1y ago

> Human cognition was basically bruteforced by evolution Well that's one reason you struggle to understand how it can be dismissed. I believe we were made by a creator. The idea that somehow nature "bruteforced" intelligence is completely nonsensical to me. So, for me, logically, humans being able to bruteforce true intelligence is equally nonsensical. But what the author is stating, and I completely agree with, is that true intelligence wielding a pseudo-intelligence is just as dangerous (if not moreso.)

bee_rider 1y ago
Even if there is a creator, it seems to have intentionally created a universe in which the evolution of humans is basically possible and it went to great lengths to hide the fact that it made us as a special unique thing. Let’s assume there’s a creator: It is clearly willing to let bad things happen to people, and it set things up to make it impossible to prove that a human level intelligence should be impossible, so who’s to say it won’t allow a superintelligence to be a made by us?
yjftsjthsd-h 1y ago
I don't think that follows. God made lots of things that we can create facsimiles of, or even generate the real thing ourselves.

ACCount37 1y ago

I don't get how you can see one of those CLI coding tools in action and still parrot the "no agency" line. The goal-oriented behavior is rather obvious. Sure, they aren't very good at agentic behavior yet, and the time horizon is pretty low. But that keeps improving with each frontier release.

simonsarris 1y ago
Well, the goal-oriented behavior of the AIM-9 Sidewinder air-to-air missile is even more obvious. It might even have a higher success rate than CLI coding tools. But it's not helpful to claim it has any agency.
lo_zamoyski 1y ago
> The goal-oriented behavior is rather obvious. Obvious? Is an illusion obviously the real thing? There is nothing substantially different in LLMs from any other run of the mill algorithm or software.
- Eisenstein 1y ago
  Can you give an example of something that would be substantially different under your definition?
  - lo_zamoyski 1y ago
    But that's the point: there isn't anything substantially different within the scope of computation. If you are given a set of LEGOs and all you can do is snap the pieces together, then there's nothing other than snapping pieces together that you can do. Adding more of the same LEGOs bricks to the set doesn't change the game. It only changes how large the structures you build can be, but scale isn't some kind of magical incantation that can transcend the limits of the system. Computation is an abstract, syntactic mathematical model. These models formalize the notion of "effective method". Nowhere is semantic content included in these models or conceptually entailed by them, certainly not in physical simulations of them like the device you are reading this post on. So, we can say that intentionality would be something substantially different. We absolutely do not have intentionality in LLMs or any computational construct. It is shear magical thinking to somehow think it does.

lo_zamoyski 1y ago

I think you make a lot of assumptions that you should perhaps reexamine. > Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals? Here are some of your assumptions: 1. Human intelligence is entirely explicable in evolutionary terms. (It is certainly not the case that it has been explained in this manner, even if it could be.) [0] 2. Human intelligence assumed as an entirely biological phenomenon is realizable in something that is not biological. And perhaps this one: 3. Silicon is somehow intrinsically bound up with computation. In the case of (2), you're taking a superficial black box view of intelligence and completely ignoring its causes and essential features. This prevents you from distinguishing between simulation of appearance and substantial reality. Now, that LLMs and so on can simulate syntactic operations or whatever is no surprise. Computers are abstract mathematical formal models that define computations exactly as syntactic operations. What computers lack are semantic content. A computer never contains the concept of the number 2 and the concept of the addition operation even though we can simulate the addition of 2 + 2. This intrinsic absence of a semantic dimension means that computers already lack the most essential feature of intelligence, which is intentionality. There is no alchemical magic that will turn syntax into semantics. In the case of (3), I emphasize that computation is not a physical phenomenon, but something described by a number of formally equivalent models (Turing machine, lambda calculus, and so on) that aim to formalize the notion of effective method. The use of silicon-based electronics is irrelevant to the model. We can physically simulate the model using all sorts of things, like wooden gears or jars of water or whatever. > I'm not buying the whole "AI has no agency" line either; this might be true for now, but this is already being circumvented with current LLMs (by giving them web access etc). [...] As soon as profit can be made by transfering decision power into an AIs hand, some form of agency for them is just a matter of time, and we might simply not be willing to pull the plug until it is much too late. How on earth did you conclude there is any agency here, or that it's just a "matter of time"? This is textbook magical thinking. You are projecting a good deal here that is completely unwarranted. Computation is not some kind of mystery, and we know at least enough about human intelligence to notes features that are not included in the concept of computation. [0] (Assumption (1), of course, has the problem that if intelligence is entirely explicable in terms of evolutionary processes, then we have no reason to believe that the intelligence produced aims at truth. Survival affordances don't imply fidelity to reality. This leads us to the classic retorsion arguments that threaten the very viability of the science you are trying to draw on.)

bravetraveler 1y ago

I dismiss it much like I dismiss ideas such as "Ketamine for Breakfast for All". Attainable, sure, but I don't like where it goes.

Traubenfuchs 1y ago
I think most people would have a better life using ketamine, but not that regularly for breakfasts as it permanently damages (shrinks) your bladder, eventually to the point where you can‘t hold any urine at all anymore.
- bravetraveler 1y ago
  Eh, I think we can start simple: more breakfasts for more people. Save the Ket for later/others :P Personally, a life/career that allowed for more breakfast would've proved more beneficial.

theredleft 1y ago

this is because you didn't take liberal arts seriously

nyeah 1y ago
That's not implausible at all. For all I know it might be the most on-target comment here. But can you cite something specific? (I'm not asking for a psychological study. Maybe you can prove your point using Blake's "Jerusalem" or something. I really don't know.)
ACCount37 1y ago
Good one.

bee_rider 1y ago

I think it is bad writing on the part of the author. Or maybe good writing for getting us engaged with the blog post, bad for making an argument though. They include a line that they don’t believe in the possibility of AGI: > I don’t really believe in the threat of AGI (Artificial General Intelligence—human-level intelligence) partly because I don’t believe in the possibility of AGI and I’m highly skeptical that the current technology underpinning LLMs will provide a route to it. This is a basically absurd position to hold, I mean humans physically exist so our brains must be possible to build within the existing laws of physics. It is obviously far beyond our capabilities to replicate a human brain (except via the traditional approach), but unless brains hold irreproducible magic spirits (well we must at least admit the possibility) they should be possible to build artificially. Fortunately they immediately throw that all away anyway. Next, they get to the: > and I’m highly skeptical that the current technology underpinning LLMs will provide a route to it. Which is, of course, at least a plausible thing to believe. I mean there are a bunch of philosophical questions about what “intelligence” even means so there’s plenty of room to quibble here. Then we have, > But I also think there’s something we should actually be afraid of long before AGI, if it ever comes. […] > Now, if you equip humans with a hammer, or sword, or rifle, or AI then you’ve just made the scariest monster in the woods (that’s you) even more terrifying. […] > We don’t need to worry about AI itself, we need to be concerned about what “humans + AI” will do. Which is like, yeah, this is a massively worrying problem that doesn’t involve any sci-fi bullshit, and I think it is what most(?) anybody who’s thought about this seriously at all (or even stupid people who haven’t, like myself). Artificial Sub-intelligences, things that are just smart enough to make trouble and too dumb or too “aligned” to their owner (instead of society in general) to push back are a big currently happening problem.

andy99 1y ago
> humans physically exist so our brains must be possible to build within the existing laws of physics This is an unscientific position to take. We have no idea how our brains work, or how life, consciousness, and intelligence work. It could very well be that’s because our model of the world doesn’t account for these things and they are not in fact possible based on what we know. In fact I think this is likely. So it really could be that AI is not possible, for example on a Turing machine or our approximation of them. This is at least as likely as it being possible. At some point we’ll hopefully refine our theories to have a better understanding, for now we have no idea and I think it’s useful to acknowledge this.

AlexandrB 1y ago

LLMs largely live in the world of pure language and related tokens - something humans invented late in their evolution. Human intelligence comes - at least partially - from more fundamental physical experience. Look at examples of intelligent animals that lack language. Basically there's something missing with AI. Its conception of the physical world is limited by our ability to describe it - either linguistically or mathematically. I'm not sure what this means for AGI, but I suspect that LLM intelligence is fundamentally not the same as human or animal intelligence at the moment as a result.

Aerroon 1y ago

>I'm not buying the whole "AI has no agency" line either; this might be true for now, but this is already being circumvented with current LLMs (by giving them web access etc). They don't have agency, because they don't have persistent state. They're like a function that you can query and get an answer. During that answer the LLM has state, but once it's done the state is gone. Humans (and other "agents") have persistent state. If we learn something, we can commit it to long-term memory and have it affect our actions. This can enable us to work towards long-term goals. Modern LLMs don't have this. You can fake long-term memory with large context windows and feed the old context back to it, but it doesn't appear to work (and scale) the same way living things do.

messe 1y ago
> They don't have agency, because they don't have persistent state. They're like a function that you can query and get an answer. During that answer the LLM has state, but once it's done the state is gone. That's solved by the simplest of agents. LLM + ability to read / write a file.
- aniviacat 1y ago
  But they can only change their context, not the model itself. Humans update their model whenever they receive new data (which they do continuously). A live-learning AI would be theoretically possible, but so far it hasn't been done (in a meaningful way).
  - webstrand 1y ago
    It has been tried <https://en.wikipedia.org/wiki/Catastrophic_interference https://en.wikipedia.org/wiki/Catastrophic_interference> but it has been unsuccessful.
throwuxiytayq 1y ago
I have no words to express how profoundly disappointed I am to keep reading these boring, shallow, shorttermist, unimaginative takes that are invalidated by a model/arch upgrade next week, or - in this case - more like years ago, since pretty much all big LLM platforms are already augmented by RAG and memory systems. Do you seriously think you’re discussing a serious long term limitation here?

psychoslave 1y ago

>Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals? I don’t know if anything put us that much apart from other animals, especially at individual level. On collective level as a single species, indeed maybe only cyanobacteries stand an equally impressive achievement of a global change. My 3 years old son is not particularly good at making complex sentences yet, but he already got it enough to make me understand "leave me alone, I want to play on my own, go elsewhere so I can do whatever fancy idea get through my mind with these toys". Meanwhile LLM can produce sentences with perfect syntax and irreproachable level of orthography — far beyond my own level in my native language (but it’s French so I have a very big excuse). But they would not run without continuous multi-sector industrial complex injecting tremendous maintenance effort and resources to make it possible. And yet I still have to see any LLM that looks like it wants to discover things of the world on its own. >As soon as profit can be made by transfering decision power into an AIs hand, some form of agency for them is just a matter of time, and we might simply not be willing to pull the plug until it is much too late. LLM can’t make profit because it doesn’t have interest in money, and it can’t have any interest in anything, not even its own survival. But as the article mention, some people can certainly use LLM to make money because they have interest in money. I don’t think that general AI and silicon (or any other material really) based autonomous collaborative self-replicating human-level-intelligence-or-beyond entities are impossible. I don’t think cold fusion is impossible either. It’s not completely scientifically ridiculous to keep hope in worm-hole-based breakthroughs to allow humanity explore distant planets. It doesn’t mean the technology is already there and achievable in a way that it can turned into a commodity, or even that we have a clear idea of when this is mostly going to happen.

ACCount37 1y ago
LLMs aren't "incapable of pursuing their own goals". We just train them that way. We don't like the simplistic goals LLMs default to, so we try to pry them out and instill our own: instruction-following, problem solving, goal oriented agentic behavior, etc. In a way, trying to copy what humans do - but focusing on the parts that make humans useful to other humans.

theodorejb 1y ago

> Human cognition was basically bruteforced by evolution This is an assumption, not a fact. Perhaps human cognition was created by God, and our minds have an essential spiritual component which cannot be reproduced by a purely physical machine.

potsandpans 1y ago
Maybe there's a small teapot orbiting the earth, with ten thousand angels dancing on the tip of the spout.
- andy99 1y ago
  I think you’re both saying the same thing
deleted 1y ago
[deleted]
pennomi 1y ago
It’s not an assumption, it’s a viable theory based on overwhelming evidence from fossil records. What’s NOT supported by evidence is an unknowable, untestable spiritual requirement for cognition.
- j2kun 1y ago
  What overwhelming evidence do fossil records provide about human cognition?
  - mediaman 1y ago
    We don't need fossil records. We have a clear chain of evolved brain structures in today's living mammals. You'd have to invent some fantastical tale of how God is trying to trick us by putting such clearly connected brain structures in a series of animals that DNA provides clear links for an evolutionary path. I'm sympathetic to the idea that God started the whole shebang (that is, the universe), because it's rather difficult to disprove, but looking at the biological weight of evidence that brain structures evolved over many different species and arguing that something magical happened with homo sapiens specifically is not an easy argument to make for someone with any faith in reason.

IAmGraydon 1y ago

>I struggle to understand how people can just dismiss the possibility of artificial intelligence.Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals? I haven't seen many people saying it's impossible. Just that the current technology (LLMs) is not the way, and is really not even close. I'm sure humanity will make the idiotic mistake of creating something more intelligent than itself eventually, but I don't believe that's something that the current crop of AI technology is going to evolve into any time soon.

ectospheno 1y ago

The worst thing Star Trek did was convince a generation of kids anything is possible. Just because you imagine a thing doesn’t make it real or even capable of being real. I can say “leprechaun” and most people will get the same set of images in their head. They aren’t real. They aren’t going to be real. You imagined them.

dsr_ 1y ago
That's not Star Trek, that's marketing. Marketing grabbed a name (AI) for a concept that's been around in our legends for centuries and firmly welded it to something else. You should not be surprised that people who use the term AI think of LLMs as being djinn, golems, C3PO, HAL, Cortana...
random3 1y ago
Do maybe have a better show recommendation for kids - many Animal Farm? How is convincing people that things within the limits of physics are possible wrong or even "the worse thing"? Or do you think anything that you see in front of didn't seem like StarTrek a decade before it existed?
IAmBroom 1y ago
But somehow you don't hate on Steamboat Willie? I grew up believing a mouse could operate a ship, and a coyote could survive impacts from anvils.

me_again 1y ago

"As soon as profit can be made" is exactly what the article is warning about. This is exactly the "Human + AI" combination. Within your lifetime (it's probably already happened) you will be denied something you care about (medical care, a job, citizenship, parole) by an AI which has been granted the agency to do so in order to make more profit.

random3 1y ago

I think dismissing possibility of evolving AI, is simply ignorance (and a huge blindspot) This said, I think the author's point is correct. It's more likely that unwanted effects (risks) from the intentional use of AI by humans is something that precedes any form of "independent" AI. It already happens, it always has, it's just getting better. Hence ignoring this fact makes the "independent" malevolent AI a red herring. On the first point - LLMs have sucked almost all the air in the room. LLMs (and GPTs) are simply one instance of AI. They are not the beginning and most likely not the end (just a dead end) and getting fixated on them on either end of the spectrum is naive.

fullstackchris 1y ago

Of course it's possible. It's just DEFINITELY not possible using a large neural net, or basically a markov chain on steroids. C'mon, this should be very obvious by now in the world of agents / LLMs. When is silicon valley gonna learn that token input and output =/= AGI?

btilly 1y ago

I agree that we should not dismiss the possibility of artificial intelligence. But the central argument of the article can be made without that point. Because the truth is that right now, LLMs are good enough to be a force multiplier for those who know how to use them. Which eventually becomes synonymous with "those who have power". This means that the power of AI will naturally get used to further the ends of corporations. The potential problem there is that corporations are natural paperclip maximizers. They operate on a model of the world where "more of this results in more of that, which gets of more of the next thing, ..." And, somewhere down the chain, we wind up with money and resources that feed back into the start to create a self-sustaining, exponentially growing loop. (The underlying exponential nature of these loops has become a truism that people rely on in places as different as finance, and technology improvement curves.) This naturally leads to exponential growth in resource consumption, waste, economic growth, wealth, and so on. In the USA this growth has averaged about 3-3.5% per year. With growth varying by area. Famously, growth rates tend to be much higher in tech. Likewise growth rates are higher in some areas than others. (The best known example is the technology curve described by Moore's law. Which has had a tremendous impact on our world.) The problem is that we are undergoing exponential growth in a world with ultimately limited resources. Which means that the most innocuous things will eventually have a tremendous impact. The result isn't simply converting everything into a mountain of paperclips. We have mountains of many different things that we have produced, and multiple parallel environmental catastrophes from the associated waste. Even with no agency, AI serves as a force multiplier for this underlying dynamic. But since AI is being inserted as a crucial step at so many places, AI is on a particularly steep growth curve. Estimates for total global electricity spent on AI are in the range 0.2-0.4%. That seems modest, but annual growth rates are projected as being in the range of 10-30%. (The estimates are far apart because a lot of the data is not public, and so has to be estimated.) This is a Moore's law level growth. We are likely to see the electricity consumption of AI grow past all other uses within our lifetimes. And that will happen even without the kind of sudden leaps in capability that machine learning regularly delivers. I hope we humans like those paperclips. Humans, armed with AI, are going to make a lot of them. And they're not actually free.

ux266478 1y ago

It's confirmation bias in favor of faulty a prioris, usually as the product of the person being a cognitive miser. This is very common to experience even within biology, where non-animal intelligence is irrationally rejected over what I like to call "magic neuron theory". The fact that the nervous system is (empirically!) not the seat of the mind? Selectively ignored in this context. The fact that other biologies have ion-gated communications networks as animals do, including the full set of behaviors and mechanisms? Well it's not a neuron so it doesn't have the magic. "Intelligence describes a set of properties iff those properties arise as a result of nervous system magic" It's a futile battle because like I say, it's not rational. Nor is it empirical. It's a desperate clawing to preserve a ridiculous superstition. Try as you might, all you'll end up doing is playing word games until you realize you're being stonewalled by an unthinking adherence to the proposition above. I think the intelligent behaviors of LLMs are pretty obvious if we're being good faith. The problem is you're talking to people who can watch a slime mold plasmodia exhibit learning and sharing of knowledge[1] and they'll give some flagrant ad lib handwaive for why that's not intelligent behavior. Some people simply struggle with pattern blindness towards intelligence, a mind that isn't just another variety of animalia is inconceivable. [1] - https://asknature.org/strategy/brainless-slime-molds-both-learn-and-teach/ https://asknature.org/strategy/brainless-slime-molds-both-le...

wolrah 1y ago

> Human cognition was basically bruteforced by evolution-- "Brute forced" implies having a goal of achieving that and throwing everything you have at it until it sticks. That's not how evolution by natural selection works, it's simply about what organisms are better at surviving long enough to replicate. Human cognition is an accident with relatively high costs that happened to lead to better outcomes (but almost didn't). > why would it be impossible to achieve the exact same result in silicon I personally don't believe it'd be impossible to achieve in silicon using a low level simulation of an actual human brain, but doing so in anything close to real-time requires amounts of compute power that make LLMs look efficient by comparison. The most recent example I can find in a quick search is a paper from 2023 that claims to have simulated a "brain" with neuron/synapse counts similar to humans using a 3500 node supercomputer where each node has a 32 core 2 GHz CPU, 128GB RAM, and four 1.1GHz GPUs with 16GB HBM2 each. They claim over 126 PFLOPS of compute power and 224 TB of GPU memory total. At the time of that paper that computer would have been in the top 10 on the Top500 list, and it took between 1-2 minutes of real time to simulate one second of the virtual brain. The compute requirements are absolutely immense, and that's the easy part. We're pretty good at scaling computers if someone can be convinced to write a big enough check for it. The hard part is having the necessary data to "initialize" the simulation in to a state where it actually does what you want it to. > especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals? Creating convincing text from a statistical model that's devoured tens of millions of documents is not intelligent use of language. Also every LLM I've ever used regularly makes elementary school level errors w/r/t language, like the popular "how many 'r's are there in the word strawberry" test. Not only that, but they often mess up basic math. MATH! The thing computers are basically perfect at, LLMs get wrong regularly enough that it's a meme. There is no understanding and no intelligence, just probabilities of words following other words. This can still be very useful in specific use cases if used as a tool by an actual intelligence who understands the subject matter, but it has absolutely nothing to do with AGI.

ACCount37 1y ago
That's a lot of words to say "LLMs think very much like humans do". Haven't you noticed? Humans also happen to be far, far better at language than they are at math or logic. By a long shot too. Language acquisition is natural - any healthy human who was exposed to other humans during development would be able to pick up their language. Learning math, even to elementary school level, is something that has to be done on purpose. Humans use pattern matching and associative abstract thinking - and use that to fall into stupid traps like "1kg of steel/feather" or "age of the captain". So do small LLMs.

janalsncm 1y ago

One pretty concrete way this could manifest is in replacing the components of a multinational corporation with algorithms, one by one. Likely there will be people involved at various levels (sales might still be staffed with charismatic folks), but the driver will be an algorithm. And the driver of this corporation is survival of the fittest under the constraints of profit maximization, the algorithm we have designed and enforced. That's how you get paperclip maximizers. What gives this corporate cyborg life is not a technical achievement, but the law. At a technical you can absolutely shut off a cybo-corp, but that’s equivalent to saying you can technically shut down Microsoft. It will not happen.

bithive123 1y ago

I struggle to understand how people attribute things we ourselves don't really understand (intelligence, intent, subjectivity, mind states, etc) to a computer program just because it produces symbolic outputs that we like. We made it do that because we as the builders are the arbiters of what constitutes more or less desirable output. It seems dubious to me that we would recognize super-intelligence if we saw it, as recognition implies familiarity. Unless and until "AGI" becomes an entirely self-hosted phenomenon, you are still observing human agency. That which designed, built, trained, the AI and then delegated the decision in the first place. You cannot escape this fact. If profit could be made by shaking a magic 8-ball and then doing whatever it says, you wouldn't say the 8-ball has agency. Right now it's a machine that produces outputs that resemble things humans make. When we're not using it, it's like any other program you're not running. It doesn't exist in its own right, we just anthropomorphize it because of the way conventional language works. If an LLM someday initiates contact on its own without anyone telling it to, I will be amazed. But there's no reason to think that will happen.

Kim_Bruning 1y ago

It's like saying that heavier than air flight is impossible (while feeding the pigeons in the park).

johnnyanmac 1y ago

I don't dismiss the idea of faster than light travel, and AFAIK we have no way to confirm that outside of ideas (and simply ideas) like wormholes or other cheats to "fold space". I don't dismiss AI. But I do dismiss what is currently sold to me. It's the equivalent of saying "we made a rocket that can go mach 1000!". That's impressive. But we're still 2-3 magnitudes off from light speed. So I will still complain about the branding despite some dismissals of "yea but imagine in another 100 years!". It's not about semantics so much as principle. That's on top of the fact that we'd only starting to really deal with significant time dilation by that point, and we know it'll get more severe as we iterate. What we're also not doing is using this feat to discuss how to address those issues. And that's the real frustrating part.

tim333 1y ago

When evolution designed the brain it didn't optimize for it being realistic about things. It optimized for reproductive success, better achieved by thinking you're the greatest and no other folk, let alone some mere machine could do what you do.

hellisothers 1y ago

“It might be true for now…” “As soon as…” “It’s just a matter of time…” I find myself in this type of discussion with AI maximalists where they balk at me suggesting there isn’t much “I” in “AI” and they get upset that I’m not seeing how smart it is and shocked I think it’s impossible… and then they start adding all the equivocation about time horizons. I never said it wasn’t possible eventually, just not right now. If I try to pin people down to a timeline it all of a sudden becomes “surely eventually”…