12 ms·

We're not Platonists, we've just learned the bitter lesson

102 points by Michelangelo11 3y ago

retrocryptid 3y ago
meh. "intelligence" and "artificial intelligence" are marketing terms. According to Nils Nilsson, McCarthy and Minsky picked the term so the Dartmouth Summer Research Project on Artificial Intelligence would not be constrained by associations with "Cybernetics." McCarthy thought it (Cybernetics) too constrained because a: it traditionally focused on analog solutions and b: he thought Norbert Wiener was a bit pushy. I asked Minsky about this a couple decades ago and it was the only time he didn't tell me I was asking a stupid question. Apparently, after the conference, the name stuck because they found it easier to get research grants with the more "neutral" sounding name. I wonder how many people would trust "AI" if it was called "distributed indexing" or "stochastic generation."
- bryan0 3y ago
  I think the article explains very well why "intelligence" is a useful and important concept. Namely because it is concise way to discuss abilities which are highly correlated without naming each ability individually.
  - morelisp 3y ago
    Somehow the rationalist community started at empty platitudes like "correlation is not causation" yet has ended up at substantive yet amazingly wrong conclusions like "correlation is identity", amazing turn, I gotta say I did not see it coming.
    - philovivero 3y ago
      But correlation is correlation. At least, for most people's definitions of the word.
    - thebooktocome 3y ago
      I’m frankly surprised Judea Pearl’s name didn’t come up in this article, they love waving him around at times like this.
- golol 3y ago
  Intelligence has a pretty clear meaning. Of course you will have a hard time nailing down a definition but that's also true for simple words like "chair" or "car". Artificial intelligence is also a pretty good name for a pretty clear concept: intelligence that has been artificially created. Any new ambiguity here comes from the ambiguity inherent in the word "artificial". Artifivial intelligence has a large range from if/else, linear regression and decision trees over ML methods to god-like superintelligence. That is not a problem, I think it's good that we have a term for this. I don't like when people try to reduce artificial intelligence to a smaller subset of these things. We should certainly come up with more precise terms in addition, but I think artificial intelligence is a very useful terminology and concept, not a marketing term.
  - cacois 3y ago
    > Any new ambiguity here comes from the ambiguity inherent in the word "artificial". Not sure I agree here. Seems to me the ambiguity comes from the word "intelligence". As in, does that if/else clause indicate "intelligence" that's been programmed in. Of course we've created more terms like Artificial General Intelligence[1] to try to clear up the muddy definition of what constitutes intelligence. I think our fundamental challenge as society begins to try to understand what AI is comes down to what "intelligence" is. As you said, we all know what it is, but can't define it easily. If I write some really clever code, is the system running it intelligent? Is some of my intelligence imbued into that system, or does it have some of its own? At what point does a stochastic process approach intelligence - when it has enough data? Does that mean babies aren't born with intelligence, but acquire it as they learn more? So many questions, good questions, necessary questions. [1] https://en.wikipedia.org/wiki/Artificial_general_intelligence https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
    - nextaccountic 3y ago
      > does that if/else clause indicate "intelligence" that's been programmed in Yes, systems built with if/else rules are intelligent, in that they reproduce some heuristic rule that someone had to think about. I think the term "expert system" is kind of appropriate for them https://en.wikipedia.org/wiki/Expert_system https://en.wikipedia.org/wiki/Expert_system Of course they are not too intelligent, but, they are more intelligent than a rock, say. (unless this rock happen to be a chip running some neural network)
- benlivengood 3y ago
  All that matters is if they are effective terms or not. That's most of the point of the article; how to constructively inquire about whether giant LLMs might be effective at solving novel problems, perhaps better than humans can, in ways that matter to humans. Since we don't have a formal model of how systems like humans work we can't just plug numbers into equations or generate proofs that LLMs and other ML systems will or will not be able to control other systems better than humans, so we revert to less-formal concepts as best as we can. The formal AI safety folks (MIRI et al) already tried the formal approach; by their own admission it's hopeless and they poured a lot of time and bright minds into it and we won't be able to formally answer questions about intelligence via decision theory or alignment before we have actually-intelligent systems interacting with us. What will happen at that point? We don't know and can only use imprecise concepts to make educated guesses. If we don't make honest attempts at this we run the risk of being blindsided by unexpectedly capable AI/ML systems. This is why a large number of AI alignment researchers recommend pausing AI/ML systems at their current ability before they have a chance at being superhuman on most tasks. I wouldn't trust "distributed indexing" or "stochastic generation" if it could beat me at any particular skill of my choosing like similar systems can beat any of us at Chess or Go or Poker.
- m463 3y ago
  This reminds me of MRI which started out as Nuclear Magnetic Resonance Imaging, but dropped "Nuclear" because of the stigma it carried.
  - merchan 3y ago
    There's a less cynical explanation for that. "NMR" sounds a lot like "enema" and you don't want to get the two mixed up.
    - yetanotherloser 3y ago
      I'd never realised that before. Thank you, I think. Deary, deary me.
- unholiness 3y ago
  By that reasoning "Chemistry" is just a marketing term. It was used mostly interchangeably with "Alchemy" until Georg Stahl and others began writing (without justification!) that "Chemistry" describes more a more rational science and "Alchemists" were more interested in wealth. These untrue claims still became a self-fulfilling prophecy, where theories, teachings, and practice of chemistry evolved out of the former ways of thinking to become what we know it as today. AI, as used today, is certainly a coherent concept, and this article has justified and described it more thoroughly than most concepts I use daily. Who cares about the shaky history if it means something real today?
morelisp 3y ago
[flagged]
- mannykannot 3y ago
  "Intelligence explosion arguments don’t require Platonism. They just require intelligence to exist in the normal fuzzy way that all concepts exist."
  - earthboundkid 3y ago
    Okay, but we have a name for people who think concepts normally exist in some fuzzy way…
    - karaterobot 3y ago
      Okay, but then (per the article) you're just as much of a Platonist if you say that Mike Tyson is stronger than your grandmother.
earthboundkid 3y ago
If you say a whale is stronger than submarine or vice versa, it will be misleading unless you unpack the sub-claims, whereas you can say a blue whale is stronger than a killer whale or that submarine X is stronger than the Titan without unpacking it. Saying ChatGPT is more intelligent than Claude makes some sense, but there's no real point in comparing the intelligence of a person vs. an LLM any more than you would ask whether a humming bird flies better or worse than a Cessna.
- donkile 3y ago
  Doesn’t a hummingbird fly better than a Cessna? Better collision avoidance, easier maneuverability and tiny size…
  - b59831 3y ago
    What about flight distance? Cargo capacity? Navigation ability?...
    - earthboundkid 3y ago
      Fuel requirements: sugar water tastes better.
  - mos_basik 3y ago
    earthboundkid is arguing that there's no point in making the claim that one flies better than the other because that claim breaks into many subclaims; some support one position and some support the other. You named some subclaims that support the position that a hummingbird flies better than a Cessna. But there are also many that support the other position: range, speed, load...
    - deleted 3y ago
      [deleted]
  - myk9001 3y ago
    I can't help but wonder how the two of them compare to an unladen swallow in terms of airspeed velocity.
koryk 3y ago
I love this dichotomy, formalists vs. nominalists. As the author argues, it's not very useful with respect to defining intelligence and LLMs. It is however a wonderful world building detail in Anathem by Neal Stephenson. Their world history revolves around two ideologies: the syntactics and the semantics. Nominalists and Formalists. I find his predictions about a society built around this duality entertaining.
thebooktocome 3y ago
The trick behind Alexander’s supposedly non-platonic correlationism is that the hierarchy of “sub-claims and sub-sub-claims” are difficult to enumerate for specific cases. Because of this it’s not immediately apparent that the bundle of correlated subclaims are filtered for relevance. All the Platonism is hiding under that rug of relevance. That’s to say, when we talk about Tyson being stronger than my grandmother, we ignore correlated subclaims such as “Tyson consumes more protein than my grandmother” or “Tyson wears a jockstrap more often than my grandmother”—these are correlated with being stronger (probably), but they are incidental to the concept of strength. When we ask ourselves what subclaims are relevant, we’re right back to pointing at essence.
toss1 3y ago
>>It’s related to Jelinek’s Law, named after language-processing AI pioneer Frederick Jelinek: “Every time I fire a linguist, the performance of the speech recognizer goes up.” The joke is that having linguists on your team means you’re still trying to hand-code in deep principles of linguistics, instead of just figuring out how to get as big a blob of intelligence as possible and throw language at it. The limit of Jelinek’s Law is OpenAI, who AFAIK didn’t use insights from linguistics at all, and so made an AI that uses language near-perfectly. It seems a small conjecture to note that the problem here is that "deep principles of linguistics" the linguists are attempting to hard-code, are in fact more like "intermediate principles of linguistics". The linguists haven't approached anywhere near the deeper layers of how the brain works on language (nevermind how the Wernicke's area coordinates w/the visual cortex to generate "I see a bird over there"). So, when they code in their principles, they only screw up the deeper relationships. Of course the LLM has zero actual understanding of any abstract meaning such as "I", "see", "bird", or "over there", but it's still better off without those hardcodings to generate the phrase in a valid or hallucinated context.
breuleux 3y ago
It seems to me that the bitter lesson is a counter-argument of sorts to intelligence explosion: the idea that stacking more layers is more effective than trying to be really clever about it may well persist through all levels of intelligence. To put it in another way, if the bottleneck in the progress of intelligence, or of technology in general, is not intelligence, then greater intelligence will be of limited utility in accelerating it.
- loandbehold 3y ago
  The bitter lesson just says that general purpose machine learning algorithms with large amount of compute work better than domain-specific algorithms. It doesn't say that general purpose algorithms cannot be optimized. General purpose machine learning system that's able to optimize itself will lead to intelligence explosion.
  - breuleux 3y ago
    I do think that there is also a "bitter lesson" in optimization: simple optimizations are usually more effective, robust and powerful than complicated ones. Also, optimization becomes progressively harder as the system trends to optimal performance, which is likely to cancel out the advantage of higher intelligence in finding them.
    - comex 3y ago
      Also, even to the extent model architecture can be optimized, in the short term, thanks to the bitter lesson, the bottleneck seems to be less “human intelligence coming up with new ideas for optimizations” and more “compute time to test ideas on sufficiently large models”. An AI that followed the same basic thought process as humans, even if its ideas were a little better than humans’ on average, probably wouldn’t help that much. On the other hand, an AI that in some sense ‘understands’ its own structure, which is to us almost entirely a black box, might be able to do much better. Consider OpenAI’s attempt to make GPT-4 explain the purpose of neurons in GPT-2; it basically didn’t work at all, but it would have opened a huge new frontier of research if it had. But that understanding ability is hard to train, given the lack of any meaningful training data. If it does show up, it could act as an accelerant. But it could just as easily not show up, at least not until AI is already comfortably superhuman in the “thinking like a human” department – in which case it won’t help us reach that point in the first place.
lukeasrodgers 3y ago
Sometimes I wish that when folks opined on such matters they would consider that there is a wealth of philosophical literature covering the same ground that generally has already anticipated your points, contradicted them, strengthened them, provided sharper terminology for discussing them, etc. In this case, I'm thinking of natural kind terms (https://plato.stanford.edu/entries/natural-kinds/ https://plato.stanford.edu/entries/natural-kinds/) and, less well known, Boyd's homeostatic property cluster theory. It is kind of amazing to me that this article has 310 comments and not a single person has raised the question natural kinds. OTOH I am heartened that people are effectively doing philosophy in the wild without getting too concerned with "whether other people have already said this before" which can be paralyzing.
- tgv 3y ago
  I have the impression that people don't take philosophy very serious, and I would venture that that is because philosophy has such a meagre empirical record, nor a solid formal base outside of mathematics and logic. This makes it practically impossible to judge the merits of a particular idea. A somewhat extreme example: does it matter what Heidegger has said about a certain topic for any discussion? Or Žižek, for that matter.
  - thebooktocome 3y ago
    Heidegger, no. Žižek said some faintly reasonable things during his pandemic period, and semi-recently published an okay introduction to Hegel in the context of contemporary life. Also the man is very hot when he takes his shirt off.
  - goatlover 3y ago
    Philosophy provides rigorous arguments for and against various ideas, and assumptions behind those ideas. These ideas usually aren't empirical, but they are ideas many people have or questions people ask. Ethics, aesthetics, the nature of knowledge, truth, what exists, how to live, etc. People who don't take philosophy very seriously still engage in those topics. They just ignore that the topics have a history of being discussed seriously and often put forth their own unexamined assumptions. Scientists aren't immune to this.
vintermann 3y ago
Intelligence, whether you use it as a fuzzy concept or a narrow one, assumes a goal. It's impossible to say if an actor acts intelligently, without assuming what it's trying to achieve. It's peak this guy, and peak "rationalists" in general, to defend the concept of "objective" intelligence, intelligence towards unspecified goals. This is what they actually care about: that there's an objective scale of intelligence, and that they're above most people on it. Everything else they believe is tangential. They choose other beliefs according to how they can strengthen the core belief, that they are the intelligent ones. (And you bet they downvote people challenging it). What goals their intelligence are directed at, they're evasive and inconsistent about, and that others might have different goals is certainly not a permissible option. In THAT area, they are platonists, no matter how fuzzy their forms are: playing the flute is the task of a professional flautist, and steering the ship of humanity is a task for professional captain smart. Where the ship should be heading is not a subject of debate.
- drdeca 3y ago
  If a process has in it a place where a representation of a goal is stored, and for a wide range of goals, if represented in this format, if stored in this location, the process would tend to effectively achieve the goal, then it seems like the “what the goal is” (within the aforementioned wide (but not necessarily universal) range of goals) and the “capability of achieving goals” (within that range of goals) would be things that could be factored out as separate properties. Do you think that that condition doesn’t occur? Or, do you think that the “the range of goals for which this works” being limited, makes the “how capable is this process at achieving goals” concept illegitimate, no matter how broad the limited range is? Also, the reason you are getting downvoted is presumably in part because you talked about being downvoted. Your accusations are false and uninteresting, so I’m only addressing the object-level topics I can extract from your comment.
  - vintermann 3y ago
    But definitively not universal. You can't pursue all goals equally well, unless you don't pursue goals at all. So when you start talking about "goal representations", you implicitly talk about a set of possible sub-goals - the set you would like to be able to represent, and the process by which you plan to pick them, implicitly defines the actual goal. You're not interesting either, "rationalist", and I think you believe a lot of false things too. But unfortunately people who should know better listen to you.
lo_zamoyski 3y ago
"Concepts are bundles of useful correlations." Is "concept" a bundle of useful correlations? Is "bundle" a bundle of useful correlations? Is "useful" a bundle of useful correlations? Is "correlation" a bundle of useful correlations? And correlations of what? Very strong emphasis on what. Quid est? From the get go, like all skepticism, this approach begins by sitting on a branch that more or less acknowledges reality and its comprehensibility, but quickly begins sawing off the very branch it is sitting on. Skepticism is always a form of selective denial, and as such, it is incoherent. It's like those people who argue passionately that there is no truth while in the lecture hall, but as soon as the clock strikes 4pm, they're worried about missing the bus and getting everybody in the room to the next lecture, or as Duhem would say, going home to kraut and pipe. "Triangularity" is not a "bundle of correlations". Triangularity is an irreducible, but analyzable whole, a form that is instantiated in the world within particulars and that we encounter in particulars and abstract from those particulars to arrive at the concept. That's what a concept is: a form abstracted from concrete particulars that exists in the intellect as a universal. Unfortunately, mechanistic metaphysics, taught and insinuated in our scientistic curricula from a young age, is a difficult intellectual habit to break for many. It is very difficult for people to uproot these obstinate presuppositions.
- prometheus76 3y ago
  They can't even see them. They don't realize they are looking through goggles.
nvm0n1 3y ago
This article segues from the failure of 80s style expert systems (in many domains) to a generic argument for x-risk and superintelligence explosions. But it doesn't address any of the obvious counter-arguments except one about the definition of intelligence, I guess because Alexander feels they're already addressed. Once you get smart enough, you can do things that make you even smarter. AI will be one of those things. We already know that bigger blobs of compute with more training data can do more things in correlated ways - frogs are outclassed by cows, chimps, and humans; village idiots are outclassed by Einstein; GPT-2 is outclassed by GPT-4. At some point we might get a blob which is better than humans at designing chips, and then we can make even bigger blobs of compute, even faster than before. But what about: 1. Limited to human performance by training data? OpenAI apparently aren't training GPT-5 because they think that research direction is tapped out. Their focus has been on augmenting this "superintelligence" with boring logic-based systems like calculators, Python interpreters, web browsers and 80s style expert systems like Wolfram Alpha. All this is suspiciously like what a human would need, not a superintelligence. It implies they don't think they can do another 2->3->4 style leap, probably due to lack of training data that would yield more advanced capabilities. 2. Bottlenecked by physical experimentation? Alexander casually asserts that if you trained an AI on circuit design it'd immediately do better, and that'd be used to build better AI chips, which in turn would yield a smarter AI ad infinitum. But Google already tried this and it just led to fraud claims, not better chips. And even if an LLM came up with an idea for a better chip, humans would still need to do the physical experimentation to figure out how to build them. 3. Bottlenecked by lack of imagination? LLMs can be "creative" in the artistic sense but there is a suspicious and very noticeable absence of them coming up with any genuinely interesting ideas. They can think faster than humans, and know immeasurably more, yet has anyone found even just one example of the sort of out-of-the-box thinking that would be required for AI to outsmart humans? Where are all these scientific breakthroughs the AI evangelists keep promising? GPT-4 is pretty damn smart but despite asking it many things, it never once came up with an idea I hadn't already had. The article's examples seem to be its own undoing. There is no such thing as strength-leading-to-more-strength in some sort of recursive loop, which is why he needs such a bizarre and artificial example. Why should we believe there is for intelligence, when the history of human intelligence is a 2000 year struggle for even quite minor improvements in cognitive ability and even that is highly debatable?
Imnimo 3y ago
>Einstein can do arithmetic better than the toddler Is Deep Blue more intelligent than Garry Kasparov?
- deely3 3y ago
  Sorry, I don't get a question. By definition Garry should score highter than Deep Blue, because Deep Blue never takes IQ test?
  - Imnimo 3y ago
    The argument made in the article is that “Is Albert Einstein smarter than a toddler?” means things like "Einstein can do arithmetic better than the toddler". But this only makes sense because the toddler and Einstein are similar systems doing the same fundamental things when they do arithmetic. We would accept that "Einstein plays chess better than a toddler" is an indication that Einstein is more intelligent, but would we accept the same from Deep Blue? If not, why not?
    - natosaichek 3y ago
      The whole point is that intelligence is a concept that points at a bundle of related characteristics. There is one indication that Deep blue is more intelligent than a toddler - it can beat the toddler at a game of chess. There are other indications that a toddler is more intelligent than deep blue - they could write better poetry, play tic-tac-toe better, play checkers better, play, well, literally _any_ other game better, do arithmetic better, etc. So on one metric DB is more intelligent. On every other metric the toddler is more intelligent.
      - Imnimo 3y ago
        Is it actually an indication that Deep Blue is more intelligent, or does it turn out that Deep Blue is succeeding at chess by means other than intelligence?
waitSo 3y ago
Everyone is a kind of essentialist in that they refuse to engage with people below their personal threshold for “person”. I find it quite puerile how so few can accept their one of billions-ness. “If you can’t encode and decode information how I do, you stink, and so does your technology or philosophy or whatever it is!!” Religion is not the metaphors and poetry. It’s a biological state of conviction in one’s own essential ness to reality.
lukev 3y ago
This is a good discussion of concepts. However, a lot of the folks worried about superintelligence miss an important division within the concept of "intelligence." Specifically with respect to the current generation of LLMs. Namely, there's a huge difference between the concept of intelligence and an intelligent agent. You could make the argument that current LLMs have some level of intelligence. But (without a lot of supporting infrastructure) they are not agents, they have no built in mechanism for having any kind of intent or even short-term memory. Because they fundamentally just generate a single token at once, they don't even exist "within time" the way we normally would think of an intelligent entity as doing... they're more like a mathematical function that you can invoke on demand. Perhaps to get an "intelligent" result, but conflating that kind of intelligence with an intelligent entity or actor seems like a mistake. To use the analogy in the OP, it would be like calling a single hydraulic piston "strong". Sure, but unless you attach it to something it's not the same kind of strong as a bodybuilder is.
- mquander 3y ago
  OK, but the hard part was the "intelligent" part, right? If you have the "intelligent" thing, and you want to get an "intelligent agent", all you have to do at most is wrap it in a shell script that runs the equivalent of an OODA loop, e.g. AutoGPT. So we should expect a "superintelligent agent" to appear pretty much the next day after a "superintelligence" is created.
  - lukev 3y ago
    Perhaps, I don't discount the possibility that an LLM could be used as a primitive as part of a larger more complex system that could rightfully be called intelligent. But a script running a LLM in an OODA loop is a lot easier to monitor and control than some inscrutable "ghost in the parameters" that a lot of AI alarmists seem worried about. You can actually see a transcript of its "thoughts", for one!
    - Footkerchief 3y ago
      We already have self-replicating scripts that can prove very hard to detect and destroy, even without the benefit of LLM-powered fuzzing and social engineering.
carapace 3y ago
Intelligence is compression. Intelligence is achieving goals. Intelligence is the process of discovering which goals are worth achieving. - - - - Q: What's GAI? A: When the computer asks, "What's in it for me?"
- prometheus76 3y ago
  I would word your statement a little differently. What you said sounds very confusing to me, as if you are conflating intelligence with a lot of other things. Intelligence is discernment, heuristics and pattern recognition. Agency is achieving goals. Wisdom is the process of discovering which goals are worth achieving.
- pixl97 3y ago
  If this is the definition then it would be incredibly stupid for humans to build an AGI system.
  - carapace 3y ago
    That second bit is a really old AI joke. It's pointing out a very important, uh, point. Yeah, it's creepy to make people just to enslave them (Asimov's Three Laws of Robotics are also the three laws of a perfect slave, so to speak.) And, yes, it's incredibly stupid to build slaves powerful enough to turn you out (Asimov has a few stories that touch on that, IIRC.)
benreesman 3y ago
This stuff drives me up the wall for a few reasons. The main one is that I’ve never heard anyone describe in any detail how the technology will work. I’ve worked in big shops running big online ML systems and oh boy, do they need a lot of diaper-changing to even stay running. If all the ops folks went on vacation it’s a “hours vs days” not “weeks vs months” question of how soon it would fall over. Some log rotate thing gets stuck, and away we go with cascading failures. So how, in some detail, do we get from big-ass mixture model of transformers to even self-operation? That’s got to be a pre-requisite for self-improvement right? inb4 “The risk is so great the details don’t matter because not impossible”, hmmm, no. Extreme tail risk isn’t interesting when addressing it comes at the cost of driving immediate, overwhelmingly likely risk of nightmare outcomes through the roof. Consolidating the alignment and steering of big models into the hands of a few CEOs, governments captured by them, or both is a clear and present danger of horrifying proportions. I take mitigate that over extreme tail risk sure as Tuesday and taxes. Then there’s the reality of AI development thus far, which is that it comes in stops and starts. Clearly the past is no guarantee of the future, but it seems a damned sight better as a prior than the log scale and a ruler methodology employed by Yud or whoever. Smart people talk like this is serious. There are even some elite practitioners who talk kinda like this (Hinton, Karpathy on Lex, some others). And I don’t want to be walking around with my head up my ass on it, so I’ll be grateful to anyone who wants to set me straight. But if there’s an explanation somewhere, by a credible practitioner, of how this could actually happen in some actual technical detail, I haven’t found it. I think that needs to be in the footnotes the next time the mainstream press starts publishing blog posts off LessWrong as consensus.
- skepticATX 3y ago
  I have noticed that the vast majority of existential risk folks I have seen have very little experience running and maintaining large systems in production. They tend to all be on the research side of things. I think that it can be hard to understand the amount of real world friction that exists until you experience it for yourself.
  - habitue 3y ago
    Conversely, being too close to the problem can cause practitioners to miss the forest for the trees. I think in this case it's a false premise though. There are plenty of real practitioners like Joshua Bengio and Geoffrey Hinton who have sounded the alarm on these topics, and no one can call them inexperienced with real systems.
slotrans 3y ago
In which Scott demonstrates that he has no idea how "strength" works, or frankly what it even is. His quality has been sliding the last couple years, which happens to coincide with his announced commitment to AI Doom Ideology. Maybe those are connected and maybe not, but there it is.
- timmytokyo 3y ago
  He's not sliding. He's been down there at the bottom all along. Scott Alexander is fond of numerous neoreactionary ideas, especially those around race and IQ (so-called "human biodiversity"). He's admitted it himself. In his own words: "I am monitoring Reactionaries to try to take advantage of their insight and learn from them. I am also strongly criticizing Reactionaries for several reasons. "First is a purely selfish reason - my blog gets about 5x more hits and new followers when I write about Reaction or gender than it does when I write about anything else, and writing about gender is horrible. Blog followers are useful to me because they expand my ability to spread important ideas and network with important people. "Second is goodwill to the Reactionary community. I want to improve their thinking so that they become stronger and keep what is correct while throwing out the garbage. A reactionary movement that kept the high intellectual standard (which you seem to admit they have), the correct criticisms of class and of social justice, and few other things while dropping the monarchy-talk and the cathedral-talk and the traditional gender-talk and the feudalism-talk - would be really useful people to have around. So I criticize the monarchy-talk etc, and this seems to be working - as far as I can tell a lot of Reactionaries have quietly started talking about monarchy and feudalism a lot less (still haven't gotten many results about the Cathedral or traditional gender). "Third is that I want to spread the good parts of Reactionary thought. Becoming a Reactionary would both be stupid and decrease my ability to spread things to non-Reactionary readers. Criticizing the stupid parts of Reaction while also mentioning my appreciation for the good parts of their thought seems like the optimal way to inform people of them. And in fact I think it's possible (though I can't prove) that my FAQ inspired some of the recent media interest in Reactionaries."
vcg3rd 3y ago
Einstein wasn't (not is) more intelligent than a toddler in the same way he was stronger than a toddler, or maybe at all. He certainly knew more. He was actually less intelligent than he, himself, was as a toddler if one is measuring at rate of knowledge acquisition rather than acquired. Once the analogical reasoning broke down I stopped reading, especially since I believe life is a prerequisite for intelligence and is bound to it, because intelligence does not equal computation and is more than Turing complete.
- s1artibartfast 3y ago
  >This announcement was usually accompanied by discussion of how belief in “intelligence” was politically suspect
andrewflnr 3y ago
> There have already been intelligence explosions... reading and writing... iodine supplementation... It's dubious to refer to these individual events as "explosions". They're just single significant changes. Any event in humans or AI you could respectably call an explosion would be a cascade (note: causally interconnected) of a large number of these individual intelligence-increasing changes. If anything, I'd argue that the buildup of communal knowledge in human civilizations over the last several thousand years, especially the last few centuries, has been an explosion of sorts. It's been a huge increase in capability for one species, and would look like a step change on a timescale that includes all Earth life. But if that's an explosion, writing was just one of the chemical reactions.
quadhome 3y ago
Because animal intelligence (including human intelligence) is a blob of neural network arranged in a mildly clever way that lets it learn whatever it wants efficiently. That’s a leap.
nologic01 3y ago
Maybe we should define intelligence as a blob of network arranged in a mildly clever way that allows it to identify intellectual fraud?
angiosperm 3y ago
Whenever this topic comes up, there is a lot of hand-wringing as if the nightmare scenario were still in some misty future. It's not. AIs already control the world. They have proven indifferent to the welfare of humans. Perhaps the most important developments in recent decades were their coming to own the majority of public media, and the high court of the one of biggest jurisdictions permitting them to bribe politicians secretly: in practice, to own pet legislators. The AIs in question are called corporations. Their working parts mostly still include (readily replaced) humans, but decreasingly so. They have learned to manufacture voting blocs to drive political movements in nominally democratic countries to support their goals. By analogy to the paperclip apocalypse, they have a built-in motivation to turn all matter and energy into profit. Where their chosen method results in some service or product useful to humans, humans may benefit, but such choices may change freely, or may fail to change when the method turns out to cause net harm. Attempts at legislation to limit their power fail as they have taken control of the legislative process. Is it too late to do anything? Probably. I for one ...
pomatius 3y ago
Tell me you've just read a philosophy Wikipedia article for 5 minutes without telling me you've just read a philosophy Wikipedia article for 5 minutes. This is not what platonism and nominalism means lol
mcguire 3y ago
Some notes: "Variation in IQ during adulthood is about 70% genetic." The proviso "during adulthood" is important: "The heritability of IQ increases with the child's age and reaches a plateau at 18–20 years old, continuing at that level well into adulthood." (Is it just me or is that a weird thing for it to do.) That being said, the Wilson Effect paper goes on to say, "It is important to specify the populations to which any results can be generalized and not misinterpret what they mean. The samples were drawn almost exclusively from Western industrial democracies. These settings have characteristic environments. Only a few of the participants were raised in real poverty or by illiterate parents, and all study participants had access to the contemporary educational programs typical of those societies. This is the domain to which we can generalize." In any case, for most of my life the variation in earring-wearing was almost 100% genetic, so there's that. "A lot of this seems to have to do with literal brain size (which is correlated with intelligence at about 0.2)..." 0.2 is not a very strong correlation. And correlations are funny. (https://www.tylervigen.com/spurious-correlations https://www.tylervigen.com/spurious-correlations) "There have already been intelligence explosions. Long ago, humans got smart enough to invent reading and writing, which let them write down training data for each other and become even smarter (in a practical sense; this might or might not have raised their literal IQ, depending on how you think about the Flynn Effect)." This is a direct contradiction to Alexander's apparent argument above, unless you consider reading and writing to have a significant genetic component. (Heritability? Oh, yeah, I'd expect reading and writing to have pretty high heritability, at least historically.) But is knowing things really what you mean by intelligence? "Later on, we invented iodine supplementation, which let us treat goiter and gain a few IQ points on a population-wide level." Fortunately, iodine supplementation is also highly heritable. "...were emphasizing how rather than magical intelligence, AI would need slow, boring work on the nitty gritty of computation..." How are LLMs doing on that whole hallucination thing? The ability to solve novel problems? P.s. A fun video: https://www.youtube.com/watch?v=UBc7qBS1Ujo https://www.youtube.com/watch?v=UBc7qBS1Ujo

meh. "intelligence" and "artificial intelligence" are marketing terms. According to Nils Nilsson, McCarthy and Minsky picked the term so the Dartmouth Summer Research Project on Artificial Intelligence would not be constrained by associations with "Cybernetics." McCarthy thought it (Cybernetics) too constrained because a: it traditionally focused on analog solutions and b: he thought Norbert Wiener was a bit pushy. I asked Minsky about this a couple decades ago and it was the only time he didn't tell me I was asking a stupid question. Apparently, after the conference, the name stuck because they found it easier to get research grants with the more "neutral" sounding name. I wonder how many people would trust "AI" if it was called "distributed indexing" or "stochastic generation."

bryan0 3y ago
I think the article explains very well why "intelligence" is a useful and important concept. Namely because it is concise way to discuss abilities which are highly correlated without naming each ability individually.
- morelisp 3y ago
  Somehow the rationalist community started at empty platitudes like "correlation is not causation" yet has ended up at substantive yet amazingly wrong conclusions like "correlation is identity", amazing turn, I gotta say I did not see it coming.
  - philovivero 3y ago
    But correlation is correlation. At least, for most people's definitions of the word.
  - thebooktocome 3y ago
    I’m frankly surprised Judea Pearl’s name didn’t come up in this article, they love waving him around at times like this.
golol 3y ago
Intelligence has a pretty clear meaning. Of course you will have a hard time nailing down a definition but that's also true for simple words like "chair" or "car". Artificial intelligence is also a pretty good name for a pretty clear concept: intelligence that has been artificially created. Any new ambiguity here comes from the ambiguity inherent in the word "artificial". Artifivial intelligence has a large range from if/else, linear regression and decision trees over ML methods to god-like superintelligence. That is not a problem, I think it's good that we have a term for this. I don't like when people try to reduce artificial intelligence to a smaller subset of these things. We should certainly come up with more precise terms in addition, but I think artificial intelligence is a very useful terminology and concept, not a marketing term.
- cacois 3y ago
  > Any new ambiguity here comes from the ambiguity inherent in the word "artificial". Not sure I agree here. Seems to me the ambiguity comes from the word "intelligence". As in, does that if/else clause indicate "intelligence" that's been programmed in. Of course we've created more terms like Artificial General Intelligence[1] to try to clear up the muddy definition of what constitutes intelligence. I think our fundamental challenge as society begins to try to understand what AI is comes down to what "intelligence" is. As you said, we all know what it is, but can't define it easily. If I write some really clever code, is the system running it intelligent? Is some of my intelligence imbued into that system, or does it have some of its own? At what point does a stochastic process approach intelligence - when it has enough data? Does that mean babies aren't born with intelligence, but acquire it as they learn more? So many questions, good questions, necessary questions. [1] https://en.wikipedia.org/wiki/Artificial_general_intelligence https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
  - nextaccountic 3y ago
    > does that if/else clause indicate "intelligence" that's been programmed in Yes, systems built with if/else rules are intelligent, in that they reproduce some heuristic rule that someone had to think about. I think the term "expert system" is kind of appropriate for them https://en.wikipedia.org/wiki/Expert_system https://en.wikipedia.org/wiki/Expert_system Of course they are not too intelligent, but, they are more intelligent than a rock, say. (unless this rock happen to be a chip running some neural network)
benlivengood 3y ago
All that matters is if they are effective terms or not. That's most of the point of the article; how to constructively inquire about whether giant LLMs might be effective at solving novel problems, perhaps better than humans can, in ways that matter to humans. Since we don't have a formal model of how systems like humans work we can't just plug numbers into equations or generate proofs that LLMs and other ML systems will or will not be able to control other systems better than humans, so we revert to less-formal concepts as best as we can. The formal AI safety folks (MIRI et al) already tried the formal approach; by their own admission it's hopeless and they poured a lot of time and bright minds into it and we won't be able to formally answer questions about intelligence via decision theory or alignment before we have actually-intelligent systems interacting with us. What will happen at that point? We don't know and can only use imprecise concepts to make educated guesses. If we don't make honest attempts at this we run the risk of being blindsided by unexpectedly capable AI/ML systems. This is why a large number of AI alignment researchers recommend pausing AI/ML systems at their current ability before they have a chance at being superhuman on most tasks. I wouldn't trust "distributed indexing" or "stochastic generation" if it could beat me at any particular skill of my choosing like similar systems can beat any of us at Chess or Go or Poker.
m463 3y ago
This reminds me of MRI which started out as Nuclear Magnetic Resonance Imaging, but dropped "Nuclear" because of the stigma it carried.
- merchan 3y ago
  There's a less cynical explanation for that. "NMR" sounds a lot like "enema" and you don't want to get the two mixed up.
  - yetanotherloser 3y ago
    I'd never realised that before. Thank you, I think. Deary, deary me.
unholiness 3y ago
By that reasoning "Chemistry" is just a marketing term. It was used mostly interchangeably with "Alchemy" until Georg Stahl and others began writing (without justification!) that "Chemistry" describes more a more rational science and "Alchemists" were more interested in wealth. These untrue claims still became a self-fulfilling prophecy, where theories, teachings, and practice of chemistry evolved out of the former ways of thinking to become what we know it as today. AI, as used today, is certainly a coherent concept, and this article has justified and described it more thoroughly than most concepts I use daily. Who cares about the shaky history if it means something real today?

morelisp 3y ago

[flagged]

mannykannot 3y ago
"Intelligence explosion arguments don’t require Platonism. They just require intelligence to exist in the normal fuzzy way that all concepts exist."
- earthboundkid 3y ago
  Okay, but we have a name for people who think concepts normally exist in some fuzzy way…
  - karaterobot 3y ago
    Okay, but then (per the article) you're just as much of a Platonist if you say that Mike Tyson is stronger than your grandmother.

earthboundkid 3y ago

If you say a whale is stronger than submarine or vice versa, it will be misleading unless you unpack the sub-claims, whereas you can say a blue whale is stronger than a killer whale or that submarine X is stronger than the Titan without unpacking it. Saying ChatGPT is more intelligent than Claude makes some sense, but there's no real point in comparing the intelligence of a person vs. an LLM any more than you would ask whether a humming bird flies better or worse than a Cessna.

donkile 3y ago
Doesn’t a hummingbird fly better than a Cessna? Better collision avoidance, easier maneuverability and tiny size…
- b59831 3y ago
  What about flight distance? Cargo capacity? Navigation ability?...
  - earthboundkid 3y ago
    Fuel requirements: sugar water tastes better.
- mos_basik 3y ago
  earthboundkid is arguing that there's no point in making the claim that one flies better than the other because that claim breaks into many subclaims; some support one position and some support the other. You named some subclaims that support the position that a hummingbird flies better than a Cessna. But there are also many that support the other position: range, speed, load...
  - deleted 3y ago
    [deleted]
- myk9001 3y ago
  I can't help but wonder how the two of them compare to an unladen swallow in terms of airspeed velocity.

koryk 3y ago

I love this dichotomy, formalists vs. nominalists. As the author argues, it's not very useful with respect to defining intelligence and LLMs. It is however a wonderful world building detail in Anathem by Neal Stephenson. Their world history revolves around two ideologies: the syntactics and the semantics. Nominalists and Formalists. I find his predictions about a society built around this duality entertaining.

thebooktocome 3y ago

The trick behind Alexander’s supposedly non-platonic correlationism is that the hierarchy of “sub-claims and sub-sub-claims” are difficult to enumerate for specific cases. Because of this it’s not immediately apparent that the bundle of correlated subclaims are filtered for relevance. All the Platonism is hiding under that rug of relevance. That’s to say, when we talk about Tyson being stronger than my grandmother, we ignore correlated subclaims such as “Tyson consumes more protein than my grandmother” or “Tyson wears a jockstrap more often than my grandmother”—these are correlated with being stronger (probably), but they are incidental to the concept of strength. When we ask ourselves what subclaims are relevant, we’re right back to pointing at essence.

toss1 3y ago

>>It’s related to Jelinek’s Law, named after language-processing AI pioneer Frederick Jelinek: “Every time I fire a linguist, the performance of the speech recognizer goes up.” The joke is that having linguists on your team means you’re still trying to hand-code in deep principles of linguistics, instead of just figuring out how to get as big a blob of intelligence as possible and throw language at it. The limit of Jelinek’s Law is OpenAI, who AFAIK didn’t use insights from linguistics at all, and so made an AI that uses language near-perfectly. It seems a small conjecture to note that the problem here is that "deep principles of linguistics" the linguists are attempting to hard-code, are in fact more like "intermediate principles of linguistics". The linguists haven't approached anywhere near the deeper layers of how the brain works on language (nevermind how the Wernicke's area coordinates w/the visual cortex to generate "I see a bird over there"). So, when they code in their principles, they only screw up the deeper relationships. Of course the LLM has zero actual understanding of any abstract meaning such as "I", "see", "bird", or "over there", but it's still better off without those hardcodings to generate the phrase in a valid or hallucinated context.

breuleux 3y ago

It seems to me that the bitter lesson is a counter-argument of sorts to intelligence explosion: the idea that stacking more layers is more effective than trying to be really clever about it may well persist through all levels of intelligence. To put it in another way, if the bottleneck in the progress of intelligence, or of technology in general, is not intelligence, then greater intelligence will be of limited utility in accelerating it.

loandbehold 3y ago
The bitter lesson just says that general purpose machine learning algorithms with large amount of compute work better than domain-specific algorithms. It doesn't say that general purpose algorithms cannot be optimized. General purpose machine learning system that's able to optimize itself will lead to intelligence explosion.
- breuleux 3y ago
  I do think that there is also a "bitter lesson" in optimization: simple optimizations are usually more effective, robust and powerful than complicated ones. Also, optimization becomes progressively harder as the system trends to optimal performance, which is likely to cancel out the advantage of higher intelligence in finding them.
  - comex 3y ago
    Also, even to the extent model architecture can be optimized, in the short term, thanks to the bitter lesson, the bottleneck seems to be less “human intelligence coming up with new ideas for optimizations” and more “compute time to test ideas on sufficiently large models”. An AI that followed the same basic thought process as humans, even if its ideas were a little better than humans’ on average, probably wouldn’t help that much. On the other hand, an AI that in some sense ‘understands’ its own structure, which is to us almost entirely a black box, might be able to do much better. Consider OpenAI’s attempt to make GPT-4 explain the purpose of neurons in GPT-2; it basically didn’t work at all, but it would have opened a huge new frontier of research if it had. But that understanding ability is hard to train, given the lack of any meaningful training data. If it does show up, it could act as an accelerant. But it could just as easily not show up, at least not until AI is already comfortably superhuman in the “thinking like a human” department – in which case it won’t help us reach that point in the first place.

lukeasrodgers 3y ago

Sometimes I wish that when folks opined on such matters they would consider that there is a wealth of philosophical literature covering the same ground that generally has already anticipated your points, contradicted them, strengthened them, provided sharper terminology for discussing them, etc. In this case, I'm thinking of natural kind terms (https://plato.stanford.edu/entries/natural-kinds/ https://plato.stanford.edu/entries/natural-kinds/) and, less well known, Boyd's homeostatic property cluster theory. It is kind of amazing to me that this article has 310 comments and not a single person has raised the question natural kinds. OTOH I am heartened that people are effectively doing philosophy in the wild without getting too concerned with "whether other people have already said this before" which can be paralyzing.

tgv 3y ago
I have the impression that people don't take philosophy very serious, and I would venture that that is because philosophy has such a meagre empirical record, nor a solid formal base outside of mathematics and logic. This makes it practically impossible to judge the merits of a particular idea. A somewhat extreme example: does it matter what Heidegger has said about a certain topic for any discussion? Or Žižek, for that matter.
- thebooktocome 3y ago
  Heidegger, no. Žižek said some faintly reasonable things during his pandemic period, and semi-recently published an okay introduction to Hegel in the context of contemporary life. Also the man is very hot when he takes his shirt off.
- goatlover 3y ago
  Philosophy provides rigorous arguments for and against various ideas, and assumptions behind those ideas. These ideas usually aren't empirical, but they are ideas many people have or questions people ask. Ethics, aesthetics, the nature of knowledge, truth, what exists, how to live, etc. People who don't take philosophy very seriously still engage in those topics. They just ignore that the topics have a history of being discussed seriously and often put forth their own unexamined assumptions. Scientists aren't immune to this.

vintermann 3y ago

Intelligence, whether you use it as a fuzzy concept or a narrow one, assumes a goal. It's impossible to say if an actor acts intelligently, without assuming what it's trying to achieve. It's peak this guy, and peak "rationalists" in general, to defend the concept of "objective" intelligence, intelligence towards unspecified goals. This is what they actually care about: that there's an objective scale of intelligence, and that they're above most people on it. Everything else they believe is tangential. They choose other beliefs according to how they can strengthen the core belief, that they are the intelligent ones. (And you bet they downvote people challenging it). What goals their intelligence are directed at, they're evasive and inconsistent about, and that others might have different goals is certainly not a permissible option. In THAT area, they are platonists, no matter how fuzzy their forms are: playing the flute is the task of a professional flautist, and steering the ship of humanity is a task for professional captain smart. Where the ship should be heading is not a subject of debate.

drdeca 3y ago
If a process has in it a place where a representation of a goal is stored, and for a wide range of goals, if represented in this format, if stored in this location, the process would tend to effectively achieve the goal, then it seems like the “what the goal is” (within the aforementioned wide (but not necessarily universal) range of goals) and the “capability of achieving goals” (within that range of goals) would be things that could be factored out as separate properties. Do you think that that condition doesn’t occur? Or, do you think that the “the range of goals for which this works” being limited, makes the “how capable is this process at achieving goals” concept illegitimate, no matter how broad the limited range is? Also, the reason you are getting downvoted is presumably in part because you talked about being downvoted. Your accusations are false and uninteresting, so I’m only addressing the object-level topics I can extract from your comment.
- vintermann 3y ago
  But definitively not universal. You can't pursue all goals equally well, unless you don't pursue goals at all. So when you start talking about "goal representations", you implicitly talk about a set of possible sub-goals - the set you would like to be able to represent, and the process by which you plan to pick them, implicitly defines the actual goal. You're not interesting either, "rationalist", and I think you believe a lot of false things too. But unfortunately people who should know better listen to you.

lo_zamoyski 3y ago

"Concepts are bundles of useful correlations." Is "concept" a bundle of useful correlations? Is "bundle" a bundle of useful correlations? Is "useful" a bundle of useful correlations? Is "correlation" a bundle of useful correlations? And correlations of what? Very strong emphasis on what. Quid est? From the get go, like all skepticism, this approach begins by sitting on a branch that more or less acknowledges reality and its comprehensibility, but quickly begins sawing off the very branch it is sitting on. Skepticism is always a form of selective denial, and as such, it is incoherent. It's like those people who argue passionately that there is no truth while in the lecture hall, but as soon as the clock strikes 4pm, they're worried about missing the bus and getting everybody in the room to the next lecture, or as Duhem would say, going home to kraut and pipe. "Triangularity" is not a "bundle of correlations". Triangularity is an irreducible, but analyzable whole, a form that is instantiated in the world within particulars and that we encounter in particulars and abstract from those particulars to arrive at the concept. That's what a concept is: a form abstracted from concrete particulars that exists in the intellect as a universal. Unfortunately, mechanistic metaphysics, taught and insinuated in our scientistic curricula from a young age, is a difficult intellectual habit to break for many. It is very difficult for people to uproot these obstinate presuppositions.

prometheus76 3y ago
They can't even see them. They don't realize they are looking through goggles.

nvm0n1 3y ago

This article segues from the failure of 80s style expert systems (in many domains) to a generic argument for x-risk and superintelligence explosions. But it doesn't address any of the obvious counter-arguments except one about the definition of intelligence, I guess because Alexander feels they're already addressed. Once you get smart enough, you can do things that make you even smarter. AI will be one of those things. We already know that bigger blobs of compute with more training data can do more things in correlated ways - frogs are outclassed by cows, chimps, and humans; village idiots are outclassed by Einstein; GPT-2 is outclassed by GPT-4. At some point we might get a blob which is better than humans at designing chips, and then we can make even bigger blobs of compute, even faster than before. But what about: 1. Limited to human performance by training data? OpenAI apparently aren't training GPT-5 because they think that research direction is tapped out. Their focus has been on augmenting this "superintelligence" with boring logic-based systems like calculators, Python interpreters, web browsers and 80s style expert systems like Wolfram Alpha. All this is suspiciously like what a human would need, not a superintelligence. It implies they don't think they can do another 2->3->4 style leap, probably due to lack of training data that would yield more advanced capabilities. 2. Bottlenecked by physical experimentation? Alexander casually asserts that if you trained an AI on circuit design it'd immediately do better, and that'd be used to build better AI chips, which in turn would yield a smarter AI ad infinitum. But Google already tried this and it just led to fraud claims, not better chips. And even if an LLM came up with an idea for a better chip, humans would still need to do the physical experimentation to figure out how to build them. 3. Bottlenecked by lack of imagination? LLMs can be "creative" in the artistic sense but there is a suspicious and very noticeable absence of them coming up with any genuinely interesting ideas. They can think faster than humans, and know immeasurably more, yet has anyone found even just one example of the sort of out-of-the-box thinking that would be required for AI to outsmart humans? Where are all these scientific breakthroughs the AI evangelists keep promising? GPT-4 is pretty damn smart but despite asking it many things, it never once came up with an idea I hadn't already had. The article's examples seem to be its own undoing. There is no such thing as strength-leading-to-more-strength in some sort of recursive loop, which is why he needs such a bizarre and artificial example. Why should we believe there is for intelligence, when the history of human intelligence is a 2000 year struggle for even quite minor improvements in cognitive ability and even that is highly debatable?

Imnimo 3y ago

>Einstein can do arithmetic better than the toddler Is Deep Blue more intelligent than Garry Kasparov?

deely3 3y ago
Sorry, I don't get a question. By definition Garry should score highter than Deep Blue, because Deep Blue never takes IQ test?
- Imnimo 3y ago
  The argument made in the article is that “Is Albert Einstein smarter than a toddler?” means things like "Einstein can do arithmetic better than the toddler". But this only makes sense because the toddler and Einstein are similar systems doing the same fundamental things when they do arithmetic. We would accept that "Einstein plays chess better than a toddler" is an indication that Einstein is more intelligent, but would we accept the same from Deep Blue? If not, why not?
  - natosaichek 3y ago
    The whole point is that intelligence is a concept that points at a bundle of related characteristics. There is one indication that Deep blue is more intelligent than a toddler - it can beat the toddler at a game of chess. There are other indications that a toddler is more intelligent than deep blue - they could write better poetry, play tic-tac-toe better, play checkers better, play, well, literally _any_ other game better, do arithmetic better, etc. So on one metric DB is more intelligent. On every other metric the toddler is more intelligent.
    - Imnimo 3y ago
      Is it actually an indication that Deep Blue is more intelligent, or does it turn out that Deep Blue is succeeding at chess by means other than intelligence?

waitSo 3y ago

Everyone is a kind of essentialist in that they refuse to engage with people below their personal threshold for “person”. I find it quite puerile how so few can accept their one of billions-ness. “If you can’t encode and decode information how I do, you stink, and so does your technology or philosophy or whatever it is!!” Religion is not the metaphors and poetry. It’s a biological state of conviction in one’s own essential ness to reality.

lukev 3y ago

This is a good discussion of concepts. However, a lot of the folks worried about superintelligence miss an important division within the concept of "intelligence." Specifically with respect to the current generation of LLMs. Namely, there's a huge difference between the concept of intelligence and an intelligent agent. You could make the argument that current LLMs have some level of intelligence. But (without a lot of supporting infrastructure) they are not agents, they have no built in mechanism for having any kind of intent or even short-term memory. Because they fundamentally just generate a single token at once, they don't even exist "within time" the way we normally would think of an intelligent entity as doing... they're more like a mathematical function that you can invoke on demand. Perhaps to get an "intelligent" result, but conflating that kind of intelligence with an intelligent entity or actor seems like a mistake. To use the analogy in the OP, it would be like calling a single hydraulic piston "strong". Sure, but unless you attach it to something it's not the same kind of strong as a bodybuilder is.

mquander 3y ago
OK, but the hard part was the "intelligent" part, right? If you have the "intelligent" thing, and you want to get an "intelligent agent", all you have to do at most is wrap it in a shell script that runs the equivalent of an OODA loop, e.g. AutoGPT. So we should expect a "superintelligent agent" to appear pretty much the next day after a "superintelligence" is created.
- lukev 3y ago
  Perhaps, I don't discount the possibility that an LLM could be used as a primitive as part of a larger more complex system that could rightfully be called intelligent. But a script running a LLM in an OODA loop is a lot easier to monitor and control than some inscrutable "ghost in the parameters" that a lot of AI alarmists seem worried about. You can actually see a transcript of its "thoughts", for one!
  - Footkerchief 3y ago
    We already have self-replicating scripts that can prove very hard to detect and destroy, even without the benefit of LLM-powered fuzzing and social engineering.

carapace 3y ago

Intelligence is compression. Intelligence is achieving goals. Intelligence is the process of discovering which goals are worth achieving. - - - - Q: What's GAI? A: When the computer asks, "What's in it for me?"

prometheus76 3y ago
I would word your statement a little differently. What you said sounds very confusing to me, as if you are conflating intelligence with a lot of other things. Intelligence is discernment, heuristics and pattern recognition. Agency is achieving goals. Wisdom is the process of discovering which goals are worth achieving.
pixl97 3y ago
If this is the definition then it would be incredibly stupid for humans to build an AGI system.
- carapace 3y ago
  That second bit is a really old AI joke. It's pointing out a very important, uh, point. Yeah, it's creepy to make people just to enslave them (Asimov's Three Laws of Robotics are also the three laws of a perfect slave, so to speak.) And, yes, it's incredibly stupid to build slaves powerful enough to turn you out (Asimov has a few stories that touch on that, IIRC.)

benreesman 3y ago

This stuff drives me up the wall for a few reasons. The main one is that I’ve never heard anyone describe in any detail how the technology will work. I’ve worked in big shops running big online ML systems and oh boy, do they need a lot of diaper-changing to even stay running. If all the ops folks went on vacation it’s a “hours vs days” not “weeks vs months” question of how soon it would fall over. Some log rotate thing gets stuck, and away we go with cascading failures. So how, in some detail, do we get from big-ass mixture model of transformers to even self-operation? That’s got to be a pre-requisite for self-improvement right? inb4 “The risk is so great the details don’t matter because not impossible”, hmmm, no. Extreme tail risk isn’t interesting when addressing it comes at the cost of driving immediate, overwhelmingly likely risk of nightmare outcomes through the roof. Consolidating the alignment and steering of big models into the hands of a few CEOs, governments captured by them, or both is a clear and present danger of horrifying proportions. I take mitigate that over extreme tail risk sure as Tuesday and taxes. Then there’s the reality of AI development thus far, which is that it comes in stops and starts. Clearly the past is no guarantee of the future, but it seems a damned sight better as a prior than the log scale and a ruler methodology employed by Yud or whoever. Smart people talk like this is serious. There are even some elite practitioners who talk kinda like this (Hinton, Karpathy on Lex, some others). And I don’t want to be walking around with my head up my ass on it, so I’ll be grateful to anyone who wants to set me straight. But if there’s an explanation somewhere, by a credible practitioner, of how this could actually happen in some actual technical detail, I haven’t found it. I think that needs to be in the footnotes the next time the mainstream press starts publishing blog posts off LessWrong as consensus.

skepticATX 3y ago
I have noticed that the vast majority of existential risk folks I have seen have very little experience running and maintaining large systems in production. They tend to all be on the research side of things. I think that it can be hard to understand the amount of real world friction that exists until you experience it for yourself.
- habitue 3y ago
  Conversely, being too close to the problem can cause practitioners to miss the forest for the trees. I think in this case it's a false premise though. There are plenty of real practitioners like Joshua Bengio and Geoffrey Hinton who have sounded the alarm on these topics, and no one can call them inexperienced with real systems.

slotrans 3y ago

In which Scott demonstrates that he has no idea how "strength" works, or frankly what it even is. His quality has been sliding the last couple years, which happens to coincide with his announced commitment to AI Doom Ideology. Maybe those are connected and maybe not, but there it is.

timmytokyo 3y ago
He's not sliding. He's been down there at the bottom all along. Scott Alexander is fond of numerous neoreactionary ideas, especially those around race and IQ (so-called "human biodiversity"). He's admitted it himself. In his own words: "I am monitoring Reactionaries to try to take advantage of their insight and learn from them. I am also strongly criticizing Reactionaries for several reasons. "First is a purely selfish reason - my blog gets about 5x more hits and new followers when I write about Reaction or gender than it does when I write about anything else, and writing about gender is horrible. Blog followers are useful to me because they expand my ability to spread important ideas and network with important people. "Second is goodwill to the Reactionary community. I want to improve their thinking so that they become stronger and keep what is correct while throwing out the garbage. A reactionary movement that kept the high intellectual standard (which you seem to admit they have), the correct criticisms of class and of social justice, and few other things while dropping the monarchy-talk and the cathedral-talk and the traditional gender-talk and the feudalism-talk - would be really useful people to have around. So I criticize the monarchy-talk etc, and this seems to be working - as far as I can tell a lot of Reactionaries have quietly started talking about monarchy and feudalism a lot less (still haven't gotten many results about the Cathedral or traditional gender). "Third is that I want to spread the good parts of Reactionary thought. Becoming a Reactionary would both be stupid and decrease my ability to spread things to non-Reactionary readers. Criticizing the stupid parts of Reaction while also mentioning my appreciation for the good parts of their thought seems like the optimal way to inform people of them. And in fact I think it's possible (though I can't prove) that my FAQ inspired some of the recent media interest in Reactionaries."

vcg3rd 3y ago

Einstein wasn't (not is) more intelligent than a toddler in the same way he was stronger than a toddler, or maybe at all. He certainly knew more. He was actually less intelligent than he, himself, was as a toddler if one is measuring at rate of knowledge acquisition rather than acquired. Once the analogical reasoning broke down I stopped reading, especially since I believe life is a prerequisite for intelligence and is bound to it, because intelligence does not equal computation and is more than Turing complete.

s1artibartfast 3y ago
>This announcement was usually accompanied by discussion of how belief in “intelligence” was politically suspect

andrewflnr 3y ago

> There have already been intelligence explosions... reading and writing... iodine supplementation... It's dubious to refer to these individual events as "explosions". They're just single significant changes. Any event in humans or AI you could respectably call an explosion would be a cascade (note: causally interconnected) of a large number of these individual intelligence-increasing changes. If anything, I'd argue that the buildup of communal knowledge in human civilizations over the last several thousand years, especially the last few centuries, has been an explosion of sorts. It's been a huge increase in capability for one species, and would look like a step change on a timescale that includes all Earth life. But if that's an explosion, writing was just one of the chemical reactions.

quadhome 3y ago

Because animal intelligence (including human intelligence) is a blob of neural network arranged in a mildly clever way that lets it learn whatever it wants efficiently. That’s a leap.

nologic01 3y ago

Maybe we should define intelligence as a blob of network arranged in a mildly clever way that allows it to identify intellectual fraud?

angiosperm 3y ago

Whenever this topic comes up, there is a lot of hand-wringing as if the nightmare scenario were still in some misty future. It's not. AIs already control the world. They have proven indifferent to the welfare of humans. Perhaps the most important developments in recent decades were their coming to own the majority of public media, and the high court of the one of biggest jurisdictions permitting them to bribe politicians secretly: in practice, to own pet legislators. The AIs in question are called corporations. Their working parts mostly still include (readily replaced) humans, but decreasingly so. They have learned to manufacture voting blocs to drive political movements in nominally democratic countries to support their goals. By analogy to the paperclip apocalypse, they have a built-in motivation to turn all matter and energy into profit. Where their chosen method results in some service or product useful to humans, humans may benefit, but such choices may change freely, or may fail to change when the method turns out to cause net harm. Attempts at legislation to limit their power fail as they have taken control of the legislative process. Is it too late to do anything? Probably. I for one ...

pomatius 3y ago

Tell me you've just read a philosophy Wikipedia article for 5 minutes without telling me you've just read a philosophy Wikipedia article for 5 minutes. This is not what platonism and nominalism means lol

mcguire 3y ago

Some notes: "Variation in IQ during adulthood is about 70% genetic." The proviso "during adulthood" is important: "The heritability of IQ increases with the child's age and reaches a plateau at 18–20 years old, continuing at that level well into adulthood." (Is it just me or is that a weird thing for it to do.) That being said, the Wilson Effect paper goes on to say, "It is important to specify the populations to which any results can be generalized and not misinterpret what they mean. The samples were drawn almost exclusively from Western industrial democracies. These settings have characteristic environments. Only a few of the participants were raised in real poverty or by illiterate parents, and all study participants had access to the contemporary educational programs typical of those societies. This is the domain to which we can generalize." In any case, for most of my life the variation in earring-wearing was almost 100% genetic, so there's that. "A lot of this seems to have to do with literal brain size (which is correlated with intelligence at about 0.2)..." 0.2 is not a very strong correlation. And correlations are funny. (https://www.tylervigen.com/spurious-correlations https://www.tylervigen.com/spurious-correlations) "There have already been intelligence explosions. Long ago, humans got smart enough to invent reading and writing, which let them write down training data for each other and become even smarter (in a practical sense; this might or might not have raised their literal IQ, depending on how you think about the Flynn Effect)." This is a direct contradiction to Alexander's apparent argument above, unless you consider reading and writing to have a significant genetic component. (Heritability? Oh, yeah, I'd expect reading and writing to have pretty high heritability, at least historically.) But is knowing things really what you mean by intelligence? "Later on, we invented iodine supplementation, which let us treat goiter and gain a few IQ points on a population-wide level." Fortunately, iodine supplementation is also highly heritable. "...were emphasizing how rather than magical intelligence, AI would need slow, boring work on the nitty gritty of computation..." How are LLMs doing on that whole hallucination thing? The ability to solve novel problems? P.s. A fun video: https://www.youtube.com/watch?v=UBc7qBS1Ujo https://www.youtube.com/watch?v=UBc7qBS1Ujo