8 ms·

Not to be too pessimistic here, but why are we talking about things like this? I get that it’s a fun thing to think about, what we will do when a great artific

Not to be too pessimistic here, but why are we talking about things like this? I get that it’s a fun thing to think about, what we will do when a great artificial superintelligence is achieved and how we deal with it, feels like we’re living in a science fiction book. But, all we’ve achieved at this point is making a glorified token predicting machine trained on existing data (made by humans), not really being able to be creative outside of deriving things humans have already made before. Granted, they‘re really good at doing that, but not much else. To me, this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders) by Altman and company, that I’m just baffled people are still going with it.

foobiekr 2y ago
It's no mystery, AI has attracted tons of grifters trying to cash out before the bubble pops, and investors aren't really good at filtering.
- sourcepluck 2y ago
  Well said. There is a mystery though still - how many people fall for it and then stay fell, and how long that goes on for. People who've followed directly a similar pattern play itself out often many times, and still, they go along. It's so puzzlingly common amongst very intelligent people in the "tech" space that I've started to wonder if there isn't a link to this ambient belief a lot of people have that tech can "change everything" for the better, in some sense. As in, we've been duped again and again, but then the new exciting thing comes along... and in spite of ourselves, we say: "This time it's really the one!" Is what we're witnessing simply the unfulfilled promises of techno-optimism crashing against the shores of social reality repeatedly?
- prasoonds 2y ago
  Are you claiming Ilya Sutskever is a grifter?
  - chrisldgk 2y ago
    I personally wouldn’t go that far, but I would say he’s at least riding the hype wave to get funding for his company, which, let’s be honest, nobody would care about if we weren’t this deep into the AI hypecycle.
- lupire 2y ago
  Why are you assigning moral agency where there may be none? These so called "grifters" are just token predictors writing business plans (prompts) with the highest computed probability of triggering $ + [large number] token pair from venture capital token predictors.
spacecadet 2y ago
Egos man. Egos.
Bengalilol 2y ago
I got flagged for less. Anyways, nice sum up of the current AI game!
adamisom 2y ago
Agree up til last paragraph: how's Altman involved? Otoh Sutskever is a true believer so that explains his Why
- chrisldgk 2y ago
  To be clear I was just bunching high profile AI founders and CEOs that can’t seem to stop talking about how dangerous the thing they’re trying to build is together. I don’t know (nor care) about Ilyas and Altmans current relationship.
ohcmon 2y ago
> glorified token predicting machine trained on existing data (made by humans) sorry to disappoint, but human brain fits the same definition
- roywiggins 2y ago
  See, this sort of claim I am instantly skeptical of. Nobody has ever caught a human brain producing or storing tokens, and certainly the subjective experience of, say, throwing a ball, doesn't involve symbols of any kind.
  - mewpmewp2 2y ago
    Any output from you could be represented as a token. It is a very generic idea. Ultimately whatever you output is because of chemical reactions that follow from the input.
    - roywiggins 2y ago
      It could be represented that way. That's a long way from saying that's how brains work. Does a thermometer predict tokens? It also produces outputs that can be represented as tokens, but it's just a bit of mercury in a tube. You can dissect a thermometer as much as you like and you won't find any token prediction machinery. There's lots of things like that. Zooming out, does that make the entire atmosphere a token prediction engine, since it's producing eg wind and temperatures that could be represented as tokens? If you need one token per particle then you're admitting that this is task is impossible. Nobody will ever build a computer that can simulate a brain-sized volume of particles to sufficient fidelity. There is a long, long distance from "brains are made of chemicals" to "brains are basically token prediction engines."
      - therobots927 2y ago
        The argument that brains are just token prediction machines is basically the same as saying “the brain is just a computer”. It’s like, well, yes in the same way that a B-21 Raider is an airplane as well as a Cessna. That doesn’t mean that they are anywhere close to each other in terms of performance. They incorporate some similar basic elements but when you zoom out they’re clearly very different things.
bbor 2y ago
Well, an entire industry of researchers, which used to be divided, is now uniting around calls to slow development and emphasize safety (like, “dissolve companies” emphasis not “write employee handbooks” emphasis). They’re saying, more-or-less in unison, that GPT3 was an unexpected breakthrough in the Frame Problem, based on Judea Pearl’s prescient predictions. If we agree on that, there are two options: 1. They’ve all been tricked/bribed by Sam Altman and company (which btw this is a company started against those specific guys, just for clarity). Including me, of course. 2. You’re not as much of an expert in cognitive science as you think you are, and maybe the scientists know something you don’t. With love. As much love as possible, in a singular era
- majormajor 2y ago
  I would read the existence of this company as evidence that the entire industry is not as united as all that, since Sutskever was recently at another major player in the industry and thought it worth leaving. Whether that's a disagreement between what certain players say and what they do and believe, or just a question of extremes... TBD.
  - bbor 2y ago
    He didn't leave because of technical reasons, he left because of ethical ones. I know this website is used to seeing this whole thing as "another iPhone moment" but I promise you it's bigger than that. Either that or I am way more insane than I know! E: Jeez I said "subreddit" maybe I need to get back to work
- dnissley 2y ago
  Are they actually united? Or is this the ai safety subfaction circling the wagons due to waning relevance in the face of not-actually-all-that-threatening ai?
  - bbor 2y ago
    I personally find that summary of things to be way off the mark (for example, hopefully "the face" you reference isn't based on anything that appears in a browser window or in an ensemble of less than 100 agents!) but I'll try to speak to the "united" question instead. 1. The "Future of Life" institute is composed of lots of very serious people who recently helped get the EU "AI Act" passed this March, and they discuss the "myriad risks and harms AI presents" and "possibly catastrophic risks". https://newsletter.futureoflife.org/p/fli-newsletter-march-2024 https://newsletter.futureoflife.org/p/fli-newsletter-march-2... 2. Many researchers are leaving large tech companies, voicing concerns about safety and the downplaying of risks in the name of moving fast and beating vaguely-posited competitors. Both big ones like Hinton and many, many smaller ones. I'm a little lazy to scrape the data together, but it's such a wide phenomenon that a quick Google/Kagi should be enough for a vague idea. This is why Anthropic was started, why Altman was fired, why Microsoft gutted their AI safety org, and why Google fired the head of their AI ethics team. We forgot about that one cause it's from before GPT3, but it doesn't get much clearer than this: > She co-authored a research paper which she says she was asked to retract. The paper had pinpointed flaws in AI language technology, including a system built by Google... Dr Gebru had emailed her management laying out some key conditions for removing her name from the paper, and if they were not met, she would "work on a last date" for her employment. According to Dr Gebru, Google replied: "We respect your decision to leave Google... and we are accepting your resignation." 3. One funny way to see this happening is to go back to seminal papers from the last decade and see where everyone's working now. Spoiler alert: not a lot of the same names left at OpenAI, or Anthropic for that matter! This is the most egregious I've found -- the RLHF paper: see https://arxiv.org/pdf/2203.02155 https://arxiv.org/pdf/2203.02155 3. Polling of AI researchers shows a clear and overhelming trend towards AGI timelines being moved up significantly. It's still a question deeply wrapped up in accidental factors like religious belief, philosophical perspective, and general valence as a person, so I think the sudden shift here should tell you a lot. https://research.aimultiple.com/artificial-general-intelligence-singularity-timing/ https://research.aimultiple.com/artificial-general-intellige... The article I just linked actually has a section where they collect caveats, and the first is this Herbert Simon quote from 1965 that clearly didn't age well: "Machines will be capable, within twenty years, of doing any work a man can do.” This is a perfect example of my overall point! He was right. The symbolists were right, are right, will always be right -- they just failed to consider that the connectionists were just as right. The exact thing that stopped his prediction was the frame problem, which is what we've now solved. Hopefully that makes it a bit clearer why I'm anxious all the time :). The End Is Near, folks... or at least the people telling you that it's definitely not here have capitalist motivations, too. If you count the amount of money lost and received by each "side" in this "debate", I think it's clear the researcher side is down many millions in lost salaries and money spent on thinktank papers and Silicon Valley polycule dorms (it's part of it, don't ask), and the executive side is up... well, everything, so far. Did you know the biggest privately-funded infrastructure project in the history of humanity was announced this year? https://www.datacenterdynamics.com/en/opinions/how-microsoft-wins/ https://www.datacenterdynamics.com/en/opinions/how-microsoft...
foolishbard 2y ago
There's a chance that these systems can actually out perform their training data and be better than the sum of their parts. New work out Harvard talks about this idea of "transcendence" https://arxiv.org/abs/2406.11741 https://arxiv.org/abs/2406.11741 While this is a new area, it would be naive to write this off as just science fiction.
- majormajor 2y ago
  It would be nice if authors wouldn't use a loaded-as-fuck word like "transcendence" for "the trained model can sometimes achieve better performance than all [chess] players in the dataset" because while certainly that's demonstrating an impressive internalization of the game, it's also something that many humans can also do. The machine, of course, can be scaled in breadth and performance, but... "transcendence"? Are they trying to be mis-interpreted?
  - Kerb_ 2y ago
    It transcends the training data, I get the usage intended but it certainly is ripe for misinterpretation
    - lupire 2y ago
      That's trivial though, conceptually. Every regression line transcends the training data. We've had that since Wisdom of Crowds.
    - sb77 2y ago
      The word for that is "generalizes" or "generalization" and it has existed for a very long time.
      - Kerb_ 2y ago
        I've been very confidently informed that these AIs are not AGIs, which makes me wonder what the "General" in AGI is supposed to mean and whether generalization is actually the benchmark for advanced intelligence. If they're not AGI, then wouldn't another word for that level of generalization be more accurate than "generalization"? It doesn't have to be "transcendence" but it seems weird to have a defined step we claim we aren't at but also use the same word to describe a process we know it does. I don't get the nuance of the lingo entirely, I guess. I'm just here for the armchair philosophy
alecco 2y ago
Because it's likely soon LLMs will be able to teach themselves and surpass humans. No consciousness, no will. But somebody will have their power. Dark government agencies and questionable billionaires. Who knows what will it enable them to do. https://en.wikipedia.org/wiki/AlphaGo_Zero https://en.wikipedia.org/wiki/AlphaGo_Zero
- roywiggins 2y ago
  Likely according to who?
  - CyberDildonics 2y ago
    Whoever needs money from investors who don't understand LLMs.
    - figers 2y ago
      ha-ha!!!!
- mjr00 2y ago
  Mind defining "likely" and "soon" here? Like 10% chance in 100 years, or 90% chance in 1 year? Not sure how a Go engine really applies. Do you consider cars superintelligent because they can move faster than any human?
  - TechDebtDevin 2y ago
    I'm with you here, but it should be noted that while the combustion engine has augmented our day to day lives for the better and our society overall, it's actually a great example of a technology that has been used to enable the killing of 100s of millions of people by those exact types of shady institutions and individuals the commenter made reference to. You don't need something "super intelligent" to cause a ton of harm.
    - O_OtheGreat 2y ago
hintymad 2y ago
> Not to be too pessimistic here, but why are we talking about things like this I also think that we merely got a very well compressed knowledge base, therefore we are far from super intelligence, and so-called safety sounds more Orwellian than having any real value. That said, I think we should take the literal meaning of what Ilya says. His goal is to build a super intelligence. Given that, albeit a lofty goal, SSI has to put safety in place. So, there, safe super intelligence
- lxgr 2y ago
  An underappreciated feature of a classical knowledge base is returning “no results” when appropriate. LLMs so far arguably fall short on that metric, and I’m not sure whether that’s possibly an inherent limitation. So out of all potential applications with current-day LLMs, I’m really not sure this is a particularly good one. Maybe this is fixable if we can train them to cite their sources more consistently, in a way that lets us double check the output?
cjk2 2y ago
I’m a miserable cynic at a much higher level. This is top level grifting. And I’ve made a shit ton of money out of it. That’s as far as reality goes.
- therobots927 2y ago
  lol same. Are you selling yet?
  - cjk2 2y ago
    Mostly holding on still. Apple just bumped the hype a little more and gave it a few more months despite MSFT’s inherent ability to shaft everything they touch. I moved about 50% of my capital back into ETFs though before WWDC in case they dumped a turd on the table.
  - nathanasmith 2y ago
    When QQQ and SMH close under the 200 day moving average I'll sell my TQQQ and SOXL repectively. Until then, party on! It's been a wild ride.
reissbaker 2y ago
I'm pretty sure "Altman and company" don't have much to do with this — this is Ilya, who pretty famously tried to get Altman fired, and then himself left OpenAI in the aftermath. Ilya is a brilliant researcher who's contributed to many foundational parts of deep learning (including the original AlexNet); I would say I'm somewhat pessimistic based on the "safety" focus — I don't think LLMs are particularly dangerous, nor do they seem likely to be in the near future, so that seems like a distraction — but I'd be surprised if SSI didn't contribute something meaningful nonetheless given the research pedigree.
- vineyardmike 2y ago
  > I don't think LLMs are particularly dangerous “Everyone” who works in deep AI tech seems to constantly talk about the dangers. Either they’re aggrandizing themselves and their work, or they’re playing into sci-fi fear for attention or there is something the rest of us aren’t seeing. I’m personally very skeptical there is any real dangers today. If I’m wrong, I’d love to see evidence. Are foundation models before fine tuning outputting horrific messages about destroying humanity? To me, the biggest dangers come from a human listening to a hallucination and doing something dangerous, like unsafe food preparation or avoiding medical treatments. This seems distinct from a malicious LLM super intelligence.
  - lupire 2y ago
    That's what Safe Super intelligence misses. Superintelligence isn't practically more dangerous. Super stupidity is already here, and bad enough.
  - zztop44 2y ago
    They reduce the marginal cost of producing plausible content to effectively zero. When combined with other societal and technological shifts, that makes them dangerous to a lot of things: healthy public discourse, a sense of shared reality, people’s jobs, etc etc But I agree that it’s not at all clear how we get from ChatGPT to the fabled paperclip demon.
    - 2y ago
TaylorAlexander 2y ago
> why are we talking about things like this? > this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders) Ilya believes transformers can be enough to achieve superintelligence (if inefficiently). He is concerned that companies like OpenAI are going to succeed at doing it without investing in safety, and they're going to unleash a demon in the process. I don't really believe either of those things. I find arguments that autoregressive approaches lack certain critical features [1] to be compelling. But if there's a bunch of investors caught up in the hype machine ready to dump money on your favorite pet concept, and you have a high visibility position in one of the companies at the front of the hype machine, wouldn't you want to accept that money to work relatively unconstrained on that problem? My little pet idea is open source machines that take in veggies and rice and beans on one side and spit out hot healthy meals on the other side, as a form of mutual aid to offer payment optional meals in cities, like an automated form of the work the Sikhs do [2]. If someone wanted to pay me loads of money to do so, I'd have a lot to say about how revolutionary it is going to be. [1] https://www.youtube.com/watch?v=1lHFUR-yD6I https://www.youtube.com/watch?v=1lHFUR-yD6I [2] https://www.youtube.com/watch?v=qdoJroKUwu0 https://www.youtube.com/watch?v=qdoJroKUwu0 EDIT: To be clear I’m not saying it’s a fools errand. Current approaches to AI have economic value of some sort. Even if we don’t see AGI any time soon there’s money to be made. Ilya clearly knows a lot about how these systems are built. Seems worth going independent to try his own approach and maybe someone can turn a profit off this work even without AGI. Tho this is not without tradeoffs and reasonable people can disagree on the value of additional investment in this space.
- lazide 2y ago
  His paycheck is already dependent on people believing this world view. It’s important to not lose sight of that.
  - greatpostman 2y ago
    Dude he’s probably worth > 1 Billion.
    - 2y ago
flockonus 2y ago
Likewise, i'm baffled by intelligent people [in such denial] still making the reductionist argument about token prediction being a banal ability. It's not. It's not very different than how our intelligence manifest.
- TaylorAlexander 2y ago
  > It's not very different than how our intelligence manifest. [citation needed]
  - esafak 2y ago
    Search for "intelligence is prediction/compression" and you'll find your citations.
01100011 2y ago
Too many people are extrapolating the curve to exponential when it could be a sigmoid. Lots of us got too excited and too invested in where "AI" was heading about ten years ago. But that said, there are plenty of crappy, not-AGI technologies that deserve consideration. LLMs can still make for some very effective troll farms. GenAI can make some very convincing deepfakes. Drone swarms, even without AI, represent a new dimension of capabilities for armies, terrorist groups or lone wolves. Bioengineering is bringing custom organisms, prions or infectious agents within reach of individuals. I wish someone in our slowly-ceasing-to-function US government was keeping a proper eye on these things.
coffeemug 2y ago
AlphaGo took us from mediocre engines to outclassing the best human players in the world within a few short years. Ilya contributed to AlphaGo. What makes you so confident this can't happen with token prediction?
- lupire 2y ago
  If solving chess already created the Singularity, why do we need token prediction? Why do we need computers that are better than humans at the game of token prediction?
  - fastball 2y ago
    Because it can be argued that intelligence is just highly sophisticated token prediction?
    - CRConrad 2y ago
      Anything "can be argued". (Just give me something interesting enough, and I'll show ya.) It could probably also "be argued" that intelligence is an angry cabbage.
- HarHarVeryFunny 2y ago
  I'm pretty sure Ilya had nothing to do with AlphaGo, which came from DeepMind. He did work for Google Brain for a few years before OpenAI, but that was before Brain and DeepMind merged. The AlphaGo lead was David Silver.
blueboo 2y ago
it's intellectual ego catnip. https://idlewords.com/talks/superintelligence.htm https://idlewords.com/talks/superintelligence.htm
joantune 2y ago
> But, all we’ve achieved at this point is making a glorified token predicting machine trained on existing data (made by humans), not really being able to be creative outside of deriving things humans have already made before. Granted, they‘re really good at doing that, but not much else. Remove token, and that's what we humans do. Like, you need to realize that neural networks came to be because someone had the idea to mimic our brains' functionality, and see where that lead to. Many skeptics at the beginning like you discredited the inventor, but he was proved wrong. LLMs shown how much more than your limited description they can achieve. We mimicked birds with airplanes, and we can outdo them. It's actually in my view very short sighted to say we can't just mimic brains and outdo them. We're there. ChatGPT is the initial little planes that flew close to the ground and barely stayed up
- lazide 2y ago
  Except it really, actually, isn’t. People don’t ‘think’ the same way, even if some part of how humans think seems to be somewhat similar some of the time. That is an important distinction. This is the hype cycle.
joe_the_user 2y ago
I actually do doubt that LLMs will create AGI but when these systems are emulating a variety of human behaviors in a way that isn't directly programmed and is good enough to be useful, it seems foolish to not take notice. The current crop of systems is a product of the transformers architecture - an innovation that accelerated performance significantly. I put the odds another changing everything but I don't think we can entirely discount the possibility. That no one understands these systems cuts both ways.
kevincox 2y ago
Even if LLM-style token prediction is not going to lead to AGI (as it very likely won't) it is still important to work on safety. If we wait until we are at the technology that will for sure lead to AGI then it is very likely that we won't have sufficient safety before we realize that it is important.
deleted 2y ago
[deleted]
GuB-42 2y ago
We already have limited "artificial superintelligences". A pocket calculator is better at calculating than the best humans, and we certainly put calculators to good use. What we call AIs are just more generic versions of tools like pocket calculators, or guns. And that's the key, it is a tool, a tool that will give a lot of power to whoever is controlling it. And that's where safety matters, it should be made so that it helps good guys more than it helps bad guys, and limit accidents. How? I don't know. Maybe people at SSI do. We already know that the 3 laws of robotics won't work, Asimov only made them to write stories about how broken they are :) Current-gen AIs are already cause for concern. They are shown to be good at bullshitting, something that bad people are already taking advantage of. I don't believe in robot apocalypse, technological singularities, etc... but some degree of control, like we do with weapons is not a bad thing. We are not there yet with AI, but we might be soon.

foobiekr 2y ago

It's no mystery, AI has attracted tons of grifters trying to cash out before the bubble pops, and investors aren't really good at filtering.

sourcepluck 2y ago
Well said. There is a mystery though still - how many people fall for it and then stay fell, and how long that goes on for. People who've followed directly a similar pattern play itself out often many times, and still, they go along. It's so puzzlingly common amongst very intelligent people in the "tech" space that I've started to wonder if there isn't a link to this ambient belief a lot of people have that tech can "change everything" for the better, in some sense. As in, we've been duped again and again, but then the new exciting thing comes along... and in spite of ourselves, we say: "This time it's really the one!" Is what we're witnessing simply the unfulfilled promises of techno-optimism crashing against the shores of social reality repeatedly?
prasoonds 2y ago
Are you claiming Ilya Sutskever is a grifter?
- chrisldgk 2y ago
  I personally wouldn’t go that far, but I would say he’s at least riding the hype wave to get funding for his company, which, let’s be honest, nobody would care about if we weren’t this deep into the AI hypecycle.
lupire 2y ago
Why are you assigning moral agency where there may be none? These so called "grifters" are just token predictors writing business plans (prompts) with the highest computed probability of triggering $ + [large number] token pair from venture capital token predictors.

spacecadet 2y ago

Egos man. Egos.

Bengalilol 2y ago

I got flagged for less. Anyways, nice sum up of the current AI game!

adamisom 2y ago

Agree up til last paragraph: how's Altman involved? Otoh Sutskever is a true believer so that explains his Why

chrisldgk 2y ago
To be clear I was just bunching high profile AI founders and CEOs that can’t seem to stop talking about how dangerous the thing they’re trying to build is together. I don’t know (nor care) about Ilyas and Altmans current relationship.

ohcmon 2y ago

> glorified token predicting machine trained on existing data (made by humans) sorry to disappoint, but human brain fits the same definition

roywiggins 2y ago
See, this sort of claim I am instantly skeptical of. Nobody has ever caught a human brain producing or storing tokens, and certainly the subjective experience of, say, throwing a ball, doesn't involve symbols of any kind.
- mewpmewp2 2y ago
  Any output from you could be represented as a token. It is a very generic idea. Ultimately whatever you output is because of chemical reactions that follow from the input.
  - roywiggins 2y ago
    It could be represented that way. That's a long way from saying that's how brains work. Does a thermometer predict tokens? It also produces outputs that can be represented as tokens, but it's just a bit of mercury in a tube. You can dissect a thermometer as much as you like and you won't find any token prediction machinery. There's lots of things like that. Zooming out, does that make the entire atmosphere a token prediction engine, since it's producing eg wind and temperatures that could be represented as tokens? If you need one token per particle then you're admitting that this is task is impossible. Nobody will ever build a computer that can simulate a brain-sized volume of particles to sufficient fidelity. There is a long, long distance from "brains are made of chemicals" to "brains are basically token prediction engines."
    - therobots927 2y ago
      The argument that brains are just token prediction machines is basically the same as saying “the brain is just a computer”. It’s like, well, yes in the same way that a B-21 Raider is an airplane as well as a Cessna. That doesn’t mean that they are anywhere close to each other in terms of performance. They incorporate some similar basic elements but when you zoom out they’re clearly very different things.

bbor 2y ago

Well, an entire industry of researchers, which used to be divided, is now uniting around calls to slow development and emphasize safety (like, “dissolve companies” emphasis not “write employee handbooks” emphasis). They’re saying, more-or-less in unison, that GPT3 was an unexpected breakthrough in the Frame Problem, based on Judea Pearl’s prescient predictions. If we agree on that, there are two options: 1. They’ve all been tricked/bribed by Sam Altman and company (which btw this is a company started against those specific guys, just for clarity). Including me, of course. 2. You’re not as much of an expert in cognitive science as you think you are, and maybe the scientists know something you don’t. With love. As much love as possible, in a singular era

majormajor 2y ago
I would read the existence of this company as evidence that the entire industry is not as united as all that, since Sutskever was recently at another major player in the industry and thought it worth leaving. Whether that's a disagreement between what certain players say and what they do and believe, or just a question of extremes... TBD.
- bbor 2y ago
  He didn't leave because of technical reasons, he left because of ethical ones. I know this website is used to seeing this whole thing as "another iPhone moment" but I promise you it's bigger than that. Either that or I am way more insane than I know! E: Jeez I said "subreddit" maybe I need to get back to work
dnissley 2y ago
Are they actually united? Or is this the ai safety subfaction circling the wagons due to waning relevance in the face of not-actually-all-that-threatening ai?
- bbor 2y ago
  I personally find that summary of things to be way off the mark (for example, hopefully "the face" you reference isn't based on anything that appears in a browser window or in an ensemble of less than 100 agents!) but I'll try to speak to the "united" question instead. 1. The "Future of Life" institute is composed of lots of very serious people who recently helped get the EU "AI Act" passed this March, and they discuss the "myriad risks and harms AI presents" and "possibly catastrophic risks". https://newsletter.futureoflife.org/p/fli-newsletter-march-2024 https://newsletter.futureoflife.org/p/fli-newsletter-march-2... 2. Many researchers are leaving large tech companies, voicing concerns about safety and the downplaying of risks in the name of moving fast and beating vaguely-posited competitors. Both big ones like Hinton and many, many smaller ones. I'm a little lazy to scrape the data together, but it's such a wide phenomenon that a quick Google/Kagi should be enough for a vague idea. This is why Anthropic was started, why Altman was fired, why Microsoft gutted their AI safety org, and why Google fired the head of their AI ethics team. We forgot about that one cause it's from before GPT3, but it doesn't get much clearer than this: > She co-authored a research paper which she says she was asked to retract. The paper had pinpointed flaws in AI language technology, including a system built by Google... Dr Gebru had emailed her management laying out some key conditions for removing her name from the paper, and if they were not met, she would "work on a last date" for her employment. According to Dr Gebru, Google replied: "We respect your decision to leave Google... and we are accepting your resignation." 3. One funny way to see this happening is to go back to seminal papers from the last decade and see where everyone's working now. Spoiler alert: not a lot of the same names left at OpenAI, or Anthropic for that matter! This is the most egregious I've found -- the RLHF paper: see https://arxiv.org/pdf/2203.02155 https://arxiv.org/pdf/2203.02155 3. Polling of AI researchers shows a clear and overhelming trend towards AGI timelines being moved up significantly. It's still a question deeply wrapped up in accidental factors like religious belief, philosophical perspective, and general valence as a person, so I think the sudden shift here should tell you a lot. https://research.aimultiple.com/artificial-general-intelligence-singularity-timing/ https://research.aimultiple.com/artificial-general-intellige... The article I just linked actually has a section where they collect caveats, and the first is this Herbert Simon quote from 1965 that clearly didn't age well: "Machines will be capable, within twenty years, of doing any work a man can do.” This is a perfect example of my overall point! He was right. The symbolists were right, are right, will always be right -- they just failed to consider that the connectionists were just as right. The exact thing that stopped his prediction was the frame problem, which is what we've now solved. Hopefully that makes it a bit clearer why I'm anxious all the time :). The End Is Near, folks... or at least the people telling you that it's definitely not here have capitalist motivations, too. If you count the amount of money lost and received by each "side" in this "debate", I think it's clear the researcher side is down many millions in lost salaries and money spent on thinktank papers and Silicon Valley polycule dorms (it's part of it, don't ask), and the executive side is up... well, everything, so far. Did you know the biggest privately-funded infrastructure project in the history of humanity was announced this year? https://www.datacenterdynamics.com/en/opinions/how-microsoft-wins/ https://www.datacenterdynamics.com/en/opinions/how-microsoft...

foolishbard 2y ago

There's a chance that these systems can actually out perform their training data and be better than the sum of their parts. New work out Harvard talks about this idea of "transcendence" https://arxiv.org/abs/2406.11741 https://arxiv.org/abs/2406.11741 While this is a new area, it would be naive to write this off as just science fiction.

majormajor 2y ago
It would be nice if authors wouldn't use a loaded-as-fuck word like "transcendence" for "the trained model can sometimes achieve better performance than all [chess] players in the dataset" because while certainly that's demonstrating an impressive internalization of the game, it's also something that many humans can also do. The machine, of course, can be scaled in breadth and performance, but... "transcendence"? Are they trying to be mis-interpreted?
- Kerb_ 2y ago
  It transcends the training data, I get the usage intended but it certainly is ripe for misinterpretation
  - lupire 2y ago
    That's trivial though, conceptually. Every regression line transcends the training data. We've had that since Wisdom of Crowds.
  - sb77 2y ago
    The word for that is "generalizes" or "generalization" and it has existed for a very long time.
    - Kerb_ 2y ago
      I've been very confidently informed that these AIs are not AGIs, which makes me wonder what the "General" in AGI is supposed to mean and whether generalization is actually the benchmark for advanced intelligence. If they're not AGI, then wouldn't another word for that level of generalization be more accurate than "generalization"? It doesn't have to be "transcendence" but it seems weird to have a defined step we claim we aren't at but also use the same word to describe a process we know it does. I don't get the nuance of the lingo entirely, I guess. I'm just here for the armchair philosophy

alecco 2y ago

Because it's likely soon LLMs will be able to teach themselves and surpass humans. No consciousness, no will. But somebody will have their power. Dark government agencies and questionable billionaires. Who knows what will it enable them to do. https://en.wikipedia.org/wiki/AlphaGo_Zero https://en.wikipedia.org/wiki/AlphaGo_Zero

roywiggins 2y ago
Likely according to who?
- CyberDildonics 2y ago
  Whoever needs money from investors who don't understand LLMs.
  - figers 2y ago
    ha-ha!!!!
mjr00 2y ago
Mind defining "likely" and "soon" here? Like 10% chance in 100 years, or 90% chance in 1 year? Not sure how a Go engine really applies. Do you consider cars superintelligent because they can move faster than any human?
- TechDebtDevin 2y ago
  I'm with you here, but it should be noted that while the combustion engine has augmented our day to day lives for the better and our society overall, it's actually a great example of a technology that has been used to enable the killing of 100s of millions of people by those exact types of shady institutions and individuals the commenter made reference to. You don't need something "super intelligent" to cause a ton of harm.
  - O_OtheGreat 2y ago

hintymad 2y ago

> Not to be too pessimistic here, but why are we talking about things like this I also think that we merely got a very well compressed knowledge base, therefore we are far from super intelligence, and so-called safety sounds more Orwellian than having any real value. That said, I think we should take the literal meaning of what Ilya says. His goal is to build a super intelligence. Given that, albeit a lofty goal, SSI has to put safety in place. So, there, safe super intelligence

lxgr 2y ago
An underappreciated feature of a classical knowledge base is returning “no results” when appropriate. LLMs so far arguably fall short on that metric, and I’m not sure whether that’s possibly an inherent limitation. So out of all potential applications with current-day LLMs, I’m really not sure this is a particularly good one. Maybe this is fixable if we can train them to cite their sources more consistently, in a way that lets us double check the output?

cjk2 2y ago

I’m a miserable cynic at a much higher level. This is top level grifting. And I’ve made a shit ton of money out of it. That’s as far as reality goes.

therobots927 2y ago
lol same. Are you selling yet?
- cjk2 2y ago
  Mostly holding on still. Apple just bumped the hype a little more and gave it a few more months despite MSFT’s inherent ability to shaft everything they touch. I moved about 50% of my capital back into ETFs though before WWDC in case they dumped a turd on the table.
- nathanasmith 2y ago
  When QQQ and SMH close under the 200 day moving average I'll sell my TQQQ and SOXL repectively. Until then, party on! It's been a wild ride.

reissbaker 2y ago

I'm pretty sure "Altman and company" don't have much to do with this — this is Ilya, who pretty famously tried to get Altman fired, and then himself left OpenAI in the aftermath. Ilya is a brilliant researcher who's contributed to many foundational parts of deep learning (including the original AlexNet); I would say I'm somewhat pessimistic based on the "safety" focus — I don't think LLMs are particularly dangerous, nor do they seem likely to be in the near future, so that seems like a distraction — but I'd be surprised if SSI didn't contribute something meaningful nonetheless given the research pedigree.

vineyardmike 2y ago
> I don't think LLMs are particularly dangerous “Everyone” who works in deep AI tech seems to constantly talk about the dangers. Either they’re aggrandizing themselves and their work, or they’re playing into sci-fi fear for attention or there is something the rest of us aren’t seeing. I’m personally very skeptical there is any real dangers today. If I’m wrong, I’d love to see evidence. Are foundation models before fine tuning outputting horrific messages about destroying humanity? To me, the biggest dangers come from a human listening to a hallucination and doing something dangerous, like unsafe food preparation or avoiding medical treatments. This seems distinct from a malicious LLM super intelligence.
- lupire 2y ago
  That's what Safe Super intelligence misses. Superintelligence isn't practically more dangerous. Super stupidity is already here, and bad enough.
- zztop44 2y ago
  They reduce the marginal cost of producing plausible content to effectively zero. When combined with other societal and technological shifts, that makes them dangerous to a lot of things: healthy public discourse, a sense of shared reality, people’s jobs, etc etc But I agree that it’s not at all clear how we get from ChatGPT to the fabled paperclip demon.
  - 2y ago

TaylorAlexander 2y ago

> why are we talking about things like this? > this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders) Ilya believes transformers can be enough to achieve superintelligence (if inefficiently). He is concerned that companies like OpenAI are going to succeed at doing it without investing in safety, and they're going to unleash a demon in the process. I don't really believe either of those things. I find arguments that autoregressive approaches lack certain critical features [1] to be compelling. But if there's a bunch of investors caught up in the hype machine ready to dump money on your favorite pet concept, and you have a high visibility position in one of the companies at the front of the hype machine, wouldn't you want to accept that money to work relatively unconstrained on that problem? My little pet idea is open source machines that take in veggies and rice and beans on one side and spit out hot healthy meals on the other side, as a form of mutual aid to offer payment optional meals in cities, like an automated form of the work the Sikhs do [2]. If someone wanted to pay me loads of money to do so, I'd have a lot to say about how revolutionary it is going to be. [1] https://www.youtube.com/watch?v=1lHFUR-yD6I https://www.youtube.com/watch?v=1lHFUR-yD6I [2] https://www.youtube.com/watch?v=qdoJroKUwu0 https://www.youtube.com/watch?v=qdoJroKUwu0 EDIT: To be clear I’m not saying it’s a fools errand. Current approaches to AI have economic value of some sort. Even if we don’t see AGI any time soon there’s money to be made. Ilya clearly knows a lot about how these systems are built. Seems worth going independent to try his own approach and maybe someone can turn a profit off this work even without AGI. Tho this is not without tradeoffs and reasonable people can disagree on the value of additional investment in this space.

lazide 2y ago
His paycheck is already dependent on people believing this world view. It’s important to not lose sight of that.
- greatpostman 2y ago
  Dude he’s probably worth > 1 Billion.
  - 2y ago

flockonus 2y ago

Likewise, i'm baffled by intelligent people [in such denial] still making the reductionist argument about token prediction being a banal ability. It's not. It's not very different than how our intelligence manifest.

TaylorAlexander 2y ago
> It's not very different than how our intelligence manifest. [citation needed]
- esafak 2y ago
  Search for "intelligence is prediction/compression" and you'll find your citations.

01100011 2y ago

Too many people are extrapolating the curve to exponential when it could be a sigmoid. Lots of us got too excited and too invested in where "AI" was heading about ten years ago. But that said, there are plenty of crappy, not-AGI technologies that deserve consideration. LLMs can still make for some very effective troll farms. GenAI can make some very convincing deepfakes. Drone swarms, even without AI, represent a new dimension of capabilities for armies, terrorist groups or lone wolves. Bioengineering is bringing custom organisms, prions or infectious agents within reach of individuals. I wish someone in our slowly-ceasing-to-function US government was keeping a proper eye on these things.

coffeemug 2y ago

AlphaGo took us from mediocre engines to outclassing the best human players in the world within a few short years. Ilya contributed to AlphaGo. What makes you so confident this can't happen with token prediction?

lupire 2y ago
If solving chess already created the Singularity, why do we need token prediction? Why do we need computers that are better than humans at the game of token prediction?
- fastball 2y ago
  Because it can be argued that intelligence is just highly sophisticated token prediction?
  - CRConrad 2y ago
    Anything "can be argued". (Just give me something interesting enough, and I'll show ya.) It could probably also "be argued" that intelligence is an angry cabbage.
HarHarVeryFunny 2y ago
I'm pretty sure Ilya had nothing to do with AlphaGo, which came from DeepMind. He did work for Google Brain for a few years before OpenAI, but that was before Brain and DeepMind merged. The AlphaGo lead was David Silver.

blueboo 2y ago

it's intellectual ego catnip. https://idlewords.com/talks/superintelligence.htm https://idlewords.com/talks/superintelligence.htm

joantune 2y ago

> But, all we’ve achieved at this point is making a glorified token predicting machine trained on existing data (made by humans), not really being able to be creative outside of deriving things humans have already made before. Granted, they‘re really good at doing that, but not much else. Remove token, and that's what we humans do. Like, you need to realize that neural networks came to be because someone had the idea to mimic our brains' functionality, and see where that lead to. Many skeptics at the beginning like you discredited the inventor, but he was proved wrong. LLMs shown how much more than your limited description they can achieve. We mimicked birds with airplanes, and we can outdo them. It's actually in my view very short sighted to say we can't just mimic brains and outdo them. We're there. ChatGPT is the initial little planes that flew close to the ground and barely stayed up

lazide 2y ago
Except it really, actually, isn’t. People don’t ‘think’ the same way, even if some part of how humans think seems to be somewhat similar some of the time. That is an important distinction. This is the hype cycle.

joe_the_user 2y ago

I actually do doubt that LLMs will create AGI but when these systems are emulating a variety of human behaviors in a way that isn't directly programmed and is good enough to be useful, it seems foolish to not take notice. The current crop of systems is a product of the transformers architecture - an innovation that accelerated performance significantly. I put the odds another changing everything but I don't think we can entirely discount the possibility. That no one understands these systems cuts both ways.

kevincox 2y ago

Even if LLM-style token prediction is not going to lead to AGI (as it very likely won't) it is still important to work on safety. If we wait until we are at the technology that will for sure lead to AGI then it is very likely that we won't have sufficient safety before we realize that it is important.

deleted 2y ago

[deleted]

GuB-42 2y ago

We already have limited "artificial superintelligences". A pocket calculator is better at calculating than the best humans, and we certainly put calculators to good use. What we call AIs are just more generic versions of tools like pocket calculators, or guns. And that's the key, it is a tool, a tool that will give a lot of power to whoever is controlling it. And that's where safety matters, it should be made so that it helps good guys more than it helps bad guys, and limit accidents. How? I don't know. Maybe people at SSI do. We already know that the 3 laws of robotics won't work, Asimov only made them to write stories about how broken they are :) Current-gen AIs are already cause for concern. They are shown to be good at bullshitting, something that bad people are already taking advantage of. I don't believe in robot apocalypse, technological singularities, etc... but some degree of control, like we do with weapons is not a bad thing. We are not there yet with AI, but we might be soon.