5 ms·

This stuff drives me up the wall for a few reasons. The main one is that I’ve never heard anyone describe in any detail how the technology will work. I’ve work

This stuff drives me up the wall for a few reasons. The main one is that I’ve never heard anyone describe in any detail how the technology will work. I’ve worked in big shops running big online ML systems and oh boy, do they need a lot of diaper-changing to even stay running. If all the ops folks went on vacation it’s a “hours vs days” not “weeks vs months” question of how soon it would fall over. Some log rotate thing gets stuck, and away we go with cascading failures. So how, in some detail, do we get from big-ass mixture model of transformers to even self-operation? That’s got to be a pre-requisite for self-improvement right? inb4 “The risk is so great the details don’t matter because not impossible”, hmmm, no. Extreme tail risk isn’t interesting when addressing it comes at the cost of driving immediate, overwhelmingly likely risk of nightmare outcomes through the roof. Consolidating the alignment and steering of big models into the hands of a few CEOs, governments captured by them, or both is a clear and present danger of horrifying proportions. I take mitigate that over extreme tail risk sure as Tuesday and taxes. Then there’s the reality of AI development thus far, which is that it comes in stops and starts. Clearly the past is no guarantee of the future, but it seems a damned sight better as a prior than the log scale and a ruler methodology employed by Yud or whoever. Smart people talk like this is serious. There are even some elite practitioners who talk kinda like this (Hinton, Karpathy on Lex, some others). And I don’t want to be walking around with my head up my ass on it, so I’ll be grateful to anyone who wants to set me straight. But if there’s an explanation somewhere, by a credible practitioner, of how this could actually happen in some actual technical detail, I haven’t found it. I think that needs to be in the footnotes the next time the mainstream press starts publishing blog posts off LessWrong as consensus.

skepticATX 3y ago
I have noticed that the vast majority of existential risk folks I have seen have very little experience running and maintaining large systems in production. They tend to all be on the research side of things. I think that it can be hard to understand the amount of real world friction that exists until you experience it for yourself.
- habitue 3y ago
  Conversely, being too close to the problem can cause practitioners to miss the forest for the trees. I think in this case it's a false premise though. There are plenty of real practitioners like Joshua Bengio and Geoffrey Hinton who have sounded the alarm on these topics, and no one can call them inexperienced with real systems.
  - throwanem 3y ago
    They're both academics. Do they have SRE experience not listed on their CVs?
    - habitue 3y ago
      This seems like splitting hairs? They totally understand how modern neural networks work, they spend all day with them.
      - adw 3y ago
        The question isn't _neural_ networks, it's _physical_ networks (and power, and cooling, and racking, and deployment, and and and). Actually marshaling enough compute to do anything is a serious real-world-moving-atoms endeavor.
        lmm 3y ago
        Or you look up the AWS APIs and call them.
        RhysU 3y ago
        I now want a short story where killer AI is stopped because it can't reach a Google Salesperson to increase its quota.
        throwanem 3y ago
        "Hammer: $5. Knowing where to hit the machine with the hammer: $9,995."
Footkerchief 3y ago
A first stop could be that the LLM becomes good enough at economic manipulation to pay humans to maintain and upgrade itself. A possible next stop is that the LLM becomes better than humans at designing new algorithms and hardware.
NikolaNovak 3y ago
One of the things that keeps me rooted is that I've worked as then employed teams of experienced DBAs to maintain "self healing databases" for last 20 years. Large but non crazy / non special databases with robust reliable technology that just need tender loving care all the time. And I'm increasingly aware as I age that I am not special and everywhere else is not that much better. So I'm with you on the dangers of self running / self improving stuff running away, vs 10k immediate risks and issues (as in, happening, not hypotheticals).
lmm 3y ago
> I’ve worked in big shops running big online ML systems and oh boy, do they need a lot of diaper-changing to even stay running. If all the ops folks went on vacation it’s a “hours vs days” not “weeks vs months” question of how soon it would fall over. Some log rotate thing gets stuck, and away we go with cascading failures. Ever worked somewhere where all the good devs have left and it's outsourced to a bunch of cheap labour? They find ways to keep it running for years. Maybe not doing the way you'd like (log rotates are stuck? Better turn off the logging system), but they make it work. Ops people like to overestimate their own importance like everyone else. > So how, in some detail, do we get from big-ass mixture model of transformers to even self-operation? That’s got to be a pre-requisite for self-improvement right? No? Model could propose a bunch of changes to its own code and let the human developers take care of deploying it. Of course the developers could theoretically review those changes carefully... but let's be real, they won't. In the worst case a human-level-intelligent AI could certainly email the outsourced ops staff and ask them to deploy tag x. More generally, it can just do whatever a new dev would do - read your wiki and figure out which commands to run from that. If there's an error, look up old email threads on it, or similar errors online, or try to match up a traceback against source code and look for common errors. We expect fresh grads to do this stuff all the time, it's really not that hard. > Then there’s the reality of AI development thus far, which is that it comes in stops and starts. Clearly the past is no guarantee of the future, but it seems a damned sight better as a prior than the log scale and a ruler methodology employed by Yud or whoever. WTF kind of logic is this? Daily weather goes up and down, so clearly extrapolating any kind of long-term trend in temperature is silly.
- benreesman 3y ago
  So there’s a middling sci-fi novel from maybe a decade ago trying to tell the AI Takeoff story called Avogadro Corp. It’s not exactly Heinlein but it’s a fun little yarn. The basic premise is that Google pardon me Avogadro Corp is doing NLP email copy-editing, but it takes too many compute resources so in desperation the TL hand wave sets it loose to secure its own budget, and being good at writing convincing emails, well, one thing leads to another. Now this is very hand-wavy and involves a lot of magical thinking and suspension of disbelief. But that’s ok, it’s a pulpy sci-fi novel for fun. Now this fictional scenario is dramatically more specific, more detailed, and frankly more believable than anything anyone has linked on this thread. And I have no issue with folks wanting to LARP this sort of thing in a capital-R Rationalist forum somewhere: it’s a premise that has fascinated sci-fi fans since Asimov wrote his 3 Laws and probably before that. But when it starts hitting the mainstream press, K Street, freaking out everyday people, and being used to turbocharge fear-driven land-grabs around intellectual property precedent? Yeah, it’s time to do better than “But what if I Have No Mouth and I Must Scream Was Real man?!?!”
andrewflnr 3y ago
Hook up the model's output to a shell. Remember it doesn't have to be engineered to modern SRE standards, it just needs to not crash. Dirty hacks are on the table. The comparison to diaper-changing is fairly illuminating, actually: even human children, let alone adults, figure out how to take care of that kind of thing, and it fades into the background of their activities. I agree that plausible mechanism is lacking in these AI risk conversations, but ops is not the hard part IMO.
BobbyTables2 3y ago
For years, I never understood cryptocurrencies. I didn’t understand why mainstream press promoted startups managing billions run by “visionary” people (barely?) old enough to rent a car. I understood the technical aspects, just not how it was being promoted. I see now there was little to understand… It is simply an end-run around financial regulation, where the biggest visionaries are frauds, exchanges are ridiculously insecure, transactions are costly yet slow, and the primary use is illicit activities… It’s the same thing lately with AI… Sure, it has slightly more merit than cryptocurrency, but it doesn’t do what people think it actually does — certainly not as well either. AI is not sentient yet people write full paragraphs to it as if it were? They even expect such in response. This is LUNACY! This is a mass delusion! Real life is not a Star Trek TNG episode with a talking computer! It’s not an accident either. Nobody wants to look like a fool when there is a lot of money at stake. So the fly by night snake oil salespeople thrive… Some technology will be developed by overly sincere people along the way, but that was never the real goal.
- Aerroon 3y ago
  I think the most interesting part of AI doomsaying is that almost all of these risks apply to biological 'systems' too. There's no reason why a human can't be born that somehow has superhuman intelligence and does all of the things these evil AI would do. Moreover, a biological superintelligence has self-replication built into the system already!
  - tristor 3y ago
    Depending on how you define biological superintelligence, these persons already exist, there's at least two people in the world (on an SD16 scale) with a 200 IQ at our current global population, my IQ is in the "above-genius" territory and there are at least ~335 people as intelligent as I am just in the United States (and that's only statistical probability, due to global labor market conditions the number is probably higher). Do you know what really smart people do? They don't do any of the things that AI Risk says will happen, they get jobs at Google (or insert other pointless white collar job on wall street) moving widgets by 1 pixel to drive an ad engagement metric by half of a percentage and get paid $500k+ a year in total compensation. Why? Because it is a relatively low effort way to fund a lifestyle that allows enjoyment of your intellect through experiencing the world in other dimensions besides the mundane of work. The non-biological aspect is the only reason why AI Risk has any credence at all /in our current state/, because AI isn't distracted by a desire to travel the world and explore nature, or to read all the great works of literature and write their own analysis of it, etc. Biological beings, at least at this point in time and in the probable sense, have emotions and therefore have desires that range beyond whatever machinations you expect from AI Risk.

skepticATX 3y ago

I have noticed that the vast majority of existential risk folks I have seen have very little experience running and maintaining large systems in production. They tend to all be on the research side of things. I think that it can be hard to understand the amount of real world friction that exists until you experience it for yourself.

habitue 3y ago
Conversely, being too close to the problem can cause practitioners to miss the forest for the trees. I think in this case it's a false premise though. There are plenty of real practitioners like Joshua Bengio and Geoffrey Hinton who have sounded the alarm on these topics, and no one can call them inexperienced with real systems.
- throwanem 3y ago
  They're both academics. Do they have SRE experience not listed on their CVs?
  - habitue 3y ago
    This seems like splitting hairs? They totally understand how modern neural networks work, they spend all day with them.
    - adw 3y ago
      The question isn't _neural_ networks, it's _physical_ networks (and power, and cooling, and racking, and deployment, and and and). Actually marshaling enough compute to do anything is a serious real-world-moving-atoms endeavor.
      - lmm 3y ago
        Or you look up the AWS APIs and call them.
        RhysU 3y ago
        I now want a short story where killer AI is stopped because it can't reach a Google Salesperson to increase its quota.
        throwanem 3y ago
        "Hammer: $5. Knowing where to hit the machine with the hammer: $9,995."

Footkerchief 3y ago

A first stop could be that the LLM becomes good enough at economic manipulation to pay humans to maintain and upgrade itself. A possible next stop is that the LLM becomes better than humans at designing new algorithms and hardware.

NikolaNovak 3y ago

One of the things that keeps me rooted is that I've worked as then employed teams of experienced DBAs to maintain "self healing databases" for last 20 years. Large but non crazy / non special databases with robust reliable technology that just need tender loving care all the time. And I'm increasingly aware as I age that I am not special and everywhere else is not that much better. So I'm with you on the dangers of self running / self improving stuff running away, vs 10k immediate risks and issues (as in, happening, not hypotheticals).

lmm 3y ago

> I’ve worked in big shops running big online ML systems and oh boy, do they need a lot of diaper-changing to even stay running. If all the ops folks went on vacation it’s a “hours vs days” not “weeks vs months” question of how soon it would fall over. Some log rotate thing gets stuck, and away we go with cascading failures. Ever worked somewhere where all the good devs have left and it's outsourced to a bunch of cheap labour? They find ways to keep it running for years. Maybe not doing the way you'd like (log rotates are stuck? Better turn off the logging system), but they make it work. Ops people like to overestimate their own importance like everyone else. > So how, in some detail, do we get from big-ass mixture model of transformers to even self-operation? That’s got to be a pre-requisite for self-improvement right? No? Model could propose a bunch of changes to its own code and let the human developers take care of deploying it. Of course the developers could theoretically review those changes carefully... but let's be real, they won't. In the worst case a human-level-intelligent AI could certainly email the outsourced ops staff and ask them to deploy tag x. More generally, it can just do whatever a new dev would do - read your wiki and figure out which commands to run from that. If there's an error, look up old email threads on it, or similar errors online, or try to match up a traceback against source code and look for common errors. We expect fresh grads to do this stuff all the time, it's really not that hard. > Then there’s the reality of AI development thus far, which is that it comes in stops and starts. Clearly the past is no guarantee of the future, but it seems a damned sight better as a prior than the log scale and a ruler methodology employed by Yud or whoever. WTF kind of logic is this? Daily weather goes up and down, so clearly extrapolating any kind of long-term trend in temperature is silly.

benreesman 3y ago
So there’s a middling sci-fi novel from maybe a decade ago trying to tell the AI Takeoff story called Avogadro Corp. It’s not exactly Heinlein but it’s a fun little yarn. The basic premise is that Google pardon me Avogadro Corp is doing NLP email copy-editing, but it takes too many compute resources so in desperation the TL hand wave sets it loose to secure its own budget, and being good at writing convincing emails, well, one thing leads to another. Now this is very hand-wavy and involves a lot of magical thinking and suspension of disbelief. But that’s ok, it’s a pulpy sci-fi novel for fun. Now this fictional scenario is dramatically more specific, more detailed, and frankly more believable than anything anyone has linked on this thread. And I have no issue with folks wanting to LARP this sort of thing in a capital-R Rationalist forum somewhere: it’s a premise that has fascinated sci-fi fans since Asimov wrote his 3 Laws and probably before that. But when it starts hitting the mainstream press, K Street, freaking out everyday people, and being used to turbocharge fear-driven land-grabs around intellectual property precedent? Yeah, it’s time to do better than “But what if I Have No Mouth and I Must Scream Was Real man?!?!”

andrewflnr 3y ago

Hook up the model's output to a shell. Remember it doesn't have to be engineered to modern SRE standards, it just needs to not crash. Dirty hacks are on the table. The comparison to diaper-changing is fairly illuminating, actually: even human children, let alone adults, figure out how to take care of that kind of thing, and it fades into the background of their activities. I agree that plausible mechanism is lacking in these AI risk conversations, but ops is not the hard part IMO.

BobbyTables2 3y ago

For years, I never understood cryptocurrencies. I didn’t understand why mainstream press promoted startups managing billions run by “visionary” people (barely?) old enough to rent a car. I understood the technical aspects, just not how it was being promoted. I see now there was little to understand… It is simply an end-run around financial regulation, where the biggest visionaries are frauds, exchanges are ridiculously insecure, transactions are costly yet slow, and the primary use is illicit activities… It’s the same thing lately with AI… Sure, it has slightly more merit than cryptocurrency, but it doesn’t do what people think it actually does — certainly not as well either. AI is not sentient yet people write full paragraphs to it as if it were? They even expect such in response. This is LUNACY! This is a mass delusion! Real life is not a Star Trek TNG episode with a talking computer! It’s not an accident either. Nobody wants to look like a fool when there is a lot of money at stake. So the fly by night snake oil salespeople thrive… Some technology will be developed by overly sincere people along the way, but that was never the real goal.

Aerroon 3y ago
I think the most interesting part of AI doomsaying is that almost all of these risks apply to biological 'systems' too. There's no reason why a human can't be born that somehow has superhuman intelligence and does all of the things these evil AI would do. Moreover, a biological superintelligence has self-replication built into the system already!
- tristor 3y ago
  Depending on how you define biological superintelligence, these persons already exist, there's at least two people in the world (on an SD16 scale) with a 200 IQ at our current global population, my IQ is in the "above-genius" territory and there are at least ~335 people as intelligent as I am just in the United States (and that's only statistical probability, due to global labor market conditions the number is probably higher). Do you know what really smart people do? They don't do any of the things that AI Risk says will happen, they get jobs at Google (or insert other pointless white collar job on wall street) moving widgets by 1 pixel to drive an ad engagement metric by half of a percentage and get paid $500k+ a year in total compensation. Why? Because it is a relatively low effort way to fund a lifestyle that allows enjoyment of your intellect through experiencing the world in other dimensions besides the mundane of work. The non-biological aspect is the only reason why AI Risk has any credence at all /in our current state/, because AI isn't distracted by a desire to travel the world and explore nature, or to read all the great works of literature and write their own analysis of it, etc. Biological beings, at least at this point in time and in the probable sense, have emotions and therefore have desires that range beyond whatever machinations you expect from AI Risk.