10 ms·

I have noticed that the vast majority of existential risk folks I have seen have very little experience running and maintaining large systems in production. The

I have noticed that the vast majority of existential risk folks I have seen have very little experience running and maintaining large systems in production. They tend to all be on the research side of things. I think that it can be hard to understand the amount of real world friction that exists until you experience it for yourself.

habitue 3y ago
Conversely, being too close to the problem can cause practitioners to miss the forest for the trees. I think in this case it's a false premise though. There are plenty of real practitioners like Joshua Bengio and Geoffrey Hinton who have sounded the alarm on these topics, and no one can call them inexperienced with real systems.
- throwanem 3y ago
  They're both academics. Do they have SRE experience not listed on their CVs?
  - habitue 3y ago
    This seems like splitting hairs? They totally understand how modern neural networks work, they spend all day with them.
    - adw 3y ago
      The question isn't _neural_ networks, it's _physical_ networks (and power, and cooling, and racking, and deployment, and and and). Actually marshaling enough compute to do anything is a serious real-world-moving-atoms endeavor.
      - lmm 3y ago
        Or you look up the AWS APIs and call them.
        RhysU 3y ago
        I now want a short story where killer AI is stopped because it can't reach a Google Salesperson to increase its quota.
        throwanem 3y ago
        "Hammer: $5. Knowing where to hit the machine with the hammer: $9,995."
      - loandbehold 3y ago
        We are talking about superhuman AGI. It would be able to do everything human is able to do. If human SRE is able to figure it out, AGI would too. That may include buying compute from cloud platforms as well as building it's own datacenters.
        throwanem 3y ago
        It's a self-sealing argument. Anything a human can do, it can do too, because it's superhuman. Anything a human can't do, it can do anyway, because it's superhuman. Any reason it can't is false prima facie, because it's superhuman. You can claim what you like, and who's to argue? It's superhuman! It's unfalsifiable, too.
        loandbehold 3y ago
        SREs being able to maintain these systems is existence proof that neural networks are able to maintain them — neural networks in the brains of human SREs. Or do you think humans evolved innate SRE skills? The idea that people like Geof Hinton are wrong because they don't understand SRE is pretty arrogant. SRE skills are relatively straightforward vs AI theory.
        throwanem 3y ago
        It's not that I think they're wrong; it's that I am not yet convinced they aren't "not even wrong."
        loandbehold 3y ago
        You just don't understand their arguments. Put some efforts into understanding these arguments first. You can start by reading Nick Bostrom's Superintelligence book. People like you are no different that climate change deniers pooh-poohing on scientists without understanding physics involved. Be humble. People raising AGI X-risk arguments are smarter than you and gave it a lot of thought. Put an honest efforts in understanding their arguments.
jimmaswell 3y ago
Imagine the real world friction overcome to evolve a human from a single celled organism. Now consider that a computer program can replicate and change itself orders of magnitude faster than a human generation. I've dealt with spaghetti systems that need such maintenance and I'm unconvinced that poor devops is a major barrier to the hypothetical of runaway AI. Devops people will probably be using AI to fix these issues soon enough if they aren't already.
jayvanguard 3y ago
I agree that a lot of the AI risk pundits are either people with no real AI experience (or very outdated, resting-on-their-laurels experience) or if they do know AI, they probably know little about the complexities of running actual large scale, business or mission critical systems. However, in their defense, the real question is whether those types of problems are something this new class of AI could potentially excel at. I've been an AI skeptic for decades, having had real world industry experience trying to get it to solve problems, and this is the first technological advance that has actually surprised me with its capabilities. AI Risk still doesn't rank above many current, pressing global problems for me personally, but it also doesn't seem ridiculous anymore.

habitue 3y ago

Conversely, being too close to the problem can cause practitioners to miss the forest for the trees. I think in this case it's a false premise though. There are plenty of real practitioners like Joshua Bengio and Geoffrey Hinton who have sounded the alarm on these topics, and no one can call them inexperienced with real systems.

throwanem 3y ago
They're both academics. Do they have SRE experience not listed on their CVs?
- habitue 3y ago
  This seems like splitting hairs? They totally understand how modern neural networks work, they spend all day with them.
  - adw 3y ago
    The question isn't _neural_ networks, it's _physical_ networks (and power, and cooling, and racking, and deployment, and and and). Actually marshaling enough compute to do anything is a serious real-world-moving-atoms endeavor.
    - lmm 3y ago
      Or you look up the AWS APIs and call them.
      - RhysU 3y ago
        I now want a short story where killer AI is stopped because it can't reach a Google Salesperson to increase its quota.
      - throwanem 3y ago
        "Hammer: $5. Knowing where to hit the machine with the hammer: $9,995."
    - loandbehold 3y ago
      We are talking about superhuman AGI. It would be able to do everything human is able to do. If human SRE is able to figure it out, AGI would too. That may include buying compute from cloud platforms as well as building it's own datacenters.
      - throwanem 3y ago
        It's a self-sealing argument. Anything a human can do, it can do too, because it's superhuman. Anything a human can't do, it can do anyway, because it's superhuman. Any reason it can't is false prima facie, because it's superhuman. You can claim what you like, and who's to argue? It's superhuman! It's unfalsifiable, too.
        loandbehold 3y ago
        SREs being able to maintain these systems is existence proof that neural networks are able to maintain them — neural networks in the brains of human SREs. Or do you think humans evolved innate SRE skills? The idea that people like Geof Hinton are wrong because they don't understand SRE is pretty arrogant. SRE skills are relatively straightforward vs AI theory.
        throwanem 3y ago
        It's not that I think they're wrong; it's that I am not yet convinced they aren't "not even wrong."
        loandbehold 3y ago
        You just don't understand their arguments. Put some efforts into understanding these arguments first. You can start by reading Nick Bostrom's Superintelligence book. People like you are no different that climate change deniers pooh-poohing on scientists without understanding physics involved. Be humble. People raising AGI X-risk arguments are smarter than you and gave it a lot of thought. Put an honest efforts in understanding their arguments.

jimmaswell 3y ago

Imagine the real world friction overcome to evolve a human from a single celled organism. Now consider that a computer program can replicate and change itself orders of magnitude faster than a human generation. I've dealt with spaghetti systems that need such maintenance and I'm unconvinced that poor devops is a major barrier to the hypothetical of runaway AI. Devops people will probably be using AI to fix these issues soon enough if they aren't already.

jayvanguard 3y ago

I agree that a lot of the AI risk pundits are either people with no real AI experience (or very outdated, resting-on-their-laurels experience) or if they do know AI, they probably know little about the complexities of running actual large scale, business or mission critical systems. However, in their defense, the real question is whether those types of problems are something this new class of AI could potentially excel at. I've been an AI skeptic for decades, having had real world industry experience trying to get it to solve problems, and this is the first technological advance that has actually surprised me with its capabilities. AI Risk still doesn't rank above many current, pressing global problems for me personally, but it also doesn't seem ridiculous anymore.