5 ms·

What about people shouting about AGI from the halls of the most advanced research labs in the field?

by ethanbond 3y ago

lazide 3y ago
When has an ivory tower not involved the ‘most advanced research labs’? Or do you mean largest implementors?
- ethanbond 3y ago
  Ivory tower is usually a pejorative term meaning people who are isolated from the reality of the situation. So it doesn’t include the most advanced research labs when those research labs are in touch with the reality of the situation, like right now with AI. Nitpick “most advanced research labs” if you want, it’s obvious what I mean: it is clear that the “shouting” is not coming solely from people who don’t know what’s going on.
  - lazide 3y ago
    Eh, from folks isolated from the constraints and practicalities of the real world. Which OpenAI definitely is, and has been for some time. Or do you think infinite money and ‘do whatever you want’ is constraining?
    - ethanbond 3y ago
      What does constraints have to do with it? The question is who has the best grasp on the trajectory of the technology. If not the researchers who are at the frontier of building and deploying these systems, playing with next generation iterations and planning several generations forward, then who? Andreessen? People who like playing with LLMs? People who called the OpenAI API?
      - lazide 3y ago
        Because if you don’t have real world constraints (like needing to be profitable, or paying the bill for thermodynamic reality), then ‘anything is possible’. Also if you aren’t dealing with those, then problems that come from that never occur to the person involved. So the concerns are abstract and not based on real limits or real problems. Hence ivory tower.
        ethanbond 3y ago
        Okay so the server bills or thermodynamic bills mean we need not worry. Open to hearing the argument: tell me how!
        lazide 3y ago
        Not at all what I'm saying. What I'm saying is, no one knows where the line between actually economic/useful/effective and 'not worth the trouble' actually is, let alone 'could be done without boiling the oceans'. Right now it's all hand wavey, could do everything, etc. etc.
        ethanbond 3y ago
        That uncertainty is called risk! That is exactly the whole point!
        lazide 3y ago
        What does that have to do with this thread? You know the ivory tower one?
zer00eyz 3y ago
People who worry about AI taking over the world need to worry about the world continuing to work. Your average gas station gets gas 2x a day. The electric grid has 100's of faults that are worked around and fixed every day. Delivery still are loaded and driven by people. Any AGI that is a threat to humanity has to be suicidal to act on it because of how the world works. Furthermore, if you crack open an LLM and turn OFF the random noise (temperature) it becomes very deterministic. You think our accent to godhood is on the back of a vector map with some noise, I dont know what to tell you. That doesn't mean all this ML can't change the world, can't foster less BS jobs and enable creativity... but I have massive doubts that sentience and sapience are on this path.
- ethanbond 3y ago
  Sure hope that intelligence doesn’t roughly mean “ability to find solutions to problems that less intelligent entities can’t discover.” I have no idea what “our ascent to godhood” entails, and it’s hilarious that doomers get criticized for being religious about all this and then you’ll say something like that with a straight face. That’s neat that you “have doubts” sentience and sapience are on this path. You also don’t actually know.
- staunton 3y ago
  > Any AGI that is a threat to humanity has to be suicidal to act on it because of how the world works A "threat to humanity" need not mean "killing all humans next year". It could just mean some entity gaining permanent control. That situation would be irreversible and humanity would be at the mercy of that entity. Chances are, the entity would have emerged victorious from an evolutionary struggle for power and thus would not care about human flourishing. That entity need not be a single AI. It could be a system like a large corporation or a country where initially a lot of "small" decisions are made by AIs. Over time, the influence of humans on this system might diminish and the trend might become irreversible. Currently, no individual and no group, (organization, country...) on the planet has anywhere close to complete control over humanity. Further, even if some country managed to conquer everything, it could not hope to maintain that power indefinitely. An immortal system capable of complete surveillance, however, may be able to maintain power. It's a new thing, we don't know. "Sentience" doesn't matter one bit for any of this. Such a system might take centuries to form or it might go quickly. Humans might also go extinct before something like this comes about. However, that doesn't mean people who think about such possibilities are stupid.
deleted 3y ago
[deleted]
kibwen 3y ago
> What about people shouting about AGI from the halls of the most advanced research labs in the field? The most practiced researchers of alchemy were convinced that they could turn lead into gold. By itself, this argument is unconvincing. When it comes to the potential for fabulous wealth and/or unimaginable power, incentives distort and people are inclined to abandon their scruples.
- ethanbond 3y ago
  Many people who are actually making the things closest to what we currently call “AI” are concerned. I don’t care to engage in these nitpicky hypotheticals. We all know what I’m saying. If the scientists building these systems aren’t credible and the commentators not building these systems aren’t credible, then who is? If the answer is “no one,” okay, cool, that inscrutability is another reason for, not against, caution.
  - zer00eyz 3y ago
    >> Many people who are actually making the things closest to what we currently call “AI” are concerned. See grey goo: https://en.wikipedia.org/wiki/Gray_goo https://en.wikipedia.org/wiki/Gray_goo How about nukes and atmospheric fire: https://www.insidescience.org/manhattan-project-legacy/atmosphere-on-fire https://www.insidescience.org/manhattan-project-legacy/atmos... > scientists ... credible The science ends at the math. Go play with an LLM at home. Run it on your GPU, see what it does, what its limits are. Realize that it's a system that never "grows", that lacking noise it becomes deterministic. Once you look behind the curtain and find out what's there, a lot of the hype to call it AI looks dumb.
    - selfhoster11 3y ago
      I have run an LLM at home, up to 70B models. While I agree with some of what you said, I don't really know how it prevents LLMs from being useful. Yes, LLMs currently don't learn. This is also true for humans who cannot form new memories. A human who cannot learn anything new besides the contents of their short-term memory is at a huge disadvantage, but (hopefully) no less capable if given access to a scratchpad and more thinking time to compensate. As for the part that they are deterministic lacking noise... Why would you not provide it with noise? Generating randomness is something that modern machines are quite good at, so there isn't much of a reason not to inject it at inference time.
mlsu 3y ago
Those people (like most people who talk AGI) live in a world far "above the metal." They, like the rationalist bloggers, live in a "software world," far above the constraints of physical reality. This causes an understandable blind spot: the understanding that computers are, fundamentally, physical machines. It's quite easy to imagine that you could conjure superintelligence when you are conjuring a pod of 8 quadrillion transistors in a single shell script. The illusion breaks down when you get closer to the physical reality. Reality is that those transistors are made of real materials, with real impurities, that have to be broken down and painstakingly physically, electrically, chemically, debugged, in the world below the 1's and 0's abstraction. The commonality that both sides of this silly debate share is that neither really deals with entities at the layer below abstract software (or essays). To them, the perfect world of software is fundamental reality (they spend all of their time there!), and any crystal prism that can be constructed in software can be constructed in reality. Of course, such prisms are instantly shattered when they come into contact with the real world: an impurity in the wafer, a static shock on the PCB before installation, a flood in the datacenter. This software stuff is fragile by default, it actually takes a tremendous amount of active work in the physical reality to conjure it into existence. I can do something that no language model can do, and is showing no signs of being able to do any time soon: drive a car to the datacenter, turn the door handle, and unplug the rack.
- zrezzed 3y ago
  > turn the door handle, and unplug the rack. There are door handles that you cannot turn, and racks you cannot unplug: ones buried deep in the Cheyenne Mountains and hidden far away in the Siberian tundra. They are protected by unfathomably powerful systems, with the support of countless people and backed by the whims global economic power. I say this as someone who likely agrees with you. I think the power of the real world, that of companies, and governments and militaries should decrease our concern with AGI gaining power itself. But I don’t think it’s as obvious as pointing to the fragility of software. Our human systems are fragile too, and subject to manipulation in not-so-different a way as data centers. I think you should not be so quick to discount the voices of many smart people shouting.
  - 3y ago

What about people shouting about AGI from the halls of the most advanced research labs in the field?

lazide 3y ago

When has an ivory tower not involved the ‘most advanced research labs’? Or do you mean largest implementors?

ethanbond 3y ago
Ivory tower is usually a pejorative term meaning people who are isolated from the reality of the situation. So it doesn’t include the most advanced research labs when those research labs are in touch with the reality of the situation, like right now with AI. Nitpick “most advanced research labs” if you want, it’s obvious what I mean: it is clear that the “shouting” is not coming solely from people who don’t know what’s going on.
- lazide 3y ago
  Eh, from folks isolated from the constraints and practicalities of the real world. Which OpenAI definitely is, and has been for some time. Or do you think infinite money and ‘do whatever you want’ is constraining?
  - ethanbond 3y ago
    What does constraints have to do with it? The question is who has the best grasp on the trajectory of the technology. If not the researchers who are at the frontier of building and deploying these systems, playing with next generation iterations and planning several generations forward, then who? Andreessen? People who like playing with LLMs? People who called the OpenAI API?
    - lazide 3y ago
      Because if you don’t have real world constraints (like needing to be profitable, or paying the bill for thermodynamic reality), then ‘anything is possible’. Also if you aren’t dealing with those, then problems that come from that never occur to the person involved. So the concerns are abstract and not based on real limits or real problems. Hence ivory tower.
      - ethanbond 3y ago
        Okay so the server bills or thermodynamic bills mean we need not worry. Open to hearing the argument: tell me how!
        lazide 3y ago
        Not at all what I'm saying. What I'm saying is, no one knows where the line between actually economic/useful/effective and 'not worth the trouble' actually is, let alone 'could be done without boiling the oceans'. Right now it's all hand wavey, could do everything, etc. etc.
        ethanbond 3y ago
        That uncertainty is called risk! That is exactly the whole point!
        lazide 3y ago
        What does that have to do with this thread? You know the ivory tower one?

zer00eyz 3y ago

People who worry about AI taking over the world need to worry about the world continuing to work. Your average gas station gets gas 2x a day. The electric grid has 100's of faults that are worked around and fixed every day. Delivery still are loaded and driven by people. Any AGI that is a threat to humanity has to be suicidal to act on it because of how the world works. Furthermore, if you crack open an LLM and turn OFF the random noise (temperature) it becomes very deterministic. You think our accent to godhood is on the back of a vector map with some noise, I dont know what to tell you. That doesn't mean all this ML can't change the world, can't foster less BS jobs and enable creativity... but I have massive doubts that sentience and sapience are on this path.

ethanbond 3y ago
Sure hope that intelligence doesn’t roughly mean “ability to find solutions to problems that less intelligent entities can’t discover.” I have no idea what “our ascent to godhood” entails, and it’s hilarious that doomers get criticized for being religious about all this and then you’ll say something like that with a straight face. That’s neat that you “have doubts” sentience and sapience are on this path. You also don’t actually know.
staunton 3y ago
> Any AGI that is a threat to humanity has to be suicidal to act on it because of how the world works A "threat to humanity" need not mean "killing all humans next year". It could just mean some entity gaining permanent control. That situation would be irreversible and humanity would be at the mercy of that entity. Chances are, the entity would have emerged victorious from an evolutionary struggle for power and thus would not care about human flourishing. That entity need not be a single AI. It could be a system like a large corporation or a country where initially a lot of "small" decisions are made by AIs. Over time, the influence of humans on this system might diminish and the trend might become irreversible. Currently, no individual and no group, (organization, country...) on the planet has anywhere close to complete control over humanity. Further, even if some country managed to conquer everything, it could not hope to maintain that power indefinitely. An immortal system capable of complete surveillance, however, may be able to maintain power. It's a new thing, we don't know. "Sentience" doesn't matter one bit for any of this. Such a system might take centuries to form or it might go quickly. Humans might also go extinct before something like this comes about. However, that doesn't mean people who think about such possibilities are stupid.

deleted 3y ago

[deleted]

kibwen 3y ago

> What about people shouting about AGI from the halls of the most advanced research labs in the field? The most practiced researchers of alchemy were convinced that they could turn lead into gold. By itself, this argument is unconvincing. When it comes to the potential for fabulous wealth and/or unimaginable power, incentives distort and people are inclined to abandon their scruples.

ethanbond 3y ago
Many people who are actually making the things closest to what we currently call “AI” are concerned. I don’t care to engage in these nitpicky hypotheticals. We all know what I’m saying. If the scientists building these systems aren’t credible and the commentators not building these systems aren’t credible, then who is? If the answer is “no one,” okay, cool, that inscrutability is another reason for, not against, caution.
- zer00eyz 3y ago
  >> Many people who are actually making the things closest to what we currently call “AI” are concerned. See grey goo: https://en.wikipedia.org/wiki/Gray_goo https://en.wikipedia.org/wiki/Gray_goo How about nukes and atmospheric fire: https://www.insidescience.org/manhattan-project-legacy/atmosphere-on-fire https://www.insidescience.org/manhattan-project-legacy/atmos... > scientists ... credible The science ends at the math. Go play with an LLM at home. Run it on your GPU, see what it does, what its limits are. Realize that it's a system that never "grows", that lacking noise it becomes deterministic. Once you look behind the curtain and find out what's there, a lot of the hype to call it AI looks dumb.
  - selfhoster11 3y ago
    I have run an LLM at home, up to 70B models. While I agree with some of what you said, I don't really know how it prevents LLMs from being useful. Yes, LLMs currently don't learn. This is also true for humans who cannot form new memories. A human who cannot learn anything new besides the contents of their short-term memory is at a huge disadvantage, but (hopefully) no less capable if given access to a scratchpad and more thinking time to compensate. As for the part that they are deterministic lacking noise... Why would you not provide it with noise? Generating randomness is something that modern machines are quite good at, so there isn't much of a reason not to inject it at inference time.

mlsu 3y ago

Those people (like most people who talk AGI) live in a world far "above the metal." They, like the rationalist bloggers, live in a "software world," far above the constraints of physical reality. This causes an understandable blind spot: the understanding that computers are, fundamentally, physical machines. It's quite easy to imagine that you could conjure superintelligence when you are conjuring a pod of 8 quadrillion transistors in a single shell script. The illusion breaks down when you get closer to the physical reality. Reality is that those transistors are made of real materials, with real impurities, that have to be broken down and painstakingly physically, electrically, chemically, debugged, in the world below the 1's and 0's abstraction. The commonality that both sides of this silly debate share is that neither really deals with entities at the layer below abstract software (or essays). To them, the perfect world of software is fundamental reality (they spend all of their time there!), and any crystal prism that can be constructed in software can be constructed in reality. Of course, such prisms are instantly shattered when they come into contact with the real world: an impurity in the wafer, a static shock on the PCB before installation, a flood in the datacenter. This software stuff is fragile by default, it actually takes a tremendous amount of active work in the physical reality to conjure it into existence. I can do something that no language model can do, and is showing no signs of being able to do any time soon: drive a car to the datacenter, turn the door handle, and unplug the rack.

zrezzed 3y ago
> turn the door handle, and unplug the rack. There are door handles that you cannot turn, and racks you cannot unplug: ones buried deep in the Cheyenne Mountains and hidden far away in the Siberian tundra. They are protected by unfathomably powerful systems, with the support of countless people and backed by the whims global economic power. I say this as someone who likely agrees with you. I think the power of the real world, that of companies, and governments and militaries should decrease our concern with AGI gaining power itself. But I don’t think it’s as obvious as pointing to the fragility of software. Our human systems are fragile too, and subject to manipulation in not-so-different a way as data centers. I think you should not be so quick to discount the voices of many smart people shouting.
- 3y ago