3 ms·

The thing which these discussions leave out are the physical aspects: - if a computer system were able to design a better computer system, how much would it co

The thing which these discussions leave out are the physical aspects: - if a computer system were able to design a better computer system, how much would it cost to then manufacture said system? How much would it cost to build the fabrication facilities necessary to create this hypothetical better computer? - once this new computer is running, how much power does it require? What are the on-going costs to keep it running? What sort of financial planning and preparations are required to build the next generation device/replacement? I'd be satisfied with a Large-Language-Model which: - ran on local hardware - didn't have a marked affect on my power bill - had a fully documented provenance for _all_ of its training which didn't have copyright/licensing issues - was available under a license which would allow arbitrary use without on-going additional costs/issues - could actually do useful work reliably with minimal supervision

Animats 2y ago
- could actually do useful work reliably with minimal supervision That's the big problem. LLMs can't be allowed to do anything important without supervision. We're still at 5-10% totally bogus results.
- Terr_ 2y ago
  I think a deeper issue is that they are essentially "attempt to extend this document based on patterns in documents you've already seen" engines. Sometimes that's valuable and exactly what you need, but problems arise when people try to treat them as some sort of magical oracle that just needs to be primed with the right text. Even "conversational LLMs are just updating a theater-style script where one of the characters happens to be described as a computer.
jodrellblank 2y ago
Skip a few generations and the machine will build itself. There’s no need for it to take lasers exploding tin to generate ultra Violet to etch patterns to make intelligence, humans don’t grow brains that way or spend billions on fabs and power plants to produce children. How it gets from here to there is a handwave, though.
- rsanheim 2y ago
  That’s a pretty enormous handwave.
  - ben_w 2y ago
    If it wasn't, it would've already happened.
  - jodrellblank 2y ago
    It is, but it isn't once we're already handwaving "superintelligence". Problem with funding a new fab or new power stations? It can invent a thing worth a lot of money, or out-gamble, or swindle or steal it. Problem with getting things done? It can pay people lots of money to do them. That covers a huge number of cases of geting from "here" to "there" if such a path is possible at all.
- nonameiguess 2y ago
  There's no guarantee it's possible. Self-replicating molecules don't necessarily have to be hydrocarbons suspended in water inside of a semi-permeable lipid membrane, but that particular solution came about because of conditions that are needed: - Some multipolar molecules that could attract or repel each other and initiate both hydrogen bonds and covalent bonds. - Movement through some kind of medium not as chaotic as gas. - A boundary between self and everything else. Industrial processes to create machines don't lend themselves to this kind of bottom-up model as they now stand. It's not at all clear that etched silicon at all is the right medium of computation if we want it to be self-replicating. Coming up with something else is a hell of an ask and a hell of a handwave, though. It would also entail getting rid of the entire impetus of Moore's law and the reason futurolists like this ever thought there would be a singularity in the first place. Other than transistor density on a silicion die, I'm not sure any other technology has ever shown consistent long-term exponential growth at some reasonably fixed exponent.
nradov 2y ago
Right. In order to design a significantly better computer system, you first need to design a better (smaller feature size) EUV lithography process which can produce decent yield at scale.
bbor 2y ago
if a computer system were able to design a better computer system, how much would it cost to then manufacture said system? I think the implication is that the primary advancements would come in the form of software. IMO it's trivially true that we're not taking full advantage of the hardware we have from a software PoV -- if we were, we wouldn't need SWEs, right? From that it should follow that self-improving software is dangerously effective. once this new computer is running, how much power does it require? What are the on-going costs to keep it running? I mean, lots, sure. But we allocate immense resources to relatively trivial luxuries in this world; I don't think there's any reason to think we can't spare some giant computers to rapidly advance our technology. In a capitalist society, it's happily/sadly pretty much guaranteed that people will figure out how to get the resources there if scientists tell them the RoI is infinity+1. I'd be satisfied with a Large-Language-Model which Those are great asks and I agree, but just to be super clear in case its not: Vinge isn't talking about chatbots, he's talking about systems with many smaller specialized subsystems. In today's parlance, gaggle of "LLMs" equipped with "tool use", or in yesterday's parlance, a "Society of Mind".
deleted 2y ago
[deleted]
ben_w 2y ago
> if a computer system were able to design a better computer system, how much would it cost to then manufacture said system? How much would it cost to build the fabrication facilities necessary to create this hypothetical better computer? Most of the computers we use today were designed by software: Feature sizes are (and have been for some time) in the realm where the Schrödinger equation matters, and more compute makes it easier to design smaller feature sizes. Similar points apply to the question of cost: it has not been constant, the power to keep x-teraflops running has decreased* while the cost to develop the successor has increased. Regarding LLMs in particular, I believe there are already models meeting all but one of your criteria — though I would argue that the missing one, "could actually do useful work reliably with minimal supervision", is by far the most important. * If I read this chart right, my phone beats the combined top 500 supercomputers when the linked article was written by a factor of ten or so: https://commons.m.wikimedia.org/wiki/File:Supercomputers-history.svg https://commons.m.wikimedia.org/wiki/File:Supercomputers-his...

Animats 2y ago

- could actually do useful work reliably with minimal supervision That's the big problem. LLMs can't be allowed to do anything important without supervision. We're still at 5-10% totally bogus results.

Terr_ 2y ago
I think a deeper issue is that they are essentially "attempt to extend this document based on patterns in documents you've already seen" engines. Sometimes that's valuable and exactly what you need, but problems arise when people try to treat them as some sort of magical oracle that just needs to be primed with the right text. Even "conversational LLMs are just updating a theater-style script where one of the characters happens to be described as a computer.

jodrellblank 2y ago

Skip a few generations and the machine will build itself. There’s no need for it to take lasers exploding tin to generate ultra Violet to etch patterns to make intelligence, humans don’t grow brains that way or spend billions on fabs and power plants to produce children. How it gets from here to there is a handwave, though.

rsanheim 2y ago
That’s a pretty enormous handwave.
- ben_w 2y ago
  If it wasn't, it would've already happened.
- jodrellblank 2y ago
  It is, but it isn't once we're already handwaving "superintelligence". Problem with funding a new fab or new power stations? It can invent a thing worth a lot of money, or out-gamble, or swindle or steal it. Problem with getting things done? It can pay people lots of money to do them. That covers a huge number of cases of geting from "here" to "there" if such a path is possible at all.
nonameiguess 2y ago
There's no guarantee it's possible. Self-replicating molecules don't necessarily have to be hydrocarbons suspended in water inside of a semi-permeable lipid membrane, but that particular solution came about because of conditions that are needed: - Some multipolar molecules that could attract or repel each other and initiate both hydrogen bonds and covalent bonds. - Movement through some kind of medium not as chaotic as gas. - A boundary between self and everything else. Industrial processes to create machines don't lend themselves to this kind of bottom-up model as they now stand. It's not at all clear that etched silicon at all is the right medium of computation if we want it to be self-replicating. Coming up with something else is a hell of an ask and a hell of a handwave, though. It would also entail getting rid of the entire impetus of Moore's law and the reason futurolists like this ever thought there would be a singularity in the first place. Other than transistor density on a silicion die, I'm not sure any other technology has ever shown consistent long-term exponential growth at some reasonably fixed exponent.

nradov 2y ago

Right. In order to design a significantly better computer system, you first need to design a better (smaller feature size) EUV lithography process which can produce decent yield at scale.

bbor 2y ago

if a computer system were able to design a better computer system, how much would it cost to then manufacture said system? I think the implication is that the primary advancements would come in the form of software. IMO it's trivially true that we're not taking full advantage of the hardware we have from a software PoV -- if we were, we wouldn't need SWEs, right? From that it should follow that self-improving software is dangerously effective. once this new computer is running, how much power does it require? What are the on-going costs to keep it running? I mean, lots, sure. But we allocate immense resources to relatively trivial luxuries in this world; I don't think there's any reason to think we can't spare some giant computers to rapidly advance our technology. In a capitalist society, it's happily/sadly pretty much guaranteed that people will figure out how to get the resources there if scientists tell them the RoI is infinity+1. I'd be satisfied with a Large-Language-Model which Those are great asks and I agree, but just to be super clear in case its not: Vinge isn't talking about chatbots, he's talking about systems with many smaller specialized subsystems. In today's parlance, gaggle of "LLMs" equipped with "tool use", or in yesterday's parlance, a "Society of Mind".

deleted 2y ago

[deleted]

ben_w 2y ago

> if a computer system were able to design a better computer system, how much would it cost to then manufacture said system? How much would it cost to build the fabrication facilities necessary to create this hypothetical better computer? Most of the computers we use today were designed by software: Feature sizes are (and have been for some time) in the realm where the Schrödinger equation matters, and more compute makes it easier to design smaller feature sizes. Similar points apply to the question of cost: it has not been constant, the power to keep x-teraflops running has decreased* while the cost to develop the successor has increased. Regarding LLMs in particular, I believe there are already models meeting all but one of your criteria — though I would argue that the missing one, "could actually do useful work reliably with minimal supervision", is by far the most important. * If I read this chart right, my phone beats the combined top 500 supercomputers when the linked article was written by a factor of ten or so: https://commons.m.wikimedia.org/wiki/File:Supercomputers-history.svg https://commons.m.wikimedia.org/wiki/File:Supercomputers-his...