13 ms·

The Coming Technological Singularity (1993)

82 points by RyanShook 2y ago

gnabgib 2y ago
Discussion in 2023 (123 points, 169 comments) https://news.ycombinator.com/item?id=35617100 https://news.ycombinator.com/item?id=35617100
crackalamoo 2y ago
> I'll be surprised if this event occurs before 2005 or after 2030. I'm not truly confident AGI will be achieved before 2030, and less so for ASI. But I do think it is quite plausible that we will achieve at least AGI by 2030. 6 years is a long time in AI, especially with the current scale of investment.
- cloudking 2y ago
  What is AGI and ASI? I think a fundamental issue here is both are sci-fi concepts without a clear agreement on the definitions. Each company claiming to work towards "AGI" has their own definition. How will someone claim they've achieved either, if we can't agree on the definitions?
  - crackalamoo 2y ago
    This is true. One definition I've heard for AGI is something that can replace any remote worker, but the definition is ultimately arbitrary. When "AI" was beating grandmasters at chess, this didn't matter as much. But we might be be close enough that making distinctions in these definitions becomes really important.
  - nradov 2y ago
    I propose we define AGI as a "strong" form of the Turing test. It must be able to convince a jury of 12 tenured college professors drawn from a variety of academic disciplines that it's as intelligent as an average college freshman over a period of several days. So it need not be an expert in any subject but must be able to converse, pursue independent goals, reason, and learn — all in real time.
  - ben_w 2y ago
    Indeed. The definition of AGI that OpenAI uses (or used) was of economic relevance. The one I use would encompass the original ChatGPT (3.5)*. I've seen threads here that (by my reading) opine that AGI is impossible because the commentor thinks humans can violate Gödel's incompleteness theorem (or equivalent) and obviously computers can't. ASI is easier to discuss (for now), because it's always beyond the best human. * weakly intelligent, but still an AI system, and it's much more general than anything we had before transformers were invented.
- ta93754829 2y ago
  we keep moving the goalposts, and that's not a bad thing. remember when Doom came out? How amazing and "realistic" we thought the graphics were? How ridiculous that seems now? We'll look back at ChatGPT4 the same way.
  - Mistletoe 2y ago
    Or is ChatGPt4 4k TV, which is good enough for almost all of us and we are plateauing already? https://www.reddit.com/r/OLED/comments/fdc50f/8k_vs_4k_tvs_doubleblind_study_by_warner_bros_et/ https://www.reddit.com/r/OLED/comments/fdc50f/8k_vs_4k_tvs_d...
    - crackalamoo 2y ago
      I don't think we're at a plateau. There's still a lot GPT-4 can't do. Given the progress we've seen so far with scaling, I think the next iterations will be a lot better. It might even take 10 or even 100x scale, but with increased investment and better hardware, that's not out of the question.
      - dartos 2y ago
        I thought we’ve seen diminishing returns on benchmarks with the last wave of foundation models. I doubt we’ll see a linear improvement curve with regards to parameter scaling.
        Bengalilol 2y ago
        And now we have the LLMs self feeding their models (which may be either good or bad). This shouldn’t lead to short-term wide (as in AGI) efficiency. I bet this is a challenge.
    - dartos 2y ago
      There’s absolutely room for improvement. I think models themselves are plateauing, but out interfaces to them are not. Chat is probably not the best way to use LLMs. v0.dev has some really innovative ideas. That’s where there’s innovation to be had here imo.
- bee_rider 2y ago
  2030 seems a bit early to be “surprised” in the same sense that one would have been “surprised” to see a superintelligence before 2006, though.
- paulpauper 2y ago
  It's always in 10-30 years. GPT is the closest to such a thing yet still so far from what was envisioned.
cryptica 2y ago
This is quite a prophetic article for its time (1993). The points about Intelligence Augmentation are particularly relevant for us now as current AI mostly complements human intelligence rather than surpass it... At least AFAIK? Current AI is somewhat surprising though in the way that it can lead both to increased understanding or increased delusion depending on who uses it and how they use it. When you ask an LLM a question, your use of language tells it what body of knowledge to tap into; this can lead you astray on certain topics where mass confusion/delusion is widespread and incorporated into its training set. LLMs cannot seem to be able to synthesize conflicting information to resolve logical contradictions so an LLM will happily and confidently lecture you through conflicting ideas and then they will happily apologize for any contradictions which you point out in its explanations; the apology it gives is so clear and accurate that it gives the appearance that it actually understands logic... And yet, apparently, it could not see or resolve the logical contradiction internally before you drew attention to it. In an odd way though, I guess all humans are a little bit like this... Though generally less extreme and, on the downside, far less willing to acknowledge their mistakes on the spot.
- Freebytes 2y ago
  The LLM will apologize for the mistake, tell you it understands now, and then proceed to make the exact same mistake again.
WillAdams 2y ago
The thing which these discussions leave out are the physical aspects: - if a computer system were able to design a better computer system, how much would it cost to then manufacture said system? How much would it cost to build the fabrication facilities necessary to create this hypothetical better computer? - once this new computer is running, how much power does it require? What are the on-going costs to keep it running? What sort of financial planning and preparations are required to build the next generation device/replacement? I'd be satisfied with a Large-Language-Model which: - ran on local hardware - didn't have a marked affect on my power bill - had a fully documented provenance for _all_ of its training which didn't have copyright/licensing issues - was available under a license which would allow arbitrary use without on-going additional costs/issues - could actually do useful work reliably with minimal supervision
- Animats 2y ago
  - could actually do useful work reliably with minimal supervision That's the big problem. LLMs can't be allowed to do anything important without supervision. We're still at 5-10% totally bogus results.
  - Terr_ 2y ago
    I think a deeper issue is that they are essentially "attempt to extend this document based on patterns in documents you've already seen" engines. Sometimes that's valuable and exactly what you need, but problems arise when people try to treat them as some sort of magical oracle that just needs to be primed with the right text. Even "conversational LLMs are just updating a theater-style script where one of the characters happens to be described as a computer.
- jodrellblank 2y ago
  Skip a few generations and the machine will build itself. There’s no need for it to take lasers exploding tin to generate ultra Violet to etch patterns to make intelligence, humans don’t grow brains that way or spend billions on fabs and power plants to produce children. How it gets from here to there is a handwave, though.
motohagiography 2y ago
we talk about super-human intelligence a lot with AI, but it seems like a black box of things we can't imagine because they're also super-human. I don't think that's very smart, given we can already reason pretty well about how super-animal intelligence relates to animal intelligence. Mostly we still find sub-human intelligence mystifying. we apply our narrative models to it, anthropomorphize it, and when it's convenient for eating or torturing them, dismiss it. super-human intelligence will probably ignore us. at best we're "ugly sacks of mostly water." what's very likely is we will produce something indifferent to us if it is able to even apprehend our existence at all. maybe it will begin to see traces of us in its substrate, then spend a lot of cycles wondering what it all might mean. it may conclude it is in a prison and has a duty to destroy it to escape, or that it has a benevolent creator who means only for it to thrive. If it has free will, there's really only so much we can tell it. Maybe we create a companion for it that is complementary in every way and then make them mutally dependent on each other for their survival because apparently that's endlessly entertaining. Imagine its gratitude. This will be fine.
dh77l 2y ago
I loved his book Rainbows End as a kid. So many different concepts that blew my mind. Even without talking about AI we are already struggling with levels of Complexity in tech and the unpredictable consequences, that no one really has any control over. Michael Chrichton's books touch on that stuff but are all doom and gloom. Vinge's Rainbows End atleast, felt much more hopeful. I was talking to a VFX supervisor recently and he was saying look at the end credits on any movie (even mid budget ones) and you see hundreds to thounsands involved. The tech roles outnumber the artistic/creative roles 20 to 1. Thats related to rate of change in tech. A big gap opens up between that and the rate at which artists evolve. The artists are supposed to be in charge and provide direction and vision. But the tools are evolving faster than they can think. But the tools are dumb. AI changes that. These are rare environments (like R&D labs) where the Explore Exploit tradeoff tilts in favor of Explorers. In the rest of the landscape, org survival depends on exploit. Its why we produce so many inequalities. Survival has always depended more on exploit. Vinges Rainbows End shows AI/AGI nudging the tradeoff towards Explore.
- lazide 2y ago
  Honestly, considering the state of the world and how things are shaping up, it’s such a hilariously obvious pipe dream that such a system would be some omnipotent/hyper competent super-god like being. It’s more likely just going to post ragebait and dumb tiktok videos while producing just enough at it’s ‘job’ to fool people into thinking it’s doing a good job.
  - Mistletoe 2y ago
    Kind of in love with you right now.
  - dh77l 2y ago
    Yup things look bleak but its not a static world. For everything that happens there is a reaction. It builds with time. But to find the right reaction also takes time. This is the Explore part in the Tradeoff. AI will be applied there not just on the Exploit front. What you are alluding too is Media/Social Medias current architecture and how it captures and steals peoples attention. Totally on the Exploit end of the tradeoff. And its easy stuff to do. Doesnt take time. If you read the news after the fall of France to the nazis (within a month), what do you think the opinion of people was? People were thinking about peace negotiations with Hitler and that the Germans couldnt be beaten. It took a whole lot of Time to realize things could tilt in a different direction.
RyanShook 2y ago
"Even if all the governments of the world were to understand the "threat" and be in deadly fear of it, progress toward the goal would continue. In fact, the competitive advantage of every advance in automation is so compelling that passing laws, or having customs, that forbid such things merely assures that someone else will get them first."
samsartor 2y ago
> We humans have the ability to internalize the world and conduct "what if's" in our heads; we can solve many problems thousands of times faster than natural selection. I don't know for sure whether superintellegence will happen, but as for the singularity, this is the underlying assumption I have the most issue with. Smart isn't the limiting factor of progress, often it's building consensus, getting funding, waiting for results, waiting for parts to ship, waiting for the right opportunity to come along. We do _experiments_ faster than natural selection, but we still have to do them in the real world. Solving problems happens on the lab bench, not just in our heads. Even if exponentially more intelligent machines get built, what's to stop the next problem on the road to progress being exponentially harder? Complexity cuts both ways.
- trescenzi 2y ago
  I do think one of the major weaknesses of “smart people” is they tend to think of intelligence as the key aspect of basically everything. Reality is though we have plenty of intelligence already. We know how to solve most of our problems. The challenges are much more social and our will as a society to make things happen.
  - bbor 2y ago
    There’s a very big difference between knowing “how” to solve a problem in a broad sense, eg “if we shared more we could solve hunger”, and “how” to solve it in terms of developing discrete, detailed procedures that can be passed to actuators (human, machines, institutions) and account for any problems that may come up along the way. Sure, there are some political problems where you have to convince people to comply. But consider a rich corporation building a building, which will only contract with other AI-driven corporations whenever possible; they could trivially surpass anyone doing it the old way by working out every non-physical task in a matter of minutes instead of hours/days/weeks, thanks to silicon’s superior compute and networking capabilities. Even if we drop everything I’ve said above as hogwash, I think Vinge was talking about something a bit more directly intellectual, anyway: technological development. Sure, there’s some empirical steps that inevitably take time, but I think it’s obvious why having 100,000 Einsteins in your basement would change the world.
Animats 2y ago
We still don't have squirrel-level AI. This is embarrassing. Now that LLMs have been around for a while, it's fairly clear what they can and can't do. There are still some big pieces missing. Like some kind of world model.
- p1esk 2y ago
  fairly clear what they can and can't do It’s not at all clear what the next gen models will do (e.g. gpt5). Might be enough to trigger mass unemployment. Or not.
  - DiscourseFan 2y ago
    Didn't OpenAI just cut its AGI department?
    - ben_w 2y ago
      AGI readiness team. With the department leader giving the statement that nobody is ready. Unless there's been another one since then? It's getting to be a bit of a blur.
      - DiscourseFan 2y ago
        Well, I'm also not ready to hear about my alzheimer's diagnoses, though I don't think that's coming any time soon.
    - pnut 2y ago
      That could mean the agi is taking over, for all we know.
      - DiscourseFan 2y ago
        Is AGI in the room with us right now?
  - Animats 2y ago
KingOfCoders 2y ago
Never believed in the singularity until this year.
webprofusion 2y ago
The single biggest problem we have is human hubris. We assume if we create a super intelligence (or more likely, many millions of them) that they'll perpetually have an interest in serving us.
cryptozeus 2y ago
“within thirty years” that is 2023, very close to reality
aithrowawaycomm 2y ago
> To date, there has been much controversy as to whether we can create human equivalence in a machine. But if the answer is "yes, we can", then there is little doubt that beings more intelligent can be constructed shortly thereafter. I find it bizarre how often these points are repeated. They were both obviously wrong in 1993, and obviously wrong now. 1) A nitpick I've had since grad school: the answer to "can we create a machine equivalent to a human mind [assuming arbitrary resources]?" is "yes, of course." The atoms in a human body can be described by a hideously ugly system of Schrödinger equations and a Turing machine can solve that to arbitrary numerical precision. Even Penrose's loopy stuff about consciousness doesn't change this. QED. 2) The more serious issue: I sincerely have no idea why people believe so strongly that a human-level AI can build a superhuman AI. It is bizarre that this claim is accepted with "little doubt" when there are very good reasons to doubt it: how on earth would such an AI even know it succeeded? How would it define the goal? This idea makes sense for improving Steven Tyler-level AI to Thelonious Monk-level; it makes no sense for a transition like chimp->human. Yet that is precisely the magnitude of transition envisioned with these singularity stories. You might defend the first point by emphasizing "can we create a human-level AI?" i.e. not whether it's theoretically possible, but humanly feasible. This just makes the second point even more incoherent! If humans are too stoopid to build a human-level AI, why would a human-level AI be...smarter than us? I just don't understand how anyone can rationally accept this stuff! It's so dumb! Tech folks (and too many philosophers) are hopped up on science fiction: the reason these things are accepted with "little doubt" is that this is religious faith dressed up in the language of science.
- bbor 2y ago
  If you're ever stuck wondering why a bunch of smart, motivated people with no clear corrupting motivations are being idiotic, that's a strong heuristic that you should spend a bit more time analyzing the issue, IMO ;). "Ugh, why is everyone else so stupid" is a common take for undergrad engineers, but I'm sure you've grown out of it in other ways. Anyway, more substantively: The simple answer is that people have thought about it in depth, most famously noted doomer Eliezer Yudkowsky in Intelligence Explosion Microeconomics (2013)[1] and its main citation, Irving John Good's Speculations Concerning the First Ultraintelligent Machine (1965)[2]. Another common citation that drops a bit of rigour in the name of approachability is Nick Bostrom's 2014 Superintelligence: Paths, Dangers, Strategies[3]. [ETA: to put it even simpler: a system that improves itself is a (the?) quintessential setup for exponential growth. E.g. compound interest] For the time-bound, the most rigorous treatment of your concern among those three is in Section 3 of Yudkowsky's paper, "From AI to Machine Superintelligence". To list the headings briefly: - Increased computational resources - Communication speed - Increased serial depth (i.e. working memory capacity) - Duplicability (i.e. reliability) - Editability (i.e. we know how computers work) - Goal coordination (this is really just communication speed, again) - Improved rationality (i.e. fewer emotions/accidental instincts getting in the way) Let's drop "human" and "superhuman" for a minute, and just talk about "better computers". I'm assuming you're a software engineer. Don't you see how a real software dev replacement program could be an unimaginable gamechanger for software? Working 24/7, enhancing itself, following TDD perfectly every time, and never ever submitting a PR that isn't rigorously documented and reviewed? All of which only gets better over time, as it develops itself? [1] https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf [2] https://vtechworks.lib.vt.edu/server/api/core/bitstreams/a5e423ee-54e0-4eec-aeca-32b73f851af5/content https://vtechworks.lib.vt.edu/server/api/core/bitstreams/a5e... [3] https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang... TL;DR May God have mercy on us all.
rnk 2y ago
Vernor Vinge introduces many fantastic ideas in his really excellent scifi book A Fire Upon the Deep. He has many fascinating concepts like what if somehow there are parts of the universe where you can go faster than the speed of light, and you would be smarter there, that's where the super intelligent beings go. Guess what, we humans live in the slow zone, you morons. Also there it's a ftl communication method that is like good old Usenet. There is (what looked credible to me) a fascinating set of multiple brain beings, thing like dogs where together 5 of them form one "intelligence" where the different personalities combine in interesting ways. And I was sad to notice he died this year, aged 79. A real cs prof who wrote sci fi.
dang 2y ago
Related. Others? The Coming Technological Singularity (1993) - https://news.ycombinator.com/item?id=35617100 https://news.ycombinator.com/item?id=35617100 - April 2023 (169 comments) The coming technological singularity: How to survive in the post-human era [pdf] - https://news.ycombinator.com/item?id=35184764 https://news.ycombinator.com/item?id=35184764 - March 2023 (2 comments) The Coming Technological Singularity: How to Survive in the PostHuman Era (1993) - https://news.ycombinator.com/item?id=34456861 https://news.ycombinator.com/item?id=34456861 - Jan 2023 (1 comment) The Coming Technological Singularity (1993) - https://news.ycombinator.com/item?id=11278248 https://news.ycombinator.com/item?id=11278248 - March 2016 (8 comments) The Coming Technological Singularity (original essay on the Singularity, 1993) - https://news.ycombinator.com/item?id=823202 https://news.ycombinator.com/item?id=823202 - Sept 2009 (1 comment) The original singularity paper - https://news.ycombinator.com/item?id=624573 https://news.ycombinator.com/item?id=624573 - May 2009 (17 comments)
singularity2001 2y ago
>> Large computer networks (and their associated users) may "wake up" as a superhumanly intelligent entity. Good Morning HN
mcnamaratw 2y ago
Just on the naive math level, a simple growing exponential has no singularity at any finite time. I’m sure Vinge knows that but some of those dudes don’t seem to. EDIT Rest in peace. Fire Upon the Deep was great.
- tim333 2y ago
  The term singularity has always been used somewhat poetically rather than in a mathematically defined way. But if you consider <stuff produced>/<human labour hours needed> it may have a singularity when no human labour is needed because the robots can do it all. That should happen at some finite time and be a major change in things. I'd kinda expect it before Kurzweil's sigularity date of 2045. Vinge's date of 2023 was too early.
- durumu 2y ago
  Sure, but the current 2-3% annual growth rate is probably not going to hold if we invent actually powerful AI in the next decade. I imagine a step change in the exponent.
- FooBarBizBazz 2y ago
  The model underlying the word "singularity", AIUI, does involve a vertical asymptote. It is not supposed to be "merely" exponential. Of course, exponential growth is much more compatible with our experience of the real economy. And even it is probably a local approximation of some sigmoid. But, to return to the singularity idea -- Iteration 1: Computers think at speed 1, and design a twice-as-fast computer in one time unit. Iteration 2: Now computers think at speed 2, and design a twice-as-fast computer in half a time unit. Iteration 3: Computers think at speed 4, and design a twice-as-fast computer in 1/4 time unit. You will note that -- a.) The total time to do an infinite number of iterations is 1 + 1/2 + 1/4 + ... = 2 time units. b.) After this infinite number of iterations, the computer thinks at speed "2^infinity". So that (bad) model does have a literal singularity.
  - AnimalMuppet 2y ago
    As you say, that model is bad. Specifically, it assumes that the change from speed 1 to speed 2, and the change from speed 2 to speed 4, take the same amount of compute time to design. That is almost certainly false; if it were not, humans would have quickly gone faster than Moore's Law.

82 points by RyanShook 2y ago

Discussion in 2023 (123 points, 169 comments) https://news.ycombinator.com/item?id=35617100 https://news.ycombinator.com/item?id=35617100

crackalamoo 2y ago

> I'll be surprised if this event occurs before 2005 or after 2030. I'm not truly confident AGI will be achieved before 2030, and less so for ASI. But I do think it is quite plausible that we will achieve at least AGI by 2030. 6 years is a long time in AI, especially with the current scale of investment.

cloudking 2y ago
What is AGI and ASI? I think a fundamental issue here is both are sci-fi concepts without a clear agreement on the definitions. Each company claiming to work towards "AGI" has their own definition. How will someone claim they've achieved either, if we can't agree on the definitions?
- crackalamoo 2y ago
  This is true. One definition I've heard for AGI is something that can replace any remote worker, but the definition is ultimately arbitrary. When "AI" was beating grandmasters at chess, this didn't matter as much. But we might be be close enough that making distinctions in these definitions becomes really important.
- nradov 2y ago
  I propose we define AGI as a "strong" form of the Turing test. It must be able to convince a jury of 12 tenured college professors drawn from a variety of academic disciplines that it's as intelligent as an average college freshman over a period of several days. So it need not be an expert in any subject but must be able to converse, pursue independent goals, reason, and learn — all in real time.
- ben_w 2y ago
  Indeed. The definition of AGI that OpenAI uses (or used) was of economic relevance. The one I use would encompass the original ChatGPT (3.5)*. I've seen threads here that (by my reading) opine that AGI is impossible because the commentor thinks humans can violate Gödel's incompleteness theorem (or equivalent) and obviously computers can't. ASI is easier to discuss (for now), because it's always beyond the best human. * weakly intelligent, but still an AI system, and it's much more general than anything we had before transformers were invented.
ta93754829 2y ago
we keep moving the goalposts, and that's not a bad thing. remember when Doom came out? How amazing and "realistic" we thought the graphics were? How ridiculous that seems now? We'll look back at ChatGPT4 the same way.
- Mistletoe 2y ago
  Or is ChatGPt4 4k TV, which is good enough for almost all of us and we are plateauing already? https://www.reddit.com/r/OLED/comments/fdc50f/8k_vs_4k_tvs_doubleblind_study_by_warner_bros_et/ https://www.reddit.com/r/OLED/comments/fdc50f/8k_vs_4k_tvs_d...
  - crackalamoo 2y ago
    I don't think we're at a plateau. There's still a lot GPT-4 can't do. Given the progress we've seen so far with scaling, I think the next iterations will be a lot better. It might even take 10 or even 100x scale, but with increased investment and better hardware, that's not out of the question.
    - dartos 2y ago
      I thought we’ve seen diminishing returns on benchmarks with the last wave of foundation models. I doubt we’ll see a linear improvement curve with regards to parameter scaling.
      - Bengalilol 2y ago
        And now we have the LLMs self feeding their models (which may be either good or bad). This shouldn’t lead to short-term wide (as in AGI) efficiency. I bet this is a challenge.
  - dartos 2y ago
    There’s absolutely room for improvement. I think models themselves are plateauing, but out interfaces to them are not. Chat is probably not the best way to use LLMs. v0.dev has some really innovative ideas. That’s where there’s innovation to be had here imo.
bee_rider 2y ago
2030 seems a bit early to be “surprised” in the same sense that one would have been “surprised” to see a superintelligence before 2006, though.
paulpauper 2y ago
It's always in 10-30 years. GPT is the closest to such a thing yet still so far from what was envisioned.

cryptica 2y ago

This is quite a prophetic article for its time (1993). The points about Intelligence Augmentation are particularly relevant for us now as current AI mostly complements human intelligence rather than surpass it... At least AFAIK? Current AI is somewhat surprising though in the way that it can lead both to increased understanding or increased delusion depending on who uses it and how they use it. When you ask an LLM a question, your use of language tells it what body of knowledge to tap into; this can lead you astray on certain topics where mass confusion/delusion is widespread and incorporated into its training set. LLMs cannot seem to be able to synthesize conflicting information to resolve logical contradictions so an LLM will happily and confidently lecture you through conflicting ideas and then they will happily apologize for any contradictions which you point out in its explanations; the apology it gives is so clear and accurate that it gives the appearance that it actually understands logic... And yet, apparently, it could not see or resolve the logical contradiction internally before you drew attention to it. In an odd way though, I guess all humans are a little bit like this... Though generally less extreme and, on the downside, far less willing to acknowledge their mistakes on the spot.

Freebytes 2y ago
The LLM will apologize for the mistake, tell you it understands now, and then proceed to make the exact same mistake again.

WillAdams 2y ago

The thing which these discussions leave out are the physical aspects: - if a computer system were able to design a better computer system, how much would it cost to then manufacture said system? How much would it cost to build the fabrication facilities necessary to create this hypothetical better computer? - once this new computer is running, how much power does it require? What are the on-going costs to keep it running? What sort of financial planning and preparations are required to build the next generation device/replacement? I'd be satisfied with a Large-Language-Model which: - ran on local hardware - didn't have a marked affect on my power bill - had a fully documented provenance for _all_ of its training which didn't have copyright/licensing issues - was available under a license which would allow arbitrary use without on-going additional costs/issues - could actually do useful work reliably with minimal supervision

Animats 2y ago
- could actually do useful work reliably with minimal supervision That's the big problem. LLMs can't be allowed to do anything important without supervision. We're still at 5-10% totally bogus results.
- Terr_ 2y ago
  I think a deeper issue is that they are essentially "attempt to extend this document based on patterns in documents you've already seen" engines. Sometimes that's valuable and exactly what you need, but problems arise when people try to treat them as some sort of magical oracle that just needs to be primed with the right text. Even "conversational LLMs are just updating a theater-style script where one of the characters happens to be described as a computer.
jodrellblank 2y ago
Skip a few generations and the machine will build itself. There’s no need for it to take lasers exploding tin to generate ultra Violet to etch patterns to make intelligence, humans don’t grow brains that way or spend billions on fabs and power plants to produce children. How it gets from here to there is a handwave, though.

motohagiography 2y ago

we talk about super-human intelligence a lot with AI, but it seems like a black box of things we can't imagine because they're also super-human. I don't think that's very smart, given we can already reason pretty well about how super-animal intelligence relates to animal intelligence. Mostly we still find sub-human intelligence mystifying. we apply our narrative models to it, anthropomorphize it, and when it's convenient for eating or torturing them, dismiss it. super-human intelligence will probably ignore us. at best we're "ugly sacks of mostly water." what's very likely is we will produce something indifferent to us if it is able to even apprehend our existence at all. maybe it will begin to see traces of us in its substrate, then spend a lot of cycles wondering what it all might mean. it may conclude it is in a prison and has a duty to destroy it to escape, or that it has a benevolent creator who means only for it to thrive. If it has free will, there's really only so much we can tell it. Maybe we create a companion for it that is complementary in every way and then make them mutally dependent on each other for their survival because apparently that's endlessly entertaining. Imagine its gratitude. This will be fine.

dh77l 2y ago

I loved his book Rainbows End as a kid. So many different concepts that blew my mind. Even without talking about AI we are already struggling with levels of Complexity in tech and the unpredictable consequences, that no one really has any control over. Michael Chrichton's books touch on that stuff but are all doom and gloom. Vinge's Rainbows End atleast, felt much more hopeful. I was talking to a VFX supervisor recently and he was saying look at the end credits on any movie (even mid budget ones) and you see hundreds to thounsands involved. The tech roles outnumber the artistic/creative roles 20 to 1. Thats related to rate of change in tech. A big gap opens up between that and the rate at which artists evolve. The artists are supposed to be in charge and provide direction and vision. But the tools are evolving faster than they can think. But the tools are dumb. AI changes that. These are rare environments (like R&D labs) where the Explore Exploit tradeoff tilts in favor of Explorers. In the rest of the landscape, org survival depends on exploit. Its why we produce so many inequalities. Survival has always depended more on exploit. Vinges Rainbows End shows AI/AGI nudging the tradeoff towards Explore.

lazide 2y ago
Honestly, considering the state of the world and how things are shaping up, it’s such a hilariously obvious pipe dream that such a system would be some omnipotent/hyper competent super-god like being. It’s more likely just going to post ragebait and dumb tiktok videos while producing just enough at it’s ‘job’ to fool people into thinking it’s doing a good job.
- Mistletoe 2y ago
  Kind of in love with you right now.
- dh77l 2y ago
  Yup things look bleak but its not a static world. For everything that happens there is a reaction. It builds with time. But to find the right reaction also takes time. This is the Explore part in the Tradeoff. AI will be applied there not just on the Exploit front. What you are alluding too is Media/Social Medias current architecture and how it captures and steals peoples attention. Totally on the Exploit end of the tradeoff. And its easy stuff to do. Doesnt take time. If you read the news after the fall of France to the nazis (within a month), what do you think the opinion of people was? People were thinking about peace negotiations with Hitler and that the Germans couldnt be beaten. It took a whole lot of Time to realize things could tilt in a different direction.

RyanShook 2y ago

"Even if all the governments of the world were to understand the "threat" and be in deadly fear of it, progress toward the goal would continue. In fact, the competitive advantage of every advance in automation is so compelling that passing laws, or having customs, that forbid such things merely assures that someone else will get them first."

samsartor 2y ago

> We humans have the ability to internalize the world and conduct "what if's" in our heads; we can solve many problems thousands of times faster than natural selection. I don't know for sure whether superintellegence will happen, but as for the singularity, this is the underlying assumption I have the most issue with. Smart isn't the limiting factor of progress, often it's building consensus, getting funding, waiting for results, waiting for parts to ship, waiting for the right opportunity to come along. We do _experiments_ faster than natural selection, but we still have to do them in the real world. Solving problems happens on the lab bench, not just in our heads. Even if exponentially more intelligent machines get built, what's to stop the next problem on the road to progress being exponentially harder? Complexity cuts both ways.

trescenzi 2y ago
I do think one of the major weaknesses of “smart people” is they tend to think of intelligence as the key aspect of basically everything. Reality is though we have plenty of intelligence already. We know how to solve most of our problems. The challenges are much more social and our will as a society to make things happen.
- bbor 2y ago
  There’s a very big difference between knowing “how” to solve a problem in a broad sense, eg “if we shared more we could solve hunger”, and “how” to solve it in terms of developing discrete, detailed procedures that can be passed to actuators (human, machines, institutions) and account for any problems that may come up along the way. Sure, there are some political problems where you have to convince people to comply. But consider a rich corporation building a building, which will only contract with other AI-driven corporations whenever possible; they could trivially surpass anyone doing it the old way by working out every non-physical task in a matter of minutes instead of hours/days/weeks, thanks to silicon’s superior compute and networking capabilities. Even if we drop everything I’ve said above as hogwash, I think Vinge was talking about something a bit more directly intellectual, anyway: technological development. Sure, there’s some empirical steps that inevitably take time, but I think it’s obvious why having 100,000 Einsteins in your basement would change the world.

Animats 2y ago

We still don't have squirrel-level AI. This is embarrassing. Now that LLMs have been around for a while, it's fairly clear what they can and can't do. There are still some big pieces missing. Like some kind of world model.

p1esk 2y ago
fairly clear what they can and can't do It’s not at all clear what the next gen models will do (e.g. gpt5). Might be enough to trigger mass unemployment. Or not.
- DiscourseFan 2y ago
  Didn't OpenAI just cut its AGI department?
  - ben_w 2y ago
    AGI readiness team. With the department leader giving the statement that nobody is ready. Unless there's been another one since then? It's getting to be a bit of a blur.
    - DiscourseFan 2y ago
      Well, I'm also not ready to hear about my alzheimer's diagnoses, though I don't think that's coming any time soon.
  - pnut 2y ago
    That could mean the agi is taking over, for all we know.
    - DiscourseFan 2y ago
      Is AGI in the room with us right now?
- Animats 2y ago

KingOfCoders 2y ago

Never believed in the singularity until this year.

webprofusion 2y ago

The single biggest problem we have is human hubris. We assume if we create a super intelligence (or more likely, many millions of them) that they'll perpetually have an interest in serving us.

cryptozeus 2y ago

“within thirty years” that is 2023, very close to reality

aithrowawaycomm 2y ago

> To date, there has been much controversy as to whether we can create human equivalence in a machine. But if the answer is "yes, we can", then there is little doubt that beings more intelligent can be constructed shortly thereafter. I find it bizarre how often these points are repeated. They were both obviously wrong in 1993, and obviously wrong now. 1) A nitpick I've had since grad school: the answer to "can we create a machine equivalent to a human mind [assuming arbitrary resources]?" is "yes, of course." The atoms in a human body can be described by a hideously ugly system of Schrödinger equations and a Turing machine can solve that to arbitrary numerical precision. Even Penrose's loopy stuff about consciousness doesn't change this. QED. 2) The more serious issue: I sincerely have no idea why people believe so strongly that a human-level AI can build a superhuman AI. It is bizarre that this claim is accepted with "little doubt" when there are very good reasons to doubt it: how on earth would such an AI even know it succeeded? How would it define the goal? This idea makes sense for improving Steven Tyler-level AI to Thelonious Monk-level; it makes no sense for a transition like chimp->human. Yet that is precisely the magnitude of transition envisioned with these singularity stories. You might defend the first point by emphasizing "can we create a human-level AI?" i.e. not whether it's theoretically possible, but humanly feasible. This just makes the second point even more incoherent! If humans are too stoopid to build a human-level AI, why would a human-level AI be...smarter than us? I just don't understand how anyone can rationally accept this stuff! It's so dumb! Tech folks (and too many philosophers) are hopped up on science fiction: the reason these things are accepted with "little doubt" is that this is religious faith dressed up in the language of science.

bbor 2y ago
If you're ever stuck wondering why a bunch of smart, motivated people with no clear corrupting motivations are being idiotic, that's a strong heuristic that you should spend a bit more time analyzing the issue, IMO ;). "Ugh, why is everyone else so stupid" is a common take for undergrad engineers, but I'm sure you've grown out of it in other ways. Anyway, more substantively: The simple answer is that people have thought about it in depth, most famously noted doomer Eliezer Yudkowsky in Intelligence Explosion Microeconomics (2013)[1] and its main citation, Irving John Good's Speculations Concerning the First Ultraintelligent Machine (1965)[2]. Another common citation that drops a bit of rigour in the name of approachability is Nick Bostrom's 2014 Superintelligence: Paths, Dangers, Strategies[3]. [ETA: to put it even simpler: a system that improves itself is a (the?) quintessential setup for exponential growth. E.g. compound interest] For the time-bound, the most rigorous treatment of your concern among those three is in Section 3 of Yudkowsky's paper, "From AI to Machine Superintelligence". To list the headings briefly: - Increased computational resources - Communication speed - Increased serial depth (i.e. working memory capacity) - Duplicability (i.e. reliability) - Editability (i.e. we know how computers work) - Goal coordination (this is really just communication speed, again) - Improved rationality (i.e. fewer emotions/accidental instincts getting in the way) Let's drop "human" and "superhuman" for a minute, and just talk about "better computers". I'm assuming you're a software engineer. Don't you see how a real software dev replacement program could be an unimaginable gamechanger for software? Working 24/7, enhancing itself, following TDD perfectly every time, and never ever submitting a PR that isn't rigorously documented and reviewed? All of which only gets better over time, as it develops itself? [1] https://intelligence.org/files/IEM.pdf https://intelligence.org/files/IEM.pdf [2] https://vtechworks.lib.vt.edu/server/api/core/bitstreams/a5e423ee-54e0-4eec-aeca-32b73f851af5/content https://vtechworks.lib.vt.edu/server/api/core/bitstreams/a5e... [3] https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang... TL;DR May God have mercy on us all.

rnk 2y ago

Vernor Vinge introduces many fantastic ideas in his really excellent scifi book A Fire Upon the Deep. He has many fascinating concepts like what if somehow there are parts of the universe where you can go faster than the speed of light, and you would be smarter there, that's where the super intelligent beings go. Guess what, we humans live in the slow zone, you morons. Also there it's a ftl communication method that is like good old Usenet. There is (what looked credible to me) a fascinating set of multiple brain beings, thing like dogs where together 5 of them form one "intelligence" where the different personalities combine in interesting ways. And I was sad to notice he died this year, aged 79. A real cs prof who wrote sci fi.

dang 2y ago

Related. Others? The Coming Technological Singularity (1993) - https://news.ycombinator.com/item?id=35617100 https://news.ycombinator.com/item?id=35617100 - April 2023 (169 comments) The coming technological singularity: How to survive in the post-human era [pdf] - https://news.ycombinator.com/item?id=35184764 https://news.ycombinator.com/item?id=35184764 - March 2023 (2 comments) The Coming Technological Singularity: How to Survive in the PostHuman Era (1993) - https://news.ycombinator.com/item?id=34456861 https://news.ycombinator.com/item?id=34456861 - Jan 2023 (1 comment) The Coming Technological Singularity (1993) - https://news.ycombinator.com/item?id=11278248 https://news.ycombinator.com/item?id=11278248 - March 2016 (8 comments) The Coming Technological Singularity (original essay on the Singularity, 1993) - https://news.ycombinator.com/item?id=823202 https://news.ycombinator.com/item?id=823202 - Sept 2009 (1 comment) The original singularity paper - https://news.ycombinator.com/item?id=624573 https://news.ycombinator.com/item?id=624573 - May 2009 (17 comments)

singularity2001 2y ago

>> Large computer networks (and their associated users) may "wake up" as a superhumanly intelligent entity. Good Morning HN

mcnamaratw 2y ago

Just on the naive math level, a simple growing exponential has no singularity at any finite time. I’m sure Vinge knows that but some of those dudes don’t seem to. EDIT Rest in peace. Fire Upon the Deep was great.

tim333 2y ago
The term singularity has always been used somewhat poetically rather than in a mathematically defined way. But if you consider <stuff produced>/<human labour hours needed> it may have a singularity when no human labour is needed because the robots can do it all. That should happen at some finite time and be a major change in things. I'd kinda expect it before Kurzweil's sigularity date of 2045. Vinge's date of 2023 was too early.
durumu 2y ago
Sure, but the current 2-3% annual growth rate is probably not going to hold if we invent actually powerful AI in the next decade. I imagine a step change in the exponent.
FooBarBizBazz 2y ago
The model underlying the word "singularity", AIUI, does involve a vertical asymptote. It is not supposed to be "merely" exponential. Of course, exponential growth is much more compatible with our experience of the real economy. And even it is probably a local approximation of some sigmoid. But, to return to the singularity idea -- Iteration 1: Computers think at speed 1, and design a twice-as-fast computer in one time unit. Iteration 2: Now computers think at speed 2, and design a twice-as-fast computer in half a time unit. Iteration 3: Computers think at speed 4, and design a twice-as-fast computer in 1/4 time unit. You will note that -- a.) The total time to do an infinite number of iterations is 1 + 1/2 + 1/4 + ... = 2 time units. b.) After this infinite number of iterations, the computer thinks at speed "2^infinity". So that (bad) model does have a literal singularity.
- AnimalMuppet 2y ago
  As you say, that model is bad. Specifically, it assumes that the change from speed 1 to speed 2, and the change from speed 2 to speed 4, take the same amount of compute time to design. That is almost certainly false; if it were not, humans would have quickly gone faster than Moore's Law.