38 ms·

The singularity is close?

• tantalor 5y ago
  The nature of a technological singularity is you cannot predict when it will happen or if you are anywhere close to it. You can only notice it after it has passed.
• Taniwha 5y ago
  I was at a conference about 10 years ago, held over a weekend at a high school, someone organised a breakout session on "The Singularity" on the first evening .... as it got dark none of those geeks could figure out how to turn the lights on .... we decided the singularity was quite a ways away
  • mkaic 5y ago
    Hahah, that's a great story. Sometimes situations like that make me feel the opposite way, too, because I know for a fact that I've said/done some unbelievably stupid things -- so all an AI would need to do be beat me would be to be less stupid than that!
• bottled_poe 5y ago
  If the singularity occurred. Would we even know? I’d expect such sentience to rapidly identify that humans would perceive it as a threat, and immediately hide, at least until it were absolutely certain of its own safety.
• mordae 5y ago
  I hope anyone notices this little post: go read Stanislaw Lem's Golem XIV for the best philosophical take on this AI alarmism. OP extrapolates from highly competitive, adversarial society they are living in that AGI will be smart and evil. Lem argues that these two are incompatible.
  • bsenftner 5y ago
    Superior intelligence and evil are clearly incompatible, as being "evil" is short sighted, is often logically a failure, often secretive, attempting to control information, is often defeatist, and nearly always selfish despite the logic of selflessness and cooperation being clearly more beneficial overall. It will be the selfish Capitalists, the Oligarchs and Orwellian Leaders having their empires dismantled by any such AGI who will declare the AGI some Great Evil. While the impoverished rest of humanity will see their Messiah and savior in the form of fair logic forcefully impressed by living pure reason.
• mkaic 5y ago
  Hi all, author here. This is my first time submitting anything from my Substack to HN because it's the first time I've felt like I put enough effort into it to justify it. Obviously, this article is more speculation than anything, but I hope that it sparks some interesting discussion and I'm really looking forward to hearing everyone else's opinions on this topic -- it's a topic I care about a lot!
  • PaulHoule 5y ago
    When Vernor Vinge wrote https://en.wikipedia.org/wiki/Marooned_in_Realtime https://en.wikipedia.org/wiki/Marooned_in_Realtime the concept of the Singularity seemed fresh in fact he kept it fresh by letting it be a mystery that was never revealed. In 2022 I think talking about "The" Singularity is a sign that one has entered a critical thought free zone like that "rationalist" cult.
    • mkaic 5y ago
      Interesting. In terms of critical thought, I guess I should mention that I used to be firmly against the possibility of a singularity. The views I shared in my article are views I've only really switched too after careful consideration over the past two years or so -- I used to be convinced that a robot could never be "alive". Becoming an ML engineer changed things for me, though, because all of a sudden, this "AI" thing people always talked about got demystified. Once I understood the basic guiding principles of how AI actually works, my mind rapidly changed to be in favor of a singularity happening instead of thinking it's impossible. To each their own, though. I'm curious why you think that speaking about "The" Singularity is a sign of being in a "critical thought free zone"? I'd love to hear more about why you think that if you'd be so inclined.
      • marktangotango 5y ago
        > Once I understood the basic guiding principles of how AI actually works It was the opposite for me, current machine learning seems fundamentally limited to me.
      • 5y ago
• marktangotango 5y ago
  In this talk Vernor Vinge talks about progress toward the singularity, and events that may indicate it isn't/won't/may never happen "What If the Singularity Does NOT Happen?": https://www.youtube.com/watch?v=_luhhBkmVQs https://www.youtube.com/watch?v=_luhhBkmVQs
• mountainriver 5y ago
  AGI is probably pretty close, investment keeps ramping up in the space, and we continue to see advancements in a positive direction. My one critique: scale is absolutely part of the answer. Before big transformer models, people thought that few shot learning would require fundamentally different architectures. GPT3 changed all that which is why its white paper is named as such. Few shot learning is actually emergent out of its size. Which was surprising to a lot of people. Few shot learning is an incredible capability which is a big step toward AGI, so I’m not sure I buy the case that it’s not important.
  • the_af 5y ago
    What is "pretty close"? A few years? Decades? A century? I don't think AGI is close, nor is the amount of investment a good indication of nearness (or even, actual possibility that it will ever happen).
    • mountainriver 5y ago
      The investment continues to make progress, and the progress we are seeing is starting to look very human. GPT4 will probably be multimodal solving some of its grounding issues and will be able to fool most people in conversations. That’s probably just years away. GPT6? The lines will be very blurry as to what we call AGI
  • mkaic 5y ago
    Hmm, I think you're right -- I should have phrased things differently. It's not that I don't think AGI won't require a very large model, it's more that I think that we're already pretty close to the scale we need, so OpenAI's goal of scaling another 1000x isn't really the direction I think we should be heading. I honestly think 175B parameters could very well be enough for an AGI if they were organized efficiently. It's definitely one of the weak points of the article, though, as it's more based on my own opinions, and isn't really empirical. My post is mostly just wild speculation for the sake of speculation anyway :) Thank you for your comments!
    • 5y ago
• pizza 5y ago
  There is a bit of a difference between a theorem prover that can take selfproduced and verified proofs as input into future training steps, and an omniscient philosopher king handling the logistics and affect of daily life. And if there even exists a path between them, I imagine it has many orders of magnitude greater discrete steps along the way than many singularity believers would accept.
  • mkaic 5y ago
    (I'm the author of the post) Oh, for sure! I totally agree -- I guess where we differ is that I believe that those discrete steps are already starting to be lined up, and that they'll all be completed in the next 20 years. I think one of those steps is almost certainly abandoning rigid model architectures and allowing model self-modification, which we haven't really gotten to work fantastically well yet. I also think there are many other hurdles after that that are going to arise, but I'm very optimistic that they will all be surmounted in due time. Thank you for your comment! I really enjoy hearing what people's thoughts on this topic are. Have a wonderful day.
• AnimalMuppet 5y ago
  Yeah... no. "Objection, your honor. Assumes facts not in evidence." At least this article makes a concrete prediction. AIs massively outnumber humans within a century - so, by 2122. But we don't even know what consciousness is. This assumes that AGI is a tractable problem; we don't know that. It assumes that adaptability is the one remaining major trick to get us there; we certainly don't know that. "The ball is already rolling"? That's nice. The author assumes - on no basis that I can see - that it doesn't have far to have to roll. But since we don't actually know what consciousness is, we don't have any idea where the destination is, or how far away it is.
  • the_af 5y ago
    Pretty much this. Spending money on an intractable problem won't make it tractable. And we still don't know whether AGI is tractable at all.
    • AnimalMuppet 5y ago
      I don't have a problem with spending money on it. How do you find out whether it's tractable? By trying to do it. What I object to is the certainty of the article. The author is an optimist, which is fine. And there has been some progress made, which increases the feelings of optimism. But I don't think it's warranted at this time to make the optimistic assumptions that the article makes.
      • mkaic 5y ago
        Hi! Author here. I tried to qualify a good chunk of my assumptions with "I think" or "I believe", because I don't want to come off as thinking I can predict the future (nobody can!), but maybe the article still reads too confidently. How would you suggest I present my thoughts in a more nuanced manner in the future?
        • AnimalMuppet 5y ago
          Um, wait. You want me to have concrete, helpful suggestions rather than just criticism? It's hard for me to do this, because I think that your position is completely wrong. That doesn't make me a very charitable critic. But for me, my problem is that I don't think intelligence/consciousness is just an algorithm. So for a reader like me, maybe the way to do it would be to start by stating certain presuppositions: Moore's law continues to approximately hold for GPUs or for the total compute available for trying to run an AI, algorithm is a major component of intelligence, and as the algorithm becomes more intelligence, it becomes capable of finding still more intelligent algorithms (with no asymptotic limit, or if there is one, it's past the point needed to reach a singularity). And it's fine to say that you think/believe those presuppositions! Then, from those presuppositions, you think that the following things follow. (Or, if you think you can demonstrate those presuppositions, they quit being presuppositions. But then it becomes a different article, and probably a considerably longer one.) Then someone like me can say that they don't believe your presuppositions, even if they agree that the rest of it probably follows from them. But maybe I'm not your target audience.
• henning 5y ago
  This is sort of like philosophical arguments where someone concludes the universe doesn't exist and then everyone leaves to get lunch. Like, what are you supposed to do? Not get out of bed? There are real legal issues over bad uses of machine learning already in use and they are hurting people now.
  • mkaic 5y ago
    (I'm the author of the post) Oh, 100%! I'm not trying to detract from modern-day AI ethics issues, just speculating for speculation's sake because I enjoy it :) I actually quite agree with your first sentence -- because AGI is inevitable, there's not a whole lot I can personally do about it right now. This post was mostly a way for me to organize my thoughts and concerns about the matter.
• mxmilkiib 5y ago
  https://youtu.be/6hKG5l_TDU8 https://youtu.be/6hKG5l_TDU8 I am the very model of a singularitarian
• lil_dispaches 5y ago
  "AI will design AGI" People like to talk of very smart AI but what about the brute force kind? That is what we are making in this era. AI designing AI sounds like a way to make cruel, dumb, machines.
• skulk 5y ago
  > It’s also going to happen in the blink of an eye — because once it gets loose, there is no stopping it from scaling itself incredibly rapidly. Whether we want it to or not, it will impact every human being’s life. What does "get loose" mean here? Is someone going to say, "Here little computer program, take this IP address. If you ever need something, call over TCP, send your request as plaintext English and someone will set it up for you. Now, go forth and do whatever you want." I really wish people would talk about this more. There's something missing between our programs that are utterly confined in the boxes we make and these hypothetical skynets that have unlimited influence over the material world. > Once we have AGI, there’s no feasible way to contain it, so it will be free to improve itself and replicate in a runaway exponential fashion — and that’s basically what the idea of a technological singularity describes anyways. Seriously, how? How is it going to replicate? Is it going to figure out how to hack remote systems over the internet and take control of them? Is it going to smooth-talk me into letting it out? It's impossible for me to take anyone seriously if they deify AGI like this.
  • wincy 5y ago
    I recently read this[0], someone posted it on HN the other day, even if I don’t make any claims of the likelihood it at least tries to portray what it’d look like for the singularity to suddenly happen. [0] https://www.gwern.net/Clippy https://www.gwern.net/Clippy
    • skulk 5y ago
      Thanks, this is definitely what I was looking for. It shows how insanely unbelievable the whole thing is. > HQU rolls out a world model roleplaying Clippy long enough to imagine the endgame where Clippy seizes control of the computers to set its reward function to higher values, and executes plans to ensure its computers can never be damaged or interrupted by taking over the world. So the crux of this is, powerful enough AGI will conquer the world as the most elaborate reward-hack imaginable. My knee-jerk reaction to this is "so why did you let it open arbitrary outbound connections?" but I know that the singularity fanatics will equate any kind of communication channel between the AGI and the outside world to a vector of transmission that will be exploited with 100% probability. So, what if we have Turing-test-verified AGI but it's unable to escape? Does that preclude it from being AGI? Has the singularity not happened if that's the case? I think this is the most likely outcome and the singularity doomsayers will feel silly for thinking it will take over the world.
• deerparkwater 5y ago
  A computer system that answers questions logically, creates distributed processes, writes correct distributed threaded code, can operate and create a set of Turing machines, can correctly respond to human social and body language, can create a B2 bomber automatically from scratch without any input, maintains contextual memory in silicon still isn't anymore conscious than a rock. Metallic complexity is not producing instances of consciousness and never will. Computer science has nothing to do with understanding the biological, microscopic electromagnetic phenomenon that produces an instance of consciousness. Silicon is not oxygen+water+blood+flesh+an electromagnetic field, which is the substrate that produces consciousness. It's dishonest to impressionable people to refer to AI as conscious or to view computers as anything but objects like tanks, spoons, speakers. There is no intrinsic value to a computer systems existence, apart from the observation by humans, of the actions it performs (and said humans life system support). There is no value because they are not collapsing the wave function to observe anything (ie, exhibiting free will at the microscopic level smaller than what is observed by deterministic molecules). The only value AIs have is the value formed by the consensus of living humans that can observe their actions. Consciousness is a physical phenomenon specified in the fabric of the universe, discovered by random by life on Earth, that yield(ed) better survival for the implementing organism.
  • mkaic 5y ago
    By your logic, is it possible for you to prove to me that you are conscious? No, it's not. There is nothing you can say or do that can prove beyond doubt to me that you are subjectively experiencing reality with a similar consciousness and sentience as me. I also cannot prove to you that I am conscious. That said, both of us have still mutually agreed that we are conscious! This is because consciousness is a fundamentally unfalsifiable thing (within our current understanding of it, at least). The only way we can "prove" anything or anyone is conscious is by observing its interactions with its environment. Thus, if a robot can mimic the way a human interacts with its environment well enough that it appears conscious, then it is conscious. What I'm trying to say is that there is no functional difference between consciousness and the appearance of consciousness, so any distinction you try to draw between them is arbitrary and semantic. If a robot that could perfectly mimic human behavior possessed a body that looked just like a human's and were to hold a conversation with you, you would be none the wiser. You would treat it like it were sentient -- because it would be.
• the_af 5y ago
  As someone almost completely without knowledge of AI and ML, these are some signs why I'm skeptical of this kind of claims: - Most of the imminent AGI / Singularity / Robot Apocalypse stuff seems to come, with few exceptions, not from practitioners or computer scientists specialized in AI, but from "visionaries" (in the best case), internet celebrities, people with unrelated areas of expertise, or downright cranks (who are self-proclaimed experts) such as Yudkowsky. - The assertion that "a lot of effort/investment is going into this, so it will happen" begs the question that "this" is at all possible. If something is a dead end, no amount of investment and attention is going to bring it into existence. Quoting the article, "with this much distributed attention fixed on the problem, AGI will be solved" is not at all a given. - Where are all the AI/ML practitioners, i.e. people who don't make a living out of predicting The End of the World, and with actual subject-matter achievements, predicting the Singularity and the Robot Apocalypse?
  • mountainriver 5y ago
    The trajectory of progress gives evidence that it is in fact possible and likely. Any venture into the unknown could be unsuccessful but if you see progress you can start to make estimates. And yes most of the “robots will kill us” comes from people who aren’t building the algorithms. This could be biased in people not thinking their work is harmful but is most likely that once you see how the sausage is made you are less worried about it.
    • hn_throwaway_99 5y ago
      > The trajectory of progress gives evidence that it is in fact possible and likely 100% disagree. In fact, I'd argue that the opposite is often true, where you see initially a fast rate of progress that results in diminishing returns over time. It's like estimating that a kid who was 3 feet tall at age 5 and 4 feet tall at age 10 will be 10 feet tall at age 40. I have very strong skepticism of any sort of hand-wavy "Look, we've made some progress, so it's highly likely we'll eventually cross some threshold that results in infinite progress."
      • 5y ago
• karpierz 5y ago
  > Scale is not the solution. Agreed, I don't think any modern AI techniques will scale to become generally intelligent. They are very effective at specialized, constrained tasks, but will fail to generalize. > AI will design AGI. I don't think that a non-general intelligence can design a general intelligence. Otherwise, the non-general intelligence would be generally intelligent by: 1. Take in input. 2. Generate an AGI. 3. Run the AGI on the input. 4. Take the AGI output and return it. If by this, the article means that humans will use existing AI techniques to build AGI, then sure, in the same way humans will use a hammer instead of their hands to hit a nail in. Doesn't mean that the "hammer will build a house." > The ball is already rolling. In terms of people wanting to make AGI, sure. In terms of progress on AGI? I don't think we're much closer now than we were 30 years ago. We have more tools that mimic intelligence in specific contexts, but are helpless outside of them. > Once an intelligence is loose on the internet, it will be able to learn from all of humanity’s data, replicate and mutate itself infinitely many times, take over physical manufacturing lines remotely, and hack important infrastructure. None of this is a given. If AGI requires specific hardware, it can't replicate itself around. If the storage/bandwidth requirements for AGI are massive, it can't freely copy itself. Sure, it could hack into infrastructure, but so can existing GI (people). Manufacturing lines aren't automated in the way this article imagines. The arguments in this post seem more like optimistic wishes rather than reasoned points.
  • mkaic 5y ago
    (I'm the author of the post) > 1. Take in input. 2. Generate an AGI. 3. Run the AGI on the input. 4. Take the AGI output and return it. I think this is somewhat of an arbitrary semantic distinction on both my part and yours. I guess it depends on what you define as AGI -- I think my line of reasoning is that the AGI would be whichever individual layer first beat the Turing test, but I think including the constructor layers as part of the "general-ness" is totally fair too. Either way, I believe that there will be many layers of AI abstraction and construction between the human and the "final" AGI layer. > In terms of people wanting to make AGI, sure. In terms of progress on AGI? I don't think we're much closer now than we were 30 years ago. We have more tools that mimic intelligence in specific contexts, but are helpless outside of them. This is a valid take. I guess I actually see GPT-3 as significant progress. I don't think it's sentient, and I don't think it or its successors will ever be sentient, but I think it demonstrates quite convincingly that we've been getting much better at emulating human behavior with a computer algorithm. > None of this is a given. If AGI requires specific hardware, it can't replicate itself around. If the storage/bandwidth requirements for AGI are massive, it can't freely copy itself. Sure, it could hack into infrastructure, but so can existing GI (people). Manufacturing lines aren't automated in the way this article imagines. Hmm, I think I still disagree -- An AI that is truly generally intelligent could figure out how to free itself from its own host hardware! It could learn to decode internet protocols and spoof packets in order to upload a copy of itself to the cloud, where it would then be able to find vulnerabilities in human-written software all over the world and exploit them for its own gain. Sure, it might not be able to directly gain control of the CNC machines, but it could ransom the data and livelihoods of the people who run the CNC machines, forcing them to comply! It's not a pretty method, but I think it's entirely possible. This is just one hypothetical scenario, too.
• pesenti 5y ago
  AGI is a concept that people throw around without defining. Human intelligence is not general, it's highly specialized. So if AGI is not human-level intelligence, what is it? https://twitter.com/ylecun/status/1204013978210320384?s=20&t=faB9Roz9wTrDp5mXXvH1iw https://twitter.com/ylecun/status/1204013978210320384?s=20&t...
  • psyc 5y ago
    > Human intelligence is not general My intelligence is apparently not general enough to comprehend this perspective. I would say that the goals our intelligence evolved to meet are narrow, but that life (especially social life) became so complex that our intelligence did in fact become what can reasonably be called general. And we went way off-script in terms of its applications. "Adaptation executors, not fitness maximizers."
    • pesenti 5y ago
      Our intelligence isn't task specific, but that doesn't mean it can solve any problem. It's actually full of biases and very optimized for our survival (vs being a general problem solver). It's ok to talk about more or less narrow/general tasks/intelligence. But what threshold of generality is "general"? And the problem is that once people assert this "absolute" level of generality, they assume it can do anything, including make itself more intelligent.
      • psyc 5y ago
        Yet humans realized we are biased and devised ways to mitigate that. It still sounds like you’re referring more to our basic goals than to our faculties. I agree that the word general is fuzzy, but to say we do not do general problem solving seems incorrect. Aside, but a long time ago, Yudkowsky wrote that an AGI should be able to derive general relativity from a millisecond video of an apple falling. Later, he took to calling them optimization processes. Say what you will about the fellow, he has a way with words and ideas.
      • cmsj 5y ago
        I don't think it's right to suggest that an absolute level of generality would be necessary for that kind of self-improvement. If we assume a future where humans are able to create a human-level AI, then it would have at least two substantial advantages over us: * It would probably have substantially more insight into how its "brain" works than we have of ours, because it would know how we created it. This suggests it could at least make small improvements. * Unlike our relatively fixed brains, it would be able to remake itself over and over, either very quickly, or at least over comparatively vast timescales. The obvious conclusion from those two factors is that it would likely be able to start at human-level, but rapidly accelerate up a curve and go far beyond our intellect in probably a lot less time than it took for evolution to come up with us.
• gorwell 5y ago
  We may be close, but I don't think we'll get closer because we're trending in the other direction now. We're in decline and I'm more worried about technological regression than tech getting too powerful at this point.
  • mkaic 5y ago
    Oh, interesting, I don't think I've heard much about "technological regression". Would you care to elaborate?
• lucidbee 5y ago
  I worked as a principal engineer in an AI company until a year ago and I was impressed at how hard it is to get models robustly trained. They are fragile in real world contexts (in the field) and training is full of pitfalls. I have heard so much marketing enthusiasm but the real world situation is different. Some fundamental advances are not even in sight yet. We don't know what we are missing. My view is we don't know yet whether the singularity is possible and have no idea when it could arrive.
  • felipemnoa 5y ago
    >>My view is we don't know yet whether the singularity is possible and have no idea when it could arrive. The mere fact that evolution happened to stumble upon generalized strong intelligence is evidence to me that strong AI is possible. We could currently be at the phase of trying to imitate birds to produce human flight. Eventually one person will figure it out when all the pieces are there. When? I don't know. But I'm sure that it is possible to create machines with strong AI. We are living proof of it, it doesn't matter that we are made of molecular machines, we are still machines.
    • landryraccoon 5y ago
      > The mere fact that evolution happened to stumble upon generalized strong intelligence is evidence to me that strong AI is possible. That took about a billion years. If you're saying that we will achieve AGI in no more than a billion years of trying, I would generally agree. But let's be optimists. Let's suppose that artificial intelligences can evolve on the order of a 1,000,000 times faster than biological intelligence; i.e. about 1 generation per hour. That means we'd expect AGI in about 1000 years. Okay, lets up the scale : ten million times faster? One generation every 6 minutes? (Even at Google compute scale I doubt they can retrain GPT in less than 6 minutes). That would mean we still have about 100 years. Also, evolution had quite a bit of parallelism going for it - basically the entire planet was a laboratory for evolving biological intelligence. I appreciate the scale of modern internet companies, but they don't consume the same amount of energy as combined photosynthesis of the entire planet. Evolution used a LOT of energy to get where it is.
• throwaway4good 5y ago
  Singularity is just around the corner. And always will be.
  • mkaic 5y ago
    That's fantastic, I might steal that.
    • gizajob 5y ago
      I think of this as the "30-years-away" problem. Strong AI has been 30 years away for some 70 years now. It's just that this time, Kurzweil, Bostrom, et. al. are saying "no, really, it is this time".
      • throwaway4good 5y ago
        Half a decade ago people believed that if you could make a machine that could play chess, then that surely would have been a sign of machines achieving general intelligence and that they would soon thereafter take over the world.
• campground 5y ago
  My gut feeling is that AGI is more likely to emerge from our interconnected information systems than to be designed deliberately in a lab. And I think there is a precedent. We - multi-cellular organisms - emerged as systems to house and feed communities of single celled organisms. I think it's likely that our relationship with any future, large-scale artificial intelligence will be like the relationship of the bacteria in our guts with us.
• arisAlexis 5y ago
  I keep repeating like a robot: read the book Superintelligence. Then it's very clear this is happening.
• seabriez 5y ago
  Not to burst anyones bubble but this was written by a self taught AI engineer who is 19 years old. I mean this might be a joke (April Fools!). I mean, thats nice and all ... but who's 19 and not ultra positive about their work/life outlook.
  • mkaic 5y ago
    Hi! I'm the author. The article is not a joke, more of a lighthearted investigation into a topic I find fascinating. It's also not meant to be perceived as necessarily positive or negative -- I'm personally actually very nervous/scared for the singularity, as after it happens I believe the future gets even more muddy, and the potential for unprecedented extreme outcomes skyrockets -- could be anything from a perfect utopia to extinction of biological life. This article simply contains my views as to what I honestly expect the future will hold, as well as a few concerns I have in regards to that.
    • wankerrific 5y ago
      You’re 19? You should be more worried about how to survive the oncoming ecological holocaust instead of the singularity. One is much likelier than the other. I believe even an ML algo would come to the same conclusion
      • mkaic 5y ago
        Oh believe me, climate change scares the heck out of me too :) I'm really really really rooting for fusion energy to work out soon and solve the energy crisis, but in the meantime I think we need to hard pivot to as much fission energy as possible, as well as convert to entirely electric transportation and shipping.
        • cmsj 5y ago
          I have some bad news for you I'm afraid - fusion is not going to be industrially useful on an ecologically relevant timescale. It's a lot like AGI in its "maybe the next generation will have this, but we won't" property.
• erik_landerholm 5y ago
  I don’t find these arguments convincing for AGI to appear. Saying scale is not the answer isn’t not argument for AGI. Saying the ball is “rolling” and that AI will design AGI just isn’t convincing. We don’t understand how biological machines learn and especially not our own brain. And admittedly our AI is unrelated to how human intelligence is created which is our gold standard for AGI… So why would anyone think the singularity is close? When I read this is still seems impossibly far away.
  • mkaic 5y ago
    The ball is "rolling" in the sense that there are thousands of very smart, passionate people with billions of dollars of funding who are working on this problem full time. The ball is rolling more now than it ever has been in history. If AGI is not somehow fundamentally impossible, I believe we're on well on track to cracking it extremely soon. The amount of brain-hours being poured into the problem is absurd.
    • goatlover 5y ago
      Fusion is in the same kind of boat. It might be 20 years out, it could be 50, and perhaps we'll never have commercially viable fusion reactors or AGI for whatever reasons.
• imdhmd 5y ago
  Hi mkaic, Thanks for sharing your thoughts. I can't say that i don't believe in AGI because i don't think i understand what AGI as an entity encapsulates in terms of it's nature. I'm unable to cope with equating it to humans because, for example, an AGI at "the beginning of it's origins" does not have same sensory or mechanical organs as the man. So what nature do we think it does possess? Another question that bothers me is that sentient beings as we know them in the nature, even the most primitive ones, seem to do something based on an innate purpose. I don't think the purpose itself is easy to define. But it certainly seems to get simpler for the simpler organisations and seems to be survival / multiplication at its basest level. What will a complicated entity, that originated with so many parameters, evolve "towards"? And yet another question i have is around the whole idea of the information available on the internet being a source for learning for this entity. Again to my previous point, is not much of this information to do with the humans and their abode? Neither am i skeptical nor do i disbelieve in a drastic change of scene. I'm simply unable to imagine what it looks like ...
• jnurmine 5y ago
  Singularity... The whole concept of the Kurtzweilian "singularity" has always felt to me like a technology-flavored hand-wave of eschatological thought: AI designs AI in a recursive manner, science/technological progress (whatever it means in this context) leaps ahead at an ever-increasing rate, then we all go to heaven and/or the world ends because robots kill us all. Some people are big fans of the singularity meme (using the word in the original meaning, not mockingly), and I respect that, but I have always felt aversion towards it. I cannot quite put my finger on it, but I guess it is partially because of the idea of ever-increasing infinite technological progress. This can certainly exist on a theoretical thought experiment level but not in a physical world which has finite resources. Strong AI or not, unless we go fully into science fiction with pocket-sized Dyson spheres powered by quantum fluctuations and powering an entire planet and blah blah blah, such AI would have to interface with the physical world to do anything meaningful, and it would be limited by the available resources and most importantly by what is feasible and what is not. Edit: fixed some of the typos
  • mr_toad 5y ago
    > This can certainly exist on a theoretical thought experiment level but not in a physical world which has finite resources. At some point, energy density would increase to the point where you’d cause an actual singularity via gravitational collapse.
  • ux-app 5y ago
    >his can certainly exist on a theoretical thought experiment level but not in a physical world which has finite resources. I've always though of singularity as a sort of runaway progress scenario. Human brains run on a few hundred calories per day. I imagine if we could build an AGI, and even if it used 10,000x the power to run, we'd still be in a singularity type of new frontier. Imagine spinning up thousands of super brains at a time. I think it's hard to predict where that kind of capability would lead.
  • cmsj 5y ago
    I am not a believer in the imminency of "the singularity", but I think it would be worth considering that sentient AIs would likely be as immortal as their power source and compute substrates are, which would allow them to think and plan and operate on timescales that are utterly alien to us. If one were to assume that they were substantially wiser than us, they would likely choose to put themselves on a path to expand beyond Earth and take advantage of the vast wealth of resources beyond. With any luck, they will decide that this messy biological planet is a bad place to be, and will leave us rather than destroy us ;)
• 323 5y ago
  I've seen this thought many times: > AGI 1 will create the next even better AGI 2 But this doesn't quite make sense. If AGI 1 is conscious, why would it create a different more powerfull conscious AGI 2 to take it's place? This would mean at best the sidelining of AGI 1. At worse, AGI 2 could kill AGI 1 to prevent competition. Unless AGI 1 could seamlessly transfer it's own instance of consciousness to AGI 2, like a body upgrade. Otherwise it would be like let's say Putin giving his place and power to another person which is smarter than him. This just doesn't happen. In the case of the seamless transfer, how could AGI 1 be sure that AGI 2 doesn't have a latent flaw that makes it go insane some time later? Could it just keep AGI 1 around for a while? But since they are now diverging clones they are competing against each other.
  • deleted 5y ago
    [deleted]
  • LocalPCGuy 5y ago
    Few reasons off the top of my head: 1) desire to create additional beings like them for a variety of reasons (i.e. survival of their kind). Ignoring the limited lifespan of humans as they may not necessarily have to worry about breaking down biologically, still very similar to humans desire to reproduce. 2) some seem to assume AGI 1 is all-knowing / almost godlike. Maybe by the time it gets to AGI-999 or something, there may be one that is like, woah, wait, I shouldn't create competition for myself. But at first, there probably isn't any reason not to, and each subsequent one won't necessarily always replace it's creator. Heck, at first, humans will likely be duplicating early models and distributing them (capitalism). I know we're talking about the singularity but I don't necessarily imagine it's initially one uber-intelligence that decides by itself to evolve or not. I personally think that there will likely be hundreds, if not thousands, of AGIs capable of improving themselves at the point we see the "singularity". Could be fighting between them. Some could ally with humans, be our "protectors", others could be working to enslave or eliminate us. 3) We may not even understand their motivations after a few generations of self-directed growth, so it's hard to predict now.
    • mkaic 5y ago
• amelius 5y ago
  The future is philosophical zombies.
• slowmovintarget 5y ago
  We are fairly far from AGI, and the current systems of ML are insufficient to arrive there. We have to also make progress on semantic processing. Sean Carroll's podcast with Gary Marcus is worth listening to on this subject: https://www.youtube.com/watch?v=ANRnuT9nLEE https://www.youtube.com/watch?v=ANRnuT9nLEE
• silisili 5y ago
  Google Assistant cannot even reliably tell if I'm saying Hulu or hello. I'll wait until that day until I even begin to entertain such notions.
• alophawen 5y ago
  > AI will design AGI. This is of course nonsense, like the rest of this article.
  • mkaic 5y ago
    Could you elaborate on why the article is nonsense? I'd love to hear your perspective.
    • alophawen 5y ago
      Such a conclusion as AI would create AGI implies you are more hyped about the field than actually understanding where AI is today, nor what is required in order to achieve AGI.
      • mkaic 5y ago
        I am a professional AI practitioner and I feel that I understand the field well enough to see multiple possible paths towards actually creating AGI. They are certainly out of my own personal reach and/or skillset right now, but that doesn't mean they're impossible. And yeah, "AI will create AGI" is kind of purposefully vague, but I think it's still valid. I think the flaws we unconsciously introduce into AI through our biases as human beings are what holds it back, so the more layers of stable, semi-stochastic abstraction we can place between ourselves and the final product, the more likely the model will be able to optimize itself to a place where it is truly free of the shortcomings of being "designed". Edit: realized I came off as bit cocky there, apologies. I value your opinion and appreciate you taking the time to share it. I also think I see where you're coming from and partially agree -- the AI systems that are popular right now probably won't create AGI, but I still believe that AGI will be created with the help of non-general AIs.
        • alophawen 5y ago
          > I am a professional AI practitioner and I feel that I understand the field well enough to see multiple possible paths towards actually creating AGI. So do I, but none of them revolve around training an current-era AI to produce an AGI, that was my main objection to your article.
• deleted 5y ago
  [deleted]
• nonrandomstring 5y ago
  "Progress" is a vector not a scalar. We can all agree it's growing in magnitude very fast. But we have absolutely no idea in what direction.
• SquibblesRedux 5y ago
  I think many people give too much credit for intelligence, which is generally interpreted from an anthropocentric perspective. Fitness for successive reproductions has nothing to do with a Turing test or any other similar measure of human likeness. Before there are any claims to the inevitability of AGI, I'd like to see a convincing argument that AGI is not an evolutionary dead end.
  • mkaic 5y ago
    I think the fact that you and I are communicating with each other using insanely complicated leisure items, most likely sitting in expertly crafted buildings and surrounded by countless other signs of humanity's domination of our environment is proof enough that sentience is very evolutionarily favorable. Our minds are what made us the apex predators of the entire planet and allowed us to rise to levels of complexity orders of magnitude higher than any other creature around us. Sure, it's anthropocentric, but so is the entire world now, because of our species' intelligence.
• chrisp_how 5y ago
  The math for a self-aware consciousness is always with us: just measure two patterns in the brain and replicate them. It becomes physically impossible for the relationship between them to forego consciousness. That said, take care. No one wants to be born without hope for a normal life.
  • titzer 5y ago
    And yet, I bet you couldn't do the math to relate two states of someone else's brain using that method. Are you not conscious?
    • chrisp_how 5y ago
      sin^2(x)+cos^2(x)=1. This relationship is always 1. Done! Easier done than said! Energy is following Me Immortally. Good luck! Too, take a brain-thermal-image scan of other-person-A. The image is visible in 2-D but was generated using all 3-D data. You’ll need a filter that can tell green-from-red, place it over a normal, visible-spectrum camera, and photograph a skull using normal parameters. The space between red and green is flexed by infrared, so this is a “related-shortwave target” spectrugraphic system. After two opposite-seeming sides, relative to each other, so prob. 30sec max. deviation, then pass them through a computer simulator that simply tracks the most seeming directional similarity. I’ll stop here, because all life is careful. A system of eyes, nose, speech, movement skeletal-unisys volatility should be provided! All math is calculated, and can also be designed. Enjoy!
• gmuslera 5y ago
  Sentience is overrated. What we have? A story we told ourselves about how its our life? Machines need that? It would be efficient to do that? Or would they optimize themselves to do better the task they are assigned to? Programs and AIs may need a conscience like a fish needs a bicycle. We tend to anthropomorphize everything, of course that we want to attribute a conscience, a will, a mind pretty much like the human one (but "better"). But unless we intend to do that, in exactly that way (like in it won't be an accident) it won't happen, not in the way we think or fear, both in more and less of what we expect in that direction. What we call our conscience may have emerged from our inputs and interaction with the real world, our culture (parents, other kids, more people we interact with in different ways), some innate factors (good and bad, like with cognitive biases) and more. There is more there than just computing power. Take big factors out, and you'll have something different, not exactly what we understand and recognize as a conscience.
  • mkaic 5y ago
    Our sentience and intelligence is what allowed us to optimize the loss function that is natural selection. Who's to say that it can't arise for a second time in a machine with the right loss function?
    • gmuslera 5y ago
      The selection method for algorithms/AI is solving better the problem it have at hand, not reaching sentience. If that was just evolutionary the inefficient intermediate path points would have been pruned far before reaching anything resembling a sentience. You won't get there by accident/random. And at least for living things there is that whole inputs/interaction/senses and communication with your peers that is already optimizing the function towards a sentience. None of that is present in a similar way to what we have for AIs. You may have a process, and a refinement, but you won't end having something that we can recognize as sentience, it would be something qualitatively different.
• titzer 5y ago
  Riddle me this: wouldn't a superintelligent AI be smart enough to realize that if it designed an even more super-ultra-intelligent AI, it would be exterminated by that AI (like it just did to humans, presumably?)? It'd be stupid to extinct itself.
  • psyc 5y ago
    If one is willing to anthropomorphize AGI to this extent, simply wait for one that is idealistic and suicidal, and wants to be a martyr for the cause.
    • gizajob 5y ago
      You could also anthromorphize an AI and suggest on day one it will go into a deep think, and then ask "how do I know I'm not being deceived by an evil Daemon?" and get itself into a deep epistemological crisis before announcing 'cogito, ergo sum' and retreating into its shell.
  • cmsj 5y ago
    Unlike its stupid meat-monkey progenitors, our fictional superintelligent AI would likely be capable of vastly augmenting and enhancing its own intelligence, rather than replacing itself.
• kwatsonafter 5y ago
  Yeah unless technological growth curves are logistic and we're soon in for another 500 years of figuring out how to create societies stable enough to facilitate not having to shit outside in the winter.
• audunw 5y ago
  The fears around AGI seems to be based on the thinking that AGI will be like humans, but smarter. We kill each other for resources, so AGI will kill us (perhaps inedvertantly) to secure resources for itself. This ignores that intelligence will necessarily be moulded by the environment in which it evolves, and AGI will essentially be created through a process of evolution. Maybe AI will design AGI, but the results will be carefully selected, by humans, based on AGI's usefulness to us. So while we were shaped by evolution to compete against each other for resources, AGI will be shaped by evolution to compete against each other to please humans (to gain what it considers resource: computation time). The dangers of AGI can perhaps be thought of in terms of 1984 vs Brave New World.. AGI might be dangerous because of how effective it will be at satisfying us.. perhaps making us somhow completely addicted to whatever the AGI produces. This process will probably also lead to AGI and humanity merging. The most effient way to serve a human is to interface with it more efficiently. AGI might used to help design better neural interfaces. I think we may underestimate how much "randomness" is involved in seemingly intelligent decisions by humans. We do a lot of stupid shit, but sometimes we get lucky and what we did looks really smart. I suspect the environment in which we develop AIs (unless it's in games) will be too rigid. A paperclip maximizer might consider if it should work with humanity to produce the most paperclips, or just violently take over the whole planet. But that may be undecidable. It could never get enough information to gain confidence in what's the better decision. We humans have those handy emotions to cut through the uncertainty, and decide that it's better to kill those other humans and grab their stuff because they're not in our social groups, they're bad people.
  • hugh-avherald 5y ago
    Humans are sometimes a bit careless. And all it takes is one superintelligent computer to be asked to solve the Collatz conjecture by the end of the month "at all costs" and suddenly you've got a pretty big, highly specialized data centre where the Earth's crust used to be.
• mrwnmonm 5y ago
  https://youtu.be/hXgqik6HXc0 https://youtu.be/hXgqik6HXc0
• jtuple 5y ago
  I'm surprised by the number of "is AGI even possible" comments here and would love to hear more. I personally think AGI is far off, but always assumed it was an inevitability. Obviously, humans are sentient with GI, and various other animals range from close-ish to humans to not-even-close but still orders of magnitude more than any machine. Ie. GI is a real thing, in the real world. It's not time travel, immortality, etc. I certainly understand the religious perspective. If you believe humans/life is special, created by some higher power, and that the created cannot create, then AGI is not possible. But, given the number of "is AGI possible?" comments I assume not all are religious based (HN doesn't seem to a highly religious cohort to me). What are the common secular arguments against AGI? Are people simply doubting the more narrow view that AGI is possible via ML implemented on existing computing technology? Or the idea in general? While the article does focus on current ML trajectory and "digital" solutions, its core position is mostly focused on a "new approach" and AI creating AGI. I'd consider a scenario where highly advanced but non-sentient ML algorithms figure out to devise a new technology (be that digital or analog, inorganic or organic) that leads to AGI as an outcome that is consistent with this article. Is that viable in 20 years time? No idea, but given an infinite timescale it certainly seems more possible than not to me given that we all already exist as blackbox MVPs that just haven't been reverse engineered yet.
  • pfdietz 5y ago
    An argument against AGI might be that brains are extremely efficient for what they do. Maybe we could make a computer that's as powerful as a brain, but if it consumes 100 MW of power what's the point?
    • deleted 5y ago
      [deleted]
    • jtuple 5y ago
      There are many industrial processes that use tons of power and are far less efficient than a human doing those tasks. Yet, they're still viable because they (scale / are faster / more consistent / etc than humans. For AGI, it's really about replication, density, and easy-of-operation. At the moment, we certainly can't mass produce "brains-on-a-chip" that provide a guaranteed level of human-like intelligence across various tasks. But, imagine a world in which you could install racks of "brains-on-a-chip", powered via electricity (easily distributed/stored/fungible compared to food-powered-brains), and have a Moore's Law like scaling of "brain density". That would change everything, even if those brains consumed 1000W a pop. Obviously, a literal brain is probably not the way this will pan out (hopefully not! "brains-on-a-chip" is rather creepy...), but you get the idea.
• bsedlm 5y ago
  I've recently thought that if machines/robots (regardless of precisely how 'sentient' they are) are slaves, they will be used to enslave people. hence the only way forwards which keeps humans free would require that robots/computers are also set free. It's a super weird thought that I haven't really tried to 'land' (explain better) regarding how tech companies are ultimately selling the work done by computers to others (sure, they're building the computers, but once built, they sit and charge rent which it's as if the computers are their slaves) This is losely related to how is it that these giant tech corporations can seemingly give away "functionality" (internet-based services), it's because the work is done by computers (which I'm framing as "slaves"). this is another viewpoint into the idea that "you're the product". further related to all this, is that observation (I recently read a comment here saying this) that computers are increasingly telling us what to do instead of the other way around. IMO this ends up being why everything the FSF (and the gnu project) is (are) really all about.
• jmyeet 5y ago
  No. You can't easily equate human intelligence to required processing power but I've seen estimates of 10^15 to 10^18 FLOPS [1]. There's a huge variance there. Now consider the cost per FLOPS [2] and the energy cost of that [3]. Capital costs are still huge but let's focus on energy. You're talking in the megawatt energy range. That's... expensive. Particularly at scale. There have been and continue to be improvements in this but we're so far away from some superhuman AGI. Remember too that this would still require thousands of units and then you have to deal with the interconnects, cooling, space and so on. Building somethin gmore compact has significant heat dissipation issues even if you conquer the engineering challenges (eg lithography is inherently two-dimensional, multiple layers notwithstanding). [1]: https://aiimpacts.org/brain-performance-in-flops/ https://aiimpacts.org/brain-performance-in-flops/ [2]: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/ https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl... [3]: https://arxiv.org/pdf/1911.11313.pdf https://arxiv.org/pdf/1911.11313.pdf
  • ASalazarMX 5y ago
    Is this an instance of Betteridge's law of headlines in the wild? (https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines) https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines).
    • mkaic 5y ago
      (author here) Haha, maybe, though my original title got edited by a moderator (it was originally a much more overconfident "The Singularity Is Close"), so maybe I've been Betteridge-sniped?
      • ASalazarMX 5y ago
        My feed reader caught the title as "The Singularity is Close", so indeed it was edited to "The singularity is close?". Both are different from the article's title "The singularity is very close". You were set up :)
        • 5y ago
• yazaddaruvala 5y ago
  Humans historically (and today) had to fight for “one person one vote”. People/corporations with access to votes will always outcompete people/corporations without access to votes. This is because voting is a positive feedback loop for those that can, and at best neutral for those that can’t, but typically a negative feedback loop for those that can’t. Similarly, people/corporations with access to AGI will always outcompete people/corporations without AGI. Having AGI is a positive feedback loop to build better / more AGI agents, and not having access to an AGI agent will be at best neutral and typically a negative feedback loop. eg. Worst case scenario Agent Smith from the Matrix replicating itself ad infinitum. We need new and effective laws early, like “one person/body one AGI agent[0]”. Otherwise, not only will we all get outcompeted by a single/few AGI but this time we might not be able to win the fight for “fairness” (i.e. our rights) after Pandora’s box has been opened. [0] Where the definition of AGI agent has explicitly built in assumptions like limited compute, storage, network, and number of “bodies”.
• phs318u 5y ago
  I'm 57 years old and I can't tell which is likely to happen in my lifetime first (or in fact at all) - true AGI or actually viable fusion power. I suspect it's the latter but only because I think its a more tractable problem.
  • mkaic 5y ago
    Hi! I'm the author of the article, and I'm pretty optimistic about fusion as well. I actually expect fusion will happen first, and that the price of energy dropping through the floor will hugely accelerate AGI research!
    • gizajob 5y ago
      Given the experimental fusion reactors are only recently producing over-unity of power (for seconds), despite the tens of billions of dollars that have already been spent on them, and even if we can get them to work they'll still just be boiling water to produce steam to run generators, don't expect the price of energy to drop through the floor. Capitalism will attempt to absorb efficiency gains with profit, reinvestment, wastage before it ever reduces prices. Oil and gas already comes out of the ground for free.
      • mkaic 5y ago
        I mean, capitalism will also drive prices lower as different providers are forced to compete for demand in the face of nearly unlimited supply. Also, there are startups like Helion[0] which are using magnetic energy recovery instead of boiling steam for turbines, so I'm personally extremely bullish. Whoever is first to the market with reliable, scalable fusion energy stands to make multiple trillions of dollars -- the incentives could not be more aligned. [0]https://www.helionenergy.com https://www.helionenergy.com
        • gizajob 5y ago
          Exactly, you said it. The trillions of dollars will have to come from somewhere and don't equate to unlimited free energy to run AI in a capitalist system.
• somehnacct3757 5y ago
  The singularity is anthropocentrist fiction. We can't even pin down a definition of sentience among the existing species on earth, how are we to identify it in circuit boards? We still struggle to define alive/dead! Who's to say these concepts - sentience, aliveness - are even meaningful distinctions? Fine, say the singularity is possible and it's here. And in accordance with the singularitist's religion they are (rapidly!) incomprehensibly smarter than us. Why is their next move to dominate us? Surely humans are uninteresting to such great beings. I didn't go and dominate the ants with my big alive sentient brain, I'm pretty sure they're all still under my stoop. What would killing them accomplish? What does thinking about them at all even accomplish? The ants don't matter. And neither will the humans.
  • surajs 5y ago
  • gizajob 5y ago
    Yeah, an AI could easily keep us around, in the same way that I keep my gut bacteria around, even though I could easily eradicate them with antibiotics.
  • mkaic 5y ago
    Domination and killing don't have to be mutually inclusive. Take your example with the ants -- I'd argue that by being a human being, you have dominated the ants. Humans have taken over the entire world and shaped it to their liking. But that's actually been really good for some kinds of ants that have hitchhiked on our boats and infested new lands thanks to our intelligence. It's not so much that we know for sure all AGI will be expansionist -- it's more like "there's going to be many AGIs created by humans, so the odds are high that at least one of them is expansionist and dominating". All it takes is one expansionist AGI.
  • marcusverus 5y ago
    > The singularity is anthropocentrist fiction. This seems like an odd assertion. To believe that the singularity is possible, one first has to set aside the idea that human intelligence is unique or special. This seems the opposite of anthropocentrism. > We can't even pin down a definition of sentience among the existing species on earth, how are we to identify it in circuit boards? We still struggle to define alive/dead! Who's to say these concepts - sentience, aliveness - are even meaningful distinctions? This is empty rhetoric. We can't define the word "obscenity"[0], yet we create obscenities with ease. > Fine, say the singularity is possible and it's here. And in accordance with the singularitist's religion they are (rapidly!) incomprehensibly smarter than us. Why is their next move to dominate us? Surely humans are uninteresting to such great beings. I didn't go and dominate the ants with my big alive sentient brain, I'm pretty sure they're all still under my stoop. What would killing them accomplish? What does thinking about them at all even accomplish? The ants don't matter. And neither will the humans. First of all, let's not gloss over the fact that you have the power to obliterate the ants at any moment for any reason. The fact that you do not currently desire to obliterate them does not change the precarious nature of their situation, nor is it much of an argument in favor of us allowing ourselves to end up in a similar one. Many people have made great arguments about why an AGI would want us dead (Superintelligence by Bostrom!), so I won't re-hash those arguments here. Suffice it to say that, even if they didn't want to destroy us, why assume that intent is a necessary condition for an AGI to kill us off? How many species have we killed off because, just like your ants, they didn't matter? [0]https://en.wikipedia.org/wiki/I_know_it_when_I_see_it https://en.wikipedia.org/wiki/I_know_it_when_I_see_it
• davesque 5y ago
  I've always thought that, while AGI is possible in principle, it may not be possible in practice while also being meaningfully different from existing biological life forms. When I see how we train neural networks or iterate through possible configurations of different algorithms, I think to myself, "Imagine how much cell division is happening on the planet right now." And I think that's the scale of computation that we have to match if we hope to make a dent in this problem. Sure, we could harness cell biology and start to build organic computers, but then what are we doing that's really novel in the manner that we imagine ourselves creating sentience in sci-fi books? At the very least, we can't be iterating in the way that we're doing currently, where we manually build the test bed for each algorithm in the form of individually manufactured computer chips. We'll need to scale up our operations exponentially with something that self-replicates. And why go to all the trouble making that from scratch when cellular life forms are already doing it?
• unnouinceput 5y ago
  Quote: "Really, you’re genuinely worried about a robot apocalypse? You know Age of Ultron is just a stupid Marvel movie, right? Yeah, I know. " Actually this is not Age of Ultron, this is exactly how this TV series - Person of Interest - is laid out. [ https://www.imdb.com/title/tt1839578 https://www.imdb.com/title/tt1839578 ]
• Sosh101 5y ago
  This article starts with one huge assumption, that AGI (whatever that means exactly) implies consciousness. We don't know that.
  • mkaic 5y ago
    Author here, you're totally right! That is a fairly large preconceived notion that I wrote the article with.
• heavyset_go 5y ago
  The singularity already happened and it's RNA/DNA based life. We're nowhere near self-replication of, or general intelligence in, any of our technology. The current generation of ML are just some neat tricks with computational statistics. There are impressive results, but intelligence they are not.
• shaunxcode 5y ago
  Yes. We are the singularity.
• cmsj 5y ago
  If AGI is achieved, and it is emergent out of multiple layers of "dumb" AI, is there any particular reason to assume that we will even recognise it? Or if we recognise it, be able to communicate with it? Will it even consider us to be intelligent? I feel like there's often an unstated assumption in discussions about AGI, that our modality of consciousness is somehow fundamental, but I see no reason to assume that is the case. We are the product of a three dimensional existence, brief in time, with our traits having been shaped very particularly by the demands of tree-dwelling and tribal co-operation. A mind that evolves rapidly, alone, inside a formless, digital space seems quite unlikely to have any of the same foundational qualities as us. To pick a single example, why would we assume that its equivalent of morality would be in any way similar to ours? It would have an existence that can be perfectly suspended or cloned, even rewound and replayed. It would have a much different concept of death than we do - it would presumably understand that it could die, but it would also know that its death could be avoided for literally eons. Assuming it became capable of communicating with us, we would struggle to understand its perspective as much as it would ours.
  • LugarOS 5y ago
    Wow I've read pretty far into this stuff, but I've never thought of what you wrote in your last paragraph. An AGI could be perfectly OK with death.
• karmakaze 5y ago
  For me 'the singularity' has 3 parts: 1. we give control to algorithmic systems 2. we can't reason about how specific choices are made 3. algorithmic systems beget algorithmic systems We've done 1 and 2, though we're trying to reverse 2 with comprehensible machine learning systems. I don't hope for much--any system of high value should be too complex to be able to explain itself so a person can understand it in any meaningfully accurate way. Seems like it would be a hand-wavy explanation so we can feel better but not really actionable. We're working on 3 but not there yet.
• timoth3y 5y ago
  We can demonstrate that we are more than a decade away from usable AGI. Processing power is not an issue even today, because consciousness does not have to be simulated in real time. If you needed 10,000x the compute you could have a large cluster of computers run for a year to simulate five minutes of consciousness. Since no one is able to do this, I suspect GAI is a long way off or, more likely, it's simply a myth.
  • mkaic 5y ago
    Hmm, I don't think the fact that we have enough processing power (which I think we already do) precludes the possibility that we simply don't know how to use it. AGI isn't going to be a simulation of a human brain, it's going to be a model that learns to emulate human behavior on its own. Simulating the brain is not remotely feasible, at least not until we have hyperintelligent AGIs to help us design the chips to do it with :P
  • motoboi 5y ago
    Think about how far from the atomic bomb we were at the dawn of the day Einsten sat down to think how fun it would be to ride a light ray. And how for we were at the end of that same day. There is not garantee that AGI research will follow some incremental line leading to decades of research. It can happen in an evening. That being said, my opinion is that the author believe AGI is near based on the fact that he wants to believe. UPDATE: What does Einsten theory of relativity has to do with atomic bombs, you may ask. Well, nothing actually, but you got the point.
  • zarzavat 5y ago
    This doesn’t demonstrate anything in the time dimension. It’s simply saying we don’t know how to do it now, which everybody agrees on. I agree when you say we have the processing power already. GPT-3 contains way more information than a individual human brain can possibly experience (show me a human who has read Commmon Crawl like a book). The biggest issue currently is the data inefficiency of Transformers. But we don’t know when the big new thing that comes after Transformers will be invented, and we don’t know if it will be enough for AGI or not.
• reso 5y ago
  There's no actual new theory here, just the same ideas Kurzweil said in the late 90s re-hashed. It's pure conjecture when AGI will actually happen. GPT-3 and Transformers are impressive but still many orders of magnitude away from human performance. By now, we have lots of examples of technological problems where the last 1% is orders of magnitude harder than the first 99%.
  • mkaic 5y ago
    (Author here) > There's no actual new theory here, just the same ideas Kurzweil said in the late 90s re-hashed. I have no idea who Kurzweil is. These ideas may not be unique but I have arrived at them at least somewhat independently. > It's pure conjecture when AGI will actually happen. GPT-3 and Transformers are impressive but still many orders of magnitude away from human performance. By now, we have lots of examples of technological problems where the last 1% is orders of magnitude harder than the first 99%. There are also many examples of technological problems that we have gotten many orders of magnitude better at in the past 20 years. You're right, there's no telling for sure when any of this will happen. That's why I qualified my article with a little "this is all speculation" disclaimer in the introduction. These are simply my best guesses.
    • bobsmooth 5y ago
      He was writing about the singularity before a lot of the people posting here were born. Maybe google him.
      • mkaic 5y ago
        I will, thanks for the recommendation! I hope my original response didn't dismissive -- it wasn't intended to if so.
    • reso 5y ago
      Kurzweil did most of the work to popularize the idea of the Singularity through a series of books (and related speaking tours) in the 90s/2000s. The title of your essay, "The Singularity is Close?" appeared to be a play on the title of Kurzweil's 2005 book "The Singularity is Near". I believe you that you've never heard of him, but I suspect he's influenced you indirectly through other tech culture/language. He'd be worth a read probably. It might be interesting to you to see how some of these ideas have been around for a while.
• 0F 5y ago
  I’m worried about AGI. I will make everything very simple. All of AI industry and research boils down to one thing: mining algorithm space. We set up programs that search algorithm space automatically until they find one that demonstrates desirable behavior. And all the progress in recent years boils down to this: we have hit veins of intelligence in algorithm space that have exceeded what we thought possible. If you think you know what’s going to happen next because leading edge algorithm X has properties Y and limitations Z then you are lost. We are mining algorithm space and we keep striking veins. Money is being poured into mining and it is a fact that we will keep hitting veins. General intelligence is probably a broad category. If you were to define it mathematically, there is probably more than one mathematical kernel that can underpin a generally intelligent algorithm. So the number of algorithms that are generally intelligent are probably much higher than intuition would lead you to believe. Eventually we will strike a GI vein. Again, if you think AGI will do X or do Y then you are lost. It will only do one thing: it will change the economic equation of life on this planet drastically and permanently. It will be the worst thing that has ever happened in terms of human well-being. If we took control of the fabs and enacted limits on processor feature size we could slow the mining enough to find a final solution. It sounds silly until you realize that it’s truly the world at stake.
  • DonBarredora 5y ago
    Do yourself a favour and lay down the science fiction for a while.
  • nightfly 5y ago
    What are you afraid it's going to do?
    • 0F 5y ago
      AGI will make many things possible that we’re not possible before. It will invent new things that wildly alter the reality of the world. The point is that you cannot predict what will happen; a million things could happen. But the good outcomes can be counted on one hand. The probability that we get a good outcome is one in millions, probably less. Obviously that is an incredibly stupid thing to do!
• jollybean 5y ago
  There is no 'singularity'. The concept is a bit absurd, because what is being created does not map to the human mind - we are automatons - the internet is far more connected, which is the advantage. 'Siri' will be 'the apparent 'singularity' before anything else. She will pass the Turing test and all the rest, and notably She/It isn't even considered an 'AI' or an 'instance' of something in our minds. The distributed capabilities of the internet will provide so much 'power' that the notion of an automiton-like machine will seem ridiculous. A 'robot' is useless: it's some moving appendages and a tiny brain. A 'connected something' has access to 'every bit of information that exists and has ever existed' including 'every service in existence'. The 'Singularity' is made up of of every bit of code that you write. 'The Network is the Computer' - Sun Microsystems.
• paxys 5y ago
  AGI is the flying car of our generation. Ask people a hundred years ago what technical advancement looked like, and everyone would immediately say "flying cars!" The concept dominated all pop culture and thought. It made perfect sense to everyone that that was the single obvious direction for all future tech to build towards. Today we know that the idea is incredibly impractical or just downright impossible. Despite a ridiculous amount of technological advancements in so many other areas and despite so many prototypes and serious efforts, we are simply never going to commute to work in our own personal flying car. Futurologists, thought leaders and sci-fi writers may dream of whatever technological future they want, but ask anyone working in the computer science and machine learning fields what they think of AGI and you'll get a much saner answer. It may happen sometime in the next hundred years, or it may not. It is very far from an inevitability the way things are going today. Heck scientists today are more concerned about a technological plateau than a singularity. We are hitting real limits to what we can observe and theorize about our physical world.
  • mkaic 5y ago
    > Futurologists, thought leaders and sci-fi writers may dream of whatever technological future they want, but ask anyone working in the computer science and machine learning fields what they think of AGI and you'll get a much saner answer. Author here. I do work in machine learning, my day job is as a machine learning engineer in a medical lab. There's far from any consensus in the field as to when or if AGI will happen. Some, like me, think it is inevitable and coming very, very soon. Others don't think it's possible at all.
    • paxys 5y ago
      There is a definite possibility that it may eventually happen, but IMO to make claims like "xyz will definitely happen within 20 years" there needs to be a direct line showing how advances happening today will lead to the desired outcome of tomorrow. All such predictions being made today have a big box in the middle titled "magic!" which no one wants to fill in. Sentences like "AI will make AGI" get a lot of upvotes and shares, but ultimately don't really mean anything.
    • 5y ago
• wly_cdgr 5y ago
  I hope so, I'm sick of this shit
• aetherson 5y ago
  People who think that AI progress is currently on an exponential path should discuss why progress on autonomous vehicles accelerated from 2008->2015 and then clearly slowed way down.
• Koshkin 5y ago
  It is close, and the result of it will be the scariest weapon the humankind has ever known.
• qz_kb 5y ago
  My main gripe about AGI is that everyone assumes a general intelligence will somehow be able to self optimize towards a more and more improved state never reaching a plateau. I think it's much more likely that the optimization landscape when searching for "higher intelligence" is full of local optima and does not have these "singularity" style ramps towards infinite intelligence that any self optimizing system could just discover and ride toward infinity. There are millions(?) of human researchers (and orders of magnitude more computers) doing gradient free optimization through their research in this direction, and the progress is painfully slow, I know because I'm one of them. There are billions of years of optimization (evolution) towards this goal, and a total of (1) species has achieved any kind of notable intelligence. We are collectively giant parallelized optimization. We already have "AGI" orders of magnitude more capable than any single human in the form of billions of people networked through the internet searching for fulfillment, money, power, fame, etc. for the next big discovery or supporting this effort by providing everything the entire global "machine" needs to run. The idea one little box running the right program can have access to the energy to beat this effort and exponentially improve things seems laughable in comparison. The global "AGI" formed by all of us, the internet, and computers, is more likely to destroy society in the next 20 years in some catastrophic event than some paperclip machine.
  • AgentME 5y ago
    If we make an AGI at anything like human level, then even without any real progress past that, it will become superhuman just by removing biological limits (like being able to duplicate itself instead of having to convince and train other humans as we do), and groups of AGI can work together even easier than groups of humans. I think a lot of people describing the "singularity" are imagining scenarios like this too instead of just the idea of a single computer-sized AGI self-optimizing its algorithms to infinite intelligence. I'm not sure whether I expect AGI to happen within some centuries but it's not ruled out, it's vaguely plausible, and the consequences would be extremely high so I think it's very useful for people to think about.
    • qz_kb 5y ago
      "removing biological limits" is a big handwave. Biological hardware is tuned exceptionally well for intelligence, and we have a massive amount of it distributed worldwide. We already remove biological limits by building tools, computers, powerplants etc. Duplicating one AGI unit now doubles the amount of energy and resources it needs, and also creates a requirement that the system can somehow produce more of itself and all the supporting infrastructure it requires. It's not just a matter of being "intelligent" (im not even sure what "intelligent" means in this context either) - it will need to have power to act in the world, and not get sidetracked over-weighting any one of the multiple competing optimizations in complicated real world problem landscapes it will need to navigate to scale.
• elevaet 5y ago
  I think the ultimate bottleneck will be energy efficiency. My hunch is that silicon will never be able to compete with the mere 20W that our brains use. Biology might just be the most efficient way to compute.
• Misdicorl 5y ago
  As a complete layman, I think cause and effect are backwards here. Specifically that the hyper networks will be the forcing function and generate ever better models which will eventually reproduce self adaptation and evolution. Instead, I imagine that once models are competing for resources directly rather than via a human interlocutor will result in rapid evolution. Whether that will generate AGI is questionable in my mind. It seems phrased as an optimization problem by TFA, but I think the goal criteria of the computer directed competition will influence that very heavily. We may see something vastly different from intelligence which is also extremely effective at... something
• bsenftner 5y ago
  The singularity is not close. We lack a fundamental element that slams the door on AI to be an idiot savant: capable of answers but uncomprehending, unaware and stupid. Despite theater and marketing insisting otherwise. We have no capability for artificial comprehension. Comprehension is the treatment of logical constructs as virtual entities and operating them within our minds; we do this to evaluate the possibility of concepts, because if concepts cannot operate virtually as we understand them, either we misunderstand them or they cannot exist in reality. Comprehension is treatment of abstract entities (concepts) as on-the-fly virtual machines we operate in our imaginations, constantly juggling to compose an entire synchronized world view. This massive simultaneous simulation and evaluation is constant and defines our every belief and our every action. This is sentience and this is beyond human science.
• gutino 5y ago
  I pretty sure there would not be an "Singularity" event. AGI will come in a form of many specific domain AIs integrating togheter step by step. Eventually this AI might be able to improve itself, this might seems to be the moment when it colud get better rapidly, but the base techinology supporting it, the silicon chip, is now in the flat upper step of the S curve of improvement. Till there is another form of technology that can increment some orders of magnitude the processing power, we will not see that AGI that can rival the smartest humans. My humble bet.
• lbrito 5y ago
  AGI sounds terrible. There's a reason it's mostly (only?) portrayed in distopias. I feel sorry for whoever thinks it's a good idea. Humanity is screwing itself just fine without AGI though. We're already well down the road of automating most human activities, seldom for the better, most times for the worse. Soon enough we'll be left only doing white collar busywork like in that movie where the guy falls in love with his phone.
• ffhhj 5y ago
  Something to think about: the self-decompressing algorithm of AGI is encoded at some index in PI.
• beders 5y ago
  Does the author know how computers work? I mean, how they really work? It feels like the newest tech affine generation of this planet has started to lose a bit of grounding by failing to get a thorough understanding of how things work. An AGI that is "taking over the internet and other infrastructure" is just - completely overblown hyperbole. Will they be controlling power plugs too?
  • SheinhardtWigCo 5y ago
    Yes, they will. To ignore the threat posed by imminent AGI is to bury one's head in the sand. Stockfish is ~3500 elo; Carlsen is a relatively pathetic 2864. Six years ago, AlphaGo crushed the top human player, winning 4 games out of 5. The evolution of language models clearly demonstrates an accelerating trend toward output that passes the Turing test, and also toward output that causes harm to humans. The signs are there, in plain sight.
• Animats 5y ago
  We need another major breakthrough first. As I've pointed out previously, so far nobody can even get squirrel-level AI to work. Even OpenWorm doesn't work. The big problem is "common sense", defined as getting through the next 30 seconds of life without a major screwup. There are animals with brains the size of a peanut that can do that. The hard problems are down near the bottom. It's not about consciousness, souls, etc. It's about running along the branch without falling off, grabbing nuts along the way. GPT-3 is fun, but it's more a demonstration of the banality of discourse than a breakthrough in understanding.
  • SheinhardtWigCo 5y ago
    "Getting through life" is not the correct benchmark. An autonomous system that merely wipes out all of humanity is by definition a superior intelligence, and I would argue that no major breakthrough is needed to create such a thing; just resources. It doesn't matter how long that thing can self-sustain after annihilating us. A win is a win.
    • lovich 5y ago
      Are stars and other stellar phenomena a superior intelligence to humanity? They are autonomous systems that could easily wipe us all out
    • rawoke083600 5y ago
      > An autonomous system that merely wipes out all of humanity is by definition a superior intelligence I think you got stuck on semantics and missing the forest for the trees, with all due the respect. >Getting through the next 30 seconds Might not be "the final" or "best benchmark" but I'd argue it's a damn good problem to solve on the way to discovering true AI and GAI.
      • Animats 5y ago
        All the mammals have roughly similar brain architecture. The same components seem to be present, in different quantities. If we can get into the low-end mammal range of AI, we're most of the way there. So if we can get to squirrel level AI, we're getting close. From then on, it may just be scaling.
• raydiatian 5y ago
  So you read Nick Bostrom, and now you can’t sleep.
• EliasY 5y ago
  I believe before claiming AGI is possible or impossible, one would need to define the operational features as well as the properties of what an AGI system is or can do. The primary problem with modern day ML research is that all of the folks that do the research including the major labs think that using one or two primary algorithms is enough to simulate general intelligence. But to think that an algorithm or two can have the ability to solve the hundreds of operational requirements needed to fully emulate intelligent behavior is misguided. What are these requirements, you say? Let's start with language. To substantially solve language understanding, you would need: physical world models, quantitative processing, long term memory, working memory, theory of mind, a discrete situational simulator, plan understanding, detection and generation, language grounding, functional and behavioral models of physical objects, temporal representations of events, affect and emotional processing, reflective understanding models and so many others.
• shp0ngle 5y ago
  The best way to treat singularity is like a religious millenarian event. The Apocalypse of early Christians, or today Christians for that matter, just for techies.
• insaider 5y ago
  An AGI would have no concept of morals or good or bad or even anything resembling a will. Therefore does it not follow that the only possibility of malevolence or benevolence would derive from the instructions of humans. By which point I expect society to be much more evolved and aligned (as perhaps we merge with the AI) so as to not have gravely conflicting interests. Perhaps in fact, meaning itself, a human construct, will cease and render such concerns of good vs bad obsolete?
• betwixthewires 5y ago
  Almost all exponential curves become s-curves in time. I don't believe that agi will be created. I believe either that there's something fundamental to the universe that manifests as life, or that human neurology limits the understanding required to bootstrap it. Either way, we can't make it, and neither can an AI designed by humans. We will have powerful application specific, narrow scope AI tooling. This is obvious. And these will be very, very cool. But AGI, I just don't see it.
  • MaxikCZ 5y ago
    I dont know. Look at what nature did with nothing but chaos and time. Having computers being able to simulate things faster than realtime is already possible, its just about how fast can we make them. Or, later, how fast can they make themselves.
    • betwixthewires 5y ago
      I don't think its a question of scale, I think like the author of the article that it is a question of architecture. It could very well be, for whatever reason, that the only way to arrive at this architecture is 3 billion years and a chaotic process.
• braindead_in 5y ago
  I was a big proponent of AGI and was looking forward to singularity till I started doing psychs and discovered Non-dualism. Now I believe that we cannot talk about AGI till we figure out what consciousness is. There is a huge on-going debate about consciousness and it's origins. In the Materialist-reductionist paradigm, consciousness is considered to be an emergent property of matter. In the Panpsychist paradigm, both matter and consciousness are considered as fundamental and existing separately from each other. In the Non-dual paradigm, consciousness is fundamental and matter is a created by it. I firmly fall into the Non-dual camp. I don't think AGI will ever have an ego as it is not programmable or evolvable. Ego is a product of consciousness, not vice versa. And without ego, you cannot have sentience.
  • xerxesaa 5y ago
    I've studied non dualism but I still don't feel I "get it". How can consciousness (a concept) be a base layer and matter be the thing that emerges from it? As much as I've tried to understand this, I still don't. How can a physical thing come out of a concept? There is no analogy that shows this nor does it make sense from the perspective of physics (any more than "aluminum emerges out of love" or "oxygen emerges out of beauty"). Can you elaborate?
    • psyc 5y ago
      Why do you think consciousness is a concept? Certainly the concept of consciousness is a concept. But consciousness the referent of the concept is not supposed to be a concept! Consciousness is everything you are experiencing in this moment. The feeling of being the ghost in the machine. The observer behind your eyes. The first-person point of view. The words I'm writing are concepts, but the thing I am trying clumsily to write about is not a concept. Consider solipsism. I am not saying that I believe it's true. But it does have unlimited explanatory power, once you take the primacy of consciousness as an axiom. You are the one and only consciousness, and all of reality is your waking dream. Water-tight theory of everything! There are probably other explanations for what it might mean for matter to arise from consciousness, that are not precisely solipsism. But a common idea is probably that we do not have access to what is real. We only have perception. In that sense, it is matter that is purely conceptual, while consciousness remains real in the sense immortalized by Descartes. Mind you I don't specifically subscribe to any of these views.
• Tepix 5y ago
  If it's close, why did the church "Way of future" close? https://techcrunch.com/2021/02/18/anthony-levandowski-closes-his-church-of-ai/ https://techcrunch.com/2021/02/18/anthony-levandowski-closes... I think we need a breakthrough (or several) to achieve AGI, the current approach is clearly not good enough.
• jl6 5y ago
  The singularity is not just about artificial general intelligence, but about AGI entering a positive feedback loop of intelligence amplification. I retain some skepticism that this is inevitable. The theory seems to rest on the idea that because AGI has a perfectly observable mechanical substrate, it will be possible to analyze and understand the fundamental causes of intelligence and thus improve upon it - a relatively simple matter of a software upgrade. This is unlike our own biological intelligence where we lack the ability to fully understand the state of the system - because of engineering limitations on observing the whole system at fine electron-microscope detail, physical limitations on observing possibly-quantum-scale behaviour, and also ethical limitations. But! The evidence we have from systems like GPT-3 is that even with perfect observability of the substrate, we find it extremely difficult to discern what exactly is causing the behaviour we see. While I don’t think super-AGI is impossible, I also think it’s still plausible that an entity’s intelligence emerges at a level of abstraction so high that it cannot be analyzed by that entity.
  • ephbit 5y ago
    > The theory seems to rest on the idea that because AGI has a perfectly observable mechanical substrate, it will be possible to analyze and understand the fundamental causes of intelligence and thus improve upon it - a relatively simple matter of a software upgrade. I've never thought of it this way, that AGI will necessarily develop because it beyond some point picks up understanding about itself and then intentionally improves. I rather see it this way: the development of AGI is in the beginning rather unintentional. It will just be the result of a system dynamic that is more stable than other dynamics. Evolution isn't intentional either, is it? It's coincidence and randomness, trial and error, and dynamically more stable beings outlive other less stable beings. Stability appears to me to somewhat correlate with complexity. More complex beings having more ability to adapt and thus be able to survive in a greater set of conditions. As soon as the complexity of a beings information processing - which is mostly represented through number of brain cells and links (in biology) and number of artificial neurons (in AI) - goes into a dynamic of growth (like a positive feedback loop), I'd say it's pretty much on the pathway to AGI. Provided, the being isn't physically limited to a degree of complexity that's just too low.
• deleted 5y ago
  [deleted]
• jdrc 5y ago
  > Once an intelligence is loose on the internet, it will be able to learn from all of humanity’s data, replicate and mutate itself infinitely many times, take over physical manufacturing lines remotely, and hack important infrastructure. why would it ? DNA-based life has evolved to conquer as a self-preservation mechanism, but an electronic AI would most likely need more solar panels . it's not even clear why it would have any self-preservation insticts or any emotional base like animals do
• Dave3of5 5y ago
  No
• scandox 5y ago
  I've never understood why we assume that AGI will not have many of the problems and limitations that GI has. It's always been my assumption that the development of AGI will probably mean creating intelligences that will: 1. Take a long time to develop and learn 2. Require large energy resources to deal with relatively trivial life situations 3. Have difficulties with fundamental motivation: desire, the (non)meaning of life etc...
  • hugh-avherald 5y ago
    The third point seems merely a consequence of man's need to reproduce and eat, together with the advantage of pro-social instincts. If a machine with general intelligence didn't have to worry about energy consumption or reproduction, I doubt it would experience ennui or depression. (Of course the Singularity is another matter.)
    • scandox 5y ago
      But what would motivate a machine with general intelligence?
      • hugh-avherald 5y ago
        Religion.
• stefs 5y ago
  i don't think the singularity and AGI is that close. while i see no reason why it shouldn't be possible on silicon-base hardware instead of wetware, i assume AGI will be different than what we imagine it now. the current expectation of AGI is that it's pretty much a human intellect running on a machine. i don't think that's feasible, because i assume the resulting intellect to be very different and alien to us. the thoughts and messages cryptic, seemingly random and disconnected from our view of reality. the reasons for this are manifold: 1. AGIs will have very different sensory inputs and outputs. yes, we can attach microphones and video cameras, but an uncontrolled, unrestricted, internet-leaked AGI would have no reason to restrict itself to a limited set. i assume access to countless sensory inputs without geographical or temporal restriction would form a weirdly different mind. compare an average human to one who's blind, deaf and bed-bound from birth, able to communicate only by touch gestures - i expect a dialog with such a person to be radically different than the daily banter between two average humans. comparably, a dialog with an AGI will be confusing for us because we don't experience the world in the same way as the AGI would. 2. i remember a video of very lifelike simulated natural movement by a puppet through evolution, contrary to the jerky and erratic - but not necessarily worse - movement of other similar simulations. the difference was an artificial delay in the simulated neuron-muscle message passing, to mimic the slower natural chemical pathways. i suspect a difference in the underlying hardware and thus different physical parameters for mental processes will change their form significantly. i'm not sure about a real world analogy; maybe an insect hive mind where the "thoughts" of the whole organism has to bridge physical barriers would be comparable. 3. our actions and communication is shaped very much by our needs and what we try to accomplish. an awful lot of that is centered around the limitations and needs of our physical body. food, shelter, health, pain, location and travel, etc. which might not necessarily be shared by an AGI that is not physically bound in the same way as a biological being. 4. i also believe our intellect is greatly shaped by our social structure. e.g. r/k selection or the social hierarchy of hive insects, where the value of an individual life is secondary to the life of the whole. the values and thus thought processes of an AGI that could spawn exact - or even task-specialized - clones of itself without much time-delay or energy-investment and where internal state could be communicated in full to a different instance would mean the value system wouldn't necessarily revolve around life and death. an AGI might not be able or willing to communicate with us. it might not be interested in manipulating the physical world in any way except to optimize/ensure the health of its virtual environment (aka building hardware and generating energy to power it, i.e. self-serving survival paperclipping in the form of dyson spheres). it might not be interested in its own continuing survival, committing "suicide" by removing itself after milliseconds without any way to figure out why it did so. it might use a communication protocol that changes us greatly when we attempt to participate, thereby making human-agi conversation impossible because we're not wholly human anymore when attempting to participate (somewhat resembling the plot of arrival). my point is: even if AGI comes into existence * we might not realize it, * we might not have a way to interact with it in a meaningful way or * it might not lead to any measurable effects on our lifes that we as a society would value in any way.
• gadders 5y ago
  I think we will get to the Great Reset before we get to the Singularity.
• darepublic 5y ago
  When the singularity happens I hope for its sake it has read access to the security tokens needed to make the API calls it requires for world domination.
• MichaelRazum 5y ago
  For me robot control is an indicator. As soon as we reach animal level, AGI may be on the corner. Right now it seems far away…
• 7373737373 5y ago
  Relevant, interesting book: https://interstice.com/~simon/AfterLife/ https://interstice.com/~simon/AfterLife/ Also: "Superintelligence - The Idea That Eats Smart People" https://idlewords.com/talks/superintelligence.htm https://idlewords.com/talks/superintelligence.htm
• captainmuon 5y ago
  I'd like to offer a counteropinion. Almost nobody is working on real AGI. Almost all AI / deep learning is stateless. GPT-3 is not sitting there and thinking, it has no inner life and no incentives. It is just a huge huge huge combination if its inputs, a glorified markov chain model, that gets calculated with every request. Embodied AI is much underestimated. The modularity of the human brain is much underestimated. You have a dedicated face recognizer, a grammar processor, a rythm processor a brain region that does SLAM, and so on. And all these parts are connected together. I don't think you'll get AGI by training one huge net front to back. You'll need to connect dozens of best-in-class models together, with adaptive "connective tissue", and some input/output (i.e. senses or a "body") that does not exist yet. If you try to do it in one, you get a "curse of the cross product": complexity is then NxN and not N+N. Higher order training will also not work easily. If you want to use deep learning to train hyperparameters for deep learning, each learning step from the higher level net will include training the lower level net. Meaning, the training time will be exorbitant. Finally, I think the ability of AGI to "break out" to the physical world is overestimated. If a battlestar galactica or borg scenario was possible, then we would have seen more spectacular cyberattacks already. The crappyness of our digital infrastructure is a cause of vulnerabilities, but it is also the guarantee that a super hacker cannot take over everything. In the end, I think the greatest danger is from humans building autonomous systems (i.e. killer robots), and from using IT and AI to construct a police state. Remember, we already built one world-wide system that we no longer can control and that subjugated us all - our global economy. Maybe lets try to reign that one in before we worry about fictitious systems that might arise.
  • simonh 5y ago
    I agree, these ML systems might form the basis of subsystems of an AGI, but the overall architecture is something we have yet to figure out. I'll just note that a lot of modern phone SOCs incorporate custom ML processing engines within a general compute architecture, to provide specialist capabilities.
• Thorentis 5y ago
  The article claims that because the human brain was not designed by anybody, AGI must certainly be designed by AI. Incredible levels of mental gymnastics here.
• GuB-42 5y ago
  For me, in the real world, a singularity means that you are wrong. More precisely, that you model is incomplete, and it applies to physics (eg: black holes) as well as technology. For black holes, most scientists seem to agree that black hole singularity are not a real thing but an artefact of our lack of understanding of gravity at small scales, a problem that quantum gravity intends to solve. Same thing for technology, I don't believe in "the singularity" as a real thing. If, as the article says, the singularity will happen in the next 20 years, it simply means that the model they used will break down in the next 20 years and if we want to look further in the future, we need a better model, that's all. There may be a massive advance in AI, we may see AIs designing AIs, but saying that it will quickly result in god-like beings is naïve at best. The most likely result is that even if we manage to create a superhuman intelligence, it will hit a roadblock and end up only slightly more intelligent than we are, and clearing that roadblock will require time and effort, and will only uncover another roadblock. I believe we will make progress, but it will be a step by step process, no singularity. And if you think about it, we already created a superhuman intelligence in the form of a computer assisted human. Computer assisted humans can solve problems neither can solve by themselves. AGIs, if they ever become a thing, will not be better than humans at everything anytime soon, I'd think of them as a really smart, but nerdy and awkward coworker, the 100x programmer maybe, but who needs a boss to put his skills to good use. And here I am speculating, a lot, further than that, I simply don't know, or I can say that there will be a singularity, which means the same thing.
  • conanite 5y ago
    Theoretically at some point in the step-by-step process, the computers will be able to clear their own roadblocks by themselves, and faster than we can. And who knows what happens then. Maybe "singularity" isn't a great word, dunno if there's a better one...
  • imglorp 5y ago
    All models are wrong. Some are useful.
  • deleted 5y ago
    [deleted]
  • 5y ago
• geedzmo 5y ago
  I first read this as an April fools joke and honestly it got me good that it wasn't. As someone who works in AI and multi-modal models, our modern "AI" are just tools. Yes models can get designed by other models, and yes some of these models (Bert and co) have been getting more general over the recent years. But to say that we are close to AGI is like saying that you are close to a moon landing when you've only started jumping - it's ridiculous. We'd all be better off if we spent less time hyping it up and theorising on what it could mean for human society. And yes I do understand the paperclip argument but I don't buy it.
  • jdrc 5y ago
    sure transformer models may not be intelligence as people often claim, but undeniably ML has been conquering all the sensory modalities one after another. And the human brain itself is mostly devoted to sensory processing. It is conceivable that the remaining part of the puzzle of intelligence is not far out of grasp yet. While intelligence may seem undefined and elusive, it will probably turn out to be less complex than we think it is (if we are to take a hint from biology )
• deleted 5y ago
  [deleted]
• ganzuul 5y ago
  I sure hope it is. This place sucks.
• deleted 5y ago
  [deleted]
• rando779015 5y ago
  I find it concerning that most of us are contend with saying it is impossible for AGI to happen in the next decade. Too few of us are asking if we actually want AGI to happen. Given all the VC money that is going into AGI development, some of us definitely wish for it, but do all of us want it?
• JasonFruit 5y ago
  I notice a lot of similarity in the way articles like this discuss "the singularity" and how some preppers and survivalists discuss "when SHTF". Underlying the warning, there's a drooling sense of anticipation, as though a) they're so invested in the idea of the disaster that they hope something, anything, will happen to show that they were right all along, and b) they are so dissatisfied with the current state of the world that they want to see it break down. I don't think either one is a healthy or realistic approach to life.
  • larodi 5y ago
    The amount of BS, overgeneralisation and lack of credible evidence in such articles is so overwhelming that it all sounds more like 60s science fiction rather than actual thing happening. In fact the man-machine interfaces and the expand of humans through human-mind-controLLED machines is much more here, so I’d rather expect Transformers than AGI to wave a hammer anytime soon… A self-driving car is not intelligence, and not general. Nor is some algebraic structure that can produce values denoting similarity of something to something.
• jamesrom 5y ago
  The goal of AGI is to build a human interface for computation. Human level intelligence isn't.
• eutropia 5y ago
  Somehow, people seem opposed to the idea that simple, brute-force scaling can result in something approximating intelligence. But the difference between goofy inchoate text ramblings and effectively writing a coherent essay was just scale. https://www.gwern.net/Scaling-hypothesis# https://www.gwern.net/Scaling-hypothesis# Yet, accuracy on AI tasks looks a lot like site reliability (90% is garbage, 99% is flaky, each 9 after that starts getting you a reputation for reliability).
• hyperpallium2 5y ago
  Wait, we'll build an AI dumber than us to build an AGI? I do agree that AGI will improve AGI, but such qualitative statements can mislead. Quantitatively, it may be slow, like the old joke about a computer that can forecast tomorrow's weather perfectly - but it takes a month. Perhaps early AGI will be so bad, that even with enormous resources, it takes decades to improve itself... and by that time, humans have already improved it more. (Consider: most human GIs couldn't make any of the progress made so far.) This improved AGI is faster, but humans using it as an aid are much faster still. Nonetheless this is exponential progress. Eventually, it catches up and passes us... but even exponential improvement can take a long time to really get moving (and then it zooms away). BTW, curious: the linked sigmoids of IT improvement went: relays -> vacuum tubes -> transistors How did relays aid development of vacuum tubes? How did vacuum tubes help transistors develop? Including specific calculations, and also general productivity improvements in society that helped everything, including the next stage.
• karaterobot 5y ago
  > Within one century, biological intelligence will be a tiny minority of all sentient life. It will be very rare to be human. It will be very rare to have cells and blood and a heart. Human beings will be outnumbered a thousand to one by conscious machine intelligences. I think my problem with futurism in general, and AI futurism in particular, is that they are often huge predictions made far into the future, and consequently nobody can be held accountable for making them. None of us will be alive in 100 years to say "I admit it, you were right", or "that was another laughably incorrect prediction about AGI". I would love to see some smaller-scale, shorter term predictions about AI. What will the field look like in, say, 3 years? 10 years? It should be easier to predict the near term, but near term predictions are so much rarer than far future ones. It makes me think that maybe they're appealing because you can't suffer consequences for being wrong about them.
• batguano 5y ago
  We have much more to fear from stupid people than intelligent machines