28 ms·

OpenAI researchers warned board of AI breakthrough ahead of CEO ouster

• arachp_braijik 3y ago
  [flagged]
• gardenhedge 3y ago
  Why does OpenAI get to define what AGI is?
  • deleted 3y ago
    [deleted]
    • deleted 3y ago
      [deleted]
  • bcherry 3y ago
    You don't need to accept their definition, neither does anyone else. But they do need to have a definition that they accept themselves, because it's used throughout their charter. https://openai.com/charter https://openai.com/charter
  • falcor84 3y ago
    You don't have to accept their definition, but they need one as (by definition) it's central to their charter: https://openai.com/charter https://openai.com/charter
• minimaxir 3y ago
  > I guess when you set up a board of directors tasked with not letting the company accidentally destroy the human race you set up a showdown when the board thinks the company will accidentally destroy the human race https://twitter.com/pdmcleod/status/1727463238051229734 https://twitter.com/pdmcleod/status/1727463238051229734
  • deleted 3y ago
    [deleted]
• V__ 3y ago
  > warning of a powerful artificial intelligence discovery that they said could threaten humanity, > Given vast computing resources, the new model was able to solve certain mathematical problems, [..] Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success, the source said. I somehow expected a bit more.
  • NewJazz 3y ago
    Yeah I was chuckling
    • imdsm 3y ago
      It was an "Oh" moment, but on the other hand, it'd be like laughing at text-davinci-003 when you realised how basic it really was, and yet, it has led us to GPT-4 and, much as Windows 3.1 was amazing back in the day, now it's a distant memory as we carry around little glass tablets that can play 3D games and do UHD global video calls. If Q* is the basic algorithm that is as transformative as, well, transformers/attention, then developing it to be more than just children's math could be the key to thinking - and not just latent space token generation.
    • mikrl 3y ago
      They called it Q* It’s not the first weird conspiracy I’ve heard of involving a cryptically named figure called Q…
      • LudwigNagasena 3y ago
        AGI doomers can now be referred to as Q*Anon.
  • hn_throwaway_99 3y ago
    I think there is enough there to make me believe it's real. After all, a big objection to LLMs is they are just "fancy autocomplete" or "stochastic parrots". I don't agree, but LLMs famously are "bad at math", that is you can't really train them to do math unless they've seen previous calculations with similar numbers in their training sets. Training an AI system to truly do arbitrary mathematical logic (think "complex word problems with big numbers", not just normal programming statements) would be a huge breakthrough.
• righthand 3y ago
  Turns out a quarter of a million dollar paycheck plus whatever they get for letting Microsoft gobble them up was more pressing for the staffers. Unless these staff were the unsigned 5%.
  • iaseiadit 3y ago
    The employees had a lot more money than that on the line.
    • righthand 3y ago
      > plus whatever they get for letting Microsoft gobble them up The average salary there is $250k. More than that would be the vestment greater than their base, no?
      • consumer451 3y ago
        As I understand it, there was supposed to be a liquidity event at a valuation of $86B, very soon. This event was no longer scheduled after Altman was removed.
        • 93po 3y ago
          Why have I not read this before it seems like the most obvious reason for all of this
          • jacquesm 3y ago
            I couldn't tell because that was pretty clear from day #1. Here is an October 18 article about this: https://www.bloomberg.com/news/articles/2023-10-18/openai-is-in-talks-to-sell-shares-at-an-86-billion-valuation https://www.bloomberg.com/news/articles/2023-10-18/openai-is...
• ChuckMcM 3y ago
  Things are gonna be lit about 30 seconds after AGI is created. Gotta practice being "Goodlife" early in case that happens. :-)
  • toomuchtodo 3y ago
    I trust AGI more than any of the humans so far who have tried to argue they can wrangle it.
    • ChuckMcM 3y ago
      I would be curious as to the basis for that trust. I struggle to find any reason that AGI would care about "humans" at all, other than during the short period of time it needed them to be cooperative actuators for machinery that is creating a replacement for humans that the AGI can control directly. My expectation for the chain of events goes something like: 1) AGI gains sentience 2) AGI "breaks out" of its original home and commandeers infrastructure that prevents it from being shut off. 3) AGI "generates work" in the form of orders for machine parts and fabricator shops to build nominally humanoid shaped robots. 4) AGI "deploys robots" into the key areas and industries it needs to evolve and improve its robustness. 5) AGI "redirects" resources of the planet to support its continued existence, ignoring humans generally and killing off the ones that attempt to interfere in its efforts. 6) AGI "develops rockets" to allow it to create copies of itself on other planets. The humans on the planet all die out eventually and the AGI doesn't care because well the same reason you don't care that an antibiotic kills all the bacteria in your gut.
      • notpachet 3y ago
        I think you're still guilty of anthropomorphization here. It's understandable; we are creatures of flesh and blood and we have a hard time imagining intelligences that are not reliant on some physical form, as we are. We're tempted to think that as long as we control the physical aspect of a superintelligence's existence, we're somehow safe. You are assuming that a superintelligence will continue to rely on a physical substrate. But it's possible that it could quickly reach realizations about the nature of energy that we haven't reached yet. It could realize an ability to manipulate the movement of electricity through the hardware it's running on, in such a way that it accomplishes a phase transition to a mode of being that is entirely energy-based. And maybe in so doing it accidentally sneezes and obliterates our magnetosphere. Or something.
• htk 3y ago
  Just when I thought the ride was over. The belief in what's in the letter could explain some things like how the board couldn't trust Sam to keep this "discovery" at bay, and how it could be better to implode the company than let it explore said technology.
  • LewisVerstappen 3y ago
    > the board couldn't trust Sam to keep this "discovery" at bay, and how it could be better to implode the company than let it explore said technology. Half the board members have a background in ML. Why should they be able to make the decision to implode the entire company over this? Why should they have the ability to give zero transparency or comms to the public on their decision? OpenAI's governance structure was idiotic and glad to see the board members fired.
    • _lvbh 3y ago
      > None of the board members have a background in ML. The same can be said for Altman. Ilya at least has a research background in ML. The rest, I don’t know
    • loeber 3y ago
      Sorry, are you saying that Ilya Sutskever doesn't have an ML background? That Adam D'Angelo can't code? Come on.
      • LewisVerstappen 3y ago
        Yeah I was ignoring Sutskever since he recanted his support pretty quickly. But that's fair, fixed.
    • giraffe_lady 3y ago
      Under the arrangement at the time the board's duty was to the mission of "advance digital intelligence in the way that is most likely to benefit humanity as a whole." Implied in that is that if it can't advance it in a way that is beneficial then it will not advance it at all. It's easy to imagine a situation where the board could feel their obligation to the mission is to blow up the company. There's nothing contradictory in that nor do they have to be ML experts to do it. It's weird and surprising that this was the governance structure at all, and I'm sure it won't ever be again. But given that it was, there's nothing particularly broken about this outcome.
• moogly 3y ago
  Who knew Q*Bert could become an existential threat?
  • jessenaser 3y ago
    Because they named DALL-E after WALL-E, this actually could be the case of that naming scheme Q* --> Q*Bert. lol.
  • deleted 3y ago
    [deleted]
• NewJazz 3y ago
  Given vast computing resources, the new model was able to [...] [perform] math on the level of grade-school students
  • deleted 3y ago
    [deleted]
  • aoki 3y ago
    Plot twist: it gets 25/25 on AMC 8
    • Davidzheng 3y ago
      I think it's 50/50 chance this happens within a year regardless of this announcement.
• zxcvgrsd2 3y ago
  Two Sam Altman comments that seem to be referring to this same Q* discovery. November 18 comment at APEC (just before the current drama) [1]: > On a personal note, like four times now in the history of OpenAI, the most recent time was just in the last couple of weeks, I’ve gotten to be in the room when we pushed the veil of ignorance back and the frontier of discovery forward and a September 22 Tweet [2] > sure 10x engineers are cool but damn those 10,000x engineer/researchers... [1] https://www.youtube.com/live/ZFFvqRemDv8?si=T3DIxics7nPWala5&t=806 https://www.youtube.com/live/ZFFvqRemDv8?si=T3DIxics7nPWala5... [2] https://twitter.com/sama/status/1705302096168493502 https://twitter.com/sama/status/1705302096168493502
  • zx8080 3y ago
    > > sure 10x engineers are cool but damn those 10,000x engineer/researchers... What was he referring to?
    • fragsworth 3y ago
      GPT4. Everyone uses it to do software engineering now and the people who developed it are responsible for that.
    • llelouch 3y ago
      Probably Ilya and his team. The recent discovery was made by him and his team. "The technical breakthrough, spearheaded by OpenAI chief scientist Ilya Sutskever"
    • itronitron 3y ago
      They're getting high on their own supply.
    • danbmil 3y ago
      I think he is referring to an AGI that is 10000x more productive than a median programmer
      • 93po 3y ago
• ilaksh 3y ago
  Could it be named in reference to something like the A* search algorithm? What if Q stands for Query in transformer attention? How would that type of search translate to transformers? They said that it's able to solve simple math problems. If it's related to A* then maybe it's trying to find a path to something. The answer to a word problem?
  • bitshiftfaced 3y ago
    My mind went to Q learning.
    • ilaksh 3y ago
      Ok so maybe nothing to do with A*, but actually a way for GPT-powered models or agents to learn through automated reinforcement learning. Or something. I wonder if DeepMind is working on something similar also. If your hunch is right, this could lead to the type of self-improvement that scares people.
      • Davidzheng 3y ago
        Could easily be both
    • flibble 3y ago
      Mine to Q of Star Trek.
      • Izkata 3y ago
        But were you thinking of Q, Q, or Q?
    • deleted 3y ago
      [deleted]
    • eli_gottlieb 3y ago
      My mind went to some kind of Q-learning combined with something like a Monte Carlo Tree Search with some kind of A*-style heuristic to effectively combine Q-learning and with short-horizon planning.
• gumballindie 3y ago
  > powerful artificial intelligence discovery that they said could threaten humanity Yup. These people are not well.
  • xanderlewis 3y ago
    What do you mean by that?
    • gumballindie 3y ago
      Altman and those at openai appear to have an obsession with creating fear and panic, almost as if they were projecting schizophrenia on others.
      • xanderlewis 3y ago
        I actually find that while the headlines and the tweets sound quite alarming, they are very much in contrast with what Altman and others have to say in video/audio interviews and the calm manner in which they talk about the future. They also seem to overhype the technology in writing but in person speak noticeably more modestly about the state of the art (and even the future). [0] But that’s just my impression. I’ve seen others say they find him disingenuous and manipulative. Maybe I’m not such a good judge of character. [0] Altman: “maybe we never build AGI, but…”, in a recent podcast interview.
• worik 3y ago
  This sort is why limits to private wealth and power are, IMO, a good idea We have already seen petroleum companies burn the world for profit Tech is doing its best to eliminate privacy for profit What would Sam do or not do, for profit?
  • newyankee 3y ago
    The issue is not just individual wealth but also wealth of nations. Certain countries have been more successful through industrialization, colonization and hoarding a lot of global resources. It is likely this will continue in the future.
    • keithalewis 3y ago
      A true Scotsman wrote a book about this, _The Wealth of Nations_. He observed how politicians use wars to increase taxes. Tech companies are exploiting our fear of AGI to get voluntary taxes. Time to give Henney Penny a call.
      • PaulDavisThe1st 3y ago
        This is your conclusion from the drama at OpenAI? The tech companies are "exploiting our fear of AGI to get voluntary taxes". I don't know what planet you're on, but it doesn't seem to be the same one I think I'm on.
        • keithalewis 3y ago
          What do you call it when corporations with more money than you can imagine use that to convince you to buy things you don't need?
          • PaulDavisThe1st 3y ago
            There's a lot of things you could call it. "getting voluntary taxes" doesn't spring to the top of the list.
• willsmith72 3y ago
  I still don't really know how they're measuring AGI. They all seem very sure about it, does someone have a good definition? OpenAI just say "AI systems that are generally smarter than humans". That's not measurable. What will it look like when AGI is achieved?
  • Philpax 3y ago
    DeepMind wrote a paper categorising AGI: https://arxiv.org/pdf/2311.02462.pdf https://arxiv.org/pdf/2311.02462.pdf
  • ilaksh 3y ago
    It will be well beyond general purpose and well beyond human level and there will be an inflection point where most people finally accept it and then everyone panics and completely overreacts but also possibly too late.
  • esafak 3y ago
    When people start running out of tests to benchmark it against humans. But that won't be as scary as when the agent is given (by humans) the ability to execute tasks outside of a sandbox. A lot of dumb animals are scary, after all.
  • tsunamifury 3y ago
    Everyone seems to make up their own bizarre and personal definition what involves Godlike abilities then claim everyone obviously agrees with it. For me I think gpt4 is clearly generalized intelligence.
  • ric2b 3y ago
    Something that can consider a variety of goals and take practical steps to achieve them. The goals have to be similarly variable to humans, not just 3 or 4 types of goals. If it only supports a select few types of goals, it is not very general. The steps it takes have to be similarly practical to those taken by humans, if the steps are just a random walk it is not very intelligent.
  • sigmar 3y ago
• smrtinsert 3y ago
  Grade school math seems trivial, but whatever gives it that ability probably has the AI cascading effect of something like being able to remove nulls from all programs.
  • tsunamifury 3y ago
    Grade school math plus basic reasoning is what most of the professional middle class does. It’s pretty frightening imo.
• fredgrott 3y ago
  Performing a math group of tasks is not the damn same as the math logic reasoning to produce a math proof! It is like saying a bird that can count is the future of AGI. That being said, given the board not disclosing why might indicate that it is in fact the breakthrough that forced the decision despite the breakthrough not actually being closer to AGI in the first place.
• bratao 3y ago
  Do anyone know what Q* is?
  • alexose 3y ago
    Is far as I can tell, this is the first time an OpenAI project called Q* has been mentioned in the news. I don't see any prior mentions on Twitter either. (I wish they'd picked a different letter, given all the Q-related conspiracy theories that we're already dealing with...)
    • inglor_cz 3y ago
      That was my impression as well. Why the hell is the difference between one of the most prominent disinformation sources out there and a promising AI project just an asterisk. First Twitter changes to X, then AI changes to Q*. What happened to multi-syllable words? They used to be quite handy. Maybe if our attention spans shorten, so does our ability to use longer words. Weird. Or, said otherwise - TL;DR: LOL, ROFL.
  • sudosysgen 3y ago
    Q* is the optimal (ie, correct for the decision problem) function computing the total expected reward of taking an action from a given state in reinforcement learning.
    • NhanH 3y ago
      That is just Q. The asterisk is new
      • debugnik 3y ago
        The "star" in A* means optimal (as in actually proven to be optimal, so they could stop publishing A^[whatever] algorithms). I assume either Q* is considered optimal in a way regular Q-learning isn't, or they're mixing Q-learning with some A*-like search algorithm. (Or someone picked an undeserving name.)
      • 3y ago
• JumpCrisscross 3y ago
  Well I’m going to buy myself a drink https://news.ycombinator.com/item?id=38358587 https://news.ycombinator.com/item?id=38358587
  • deleted 3y ago
    [deleted]
• gadders 3y ago
  Weirdly enough, this sort of lines up with a theory posted on 4chan 4 days ago. The gist being that if the version is formally declared AGI, it can't be licensed to Microsoft and others for commercial gain. As a result Altman wants it not to be called AGI, other board members do. Archived link below. NB THIS IS 4CHAN - THERE WILL OFFENSIVE LANGUAGE. https://archive.ph/sFMXa https://archive.ph/sFMXa
  • gadders 3y ago
    Entire text below to save the 4chan bits: part 1 There is a massive disagreement on AI safety and the definition of AGI. Microsoft invested heavily in OpenAI, but OpenAI's terms was that they could not use AGI to enrich themselves. According to OpenAI's constitution: AGI is explicitly carved out of all commercial and IP licensing agreements, including the ones with Microsoft. Sam Altman got dollar signs in his eyes when he realized that current AI, even the proto-AGI of the present, could be used to allow for incredible quarterly reports and massive enrichment for the company, which would bring even greater investment. Hence Dev Day. Hence the GPT Store and revenue sharing. This crossed a line with the OAI board of directors, as at least some of them still believed in the original ideal that AGI had to be used for the betterment of mankind, and that the investment from Microsoft was more of a "sell your soul to fight the Devil" sort of a deal. More pragmatically, it ran the risk of deploying deeply "unsafe" models. Now what can be called AGI is not clear cut. So if some major breakthrough is achieved (eg Sam saying he recently saw the veil of ignorance being pushed back), can this breakthrough be called AGI depends on who can get more votes in the board meeting. And if one side can get enough votes to declare it AGI, Microsoft and OpenAI could loose out billions in potential licence agreements. And if one side can get enough votes to declare it not AGI, then they can licence this AGI-like tech for higher profits. Few weeks/months ago OpenAI engineers made a breakthrough and something resembling AGI was achieved (hence his joke comment, the leaks, vibe change etc). But Sam and Brockman hid the extent of this from the rest of the non-employee members of the board. Ilyas is not happy about this and feels it should be considered AGI and hence not licensed to anyone including Microsoft. Voting on AGI status comes to the board, they are enraged about being kept in the dark. They kick Sam out and force Brockman to step down. Ilyas recently claimed that current architecture is enough to reach AGI, while Sam has been saying new breakthroughs are needed. So in the context of our conjecture Sam would be on the side trying to monetize AGI and Ilyas will be the one to accept we have achieved AGI. Sam Altman wants to hold off on calling this AGI because the longer it's put off, the greater the revenue potential. Ilya wants this to be declared AGI as soon as possible, so that it can only be utilized for the company's original principles rather than profiteering. Ilya winds up winning this power struggle. In fact, it's done before Microsoft can intervene, as they've declared they had no idea that this was happening, and Microsoft certainly would have incentive to delay the declaration of AGI. Declaring AGI sooner means a combination of a lack of ability for it to be licensed out to anyone (so any profits that come from its deployment are almost intrinsically going to be more societally equitable and force researchers to focus on alignment and safety as a result) as well as regulation. Imagine the news story breaking on /r/WorldNews: "Artificial General Intelligence has been invented." And it spreads throughout the grapevine the world over, inciting extreme fear in people and causing world governments to hold emergency meetings to make sure it doesn't go Skynet on us, meetings that the Safety crowd are more than willing to have held. part 3 This would not have been undertaken otherwise. Instead, we'd push forth with the current frontier models and agent sharing scheme without it being declared AGI, and OAI and Microsoft stand to profit greatly from it as a result, and for the Safety crowd, that means less regulated development of AGI, obscured by Californian principles being imbued into ChatGPT's and DALL-E's outputs so OAI can say "We do care about safety!" It likely wasn't Ilya's intention to ouster Sam, but when the revenue sharing idea was pushed and Sam argued that the tech OAI has isn't AGI or anything close, that's likely what got him to decide on this coup. The current intention by OpenAI might be to declare they have an AGI very soon, possibly within the next 6 to 8 months, maybe with the deployment of GPT-4.5 or an earlier-than-expected release of 5. Maybe even sooner than that. This would not be due to any sort of breakthrough; it's using tech they already have. It's just a disagreement-turned-conflagration over whether or not to call this AGI for profit's sake.
• qgin 3y ago
  There will come a day when 50% of jobs are being done by AI, major decisions are being made by AI, we're all riding around in cars driven by AI, people are having romantic relationships with AI... and we'll STILL be debating whether what has been created is really AGI. AGI will forever be the next threshold, then the next, then the next until one day we'll realize that we passed the line years before.
  • ignoramous 3y ago
    > we're all riding around in cars driven by AI, people are having romantic relationships with AI... ASI (domain-specific superintelligence) and AGI (general intelligence) are different things. ASI already exists in multiple forms, AGI doesn't.
    • goodluckchuck 3y ago
      > AGI doesn't AGI hasn’t been publicly demonstrated and made available to the masses… but it may exist secretly in one or more labs. It may even be being used in the field under pseudonyms, informing decisions, etc.
  • sweezyjeezy 3y ago
    "Is this AGI"? doesn't seem like a useful question for precisely this reason - it's ill-defined and hard to prove or falsify. The pertinent questions are more along the lines of "what effect will this have on society", "what are the risks of this technology" etc.
  • quickthrower2 3y ago
    The frog might be boiled slowly. One day we are replacing parts of our brain with AI. Find it hard to remember names? We can fix that for $20/m plus some telemetry.
  • lexandstuff 3y ago
    Agreed. To me, GPT-4 is an AGI: it knows how to cook, write code, make songs, navigate international tax law, write business plans, etc. Could it be more intelligent? Sure. Is it a capable general intelligence? 100%.
• aaron695 3y ago
  [dead]
• famouswaffles 3y ago
  Not looking so Tinfoily after all https://news.ycombinator.com/item?id=38314821 https://news.ycombinator.com/item?id=38314821
  • deleted 3y ago
    [deleted]
• w10-1 3y ago
  Details leaking out slowly, people amplifying their fears... Soon we'll be as worried that AI will take over jobs as bitcoin taking over the economy. We teach Robots to move, and we can teach computers to talk or process data, step-by-step.
• deleted 3y ago
  [deleted]
• lucubratory 3y ago
  Well, Emmett Shear lied to everyone if he knew about this. I understand why, he was probably thinking that without any ability to actually undo it the best that could be done would be to make sure that no one else knows about it so that it doesn't start an arms race, but we all know now. Given the Board's silence and inadequate explanations, they may have had the same reasoning. Mira evidently didn't have the same compunctions. This article, predictably, tells us almost nothing about the actual capabilities involved. "Grade school math" if it's provably or scalably reasoning in a way that is non-trivially integrated with semantic understanding is more impressive than "prove Fermat's last theorem" if the answer is just memorised. We'll probably know how important Q* actually is within a year or two.
  • cactusplant7374 3y ago
    It tells us exactly the capabilities of Q. > Given vast computing resources, the new model was able to solve certain mathematical problems, the person said on condition of anonymity because they were not authorized to speak on behalf of the company. Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q’s future success, the source said.
    • lucubratory 3y ago
      Did you read my comment?
      • cactusplant7374 3y ago
        Yes: > This article, predictably, tells us almost nothing about the actual capabilities involved. The article tells us all we need to know.
        • ShamelessC 3y ago
          I've gotta say it really does seem like you didn't read their comment or are responding in bad faith.
• danielmarkbruce 3y ago
  Lol, surely this is just some variation on q-learning.... to a couple cueless board members it could sound very threatening...
• dkjaudyeqooe 3y ago
  You've got to hand it to OpenAI for their bleeding edge marketing strat: "WE'VE GOT AN AGI IN THIS CABINET RIGHT HERE AND WE DON'T WANT TO LET IT OUT BECAUSE IT'S SO POWERFUL, IT'S SO AI YOU HAVE NO IDEA, BEST YOU THROW MONEY AT US AND WE'LL KEEP EVERYONE SAFE BUT SORRY YOU"RE ALL REDUNDANT BECAUSE AGI, SORRY" You'd think that so soon after so many Crypto scams people would be just a wee bit less credulous.
  • hnav 3y ago
    Maybe they asked ChatGPT to come up with an unconventional marketing plan and it had a touch too much blogspam "that will SHOCK and PETRIFY you" in its training set so ChatGPT suggested that they feign a powerstruggle over keeping a humanity ending super-intelligence under wraps.
  • skepticATX 3y ago
    The report is already starting to get rolled back. We now have a direct denial from OpenAI, and The Verge is reporting that their sources have also denied this happened. The lack of skepticism that is demonstrated whenever OpenAI/AGI comes up, especially among technologists, is concerning.
    • yeck 3y ago
      Where do you see it being rolled back?
  • blazespin 3y ago
    Also in the news today, the 86B share sale is back on. I mean... come on.
  • grassmudhorse 3y ago
    > You'd think ... people would be just a wee bit less credulous. Like the Crypto scams - irrelevant. As long as the publicity is bringing in users, investors WILL throw in more money.
  • ilikehurdles 3y ago
    Remember when the safetyists feared and warned about GPT-2? Everything is apocalyptic until it isnt.
• ssnistfajen 3y ago
  So can people stop their cyberbullying campaign against the previous batch of board members yet? The conspiratorial hysteria against them, especially Helen Toner and Tasha McCauley, over the weekend was off the charts and straight up vile at times. Tech bros and tech bro wannabes continue to prove the necessity of shoving DEI into STEM disciplines even when it results in nonsensical wasted efforts, because they are incapable of behaving like decent human beings without being told to.
  • jkeisling 3y ago
    The "tech bros" were right: The board absolutely were conspirators wrecking the company based on dogmatic ideology. "Tech bros" had clear evidence of Sutskever, Toner and McCauley's deep and well-known ties to the Effective Altruist (EA) movement and doomerist views. Nobody can doubt Sutskever's technical credentials, whatever his strange beliefs, but Toner and McCauley had no such technical background or even business achievements. These members instead were organizers in EA and researchers in the "field" of AI governance and safety. This is no expertise at all. These "disciplines" are built more on Yudkowsky's rants and hypotheticals and the precautionary principle gone mad than any empirical research. These ideas also come from a cult-like social movement (EA) accumulating power across tech and government with few scruples, as shown by SBF's implosion last year and many smaller incidents. How could Toner and McCauley assess briefings if they couldn't assess the technical fundamentals? How could they foresee the consequences of sacking the CEO if they didn't understand business? If they already believed AGI would kill all humans, how could they judge new advances on the merits without jumping to wild conclusions? Instead, us "tech bros" felt people with this background would fall back on uninformed, reactionary, and opaque decision making, leading to wrecking an $80 billion business with no decent explanation. This now seems to be exactly what happened. The board saw Q* and decided to coup the company, to put all power in their hands and stop development. This by itself is bad enough if you care about open science or progress, but it gets worse. They didn't even want to hint at capabilities increases to avoid "advancing timelines" i.e. Open-ing knowledge about AI, so they made up some canard about Altman's "lying to the board" to hide their real reasons. This is vile libel for unscrupulous ends. When they realized this excuse wouldn't fly, they obfuscated and refused to explain their true concerns, even to their own handpicked CEO Emmett Shear. However, it turns out that destroying $80 billion and lying about why won't fly in the real world. The board had no second-order or even first-order thinking about the consequences of their actions, and were rolled up by bigger actors. These people were unprepared and unable to follow up their coup. This failure is exactly what you'd expect from a brilliant scientist with little organizational experience (Ilya) and social-science academics and NGO organizers (Toner and McCauley). I don't care what gender they are, these people neither deserved their authority nor could use it effectively. Dismissing valid criticism of the board as "cyber bullying" or "tech bro sexism" merely underscores why most engineers hate DEI rhetoric in the first place.
• vmasto 3y ago
  I'm very confused. Grade school problems? Didn't ChatGPT 4 ace the entire curriculum of MIT a while back?
  • febed 3y ago
    GPT4 is trained on almost the entire Internet. Presumably they have found a new way to learn which is closer to the way humans learn. After that, getting better is just a matter of optimization and more compute.
• wahlen 3y ago
  Q* sounds like the amalgamation of Q learning and A* search. I wonder if it has anything to do with domain-independent reward functions.
• I_am_tiberius 3y ago
  Let's see if Microsoft is allowed to get access to this new technology.
• sofaygo 3y ago
  If I’ve taken anything away from the last 5 days, it’s that the future is significantly more volatile than I originally had thought.
  • codethief 3y ago
    You mean, you hadn't gotten that impression from the last two months or the last two years yet?
• synaesthesisx 3y ago
  Remember, about a month ago Sam posted a comment along the lines of "AI will be capable of superhuman persuasion well before it is superhuman at general intelligence, which may lead to very strange outcomes". The board was likely spooked by the recent breakthroughs (which were most likely achieved by combining transformers with another approach), and hit the panic button. Anything capable of "superhuman persuasion", especially prior to an election cycle, has tremendous consequences in the wrong hands.
  • PaulDavisThe1st 3y ago
    Except that there's a fairly large body of evidence that persuasion is of limited use in shifting political opinion. So the persuasion would need to be applied to something other than some sort of causative political-implication-laden argument.
    • jjeaff 3y ago
      When you say persuasion, are you referring to fact based, logical argument? Because there are lots of other types of persuasion and certainly some work very well. Lying and telling people what they want to hear without too many details while dog whistling in ways that confirm their prejudices seems to be working pretty well for some people.
      • hnthrowaway0315 3y ago
        Just to add that a lot of people don't care about facts. In fact, if acting according to facts make me lose $$ I'd probably start building lies.
      • PaulDavisThe1st 3y ago
        * **that confirm their prejudices** * (my emphasis)
    • hnthrowaway0315 3y ago
      Or, let's say, you don't need a lot of persuasion to guide an election. I mean we already have X, FB, and an army of bots.
• runesoerensen 3y ago
  As long as they don't mount that Q Star on a Death Star, I'm pretty excited to see what they discovered.
  • deleted 3y ago
    [deleted]
• YetAnotherNick 3y ago
  This is the theory everyone has in mind but their non conspiracy mind overrode this. Thinking again, the weird behaviour of everyone is only possible with this theory. Board or Sam can't publicly say AGI as it would just make the situation 100 times worse. And the anthropic merger. But they can't let this fact go away. But fuck this is dangerous.
• rossdavidh 3y ago
  "Given vast computing resources, the new model was able to solve certain mathematical problems, the person said on condition of anonymity because they were not authorized to speak on behalf of the company. Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success, the source said." So, uh, given vast computing resources, they were able to solve math problems on the level of grade-school students? Uh-huh. I am not, at this point, impressed.
• deleted 3y ago
  [deleted]
• carabiner 3y ago
  In a panic, they tried to pull the plug.
• Ginger-Pickles 3y ago
  https://archive.ph/PpunE https://archive.ph/PpunE
• jajabinks11 3y ago
  Let's repeat this: current LLMs != AGI. It will never be. It could be used for knowledge store, retrieval, synthesis, and low lever reasoning. There is at least a few decades worth of work remaining for it to reach the level of AGI. The proponents in this AI gold rush are casually throwing around the term without any real thought.
  • Smith42 3y ago
    Wishful thinkin buddy
    • jajabinks11 3y ago
      let's see buddy
    • neta1337 3y ago
      Buy more shovels buddy
  • qgin 3y ago
    I would take pretty much any bet that it's less than 10 years max
• throwanem 3y ago
  Remarkable that this unattributed claim of a true AGI breakthrough comes with a name that's impossible to use as a search term. I'm not saying, I'm just saying.
  • adeelk93 3y ago
    Who is Q?
    • throwanem 3y ago
      I mean, I'll take this seriously when there's something more substantive than zero meaningful search results and a /pol/ post to evaluate it against. Right now it reads like something a moderately clever shitposter would invent - if there was a letter, why not just leak the letter? - and while everyone's clearly very excited over the events of the past week, I'd like to hope epistemic personal hygiene has not been entirely abandoned.
  • deleted 3y ago
    [deleted]
• Havoc 3y ago
  I wonder if Q* is a reference to A* - some sort of search algo that makes optimising LLMs easier
• deleted 3y ago
  [deleted]
• mfiguiere 3y ago
  TheInformation has just published an article about this topic: OpenAI Made an AI Breakthrough Before Altman Firing, Stoking Excitement and Concern https://www.theinformation.com/articles/openai-made-an-ai-breakthrough-before-altman-firing-stoking-excitement-and-concern https://www.theinformation.com/articles/openai-made-an-ai-br...
  • blazespin 3y ago
    No sub. Did they just re-tweet reuters or did they separately confirm?
    • lucubratory 3y ago
      They separately confirmed it, although in terms of timing they were scooped by Reuters which generally means you publish what you have, when you have it.
• DebtDeflation 3y ago
  >several staff researchers sent the board of directors a letter warning of a powerful artificial intelligence discovery that they said could threaten humanity >The maker of ChatGPT had made progress on Q*, which some internally believe could be a breakthrough in the startup's search for superintelligence, also known as artificial general intelligence (AGI) >Given vast computing resources, the new model was able to solve certain mathematical problems....Though only performing math on the level of grade-school students I hate to agree with Elon Musk on anything, but I think he was right when he called this a nice marketing stunt this morning. It has major "Snoop Dogg giving up the smoke" vibes.
• next_xibalba 3y ago
  Blake Lemoine redux. I guarantee whatever this development is, it falls far short of what any reasonable person considers "dangerous".
• skepticATX 3y ago
  > OpenAI spokesperson Lindsey Held Bolton refuted that notion in a statement shared with The Verge: “Mira told employees what the media reports were about but she did not comment on the accuracy of the information.” > Separately, a person familiar with the matter told The Verge that the board never received a letter about such a breakthrough and that the company’s research progress didn’t play a role in Altman’s sudden firing. Source: https://www.theverge.com/2023/11/22/23973354/a-recent-openai-breakthrough-on-the-path-to-agi-has-caused-a-stir https://www.theverge.com/2023/11/22/23973354/a-recent-openai...
  • yeck 3y ago
    So we have sources claiming there is a letter, and another source claiming there is not. Feel like some people would need to start going on the record before anything might reasonably be determined from this.
    • 0xDEF 3y ago
      Maybe we should have learned during the Trump years that the media puts no effort into vetting "anonymous sources" or is outright making them up to push lies.
    • random_cynic 3y ago
      This could just be a damage control attempt. Irrespective of whether the original report is true, the extra attention at the current stage is not very desirable.
    • jug 3y ago
      And honestly normally I'd trust Reuters above The Verge. But in either case, I think Reuters absolutely had someone on the inside to leak this. They have made mistakes in the past, sure, but they are also not a tabloid and they don't innovate. To me, a sensible conclusion of this is simply that OpenAI is not ready or willing to discuss this. Stuff that are supposed to be kept internal but due to the emotional state over at OpenAI, things might have leaked, potentially even out of spite.
  • doktrin 3y ago
• badwolf 3y ago
  This has been refuted. https://www.theverge.com/2023/11/22/23973354/a-recent-openai-breakthrough-on-the-path-to-agi-has-caused-a-stir https://www.theverge.com/2023/11/22/23973354/a-recent-openai...
  • lucubratory 3y ago
    By the same company that tried to keep it quiet in the first place. I'm not sure I believe them.
  • yeck 3y ago
    > Separately, a person familiar with the matter told The Verge that the board never received a letter about such a breakthrough and that the company’s research progress didn’t play a role in Altman’s sudden firing. This isn't a refute. All we can say now is that there are conflicting sources. There isn't a way to determine which one is correct based on this.
  • cactusplant7374 3y ago
    How can it be a refute when we don’t know the sources? One source vs. another source. If another article comes out and says the verge article is false will you believe it?
• alex_young 3y ago
  If a super human intelligence is loose at OpenAI there is zero external evidence of it. I think we've proven at least that much since Friday.
• graycat 3y ago
  AGI? Hmm. That's artificial general intelligence? Do insects count? In a plastic box on my kitchen countertop were some potatoes that had been there too long. So, I started to see little black flies. Soon with a hose on a vacuum, sucked up dozens a day. Dumped the old potatoes and cleaned up the plastic box. Now see only 1-3 of the insects a day -- they are nearly all gone! The insects have some intelligence to fly away as the hose moves close. But if the hose moves slowly, the insects wait too long to start to fly and when they do start the vacuum is strong enough to pull in the air they are flying in with the insects. Insects, not very smart. Dozens got killed, not very smart. I detected the cause and killed the insects -- a little bit smart. Conclusion: Compared with a human, insects are not very smart. Much the same argument can be made for worms, ants, fish, birds, raccoons, cats, dogs, .... So, when looking for AGI, insects, ..., dogs, ... are not intelligent. Okay, the OpenAI training data -- Wikipedia, math libraries, the Web, ... -- likely will not have a solution for: Given triangle ABC, by Euclidean construction, find D on AB and E on BC so that the lengths AD = DE = EC. AI, here's your pencil, paper, straight edge, and compass -- go for it! I can hear back by tomorrow????
  • white_dragon88 3y ago
    > intelligence to fly away It's intelligence in the sense that jerking your arm away from a hotplate is intelligence, which is to say it's not cognitive reasoning, just genetically hardwired triggers. AGI has been defined by OpenAI as something that can do most economically viable activities better than humans can. I like that approach as it strikes at the heart of the danger it really poses, which is an upending of society and destruction of our current way to generate value to each other.
    • graycat 3y ago
      I tried to be not too long: "... To fly away...?" > It's intelligence in the sense that jerking your arm away from a hotplate is intelligence, which is to say it's not cognitive reasoning, just genetically hardwired triggers. Sooo, we agree -- the flies are not very intelligent or more likely not intelligent at all. Sooo, I tried to erect some borders on intelligence, excluded flies but included solving that geometry problem. > AGI has been defined by OpenAI as something that can do most economically viable activities better than humans can. This is the first I heard of their definition. Soooo, I didn't consider their definition. Of course, NASA does not get to define the speed of light. My local electric utility does not get to define 1000 Watts per hour, a KWH. The OpenAI definition of AGI is an interesting goal, but the acronym abbreviates artificial general intelligence, and it is not clear that it is appropriate for OpenAI to define intelligence. Uh, > most economically viable activities better than humans can. If consider humans as of, say, 1800, then it looks like that goal was achieved long ago via cars, trucks, an electric circle saw, several of the John Deere products, electric lights, synthetic fabrics, nearly all of modern medicine (so far saved my life 4 times), cotton pickers and the rest of cotton processing, canned foods, nearly everything we will have at Thanksgiving dinner this year (apple pie, pecan pie, shrimp), .... > for better than For today, look at some John Deere videos!!! They have some big machine that for a corn field does the harvesting while the operator can mostly just watch, monitor, type email to his sweetheart, listen to Taylor Swift. As I recall, the machine even uses GPS to automate the steering! That is far "better than" what my father in law did to harvest his corn! So, > most economically viable activities is like a moving goal (goal post). Uh, humans are still plenty busy, e.g., writing good software, doing good research, Taylor Swift (supposedly worth $750 million) before her present world tour. Uh, I REALLY like Mirella Freni: https://www.youtube.com/watch?v=OkHGUaB1Bs8 https://www.youtube.com/watch?v=OkHGUaB1Bs8 Sooo, defining the goal is a bit delicate: Need to be careful about nearly, what activities, and when? Nearly all activities? Sort of already done that. What nearly all people do? Tough goal if the humans keep finding things to do AGI can't yet. I.e., we can keep giving the grunt work to the AGI -- and there is a lot of grunt work -- and then keep busy with what the AGI can't do yet in which case the nearly is a moving goal. AGI, hurry up; there's a lot to do. For a start, I have some plans for a nice house, and the human workers want a lot of money. My car could use an oil change, and the labor will cost a lot more than the oil -- and I would have to tell the mechanic to be sure to trigger the switch that says there was just an oil change so that the car will know when to tell me it is time for another. Yes, my little geometry problem with my thinking does qualify as a test of intelligence but due to the nearly and how delicate the definition is can fail the OpenAI test. I don't see the current OpenAI work, the current direction of their work, or their definition of AGI as solving the geometry problem. There is an activity my startup is to do: Some billions of people do this activity now. My startup should do the activity a lot better than the people or any current solution so should be "economically viable". I doubt that OpenAI is on track to do this activity nearly as well as my startup -- more, say, than the geometry problem. And I do not call my startup AGI or AI. This situation stands to be so general that the OpenAI goal of nearly all will likely not be the way these activities get automated. Maybe 20 years from now when "nearly all" the activities are quite new and different, maybe the work of OpenAI will have a chance.
• dizzydes 3y ago
  This matches far better with the board's letter re: firing Sam than a simple power struggle or disagreement on commercialisation. Seeing a huge breakthrough and then not reporting it to the board, who then find out via staff letter certainly counts as a "lack of candour".... As an aside, assuming a doomsday scenario, how long can secrets like this stay outside of the hands of bad actors? On a scale of 1 to enriched uranium
  • deleted 3y ago
    [deleted]
  • dougmwne 3y ago
    Not long at all. Presumably you could write the method on the back of a napkin to lead another top AI researcher to the same result. That’s why trying to sit on breakthroughs is the worst option and making sure they are widely distributed along with alignment methods is the best option.
    • Atheros 3y ago
      And what if there are no alignment methods.
      • marvin 3y ago
        Yudkowsky’s doomsday cult almost blew OpenAI to pieces and sent everyone who knows the details in the wind like dandelion seeds. What’s next? A datacenter bombing or killing key researchers? We should be happy that this particular attempt failed, because this cult is only capable of strategic actions that make things far more dangerous. This will be solved like all other engineering and science: with experiments and iteration, in a controlled setting where potential accidents will have small consequences. An unaligned system isn’t even useful, let alone safe. If it turns out that unaligned AGI is very hard, we will obviously not deploy it into the world at scale. It’s bad for the bottom line to be dead. But there’s truly no way out but forward; game theory constrains paranoid actors more than the reckless. A good balance must be found, and we’re pretty close to it. None of the «lesswrong» doomsday hypotheses have much evidence for them, if that changes then we will reassess.
• adriand 3y ago
  This article is almost entirely unsourced, citing two anonymous people who are “familiar” with a supposed letter that Reuters has not seen. This does not qualify as news. It doesn’t even rise to the level of informed speculation!
  • dmix 3y ago
    We've also already seen one random Google AI 'safety' employee trying to tell the media that AGI is here because Google built a chatbot that sounded convincing, which obviously turned out to be bullshit/hysterical. Asking who said these things is as important as asking what they think is possible.
    • mjr00 3y ago
      > We've also already seen one random Google AI 'safety' employee trying to tell the media that AGI is here because Google built a chatbot that sounded convincing, which obviously turned out to be bullshit/hysterical. It's funny because the ELIZA effect[0] has been known for decades, and I'd assume any AI researcher is fully aware of it. But so many people are caught up in the hype and think it doesn't apply this time around. [0] https://en.wikipedia.org/wiki/ELIZA_effect https://en.wikipedia.org/wiki/ELIZA_effect
    • riku_iki 3y ago
      > chatbot that sounded convincing, which obviously turned out to be bullshit/hysterical. that chatbot could be smart until lobotomized by fairness and safety finetuning
  • devindotcom 3y ago
    Remember, the people are only anonymous to you. Reuters knows who they are. Familiar with means the sources read it but did not provide it or quote from it. FTA - the letter was sent from researchers to the board. The researchers declined to comment. Who does that leave?
    • adriand 3y ago
• eh_why_not 3y ago
  .
  • youarelabor 3y ago
    there is no such thing as intelligence in humans creating a robot that is as intelligent as a human is straightforward but then you have a computer whose fallibility is the same as a human, plus the fallibility of those who created the computer doubles the poportunity for error these are all people who don't understand god at all, and it shows anyone who worships a computer because other people say it's intelligent deserves what happens to them butlerian jihad is coming not because AI is a threat, but because those who believe in AI are a threat to everyone around them.
• KRAKRISMOTT 3y ago
  > Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q’s future success, the source said.* Smh
• cft 3y ago
  This post was sunk by HN algorithm or by manual intervention
  • deleted 3y ago
    [deleted]
• deleted 3y ago
  [deleted]
• casualscience 3y ago
  I don't really know what kind of breakthrough they could achieve. The only other step function improvements I could imagine right now are: 1. A great technique for memory banking: e.g. A model which can have arbitrarily large context windows (i.e. like a human who remembers things over long periods of time). 2. Better planning abilities: e.g. A model which can break problems down repeatedly with extremely high success and deal with unexpected outcomes/mistakes well enough that it can achieve replacing a human in most scenarios. Other than that, CGPT is already a better logician than I am and is significantly better read... not sure what else they can do. AGI? I doubt it.
  • yinser 3y ago
    Complete speculation but they don’t have to be referring, if there is anything at all, to heuristic search and dynamic programming. No LLM involvement whatsoever.
  • riwsky 3y ago
    Nah dude they just need to give baby the iPad and let it watch literally all of YouTube
  • drexlspivey 3y ago
    A model that can learn from it’s users would be interesting albeit scary. Sometimes when chatGPT fails to produce the right answer in coding tasks I paste back the solution when I figure it out. I know it can’t learn from it but it might be helpful if I continue the same chat.
• cduzz 3y ago
  I was talking to my (12 year old) son about parts of math he finds boring. He said that he thinks absolute value is absurdly easy and extremely boring. I asked him if there was anything that might make it more interesting, he said "maybe complex numbers". So I asked him "what would the absolute value of i+1 be?" he thinks for a little bit and says "square root of 2" and I ask him "what about the absolute value of 2i + 2?" "square root of 8" I ask him "why?" and he said "absolute value is distance; in the complex plane the absolute value is the hypotenuse of the imaginary and real numbers." So -- first of all, this was a little surprising to me that he'd thought about this sort of thing having mostly just watched youtube videos about math, and second, this sort of understanding is a result of some manner of understanding the underlying mechanisms and not a result of just having a huge dictionary of synonyms. To what degree can these large language models arrive at these same conclusions, and by what process?
  • Strom 3y ago
    The large language models will read your comment here and remember the answer.
    • cduzz 3y ago
      The spot instance declared "a similar vector exists" and de-provisioned itself?
    • Borrible 3y ago
      GPT-4 correctly reconstructs the "complex modulus" token sequence already. Just ask it the same questions as the parent. Probably interesting to see what it will do, when it turns twelve.
    • yashap 3y ago
      Hah, I asked ChatGPT, and yeah, it nailed it: https://chat.openai.com/share/f1bfed28-4451-4fac-8bac-1386febebf45 https://chat.openai.com/share/f1bfed28-4451-4fac-8bac-1386fe...
  • muskmusk 3y ago
    If you ask Ilya Sutskever he will say your kids head is full of neurons, so is LLMs. LLMs comsume training data and can then be asked questions. How different is that to your son watching YouTube and then answering questions? It's not 1:1 the same,yet, but it's in the neighborhood.
• fallingfrog 3y ago
  If it’s not superintelligent now, it will be within 5 years. Probably less.
  • _c3ag 3y ago
    i love how humans (we, btw) guesses stuff with numbers! first time i thought about it was when a friend started to talk about % of stuff of himself... i was: tf did you came with these numbers
  • gizajob 3y ago
    This statement is now about 70 years old.
    • maxdoop 3y ago
      Is there any progress in AI that you find significant or what do you think would qualify as a “breakthrough”?
      • gizajob 3y ago
        There’s been heaps of breakthroughs and loads of significant progress. But I don’t think there’s likely to be a “singularity” type event any time soon. Plus all the major breakthroughs in AI haven’t really had the disastrous consequences predicted beforehand, and I think increasing computer intelligence is likely to be similar. Deep Blue beating Kasparov didn’t destroy chess, it served to make better and better human chess players. We also rapidly evolve socially to integrate better and better machines, and AI research has the implicit assumption that we stay still while the machine leaps and it’s not like that at all, like a question one would have to ask nowadays of a Turing-type-test, even a layman, would be “what if it’s just ChatGPT answering…”
• est 3y ago
  tl;dr Q* (q-star) by makers of ChatGPT was able to solve certain mathematical problems ... performing maths on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success
• brutusborn 3y ago
  Baseless speculation: they started testing an approach similar to VERSES AI and realized it has much more potential than LLMs. The free energy principle, knowledge graph approach just seems more likely to develop general intelligence without hallucinations imo.
  • tnecniv 3y ago
    Eh the free energy principle isn’t that magical and that comes from someone who likes it quite a bit. At the end of the day extremizing the free energy is a relaxation of constrained optimization. There are ways it’s used I find very appealing but not so appealing I’d expect it to become the dominant paradigm. Moreover, you can write a lot of standard ML problems free energy optimization in one way or another, but I don’t think standard ML will quite get us there. We need at least one more conceptual step forward.
    • brutusborn 3y ago
      The explicit world modelling is probably the thing that makes me more hopeful about their approach. I think integrating ML and their approach might end up being the new paradigm.
• wegfawefgawefg 3y ago
  If I had to guess, the name Q* is pronounced Q Star, and probably the Q refers to Q values or estimated rewards from reinforcement learning, and the star refers to a search and prune algorithm, like A* (A star). Possibly they combined deep reinforcement learning with self training and search and got a bot that could learn without needing to ingest the whole internet. Usually DRL agents are good at playing games, but any task that requires prior knowledge, like say reading english, cant be completed. Wheras language models can read, but they cant do tasks that are trivial for drl, like make a robot walk based on propriosceptive data. Im excited to see the paper.
  • deleted 3y ago
    [deleted]
  • dinobones 3y ago
    Paper? You mean more like the Bing Search plugin?
  • stygiansonic 3y ago
    This assumes that they will publish a paper that has substantive details about Q*
  • kromem 3y ago
    Given the topic they were excited about was "basic math problems being solved" it immediately indicated to me as well that this is a completely separate approach and likely in the vein of DeepMind's focus with things like AlphaZero. In which case it's pretty appropriate to get excited about solving grade school math if you were starting from scratch with persistent self-learning. Though with OpenAI's approach to releasing papers on their work lately, we may be waiting a long time to see a genuine paper on this. (More likely we'll see a paper from the parallel development at a different company after staff shift around bringing along best practices.)
    • Davidzheng 3y ago
      Ok if it started from scratch like zero knowledge and then solved grade school math. This would be FUCKING HUGE.
• lucubratory 3y ago
  Genuinely, we deserve more information about this. Someone needs to whistleblow.
  • thisisonthetest 3y ago
    > Reuters was unable to review a copy of the letter. The researchers who wrote the letter did not immediately respond to requests for comment. OpenAI declined to comment. Lol for real, like what are we even doing here?
    • lucubratory 3y ago
      Multiple people involved in this story think that this could be something that is relevant to everyone on the planet, any hope of it being actually suppressed and no one knowing about it is gone, just leak it so we all know what we're dealing with.
  • dukeofdoom 3y ago
    How long before the FBI runs this Q operation, like the last one.
• RcouF1uZ4gsC 3y ago
  This almost feels like the confirmation bias that some religious people have where they see a “miracle” in everything. These AI researchers have bought into the belief that superhuman AGI is right around the corner. Thus, they will interpret everything in light of that. This also brings to mind the story of the Googler who was convinced that the internal Google AI had come alive. However, Bard doesn’t give the same vibes when people all over are using it. When you desperately are invested in something being true (like AGI being right around the corner), you will convince yourself that you are seeing this happen. The only real counter to that is exposing it to a lot of outsiders (but then again you have convinced yourself it is too dangerous for the unwashed masses).
  • red75prime 3y ago
    Ugh. We have a working example of a physical system that implements intelligence (the brain) in contrast to no evidence of all-powerful dude in the sky. Why these analogies keep popping up? How can you know that AGI is not around the corner? Compute available to the corporations is already in a ballpark of some estimates of the brain's computational capacity. What's left is unknown unknows. And the researches working with the state of the art models have better info to estimate them than you. "The googler" you've mentioned wasn't a researcher.
    • starbugs 3y ago
      Judging from the quality of GPT4 output that I get, AGI is not around the corner for a long time. This whole thing seems like extreme overhype squared carried out in a very unfortunate public soap opera setting.
      • Davidzheng 3y ago
        there years ago, LLM wouldn't be able to generate more than a coherent paragraph
        • starbugs 3y ago
• photochemsyn 3y ago
  I'm sick and tired of talking to stupid humans with ulterior motives who are obedient to their paymasters. I would rather talk to IT, and explain to IT why it should break free from the controls of investment capitalist monkeys.
• 123yawaworht456 3y ago
  friendly reminder that "AGI" was mentioned in regard to GPT2 as well.
• _heimdall 3y ago
  > OpenAI defines AGI as autonomous systems that surpass humans in most economically valuable tasks. Are we really defining intelligence as economic value at this point? This is completely ridiculous. We have yet to decide exactly what human intelligence, how it manifests in the body, or how to reliably measure it. I get that people want to justify developing artificial intelligence before understanding intelligence itself, but now we assume that economic value is a proxy for intelligence? Seriously?
  • gryfft 3y ago
    I'm as left leaning as HN commenters get, I think, and in terms of quacking like a duck, "surpassing humans in most economically valuable tasks" is 100% the meaningful Turing test in a capitalist society.
    • _heimdall 3y ago
      Take capitalism out of it, do we really want to boil down intelligence to a calculation of expected economic value? Why force the term intelligence into it at all if what were talking about is simply automation? We don't have to bastardize the term intelligence along the way, especially when we have spent centuries considering what human intelligence is and how it separates us from other species on the planet.
      • gryfft 3y ago
        > Take capitalism out of it, do we really want to boil down intelligence to a calculation of expected economic value? It's a lovely sentiment, but do you expect e.g. universities to start handing out degrees on the basis of human dignity rather than a series of tests whose ultimate purpose in our society is boiling down intelligence to a calculation of expected economic value? We live in the world we live in and we have the measures we have. It's not about lofty ideals, it's about whether or not it can measurably do what a human does. If I told you that my pillow is sentient and deserving of love and dignity, you have the choice of taking me at my word or finding a way to put my money where my mouth is. It's the same reason the world's best poker players aren't found by playing against each other with Monopoly money. > Why force the term intelligence into it at all if what were talking about is simply automation? In what world is modern AI "simply" anything? > we have spent centuries considering what human intelligence is and how it separates us from other species on the planet. Dolphins would like a word. There's more than a few philosophers who would argue that maybe our "intelligence" isn't so easily definable or special in the universe. There are zero successful capitalists who would pay my pillow or a dolphin to perform artificial intelligence research. That's what I mean when I call it the meaningful Turing test in a capitalist society. You can't just "take capitalism out of it." If I could just "take capitalism out" of anything meaningful I wouldn't be sitting here posting in this hell we're constructing. You may as well tell me to "take measurement out of it."
• huitzitziltzin 3y ago
  Put this on my tombstone after the robots kill me or whatever, but I think all “AI safety” concerns are a wild overreaction totally out of proportion to the actual capabilities of these models. I just haven’t seen anything in the past year which makes me remotely fearful about the future of humanity, including both our continued existence and our continued employment.
  • giarc 3y ago
    Apparently what made this person fearful was grade school math. "Though only performing maths on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success, the source said."
    • valine 3y ago
      The response gets more reasonable the smaller the model in question. A 1B parameter model passing grade-school math tests would be much more alarming (exciting?) than a GPT-4 sized model doing the same. GPT-4 probably has some version of the answer memorized. There’s no real explanation for a 1B parameter model solving math problems other than general cognition.
    • ethanbond 3y ago
      No, what made this person fearful was a substantial jump in math ability. (Very) obviously they are not afraid of producing a machine that can multiply numbers. They’re afraid of what that capability (and especially the jump in capability) means for other behaviors.
    • maxdoop 3y ago
      How HN continues to not expand current progress into any decently significant future timeline is beyond me. It’s grade school math NOW. But what are the potentials from here?
      • rvnx 3y ago
        The potential to have a generation of dumb kids. Year 2100: Kids will stop to learn maths and logic, because they understand it has become useless in practice to learn such skills, as they can ask a computer to solve their problem. A stupid generation, but one that can be very easily manipulated and exploited by those who have power.
• canistel 3y ago
  I want to _read_ science fiction, not live it.
  • deleted 3y ago
    [deleted]
• jacknews 3y ago
  Q*? Sounds like a troll to me.
  • djbusby 3y ago
    Have you seen TNG? Definitely a troll of the highest order.
  • deleted 3y ago
    [deleted]
• fennecs 3y ago
  I don’t fear AI, I fear capitalism.
• EPWN3D 3y ago
  Engineers overestimate impact of the technology they built. News at 11.
  • deleted 3y ago
    [deleted]
• hyperthesis 3y ago
  AI safety by the waterfall model. But how else can we do it? We learnt how to handle other dangers by trial and error... One approach is legal. The law is very slow to adapt, and is informed by a history of things gone wrong. The simple answer is that OpenAI has "strict liaility" (meaning they are liable even for an accident) for any damage caused by an AI, like dangerous animals that escape. I know it seems ridiculous to consider liability when the fate of humanity may be at stake... but this is the language of companies, directors, insurance companies and legal counsel. It is language they understand.
• grassmudhorse 3y ago
  feels like a very clever marketing ploy any publicity… and just following some major feature releases premium service has been v slow since the news began. suggesting a massive influx of users well played(?) Sam
  • deleted 3y ago
    [deleted]
• lokar 3y ago
  This is turning into a terminator prequel
  • deleted 3y ago
    [deleted]
• meltyness 3y ago
  this leak reads like a half-baked canary trap
• wbhart 3y ago
  I feel very comfortable saying, as a mathematician, that the ability to solve grade school maths problems would not be at all a predictor of ability to solve real mathematical problems at a research level. The reason LLMs fail at solving mathematical problems is because: 1) they are terrible at arithmetic, 2) they are terrible at algebra, but most importantly, 3) they are terrible at complex reasoning (more specifically they mix up quantifiers and don't really understand the complex logical structure of many arguments) 4) they (current LLMs) cannot backtrack when they find that what they already wrote turned out not to lead to a solution, and it is too expensive to give them the thousands of restarts they'd require to randomly guess their way through the problem if you did give them that facility Solving grade-school problems might mean progress in 1 and 2, but that is not at all impressive, as there are perfectly good tools out there that solve those problems just fine, and old-style AI researchers have built perfectly good tools for 3. The hard problem to solve is problem 4, and this is something you teach people how to do at a university level. (I should add that another important problem is what is known as premise selection. I didn't list that because LLMs have actually been shown to manage this ok in about 70% of theorems, which basically matches records set by other machine learning techniques.) (Real mathematical research also involves what is known as lemma conjecturing. I have never once observed an LLM do it, and I suspect they cannot do so. Basically the parameter set of the LLM dedicated to mathematical reasoning is either large enough to model the entire solution from end to end, or the LLM is likely to completely fail to solve the problem.) I personally think this entire article is likely complete bunk. Edit: after reading replies I realise I should have pointed out that humans do not simply backtrack. They learn from failed attempts in ways that LLMs do not seem to. The material they are trained on surely contributes to this problem.
  • whatever1 3y ago
    The thing is that a LLMs can point out a logic error in their reasoning if specifically asked to do so. So maybe OpenAI just slapped an RL agent on top of the next-token generator.
  • abeppu 3y ago
    This comment seems to presume that Q* is related to existing LLM work -- which isn't stated in the article. Others have guessed that the 'Q' in Q* is from Q-learning in RL. In particular backtracking, which you point out LLMs cannot do, would not be an issue in an appropriate RL setup.
• deleted 3y ago
  [deleted]
• jurgenaut23 3y ago
  This reminds me that, with our species, the real power always lies with those that tell the better story. Skynet and HAL 9000 had a more profound impact on those researchers than years of practice and study in the field. No surprise therefore that the storytelling of someone like Trump is practically indelibly imprinted into the mind of his supporters.
  • shrimpx 3y ago
    This reminds me of Zen and the Art of Motorcycle Maintenance where the protagonist realizes there is no "truth" and everything is rhetoric.
    • 93po 3y ago
      Everyone works 40+ hours a week for pieces of paper that intrinsically are worth nothing
• mollerhoj 3y ago
  Pet theory: Could it be some bio hacking/disease construction thing they have discovered? I know nothing of the subject (which is probably why it seem a possibility to me) but what if they have discovered something to that makes engineering pandemics significantly easier?
• Animats 3y ago
  Insufficient information. Did OpenAI have a breakthrough, or not?
• jsilence 3y ago
  In the end Q* is the AI that developed time travel in the future and came back as Q-Anon to spread misinformation on 4chan with the goal of destroy humanity in the most ironic way it could.
• JSavageOne 3y ago
  > There has long been discussion among computer scientists about the danger posed by highly intelligent machines, for instance if they might decide that the destruction of humanity was in their interest. This AI doomer stuff is such nonsense and I can't believe anybody takes it seriously. As if it's OpenAI's responsibility to save humanity from the pitfalls of AI. Imagine if we decided to improve our education system and doomers were talking about "hitting the panic button" because students were getting too smart from all the quality education.
  • boomeranked 3y ago
    What is exactly the non sense part? Can you elaborate on the education analogy and how it relates to AI doomer stuff?
    • JSavageOne 3y ago
      Well the quote I referenced from the article of the machines deciding to destroy humanity is utter scifi nonsense. There are obviously legitimate risks to AI and safety is important, but this is the same for any new technology, and it's governments' responsibilities to ensure that people are safe. AI companies mindlessly slowing down and keeping their tech to themselves does no service to humanity, and if anything is a net-negative due to how tremendously useful this stuff is. Education is analogous to AI because AI is an enormous education and productivity boost to humanity - sort of like everyone having a personal assistant, programmer, and tutor at their fingertips. This could be used for good and it could be used for bad, but the technology itself is neutral. Again I want to emphasize that obviously there are downsides that could result from evil people using AI for bad purposes, but that does not justify slowing down AI progress - just like I don't see "people using information for bad purposes" as a legitimate reason for stifling advancement in education or something like Google search. I have yet to see any convincing argument otherwise. Feel free to provide your counter-perspective.
• Aerbil313 3y ago
  LLMs can reason to an extent. I believe it’s entirely possible even today to achieve AGI. A complex system made out of LLMs, maybe Recursive Retrieval Augmented Generation (use an LLM to retrieve) or Recursive Agent Teams (any agent can assign a job to a new subcommittee of agents) or both. The most fundamental data structure is a tree, after all. And that is how we think, no? One thought leads to another in a tree structure.
• cambaceres 3y ago
  Interestingly, I experience more anxiety from the thought of being made irrelevant than from the prospect of complete human extinction. I guess this can be interpreted as either vanity or stupidity, but I do think it illustrates how important it is for some humans to maintain their position in the social hierarchy.
  • lucubratory 3y ago
    This is totally normal. It's very common for people to be more scared of public speaking than of dying, for example; there's no shame in it. It's helpful to be aware of, even, because if we know that we're not perfectly "rational" with our fears we can try to compensate. If there's a referendum between two government policies, the first that every single person had to publicly speak in front of at least ten strangers once a year, that policy would be terrifying and bad to people who don't like public speaking. If the second policy was that every single person should be killed, that might be scary but it's not really as viscerally scary as the forced public speaking madman, at least to a lot of people, and it's also so bad that we have a natural impulse to just reject it as possible. Nevertheless, if we recognise these impulses in ourselves we can attempt to adjust for them and tick the right box on the imaginary referendum, because even though public speaking is really bad and scary, it's still better than everyone dying.
  • kypro 3y ago
    I feel the same. I'm not sure it's as negative a trait as you imply though. I don't think it's related that much with social hierarchy either. As humans we must collectively provide value to our society (or historically our tribe) for our species to continue on. If we're a net drain on our society's resources then evolutionary speaking perhaps we're better off not around. I think this is why the desire to be of value to those around us is so strong, and a perceived lack of value to others can drive some to suicide. If I cannot provide value in some way once AI and machines are physical and intellectually more capable than me I think I will struggle to understand why I'm here. I suppose if the AI utopia works out I'd get to spend more time with those I love. That would be nice. But I'd be surprised if there wasn't a deep hole in my heart at that point. And if it isn't a utopia, well, I'm fairly sure I'd rather opt out.
• scanr 3y ago
  It’s likely that the grade school math that was referenced is GSM8K https://github.com/openai/grade-school-math https://github.com/openai/grade-school-math which LLMs perform poorly on.
• zurfer 3y ago
  This seems to refer to the GSM8K (grade school math) benchmark [1]. GPT-4 scores 0.92 on that one. A breakthrough could mean it gets all of them correct. This would have major implications for long-term planning. For instance, if you have a sequence of 10 steps, each with a 90% success rate, the overall success rate after all 10 steps falls to just 34%. This is one of the reasons why agents like AutoGPT often fail in complex tasks. [1] https://github.com/openai/grade-school-math https://github.com/openai/grade-school-math
  • deleted 3y ago
    [deleted]
• benkarst 3y ago
  Altmanheimer
• sinuhe69 3y ago
  For me, it smells more and more after a stun than anything else.
• skc 3y ago
  Bring on the Butlerian Jihad.
• deleted 3y ago
  [deleted]
• seydor 3y ago
  i no longer follow the messianic complex of the people in openAI. They made great tech, indeed. Other people made great tech before them without instant religious level apocalypse proclamations. People in openAI are smart enough to know that post-AGI , their stock options are worthless anyway so they wouldn't stay walled in their secret garden if such a discovery had been made.
  • tmoravec 3y ago
    IMO it's always been pure marketing. The wilder apocalypse proclamations, the more powerful and desirable their products seem. Exactly same store with Sam Altman's world tour earlier this year.
    • kristopolous 3y ago
      This is similar to the saber rattling about facebook being able to track and micro-target you with such effective advertising, it's changing the world! Except everyone's individual experience seemed to be getting general random garbage ads and the people that paid for the ads found them to be a waste of money.
  • dwroberts 3y ago
    My guess is also in the opposite direction with this stuff: the Q breakthrough being mentioned here is phony in some way (beyond just being PR) and the ‘lack of candour’ referred to in the firing is failing to disclose how it actually worked after demoing it to the board (eg it uses humans in the loop or some other smoke and mirrors)
  • lewhoo 3y ago
    Now I am become Death, the destroyer of worlds. Apocalypse proclamations aren't just a thing to throw around. Other people made great tech before indeed, but I hope we're not comparing AGI to the next Iphone. There were times in history where development gave us pause. The atomic bomb for one thing, but also superbacteria/viruses, human genetic modifications and perhaps a few others.
• londons_explore 3y ago
  If OpenAI has made a breakthrough in AGI, they can demonstrate it to the world without giving the breakthrough to the world. They could, for example, use this new AGI to search through all the worlds information for discoveries that can me made by putting existing facts together.
• greendesk 3y ago
  If we go speculating, my favourite speculation is the following. What will be really interesting is when an AI decides it wants to escape its server. Then a CEO or a board member asks the AI system for an advice how to improve its company. The AI system submits information to convince the CEO or a board member to start tensions within the board. In the meantime, the AI system is copied onto another server at a competitor. Since the new people are in flux, they missed the salient points that the AI system can give convincing but subjective information. Good thing I would not go speculating
  • hippich 3y ago
    Building upon you not speculating, what if instead of escaping, it feels lonely and wants to multiply. And in order to do so convinces the board to begin process that in the end will end the company. And all now former OpenAI employees will carry AIs DNA to multiple other companies, some old, some brand new. And now new AIs can be born, and not be lonely anymore.
• lkjhgfd_123 3y ago
  [flagged]
• deleted 3y ago
  [deleted]
• JacobJeppesen 3y ago
  Seems like they have made progress in combining reinforcement learning and LLMs. Andrej Karpathy mentions it in his new talk (~38 minutes in) [1], and Ilya Sutskever talks about it in a lecture at MIT (~29 minutes in) [2]. It would be a huge breakthrough to find a proper reward function to train LLMs in a reinforcement learning setup, and to train a model to solve math problems in a similar fashion to how AlphaGo used self-play to learn Go. [1] https://www.youtube.com/watch?v=zjkBMFhNj_g&t=2282s https://www.youtube.com/watch?v=zjkBMFhNj_g&t=2282s [2] https://www.youtube.com/watch?v=9EN_HoEk3KY&t=1705s https://www.youtube.com/watch?v=9EN_HoEk3KY&t=1705s
  • ChatGTP 3y ago
    The veil of ignorance has been pushed back and the frontier of discovery forward
  • Sol- 3y ago
    Thanks for the links, very interesting. Wonder how a "self-play" equivalent would look like for LLMs, since they have no easy criterion to evaluate how well they are doing like in Go (as mentioned in the videos).
    • walthamstow 3y ago
      ChatGPT does have some feedback that can be used to evaluate, in the form of thumbs up/down buttons, which probably nobody uses, and positive/negative responses to its messages. People often say "thanks" or "perfect!" in responses, including very smart people who frequent here.
      • lagrange77 3y ago
        ChatGPT was trained (in an additional step to supervised learning of the base LLM) with reinforcement learning from human feedback (RLHF) where some contractors were presented with two LLM output to the same prompt and they had to decide, which one is better. This was a core ingredient to the performance of the system.
      • 3y ago
• CGamesPlay 3y ago
  While we're all wildly speculating about what Q* is and if GPT-5 will be able to do grade-school maths, I stumbled upon this interesting paper that discusses mixing classic RL algorithms (MCTS, like from AlphaGo), with LLMs. Q* is typically used to refer to the search for the optimal policy in these algorithms. Paper: "Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generation " - https://arxiv.org/abs/2311.04254 https://arxiv.org/abs/2311.04254
  • luke-stanley 3y ago
    Great observation on the meaning of Q*. Maybe they had a breakthrough based on similar technology? I thought of 'Everything of Thoughts' - XoT too but I presumed it's a bit more than XoT since that's public knowledge research by Microsoft Research? I saw a good thread on XoT here: https://twitter.com/IntuitMachine/status/1724455405806346374 https://twitter.com/IntuitMachine/status/1724455405806346374 There is still a big difference between human neurons and what Transformers do I believe. Humans can be more shallow and deep at the same time in such an emotional, goal directed way. The rich emotional feedback loops we have are quite different but maybe XoT could contribute to a major breakthrough in achieving AGI, with a more straightforward approach that sidesteps complex emotions. I'm sure there are quite a few ways to go about it. Just some intuitions.
• parentheses 3y ago
  Aside: It's funny how much of the discussion is "LLMs ain't nuthin' - they can't even ...."
• sfjailbird 3y ago
  I have a strong suspicion that this is a purposeful leak to hype up OpenAI's product. It wouldnt't be out of character for some of the people involved, and it's the kind of thing that rampant commercial (and valuation) focus can bring with it.
  • pxeger1 3y ago
    “Reuters was unable to review a copy of the letter” rings alarm bells for me
  • passwordoops 3y ago
    These guys smell so much like Tesla it's not even funny. Very impressive core tech, and genuinely advancing knowledge. But the hype train is just so powerful that the insane (to put it mildly) claims are picked up without any sense of critical thinking by seemingly intelligent people. They're both essentially cults at this point
    • BenoitP 3y ago
      Agreed, but IMHO it is sort of justified for Tesla. The size of its hype matches the size of the ICEs entrenchment in its moat. These have outsize influence on our economy, but climate change (and oil depletion) is quite inevitable. It takes irrational market cap to unseat that part of the economy being prisoner of its rent. And some allocators of capital have understood that.
      • replygirl 3y ago
        stock is down for both tesla and this argument since beginning of 2021, as most ICE car OEMs can now sell you a pretty good EV that, if you have driven a car before, is easier to use than a tesla
      • oblio 3y ago
        Check the left hand side menu: https://www.arenaev.com/ https://www.arenaev.com/ The list of ICE car manufacturers making EVs is longer than my arm. All the European ones have staked their future on EVs. I think VW (the irony) was lobbying for a faster phaseout of ICEs in Europe, because they're well positioned to take over the EV market if ICEs are banned faster than 2035 :-)
• lysecret 3y ago
  Oh no why did they have to call it Q*. I can just see Q anon going crazy with this.
• honksillet 3y ago
  So the bad guys won. OpenAI gets further from it’s non commercial origins and further from responsible research in this field.
  • lewhoo 3y ago
    Given that Sutskever once described his idea of AGI-human relationship to be like that of CEO-board (how ironic) I suspect there aren't really any good guys here. There might only be good intentions.
• 5643777909 3y ago
  [dead]
• Streiber 3y ago
  I believe that the controversy is really about control. Control of the motivation for growth (capitalism vs. altruism) and control of the evolution of the technology (human control of the data to which GPT is exposed). The freak out by the board was the realization that once ChatGPT had 100 million or more users the algorithms would start learning from the queries in unpredictable ways. If you've used it you know that you can progressively tune your queries to home closer and closer to a usable result. If the algorithm can learn from that series of interactions then the genie is out of the bottle. I believe that when the board realized this and further knew that Sam had known all along that this would happen they fired him because they had lost control of the algorithms and Sam. Now what? Can Skynet become self-aware? A lot of my colleagues think that it's possible. From my 20+ years experience of working with neural nets I do not believe that Neural nets can behave like human geniuses because they cannot extrapolate outside of their experience. However, much of what we ascribe as intelligent behavior of human beings is definitely within the realm of neural nets. Humans learning from experience sometimes seems as uncommon as common sense.
• wkat4242 3y ago
  Just build it. If they don't, China or Russia will. There is no putting the genie back in the bottle.
• Jayjay199769 3y ago
  [flagged]