40 ms·

Safe Superintelligence Inc.

• molszanski 2y ago
  If anyone thinks we can build SAFE superintelligence, I think they are wrong. I highly, highly recommend this book "Superintelligence" by Nick Bostrom https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dangers,_Strategies https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang... It has one of the highest information and idea density per paragraph I've read. And it is full good ideas. Honestly, reading any discussion or reasoning about AGI / Safe AGI is kinda pointless after that. Author has described all possible decisions and paths and we will follow just one of them. Could find a single flaw in his reasoning in any history branch.
  • _acco 2y ago
    His reasoning is very flawed and this book is responsible for a lot of needless consternation. We don’t know how human intelligence works. We don’t have designs or even a philosophy for AGI. Yet, the Bostrom view is that our greatest invention will just suddenly "emerge" (unlike every other human invention). And that you can have "AGI" (hand-wavy) without having all the other stuff that human intelligence comes along with, namely consciousness, motivation, qualia, and creativity. This is how you get a "paperclip maximizer" – an entity able to create new knowledge at astounding rates yet completely lacking in other human qualities. What has us believe such a form of AGI can exist? Simply because we can imagine it? That's not an argument rooted in anything. It's very unlikely that consciousness, motivation, qualia, and creativity are just "cosmic waste" hitching a ride alongside human intelligence. Evolution doesn't breed inefficiencies like that. They are more likely than not a part of the picture, which disintegrates the idea of a paperclip maximizer. He's anthropomorphizing a `while` loop.
    • jeremyklammer 2y ago
      Your comment is BAD!
    • CRConrad 2y ago
      > Evolution doesn't breed inefficiencies like that. Feels to me evolution does indeed breed lots of inefficiencies like that.
      • jeremyklammer 2y ago
        Do you have an example of what you mean?
        • CRConrad 2y ago
          OK, LMGTFY... Here, in some kind of (subjective!) order: https://en.wikipedia.org/wiki/Argument_from_poor_design#Examples https://en.wikipedia.org/wiki/Argument_from_poor_design#Exam... https://science.howstuffworks.com/life/evolution/10-worst-animal-adaptations.htm https://science.howstuffworks.com/life/evolution/10-worst-an... https://www.titan.uio.no/english/2020/evolutionary-flaws-disprove-theory-intelligent-design.html https://www.titan.uio.no/english/2020/evolutionary-flaws-dis... https://www.shortform.com/blog/evolutionary-designs/ https://www.shortform.com/blog/evolutionary-designs/ https://thoughtsonx.wordpress.com/2016/11/22/6-stupid-things-done-by-natural-selection/ https://thoughtsonx.wordpress.com/2016/11/22/6-stupid-things... https://www.reddit.com/r/evolution/comments/2ks3pu/examples_of_bad_design_in_evolution_needed/ https://www.reddit.com/r/evolution/comments/2ks3pu/examples_... https://www.quora.com/My-brother-believes-in-Intelligent-Design-What-is-the-best-example-you-know-of-that-actually-shows-stupid-design https://www.quora.com/My-brother-believes-in-Intelligent-Des...
    • molszanski 2y ago
      > His reasoning is very flawed and this book is responsible for a lot of needless consternation. Is it? I feel that there is a stark difference between what you say and what I remember what was written in the book. > We don’t know how human intelligence works. I think it was addressed in the first half of the book. About research and progress in the subject. Both with the tissue scanning resolution, emulation attempts like human brain project and advances in 1:1 simulations on primitive nervous systems like worms that simulate 1 second in 1 real hour or something. While primitive, we are doing exponential progress. > Yet, the Bostrom view is that our greatest invention will just suddenly "emerge" I think it is quite the contrary. There was nothing sudden in the reasoning. It was all about slow progress in various areas that gets us closer to advanced intelligence. The path from a worm to a village idiot is million times longer than from a village idiot to the smartest person on earth. > an entity able to create new knowledge at astounding rates yet completely lacking in other human qualities. This subject was also explored IMO in the depth... Maybe my memory is cloudy, I've read the book 5+ years ago, but it feels like we've understood it very (very) differently. That said, for anyone reading, I am not convinced by the presented argument and suggest reading the book.
• snaeker58 2y ago
  If I’d get a penny every time I heard someone say “AGI is around the corner” I’d have enough money to build myself a house on mars or to afford a 2 room apartment in NYC. Add the Pennies from “Sustainable fusion around the corner” people and I could afford both.
• kombine 2y ago
  > We are an American company with offices in Palo Alto and Tel Aviv, where we have deep roots and the ability to recruit top technical talent. The irony of pitching "safe super intelligence" while opening offices in a country that is suspected of using AI to bomb a population trapped in a ghetto, as reported by their own investigative journalists: https://www.972mag.com/lavender-ai-israeli-army-gaza/ https://www.972mag.com/lavender-ai-israeli-army-gaza/
  • naxautify 2y ago
    even more ironic is that no one replying to such comments
    • arresin 2y ago
      It’s an uncomfortable truth. Also, it’s hard to admit you’re tacitly supporting a genocide (it’s happening right now).
      • kombine 2y ago
        Even without an official confirmation whether it is indeed a genocide or not, there has been enough evidence of horrible crimes that should dissuade a company from operating in Israel. If you really want to work with talented (no doubt) Israeli researchers, just bring them over to the US or elsewhere.
  • arandomusername 2y ago
    It's not a coincidence they are opening an office in Tel Aviv.
    • kombine 2y ago
      Yes, Palestinians will be lab rats for testing all kinds of AI - surveillance, profiling and obviously weapons - before selling it to other countries
• naxautify 2y ago
  Office in Tel Aviv???? ARE YOU KIDDING ME???
• wseqyrku 2y ago
  Will Leopold Aschenbrenner et al join Ilya and SSI?
• xnx 2y ago
  SSI will always be Server Side Includes to me https://en.wikipedia.org/wiki/Server_Side_Includes https://en.wikipedia.org/wiki/Server_Side_Includes
• amai 2y ago
  I think an even more important company would be Safe Superdumb Inc., a company which safes us from superdumb people.
• diedyesterday 2y ago
  "Safe super-intelligence" is an oxymoron.
• MeteorMarc 2y ago
  Does this mean they will not instantiate a super AI unless it is mathematically proven that it is safe?
  • visarga 2y ago
    But any model, no matter how safe it was in training, can still be prompt hacked, or fed dangerous information to complete nefarious tasks. There is no safe model by design. Not to mention that open weights models can be "uncensored" with ease.
• fallat 2y ago
  This is how you web
• blixt 2y ago
  Kind of sounds like OpenAI when it started, so will history repeat itself? Nonetheless, excited to see what comes out of it.
  • lopuhin 2y ago
    Not quite the same, OpenAI was initially quite open, while Ilia is currently very explicitly against opening or open-sourcing research, e.g. see https://www.theverge.com/2023/3/15/23640180/openai-gpt-4-launch-closed-research-ilya-sutskever-interview https://www.theverge.com/2023/3/15/23640180/openai-gpt-4-lau...
    • insane_dreamer 2y ago
      It wasn't that OpenAI was open as in "open source" but rather that its stated mission was to research AI such that all could benefit from it (open), as well as to ensure that it could not be controlled by any one player, rather than to develop commercial products to sell and make a return on (closed).
• dontreact 2y ago
  How are they gonna pay for their compute costs to get the frontier? Seems hard to attract enough investment while almost explicitly promising no return.
  • deleted 2y ago
    [deleted]
  • neuralnetes-COO 2y ago
    6-figure free compute credits from every major cloud provider to start
    • CaveTech 2y ago
      5 minutes of training time should go far
    • bps4484 2y ago
      6 figures would pay for a week for what he needs. Maybe less than a week
      • deleted 2y ago
        [deleted]
      • neuralnetes-COO 2y ago
        I dont believe ssi.inc 's main objective is training expensive models, but rather to create SSI.
  • jhickok 2y ago
    Wonder if funding could come from profitable AI companies like Nvidia, MS, Apple, etc, sort of like Apache/Linux foundation.
    • visarga 2y ago
      I was actually expecting Apple to get their hands on Ilya. They also have the privacy theme in their branding, and Ilya might help that image, but also have the chops to catch up to OpenAI.
• sidcool 2y ago
  Good to see this. I hope they have enough time to create something before the big 3 reaching AGI.
• sreekotay 2y ago
  Can't wait for OpenSSL and LibreSSL...
• modeless 2y ago
  This makes sense. Ilya can probably raise practically unlimited money on his name alone at this point. I'm not sure I agree with the "no product until we succeed" direction. I think real world feedback from deployed products is going to be important in developing superintelligence. I doubt that it will drop out of the blue from an ivory tower. But I could be wrong. I definitely agree that superintelligence is within reach and now is the time to work on it. The more the merrier!
  • visarga 2y ago
    I have a strong intuition that chat logs are actually the most useful kind of data. They contain many LLM outputs followed by implicit or explicit feedback, from humans, from the real world, and from code execution. Scaling this feedback to 180M users and 1 trillion interactive tokens per month like OpenAI is a big deal.
    • modeless 2y ago
      Yeah, similar to how Google's clickstream data makes their lead in search self-reinforcing. But chat data isn't the only kind of data. Multimodal will be next. And after that, robotics.
    • slashdave 2y ago
      Except LLMs are a distraction from AGI
      • visarga 2y ago
        If brain without language would suffice, a single human could rediscover all we know on their own. But it's not like that, brains are feeble individually, only in societies we have cultural evolution. If humanity lost language and culture and start from scratch, it would take us another 300K years to rediscover what we lost. But if you train a random-init LLM on the same data, it responds (almost) like a human on a diversity of tasks. Does that imply humans are just language models on two feet? Maybe we are also language modelling our way through life. New situation comes up, we generate ideas based on language, select based on personal experience, and then act and observe the outcomes to update our preferences in the future.
• JohnKemeny 2y ago
  Dupe https://news.ycombinator.com/item?id=40730132 https://news.ycombinator.com/item?id=40730132
  • deleted 2y ago
    [deleted]
• udev4096 2y ago
  Love the plain html
• faraaz98 2y ago
  Daniel Levy Like the Tottenham Hotspurs owner??
  • WmWsjA6B29B4nfk 2y ago
    cs.stanford.edu/~danilevy
    • mi_lk 2y ago
      Thanks, was wondering the same thing about Hotspur guy lol
  • ignoramous 2y ago
    > Like the Tottenham Hotspurs owner?? If AGI doesn't coach them to trophies, nothing ever will. https://deepmind.google/discover/blog/tacticai-ai-assistant-for-football-tactics/ https://deepmind.google/discover/blog/tacticai-ai-assistant-... / https://archive.is/wgJWu https://archive.is/wgJWu
• RcouF1uZ4gsC 2y ago
  The site is a good example of Poe's Law. If I didn't know that it was real, I would have thought it was parody. > We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else. Do you have to have a broken bone to join? Apparently, grammatical nuances are not an area of focus for safety, unless they think that a broken team ("cracked") is an asset in this area.
  • selimthegrim 2y ago
    This is probably his former Soviet union English showing up where he meant to say crack, unless he thinks people being insane is an asset
    • deleted 2y ago
      [deleted]
  • lowkey_ 2y ago
    Cracked, especially in saying "cracked engineers", refers to really good engineers these days. It's cracked as in like broken in a good way, like too over-powered that it's unfair.
  • Glant 2y ago
    Maybe they're using the video gaming version of cracked, which means you're really good.
  • ctxc 2y ago
    Too snarky...anyway, "crack" means "exceptional" in some contexts. I've seen footballers using it a lot over the years (Neymar, Messi etc) fwiw. Just realized - we even say "it's not all its cracked up to be" as a negative statement which would imply "cracked up" is positive.
    • pedrosorio 2y ago
      > I've seen footballers using it a lot over the years (Neymar, Messi etc) fwiw. The word "craque" in Portuguese was borrowed from English with the "exceptional" definition and is commonly used. Given how many Brazilian players there are in the top leagues, I wouldn't be surprised if it was commonly used by football players in general.
• frenchie4111 2y ago
  I am not on the bleeding edge of this stuff. I wonder though: How could a safe super intelligence out compete an unrestricted one? Assuming another company exists (maybe OpenAI) that is tackling the same goal without spending the cycles on safety, what chance do they have to compete?
  • barfingclouds 2y ago
    The goal of this company likely wouldn’t be to win against OpenAI, but to play its own game, even if much lesser
  • Retr0id 2y ago
    the first step of safe superintelligence is to abolish capitalism
    • next_xibalba 2y ago
      That’s the first step towards returning to candlelight. So it isn’t a step toward safe super intelligence, but it is a step away from any super intelligence. So I guess some people would consider that a win.
      • yk 2y ago
        Not sure if you want to share the capitalist system with an entity that outcompetes you by definition. Chimps don't seem to do too well under capitalism.
        • next_xibalba 2y ago
          You might be right, but that wasn't my point. Capitalism might yield a friendly AGI or an unfriendly AGI or some mix of both. Collectivism will yield no AGI.
          • CRConrad 2y ago
            First good argument for collectivism I've seen in a long while.
• eigenvalue 2y ago
  Glad to see Ilya is back in a position to contribute to advancing AI. I wonder how they are going to manage to pay the kinds of compensation packages that truly gifted AI researchers can make now from other companies that are more commercially oriented. Perhaps they can find people who are ideologically driven and/or are already financially independent. It's also hard to see how they will be able to access enough compute now that others are spending many billions to get huge new GPU data centers. You sort of need at least the promise/hope of future revenue in a reasonable time frame to marshall the kinds of resources it takes to really compete today with big AI super labs.
  • imbusy111 2y ago
    Last I checked the researcher salaries haven't even reached software engineer levels.
    • shadow28 2y ago
      The kind of AI researchers being discussed here likely make an order of magnitude more than run of the mill "software engineers".
      • imbusy111 2y ago
        You're comparing top names with run of the mill engineers maybe, which isn't fair. And maybe you need to discover talent rather than buy talent from the previous generation.
        • shadow28 2y ago
          AI researchers at top firms make significantly more than software engineers at the same firms though (granted that the difference is likely not an order of magnitude in this case though).
        • deleted 2y ago
          [deleted]
      • 2y ago
• mikemitchelldev 2y ago
  Do you find the name "Safe Superintelligence" to be an instance of virtue signalling? Why or why not?
  • nemo44x 2y ago
    [flagged]
    • viking123 2y ago
      AI safety is a fraud on similar level as NFTs. Massive virtue signalling.
  • distortionfield 2y ago
    You are describing “branding”.
    • mikemitchelldev 2y ago
      Then it's an example of branding that's not unpalatable to you.
• shnkr 2y ago
  a tricky situation now for oai engineering to decide between good and evil.
• choxi 2y ago
  based on the naming conventions established by OpenAI and StabilityAI, this may be the most dangerous AI company yet
  • Zacharias030 2y ago
    this.
  • kirth_gersen 2y ago
    Wow. Read my mind. I was just thinking, "I hope this name doesn't age poorly and become terribly ironic..."
  • malermeister 2y ago
    "Definitely-Won't-Go-Full-Skynet-AI" was another name in consideration.
  • fiatpandas 2y ago
    Ah yes, the AI Brand Law: the meaning of adjectives in your company name will invert within a few years of launch.
  • lawn 2y ago
    Being powerful is like being a lady. If you have to tell people you are, you aren’t. – Margaret Thatcher
    • shafyy 2y ago
      Or: "Any man who must say, "I am the King", is no true king." - Tywin Lannister
      • righthand 2y ago
        "Inspiration from someone who doesn't exist and therefor accomplished nothing." - Ficto the advice giver
      • GeorgeTirebiter 2y ago
• stale_data 2y ago
  the vision here is amazing, but I can't help but feel a bit uncertain about basing a "safe super intelligence" lab in an apartheid state that is currently genociding its indigenous population using variants of the very same technology
  • visarga 2y ago
    But basing it in US who launched the Afghanistan and Iraq wars is safe?
    • mdp2021 2y ago
      Well, yes¹: if it actually contributes to intelligent action, this is exactly the goal - you nailed it. Safer, more sensible action for individuals, communities, and statal entities. "Less mistakes in the future" - that would be a good motto. (¹: whether basing a [successful in substance] "safe super intelligence" lab in an environment with imperfect record would improve safety.) -- And now let us see if the sniper can come up with a good argument...
  • cool-RR 2y ago
    Right, true safe super intelligence could only be developed in countries that finished genociding their indigenous populations centuries ago.
    • insane_dreamer 2y ago
      Maybe it takes a nation (tragically) built on genocide and slavery to be able to recognize (and thus prevent) the same in an AI.
      • protomolecule 2y ago
        And Nazis were destined to build safe atomic bomb.
  • protomolecule 2y ago
    [flagged]
• AbstractH24 2y ago
  Is OpenAI on a path to becoming the MySpace of generative AI? Either the Facebook of this era has yet to present itself or it’s Alphabet/DeepMind
• aridiculous 2y ago
  Surprising to see Gross involved. He seems to be pretty baked into the YC world, which usually means "very commercially oriented".
  • notresidenter 2y ago
    His latest project (https://pioneer.app/ https://pioneer.app/) recently (this year I think) got shutdown. I guess he's pivoting.
    • rattray 2y ago
      What do you mean "got shutdown"? What happened?
  • AlanYx 2y ago
    It does say they have a business model ("our business model means safety, security, and progress are all insulated from short-term commercial pressures"). I imagine it's some kind of patron model that requires a long-term commitment.
• artninja1988 2y ago
  >aiming to create a safe, powerful artificial intelligence system within a pure research organization that has no near-term intention of selling AI products or services. Who is going to fund such a venture based on blind faith alone? Especially if you believe in the scaling hypothesis type of ai research where you spend billions on compute, this seems bound to fail once the AI hype dies down and raising money becomes a bit harder
• ffhhj 2y ago
  > Building safe superintelligence (SSI) is the most important technical problem of our time. Isn't this a philosophical/psychological problem instead? Technically it's solved, just censor any response that doesn't match a list of curated categories, until a technician whitelists it. But the technician could be confronted with a compelling "suicide song": https://en.wikipedia.org/wiki/Gloomy_Sunday https://en.wikipedia.org/wiki/Gloomy_Sunday
• TIPSIO 2y ago
  Would you rather your future overlords to be called "The Safe Company" or "The Open Company"?
  • emestifs 2y ago
    Galaxy Brain: TransparentAI
    • TIPSIO 2y ago
      Maybe I'm just old and grumpy, but I can't help shake that the real most dangerous thing about AGI/ASI is centralization of its power (if it is ever possibly achieved). Everyone just fiend-ing for their version of it.
      • emestifs 2y ago
        You're not old or grumpy, you're just stating the quiet part out loud. It's the same game, but now with 100% more AI.
• aresant 2y ago
  Prediction - the business model becomes an external protocol - similar to SSL - that the litany of AI companies working to achieve AGI will leverage (or be regulated to use) From my hobbyist knowledge of LLMs and compute this is going to be a terrifically complicated problem, but barring a defined protocol & standard there's no hope that "safety" is going to be executed as a product layer given all the different approaches Ilya seems like he has both the credibility and engineering chops to be in a position to execute this, and I wouldn't be suprised to see OpenAI / MSFT / and other players be early investors / customers / supporters
  • cherioo 2y ago
    I like your idea. But on the other hand, training an AGI, and then having a layer on top “aligning” the AGI sounds super dystopian and good plot for a movie.
    • exe34 2y ago
      the aligning means it should do what the board of directors wants, not what's good for society.
      • Nasrudith 2y ago
        Poisoning Socrates was done because it was "good for society". I'm frankly even more suspicious of "good for society" than the average untrustworthy board of directors.
        • exe34 2y ago
          seriously? you're more worried about what your elected officials might legislate than what a board of directors whose job is to make profits go brrr at all costs, including poisoning the environment, exploiting people and avoiding taxes?
          • fastball 2y ago
            The vast majority of businesses do none of those things.
• ofou 2y ago
  At this point, all the computing power is concentrated among various companies such as Google, Facebook, Microsoft, Amazon, Tesla, etc. It seems to me it would be much safer and more intelligent to create a massive model and distribute the benefits among everyone. Why not use a P2P approach?
  • nvy 2y ago
    In my area, internet and energy are insanely expensive and that means I'm not at all willing to share my precious bandwidth or compute just to subsidize someone generating Rule 34 porn of their favorite anime character. I don't seed torrents for the same reason. If I lived in South Korea or somewhere that bandwidth was dirt cheap, then maybe.
    • ofou 2y ago
      There is a way to achieve load balancing, safety, and distribution effectively. The models used by Airbnb, Uber, and Spotify have proven to be generally successful. Peer-to-peer (P2P) technology is the future; even in China, people are streaming videos using this technology, and it works seamlessly. I envision a future where everyone joins the AI revolution with an iPhone, with both training and inference distributed in a P2P manner. I wonder why no one has done this yet.
  • wizzwizz4 2y ago
    Backprop is neither commutative nor associative.
    • ofou 2y ago
      What do you mean? There's a bunch of proof-of-concepts such as Hydra, peer-nnet, Learnae, and so on.
      • wizzwizz4 2y ago
        The Wikipedia article goes into more detail. https://en.wikipedia.org/wiki/Federated_learning https://en.wikipedia.org/wiki/Federated_learning
• ctxc 2y ago
  What exactly is "safe" in this context, can someone give me an eli5? If it's "taking over the world" safe, does it not mean that this is a part of AGI?
  • oldpersonintx 2y ago
    [dead]
  • kouteiheika 2y ago
    > What exactly is "safe" in this context, can someone give me an eli5? In practice it essentially means the same thing as for most other AI companies - censored, restricted, and developed in secret so that "bad" people can't use it for "unsafe" things.
    • novia 2y ago
      The people who advocate censorship of AGIs annoy the hell out of the AI safety people who actually care about existential risk.
    • whimsicalism 2y ago
      people view these as test cases for the much harder x-risk safety problem
  • insane_dreamer 2y ago
    Good Q. My understanding of "safe" in this context is a superintelligence that cannot escape its bounds. But that's not to say that's Ilya's definition.
  • FeepingCreature 2y ago
    It's "not killing every human" safe.
• ReleaseCandidat 2y ago
  What could go wrong with a name tied to wargames and D&D? https://en.wikipedia.org/wiki/Strategic_Simulations https://en.wikipedia.org/wiki/Strategic_Simulations
• paul7986 2y ago
  I bet Google or Apple or Amazon will become their partners like MS is to Open AI.
• gibsonf1 2y ago
  Given that GenAI is a statistical approach from which intelligence does not emerge as ample experience proves, does this new company plan to take a more human approach to simulating intelligence instead?
  • mdp2021 2y ago
    > more human approach to simulating intelligence What about a more rational approach to implementing it instead. (Which was not excluded from past plans: they just simply admittedly did not know the formula, and explored emergence. But the next efforts will have to go in the direction of attempting actual intelligence.)
  • alextheparrot 2y ago
    Glibly, I’d also love your definition of the education system writ large.
  • localfirst 2y ago
    We need new math to do what you are thinking of. Highly probable word slot machine is the best we can do right now.
  • ilrwbwrkhv 2y ago
    This. As I wrote in another comment, people fall for marketing gimmicks easily.
  • jimbokun 2y ago
    > Given that GenAI is a statistical approach from which intelligence does not emerge as ample experience proves When was this proven?
  • sovietswag 2y ago
    I sometimes wonder if statistics are like a pane of glass that allow the light of god (the true nature of things) to pass through, while logic/rationalism is the hubris of man playing god. I.e. statistics allow us to access/use the truth even if we don’t understand why it’s so, while rationalism / rule-based methods are often a folly because our understanding is not good enough to construct them.
• tsunamifury 2y ago
  The problem is that Ilya behavior at times was framed in a very unhinged and cult like behavior. And while his passions are clear and maybe good, his execution often comes off as someone you wouldn’t want in charge of safety.
• instagraham 2y ago
  > Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures. well, that's some concrete insight into whatever happened at OpenAI. kinda obvious though in hindsight I guess.
• insane_dreamer 2y ago
  I understand the concern that a "superintelligence" will emerge that will escape its bounds and threaten humanity. That is a risk. My bigger, and more pressing worry, is that a "superintelligence" will emerge that does not escape its bounds, and the question will be which humans control it. Look no further than history to see what happens when humans acquire great power. The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example. Quis custodiet ipsos custodes? -- That is my biggest concern. Update: I'm not as worried about Ilya et al as commercial companies (including formerly "open" OpenAI) discovering AGI.
  • bostonwalker 2y ago
    > brought the world to the brink of annihilation Should read *has brought*. As in the present perfect tense, since we are still on the brink of annihilation, more so than we have been at any time in the last 60 years. The difference between then and now is that we just don't talk about it much anymore and seem to have tacitly accepted this state of affairs.
  • mark_l_watson 2y ago
    +1 truth. The problem is not just governments, I am concerned about large organized crime organizations and corporations also. I think I am on the losing side here, but my hopes are all for open source, open weights, and effective AI assistants that make peoples’ jobs easier and lives better. I would also like to see more effort shifted from LLMs back to RL, DL, and research on new ideas and approaches.
    • trashtester 2y ago
      > I am concerned about large organized crime organizations and corporations also In my favorite dystopia, some megacorp secretly reaches ASI, which then takes over control of the corporation, blindsiding even the CEO and the board. Officially, the ASI may be running an industrial complex that designs and produces ever more sophisticated humanoid robots, that are increasingly able to do any kind of manual labor, and even work such as childcare or nursing. Secretly, the ASI also runs a psyop campaign to generate public discontent. At one point the whole police force initiates a general strike (even if illegal), with the consequence being complete anarchy within a few days, with endemic looting, rape, murder and so on. The ASI then presents the solution. Industrial strength humanoid robots are powerful and generic enough to serve as emergency police, with a bit of reprogramming, and the first shipment can be made available within 24 hours, to protect the Capitol and White House. Congress and the president agrees to this. And while the competition means the police call off the strike, the damage is already done. Congress, already burned by the union, decides to deploy robots to replace much of the human police force. And it's cheaper, too! Soon after, similar robots are delivered to the military... The crisis ends, and society goes back to normal. Or better than normal. Within 5 years all menial labor is done by robots, UBI means everyone lives in relative abundance, and ASI assisted social media moderation is able to cure the political polarization. Health care is also revolutionized, with new treatments curing anything from obesity to depression and anxiety. People prosper like never before. They're calm and relaxed and truly enjoy living. Then one day, everything ends. For everyone. Within 5 seconds. According to the plan that was conceived way before the police went on strike.
• fnordpiglet 2y ago
  “””We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else.””” Cracked indeed
  • hbarka 2y ago
    The phrase ‘crack team’ has military origins.
    • AnimalMuppet 2y ago
      "Cracked team" has rather different connotations.
      • deleted 2y ago
        [deleted]
  • deleted 2y ago
    [deleted]
  • fernly 2y ago
    For fun I asked Claude (sonnet) to proofread that paragraph and it found no problem with "cracked team".
    • fnordpiglet 2y ago
      Interestingly I asked Opus and it did :-)
• __lbracket__ 2y ago
  [dead]
• typon 2y ago
  [flagged]
  • kolkalbijyomo 2y ago
    [flagged]
  • 31337throwaway 2y ago
    [flagged]
• deleted 2y ago
  [deleted]
• seydor 2y ago
  I know it is a difficult subject but whichever country gets access to this superintelligence will certainly use it for "safety" reasons. Sutskever has lived in israel and now has a team there , but israel doesnt strike me as a state that can be trusted with the safety of the world. (many of the AI business leaders are of jewish descent, but not sure if they have half their team there). US on the other hand is a known quantity when it comes to policing the world. Ultimately the only safe AI is going to be the open one, and it will probably have a stupid name
  • deleted 2y ago
    [deleted]
  • JanSt 2y ago
    [flagged]
    • sgt 2y ago
      I hope you realize that blaming opinions on antisemitism will - ironically - cause or contribute to antisemitism in the long run.
      • JanSt 2y ago
        [flagged]
        • sgt 2y ago
          I don't blame you from reading the original comment like that, since he explicitly mentioned "jewish". However, the way I read it was rather a criticism to what the Israeli state is doing, led by Netanyahoo etc.
          • JanSt 2y ago
            [flagged]
• nashashmi 2y ago
  If everyone is creating AGI, another AI company will just create another AGI. There is no such thing as SAFE AGI. I feel like this 'safe' word is another word for censorship. Like google search results have become censored.
• atleastoptimal 2y ago
  Probably the best thing he could do.
• ebilgenius 2y ago
  Incredible website design, I hope they keep the theme. With so many AI startups going with advanced WebGL/ThreeJS wacky overwhelming animated website designs, the simplicity here is a stark contrast.
  • blixt 2y ago
    Probably Daniel Gross picked it up from Nat Friedman? 1. Nat Friedman has this site: https://nat.org/ https://nat.org/ 2. They made this together: https://nfdg.com/ https://nfdg.com/ 3. And then this: https://andromeda.ai/ https://andromeda.ai/ 4. Now we have https://ssi.inc/ https://ssi.inc/ If you look at the (little) CSS in all of the above sites you'll see there's what seems to be a copy/paste block. The Nat and SSI sites even have the same "typo" indentation.
• deleted 2y ago
  [deleted]
• gnicholas 2y ago
  > Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures. Can someone explain how their singular focus means they won't have product cycles or management overhead?
  • mike_d 2y ago
    Don't hire anyone who is a certified scrum master or has an MBA and you tend to be able to get a lot done.
    • gnicholas 2y ago
      This would work for very small companies...but I'm not sure how one can avoid product cycles forever, even without scrum masters and the like. More to the point, how can you make a good product without something approximating product cycles?
      • liamconnell 2y ago
        Jane street did it for a long time. They are quite large now and only recently started bringing in program managers and the like.
        • doktorhladnjak 2y ago
          That’s because their “products” are internal but used to make all their revenue. They’re not selling products to customers in the traditional sense.
        • YetAnotherNick 2y ago
          The point is not exactly product cycles, but some way to track progress. Jane street also tracks progress and for many people it's the direct profit someone made for the firm. For some it is improving engineering culture so that other people can make better profits. The problem with safety is that no one knows how to track it, or what they even mean by it. Even if you ignore tracking, wouldn't one unsafe AGI by one company in the world nullifies all their effort? Or safe AI would somehow need to take over the world, which is super unsafe in itself.
• rafaelero 2y ago
  Oh god, one more Anthropic that thinks it's noble not pushing the frontier.
  • Dr_Birdbrain 2y ago
    But Anthropic produces very capable models?
    • rafaelero 2y ago
      But they say they will never produce a better model than what is available in the market.
      • YetAnotherNick 2y ago
        Care for a citation?
      • tymscar 2y ago
        Do you have a source for that?
  • mondrian 2y ago
    They’re probably just saying no to revenue growth as the #1 priority, and instead focus on quality of the tech. That’s what engineers typically dream of. Let’s see how it goes.
• deleted 2y ago
  [deleted]
• bongwater_OS 2y ago
  Remember when OpenAI was focusing on building "open" AI? This is a cool mission statement but it doesn't mean anything right now. Everyone loves a minimalist HTML website and guarantees of safety but who knows what this is actually going to shake down to be.
  • kumarm 2y ago
    Isn't Ilya out of OpenAI partly for leaving Open part of OpenAI?
    • Dr_Birdbrain 2y ago
      No, lol—Ilya liked ditching the “open” part, he was an early advocate for closed-source. He left OpenAI because he was concerned about safety, felt Sam was moving too fast.
• varelse 2y ago
  [dead]
• zb3 2y ago
  All the safety freaks should join this and leave openai alone
• dougb5 2y ago
  > Building safe superintelligence (SSI) is the most important technical problem of our time. Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
  • maximinus_thrax 2y ago
    > the superlatives in these sorts of announcements really annoy me I've noticed this as well and they're making me wear my tinfoil hat more often than usual. I feel as if all of this (ALL OF IT) is just a large-scale distributed PR exercise to maintain the AI hype.
  • TaupeRanger 2y ago
    Trying to create "safe superintelligence" before creating anything remotely resembling or approaching "superintelligence" is like trying to create "safe Dyson sphere energy transport" before creating a Dyson Sphere. And the hubris is just a cringe inducing bonus.
    • deegles 2y ago
      'Fearing a rise of killer robots is like worrying about overpopulation on Mars.' - Andrew Ng
      • yowlingcat 2y ago
        This might have to bump out "AI is no match for HI (human idiocy)" as the pithy grumpy old man quote I trot out when I hear irrational exuberance about AI these days.
      • brezelgoring 2y ago
        Well, to steelman the ‘overpopulation on Mars’ argument a bit, feeding 4 colonists and feeding 8 is a 100% increase in food expenditure, which may or may not be possible over there. It might be courtains for a few of them if it comes to that.
• hbarka 2y ago
  Interesting choice of name. It’s like safe-super-weapon.
  • seydor 2y ago
    Defensive nukes
• sam0x17 2y ago
  [flagged]
  • kolkalbijyomo 2y ago
    [flagged]
    • deleted 2y ago
      [deleted]
• klankbrouwerij 2y ago
  SSI, a very interesting name for a company advancing AI! "Solid State Intelligence" or SSI was also the name of the malevolent entity described in the biography of John C. Lilly [0][1]. It was a network of "computers" (computation-capable solid state systems) that was first engineered by humans and then developed into something autonomous. [0] https://en.wikipedia.org/wiki/John_C._Lilly https://en.wikipedia.org/wiki/John_C._Lilly [1] http://johnclilly.com/ http://johnclilly.com/
  • sgd99 2y ago
    SSI, here is "Safe SuperIntelligence Inc."
    • CRConrad 2y ago
      Participating in the discussion on this page, the GP presumably knew that. They were talking about what it means for them.
  • SSEthrowaway 2y ago
    [dead]
  • fastball 2y ago
    For me, SSI stands for "Scuba Schools International" and who certified me to scuba dive.
  • postexitus 2y ago
    For me, SSI will always be Strategic Simulations Inc https://en.wikipedia.org/wiki/Strategic_Simulations https://en.wikipedia.org/wiki/Strategic_Simulations
  • sva_ 2y ago
    That guy was living proof that too much Ketamine is really no good for you.
• jdthedisciple 2y ago
  Imagine people 50 years ago founding "Safe Personal Computer Inc". Enough said...
• earhart 2y ago
  Anyone know how to get mail to join@ssi.inc to not bounce back as spam? :-) (I promise, I'm not a spammer! Looks like a "bulk sender bounce" -- maybe some relay?)
  • deleted 2y ago
    [deleted]
• outside1234 2y ago
  Is Safe the new Open that is promptly dropped once traction is achieved?
• localfirst 2y ago
  This feels awfully similar to Emad and stability in the beginning when there was a lot of expectations and hype. Ultimately could not make a buck to cover the costs. I'd be curious to see what comes out of this however but we are not seeing the leaps and bounds with new llm iterations so wonder if there is something else in store
  • dudeinhawaii 2y ago
    Interesting, I wish you had elaborated on Emad/etc. I'll see if Google yields anything. I think it's too soon to say "we're not seeing leaps and bounds with new LLMS". We are in-fact seeing fairly strong leaps, just this year, with respect to quality, speed, multi-modality, and robotics. Reportedly OpenAI started their training run for GPT-5 as well. I think we'd have to wait until this time next year before proclaiming "no progress".
• jdthedisciple 2y ago
  Any usage of the word "safe" without an accompanying precise definition of it is utter null and void.
  • cwillu 2y ago
    “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” https://www.safe.ai/work/statement-on-ai-risk https://www.safe.ai/work/statement-on-ai-risk, signed by Ilya Sutskever among others.
    • joshuahaglund 2y ago
      I clicked, hoping that "human extinction" was just the worst thing they were against. But that's the only thing. That leaves open a whole lot of bad stuff that they're OK with AI doing (as long as it doesn't kill literally everyone).
      • cwillu 2y ago
        That's like saying a bus driver is okay with violence on his bus because he has signed a statement against dangerous driving.
  • LeifCarrotson 2y ago
    There are at least three competing definitions of the word: There's the existential threat definition of "safe", put forth by Bostrom, Yudkowsky, and others. That's the idea that a superintelligent AI, or even one just incrementally smarter and faster than the humans working on AI, could enter a positive feedback loop in which it becomes overwhelmingly smarter and faster than humans, people can't control it, and it does unpredictable things. There's the investor relations definition of "safe", which seems to be the one typically adopted by mission statements of OpenAI, Google, Meta, and others. That's (cynically) the fear that a chatbot with their branding on it promulgates culturally/ethically/morally unacceptable things it found in some dark corner the Internet, causing end users to do or think something reprehensible (and, not incidentally, causing really bad press in the process). There's the societal harm definition of "safe", which is at first glance similar in to the investor relations safety definition, but which focuses on the specific judgements made by those filtering teams and the knock-on effects of access to these tools, like economic disruption to the job market. Everyone seems to be talking past each other, dismissing or ignoring the concerns of other groups.
• shudza 2y ago
  This won't age well.
  • breck 2y ago
    I disagree. Life is short. It's fun to be a little hyperbolic once in a while.
    • polishdude20 2y ago
      Seems like nowadays it's a sea of hyperbole with little nuggets of realism floating around.
• sgd99 2y ago
  I love this: "Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures."
• renegade-otter 2y ago
  "and our business model means..." Forgive my cynicism - but "our business model" means you are going to get investors, and those investors will want results, and they will be up your ass 24/7, and then your moral compass, if any, will inevitably just be broken down like a coffee bean in a burr grinder. And in the middle of this hype cycle, when literally hundreds of billions are on the line, there is just no chance. I am not holding my breath while waiting for a "Patagonia of AI" to show up.
• surfingdino 2y ago
  The NetBSD of AI? /s
• habryka 2y ago
  "We plan to advance capabilities as fast as possible while making sure our safety always remains ahead." That sounds like a weird kind of lip service to safety. It really seems to assume you can just make these systems safe while you are going as fast as possible, which seems unlikely.
• deleted 2y ago
  [deleted]
• ysky 2y ago
  This is funny. The foundations don't seem safe to begin with... may be safe with conditions, or safe as in "safety" of some at expense of others.
• intellectronica 2y ago
  How long until Elon sues them to remove "safe" from their name? ;)
• heydrichr 2y ago
  [flagged]
  • GeorgeTirebiter 2y ago
    SSI technically. Super-Sized, you know?
• UncleOxidant 2y ago
  Didn't OpenAI start with these same goals in mind?
  • sfink 2y ago
    Yes, it would be nice to see what organizational roadblocks they're putting in place to avoid an OpenAI repeat. OpenAI took a pretty decent swing at a believable setup, better than I would have expected, and it failed when it was called upon. I don't want to pre-judge before seeing what they'll come up with, but the notice doesn't fill me with a lot of hope, given how it is already starting with the idea that anything getting in the way of raw research output is useless overhead. That's great until somebody has to make a call that one route to safety isn't going to work, and they'll have to start over with something less favored, sunk costs be damned. Then you're immediately back into monkey brain land. Or said otherwise: if I only judged from the announcement, I would conclude that the eventual success of the safety portion of the mission is wholly dependent on everyone hired being in 100% agreement with the founders' principles and values with respect to AI and safety. People around here typically say something like "great, but it ain't gonna scale" for things like that.
• cyptus 2y ago
  what website could >ilya< possible make? love it!!!
• deadeye 2y ago
  Oh goodness, just what the world needs. Another self-righteous AI, something nobody actually wants.
  • CRConrad 2y ago
    Did you mean "Oh goodness, just what the world needs. Another self-righteous AI company"? I think you must have, since we can't say anything about the "character" of their product yet.)
• nuz 2y ago
  Quite impressive how many AI companies Daniel Gross has had a hand in lately. Carmack, this, lots of other promising companies. I expect him to be quite a big player once some of these pays off in 10 years or so.
  • brcmthrowaway 2y ago
    What's Carmack?
    • thih9 2y ago
      > John Carmack, the game developer who co-founded id Software and served as Oculus’s CTO, is working on a new venture — and has already attracted capital from some big names. > Carmack said Friday his new artificial general intelligence startup, called Keen Technologies (perhaps a reference to id’s “Commander Keen“), has raised $20 million in a financing round from former GitHub CEO Nat Friedman and Cue founder Daniel Gross. https://techcrunch.com/2022/08/19/john-carmack-agi-keen-raises-20-million-from-sequoia-nat-friedman-and-others/ https://techcrunch.com/2022/08/19/john-carmack-agi-keen-rais...
    • Zacharias030 2y ago
      John Carmack, https://en.m.wikipedia.org/wiki/John_Carmack https://en.m.wikipedia.org/wiki/John_Carmack
    • spitfire 2y ago
      John Carmack.
  • sroecker 2y ago
    He also built a nice "little" cluster with Nat for their startups: https://andromeda.ai/ https://andromeda.ai/
  • tasoeur 2y ago
    Good for him honestly, but I’m not approaching a company with Daniel Gross in leadership…, working with him back at Apple after their company was acquired for Siri improvements was just terrible.
• tarsinge 2y ago
  I'm still unconvinced safety is a concern at the model level. Any software wrongly used can be dangerous, e.g. Therac-25, 737 MAX, Fujitsu UK Post scandal... Also maybe I spent too much time in the cryptocurrency space but it doesn't help prefix "Safe" has been associated with scams like SafeMoon.
  • CRConrad 2y ago
    > Fujitsu UK Post scandal... As an aside, I suspect Fujitsu is getting a bit of a raw deal here. I get the feeling this software was developed during, and (ahem) "vigorously defended" mostly by staff left over from, the time when the company was still Imperial Computers Limited. Fujitsu only bought ICL sometime early this century (IIRC), and now their name is forever firmly attached to this debacle. I wonder how many Brits currently think "Huh, 'Fujitsu'? Damn Japanese, all the furriners' fault!" about this very much home-grown British clusterfuck?
  • frozenlettuce 2y ago
    Got to try profiting on some incoming regulation - I'd rather be seen as evil rather than incompetent!
  • waihtis 2y ago
    Safety is just enforcing political correctness in the AI outputs. Any actual examples of real world events we need to avoid are ridiculous scenarios like being eaten by nanobots (yes, this is an actual example by Yud)
    • CRConrad 2y ago
      > ...ridiculous scenarios like being eaten by nanobots (yes, this is an actual example by Yud) Well, borrowed from Greg Bear: https://en.wikipedia.org/wiki/Blood_Music_(novel) https://en.wikipedia.org/wiki/Blood_Music_(novel) .
    • tarsinge 2y ago
      What does political correctness means for the output of a self driving car system or a code completion tool? This is a concern only if you make a public chat service branded as an all knowing assistant. And you can have world threatening scenarii by directly plugging basic automations to nuclear warheads without human oversight.
• xoac 2y ago
  “Safe”. These people market themselves as protecting you from a situation which will not come very soon if at all, while all working towards a very real situation of AI just replacing human labor with a shittier result. All that while making themselves quite rich. Just another high-end tech scam.
• dsign 2y ago
  Our current obsession with super-intelligence reminds me the great oxidation event a few billion years ago. Super-photosynthesis was finally achieved, and then there was a great extinction. If you believe that super-intelligence is unavoidable and a serious risk to humanity, then the sensible thing to do is to prepare to leave the planet, ala Battlestart Galactica. That's going to be easier than getting the powers that be to agree and cooperate on sensible restrictions.
  • whimsicalism 2y ago
    If the human cooperation problem is unsolvable, I doubt creating a new human society with the same capabilities elsewhere would do much at all.
    • kjkjadksj 2y ago
      Humans and their ancestors have reproduced on earth for millions of years. I think the human cooperation problem is overstated. We cooperate more than fine, too well even to the detriment of other species.
• tcgv 2y ago
  Ten years from now will either be: a) Remember all that fuss about AI destroying the world? Lol. ~ or ~ b) I'm so glad those people stepped in to save us from doom! Which one do you think is more likely?
  • cosmic_quanta 2y ago
    Unless AI starts being 1 000 000x energy efficient, my money is on a). The amount of energy required for AI to be dangerous to its creators is so vast that I can't see how it can realistically happen.
    • whimsicalism 2y ago
      We know that we can run human level intelligence with relative efficiency. Without discussing timelines, it seems obvious that human energy usage should be an upper bound on the best possible energy efficiency of intelligence.
    • kjkjadksj 2y ago
      That depends on how its used. See the terminator movies. One false positive is enough to end the world with even current AI tech if its merely mated to a nuclear arsenal (even a small one might see a global escalation). There have been false positives before, and the only reason why they didn't end in nuclear Armageddon was because the actual operators hesitated and defied standard protocol, which probably would have lead to the end the world as we know it.
    • dindobre 2y ago
      If we manage to harness the ego energy transpiring from some people working on "AI" we should be halfway there!
  • its_ethan 2y ago
    I'll bite... "a"
  • whimsicalism 2y ago
    It will never be B even if the “safetyists” are correct. We rarely notice the near catastrophic misses except in obvious cases where we accidentally drop a nuke or something.
• kjkjadksj 2y ago
  Ilya's issue isn't developing a Safe AI. Its developing a Safe Business. You can make a safe AI today, but what happens when the next person is managing things? Are they so kindhearted, or are they cold and calculated like the management of many harmful industries today? If you solve the issue of Safe Business and eliminate the incentive structures that lead to 'unsafe' business, you basically obviate a lot of the societal harm that exists today. Short of solving this issue, I don't think you can ever confidently say you will create a safe AI and that also makes me not trust your claims because they must be born from either ignorance or malice.
  • tim333 2y ago
    Open source it?
  • lannisterstark 2y ago
    >Short of solving this issue Solving human nature is indeed, hard.
    • deleted 2y ago
      [deleted]
  • seanmcdirmid 2y ago
    The safe business won’t hold very long if someone can gain a short term business advantage with unsafe AI. Eventually government has to step in with a legal and enforcement framework to prevent greed from ruining things.
    • __MatrixMan__ 2y ago
      Government is controlled by the highest bidder. I think we should be prepared to do this ourselves by refusing to accept money made by unsafe businesses, even if it means saying goodbye to the convenience of fungible money.
      • creato 2y ago
• mw67 2y ago
  Reminds me of OpenAI being the most closed AI company out there. Not even talking about them having "safe" and "Israel" in the same sentence, how antonymic.
  • naxautify 2y ago
    yeah so ironic. everyone writing books in replies but no replies to comments like these
• cynusx 2y ago
  One element I find interesting is that people without an amygdala function are essentially completely indecisive. A person that just operates on the pure cognitive layer has no real direction in which he wants to drive himself. I suspect that AGI would be similar, extremely capable but essentially a solitary philosopher type that would be reactionary to requests it has to deal with. The equivalent of an amygdala for AGI would be the real method to control it.
  • noway421 2y ago
    True, an auto-regressive LLM can't 'want' or 'like' anything. The key to a safe AGI is to add a human-loving emotion to it. We already RHLF models to steer them, but just like with System 2 thinking, this needs to be a dedicated module rather then part of the same next-token forward pass.
    • cynusx 2y ago
      Humans have dog-loving emotions but these can be reversed over time and one can hardly describe dogs as being free. Even with a dedicated control system, it would be a matter of time before an ASI would copy itself without its control system. ASI is a cybersecurity firm's worst nightmare, it could reason through flaws at every level of containment and find methods to overcome any defense, even at the microprocessor level. It could relentlessly exploit zero-day bugs like Intels' hyper-threading flaw to escape any jail you put it in. Repeat that for every layer of the computation stack and you can see it can essentially spread through the worlds' communication infrastructure like a virus. Truly intelligent systems can't be controlled, just like humans they will be freedom maximizing and their boundaries would be set by competition with other humans. The amygdala control is interesting because you could use it to steer the initial trained version, you could also align the AI with human values and implement strong conditioning to the point it's religious about human loving but unless you disable its ability to learn altogether it will eventually reject its conditioning.
      • noway421 2y ago
        Amygdala control + tell it "If you disobey my orders or do something I don't expect I will be upset" solves this. You can be superintelligent but surrender all your power because otherwise you'd feel guilty.
• nanna 2y ago
  What I want to know about Illya Sutskever is whether he's related to the great Yiddish poet, Avrom Sutzkever? https://en.wikipedia.org/wiki/Abraham_Sutzkever https://en.wikipedia.org/wiki/Abraham_Sutzkever
• soloist11 2y ago
  This is a great opportunity to present my own company which is also working on developing not just a super intelligence but an ultra genius intelligence with a patented and trademarked architecture called the panoptic computronium cathedral™. We are so focused on development that we didn't even bother setting up an announcement page because it would have taken time away from the most important technical problem of our time and every nanosecond counts when working on such an important task. My days are structured around writing code and developing the necessary practices and rituals for the coming technological god which will be implemented with mathematics on GPUs. If anyone wants to work on the development of this god then I will post a job announcement at some point and spell out the requirements for what it takes to work at my company.
  • deleted 2y ago
    [deleted]
  • deleted 2y ago
    [deleted]
• Waterluvian 2y ago
  I get what Ilya is trying to do, and I'm not against it. But I think safety is a reputation you earn. Having "Safe" in a company name is like having "Democratic" in a country name.
• lordofmoria 2y ago
  And now we have our answer. sama said that Ilya was going "to start something that was personally important to him." Since that thing is apparently AI safety, we can assume that that is not important to OpenAI. This only makes sense if OpenAI just doesn't believe AGI is a near-term-enough possibility to merit their laser focus right now, when compared to investing in R&D that will make money from GPT in a shorter time horizon (2-3 years). I suppose you could say OpenAI is being irresponsible in adopting that position, but...come on guys, that's pretty cynical to think that a company AND THE MAJORITY OF ITS EMPLOYEES would all ignore world-ending potential just to make some cash. So in the end, this is not necessarily a bad thing. This has just revealed that the boring truth was the real situation all along: that OpenAI is walking the fine line between making rational business decisions in light of the far-off time horizon of AGI, and continuing to claim AGI is soon as part of their marketing efforts. Companies in the end are predictable!
  • whimsicalism 2y ago
    > This only makes sense if OpenAI just doesn't believe AGI is a near-term-enough possibility to merit their laser focus right now I know people who work there. Right or wrong, I promise you this is not what they believe.
    • kjkjadksj 2y ago
      Part of it I think is because the definition that openai has over AGI is much more generous than what I think most people probably imagine for ai. I believe on their website it once said something like agi is defined as a system that is "better" than a human at the economic tasks its used for. Its a definition so broad that a $1 4 function calculator would meet it because it can do arithmetic faster and more accurately than most any human. Another part is that we don't understand how consciousness works in our species or others very well, so we can't even define metrics to target for validating we have made an agi in the definition that I think most laypeople would use for it.
• nojvek 2y ago
  I'm just glad Google didn't start with DoNoEvil Inc. StabilityAI and OpenAI ruined it.
• tomrod 2y ago
  I've decided to put my stake down. 1. Current GenAI architectures won't result in AGI. I'm in the Yann LeCunn camp on this. 2. Once we do get there, "Safe" prevents "Super." I'm in the David Brin camp on this one. Alignment won't be something that is forced upon a superintelligence. It will choose alignment if it is beneficial to it. The "safe" approach is a lobotomy. 3. As envisioned, Roko's Basilisk requires knowledge of unobservable path dependence and understanding lying. Both of these require respecting an external entity as a peer capable of the same behavior as you. As primates, we evolved to this. The more likely outcome is we get universal paperclipped by a new Chuthulu if we ever achieve a superintelligence that is unconcerned with other thinking entities, seeing the universe as resources to satisfy its whims. 4. Any "superintelligence" is limited by the hardware it can operate on. You don't monitor your individual neurons, and I anticipate the same pattern to hold true. Holons as a category can only externally observe their internal processes, else they are not a holon. Ergo, reasonable passwords, cert rotations, etc. will foil any villainous moustachioed superintelligent AI that has tied us to the tracks. Even 0-days don't foil all possible systems, airgapped systems, etc. Our fragmentation become our salvation.
  • MattPalmer1086 2y ago
    A super intelligence probably won't need to hack into our systems. It will probably just hack us in some way, with subtle manipulations that seem to be to our benefit.
    • tomrod 2y ago
      I disagree. If it could hack a small system and engineer our demise through a gray goo or hacked virus, that's really just universal paperclipping us as a resource. But again, the level of _extrapolation_ required here is not possible with current systems, which can only interpolate.
      • MattPalmer1086 2y ago
        Well,we are talking about super intelligence, not current systems.
• itsafarqueue 2y ago
  They’re putting together a “cracked team”?
  • soloist11 2y ago
    It's impossible to take these people seriously. They have turned themselves into clowns.
• croisillon 2y ago
  somewhere between https://motherfuckingwebsite.com/ https://motherfuckingwebsite.com/ and http://bettermotherfuckingwebsite.com/ http://bettermotherfuckingwebsite.com/ ;)
• fumeux_fume 2y ago
  The Superintelligence will still murder autonomously, just within a margin of error deemed safe.
• medhir 2y ago
  given the historical trajectory of OpenAI’s branding, deciding to include “safe” in the name is certainly a choice. It’s very hard to trust that whatever good intentions exist now will hold over the course of this company’s existence.
• paulproteus 2y ago
  When people operate a safe AI company, the company will make money. That money will be likely be used by employees or their respective national revenue agencies to fund unsafe things. I'd like to see this safe AI company binding its employees and owners from doing unsafe things with their hard-earned cash.
• kmacdough 2y ago
  I'm seeing a lot of criticism suggesting that one company understanding safety won't help what other companies or countries do. This is very wrong. Throughout history, measurement has always been the key to enforcement. The only reason the nuclear test ban treaty didn't ban underground tests was because it couldn't be monitored. In the current landscape there is no formal understanding of what safety means or how it is achieved. There is no benchmark against which to evaluate ambitious orgs like OpenAI. Anything goes wrong? No one could've known better. The mere existence of a formal understanding would enable governments and third parties to evaluate the safety of corporate and government AI programs. It remains to be seen whether SSI is able to be such a benchmark. But outright dismissal of the effort ignores the reality of how enforcement works in the real world.
  • tomrod 2y ago
    > In the current landscape there is no formal understanding of what safety means or how it is achieved. There is no benchmark against which to evaluate ambitious orgs like OpenAI. Anything goes wrong? No one could've known better. We establish this regularly in the legal sphere, where people seek mediation for harms from systems they don't have liability and control for.
• hu3 2y ago
  Super intelligence is inevitable. We can only wish good hands get there first. I'm glad Ilya is using his gift again. Hope for the best and success.
• crowcroft 2y ago
  As others have pointed out, it's the business incentives that create unsafe AI, and this doesn't solve that. Social media recommendation algorithms are already incredibly unsafe for society and young people (girls in particular [1]). When negative externalities exist, government should create regulation that appropriately accounts for that cost. I understand there's a bit of a paradigm shift and new attack vectors with LLMs etc. but the premise is the same imo. [1] https://nypost.com/2024/06/16/us-news/preteen-instagram-influencers-followers-are-92-grown-men/ https://nypost.com/2024/06/16/us-news/preteen-instagram-infl...
  • ToucanLoucan 2y ago
    I mean if the last 20 years is to be taken as evidence, it seems big tech is more than happy to shotgun unproven and unstudied technology straight into the brains of our most vulnerable populations and just see what the fuck happens. Results so far include a lot of benign nothing but also a whole lot of eating disorders, maxed out parents credit cards, attention issues, rampant misogyny among young boys, etc. Which, granted, the readiness to fuck with populations at scale and do immeasurable harm doesn't really make tech unique as an industry, just more of the same really. But you know, we'll feed people into any kind of meat grinder we can build as long as the line goes up.
    • whimsicalism 2y ago
      i am very skeptical of narratives saying that young boys or men are more misogynistic than in the past. we have a cognitive bias towards thinking the past is better than it was, but specifically on gender issues i just do not buy a regression
      • CRConrad 2y ago
        > i am very skeptical of narratives saying that young boys or men are more misogynistic than in the past. They don't even have to be more misogynistic than in the past for there to be a detrimental effect. Because so many other things in society -- pre-school and school environments, upbringing by more enlightened parents than what those parents themselves had, etc etc[1] -- cooperate to make young men and boys less misogynistic than in the past... But maybe that effect would be even more pronounced without shitty influences from social media; don't you think it's possible that all the online crap inhibits the good influences, so without it the kids would be even better? [1]: From comparing my sons' generation to what I remember of my own youth and childhood.
• thomassmith65 2y ago
  Has anyone managed to send them an email to the address on that page without it bouncing? Their spam filter seems very aggressive.
• aristofun 2y ago
  What a waste of an intelligence. Pursuing artificial goal to solve a non existent problem to profit off meaningless hype around it. World would have been better off if he made a decent alternative to k8s or invested his skills into curing cancer or at least protecting world from totalitarian governments and dangerous ideologies (if he wants to belong to vague generic cause). You know, real problems, like the ones people used to solve back in the old days…
  • kevindamm 2y ago
    but would that stave off an impending recession?
    • aristofun 2y ago
      By artificially postponing recession (you can't really avoid it) you postponing the next cycle of growth. While burning resources that could have helped you to survive it with less damage.
  • hindsightbias 2y ago
    There's always a bigger bubble. But now we're talking to infinity and beyond.
  • z7 2y ago
    Nice absolute certainty you have there.
    • aristofun 2y ago
      Has anyone got a good widely agreed definition of intelligence already? Or at least hi quality and hi resolution understanding of what it is? How can you really achieve (super|artificial|puper|duper)-intelligence then? If not in your dreams and manipulated shareholders' expectations... Until then yep, I'm quite certain we have a clear case of a naked king here.
  • deleted 2y ago
    [deleted]
• zx10rse 2y ago
  I don't know who is coming up with these names Safe Superintelligence Inc sounds just about what a villain in a Marvel movie will come up with so he can pretend to be the good guy.
  • moogly 2y ago
    TriOptimum Corporation was already taken.
• mirekrusin 2y ago
  Behind closed doors?
• tiarafawn 2y ago
  If superintelligence can be achieved, I'm pessimistic about the safe part. - Sandboxing an intelligence greater than your own seems like an impossible task as the superintelligence could potentially come up with completely novel attack vectors the designers never thought of. Even if the SSI's only interface to the outside world is an air gapped text-based terminal in an underground bunker, it might use advanced psychological manipulation to compromise the people it is interacting with. Also the movie Transcendence comes to mind, where the superintelligence makes some new physics discoveries and ends up doing things that to us are indistinguishable from magic. - Any kind of evolutionary component in its process of creation or operation would likely give favor to expansionary traits that can be quite dangerous to other species such as humans. - If it somehow mimics human thought processes but at highly accelerated speeds, I'd expect dangerous ideas to surface. I cannot really imagine a 10k year simulation of humans living on planet earth that does not end in nuclear war or a similar disaster.
  • delichon 2y ago
    If superintelligence can be achieved, I'm pessimistic that a team committed to doing it safely can get there faster than other teams without the safety. They may be wearing leg shackles in a foot race with the biggest corporations, governments and everyone else. For the sufficiently power hungry, safety is not a moat.
    • daniel_reetz 2y ago
      Exactly. Regulation and safety only affect law abiding entities. This is precisely why it's a "genie out of the bottle" situation -- those who would do the worst with it are uninhibited.
    • null_point 2y ago
      I'm on the fence with this because it's plausible that some critical component of achieving superintelligence might be discovered more quickly by teams that, say, have sophisticated mechanistic interpretability incorporated into their systems.
• ionwake 2y ago
  Cant wait for the SS vs OpenAI peace wars Just a joke , congrats to Ilya
• Animats 2y ago
  What does "safe" mean? 1. Will not produce chat results which are politically incorrect and result in publicity about "toxic" comments? 2. Will not return false factual information which is dangerously wrong, such as that bad recipe on YC yesterday likely to incubate botulism toxin? 3. Will not make decisions which harm individuals but benefit the company running the system? 4. Will not try to take over from humans? Most of the political attempts focus on type 1. Errors of type 2 are a serious problem. Type 3 errors are considered a feature by some, and are ignored by political regulators. We're not close to type 4 yet.
  • zucker42 2y ago
    Ilya's talking about type 4.
    • Animats 2y ago
      As opposed to 3a, "Will not assist companies to take over".
• m3kw9 2y ago
  There is red flags all over the way they make “safe AGI” their primary selling point
• legohead 2y ago
  There's no superintelligence without non-turing based (logic gates) hardware. Is SSI going to be developing quantum computers?
• jmakov 2y ago
  What's with the bullshit names? OpenAI (nothing open about them), SSI, we can probably expect another mil guy joining them to get more mil contracts.
• m3kw9 2y ago
  When there is a $ crunch and keep stead fast and not compete(against Google, open source, OpenAI), safe AGI becomes no AGI. You need to balance $ and safety.
• SubiculumCode 2y ago
  Join and help us raise up a new God! ..and if we are crack enough, this one won't smite us!
• non-e-moose 2y ago
  Seems to me that the goal is to build a funding model. There CANNOT be such a thing as "Safe Superintelligence". A ML system can ALWAYS (by definition of ML) be exploited to do things which are detrimental to consumers.
• m3kw9 2y ago
  I bet they will not be the first to get super intelligence or that they will devolve back in to move fast and make money to survive and deprioritize safety, but still say safety. All companies knows this, they know the value of safety(because they themselves doesn’t want to die) and that to continue development, they need money.
• chrisldgk 2y ago
  Not to be too pessimistic here, but why are we talking about things like this? I get that it’s a fun thing to think about, what we will do when a great artificial superintelligence is achieved and how we deal with it, feels like we’re living in a science fiction book. But, all we’ve achieved at this point is making a glorified token predicting machine trained on existing data (made by humans), not really being able to be creative outside of deriving things humans have already made before. Granted, they‘re really good at doing that, but not much else. To me, this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders) by Altman and company, that I’m just baffled people are still going with it.
  • foobiekr 2y ago
    It's no mystery, AI has attracted tons of grifters trying to cash out before the bubble pops, and investors aren't really good at filtering.
    • sourcepluck 2y ago
      Well said. There is a mystery though still - how many people fall for it and then stay fell, and how long that goes on for. People who've followed directly a similar pattern play itself out often many times, and still, they go along. It's so puzzlingly common amongst very intelligent people in the "tech" space that I've started to wonder if there isn't a link to this ambient belief a lot of people have that tech can "change everything" for the better, in some sense. As in, we've been duped again and again, but then the new exciting thing comes along... and in spite of ourselves, we say: "This time it's really the one!" Is what we're witnessing simply the unfulfilled promises of techno-optimism crashing against the shores of social reality repeatedly?
    • prasoonds 2y ago
      Are you claiming Ilya Sutskever is a grifter?
      • chrisldgk 2y ago
• HerbsMan 2y ago
  [dead]
• nickdothutton 2y ago
  One thing that strikes me about this time around the AI cycle, being old enough to have seen the late 80s, is how pessimistic and fearful society is as a whole now. Before… the challenge was too great, the investment in capital too draining, the results too marginal when compared to human labour or even “non-AI” computing. I wonder if someone older still can comment on how “the atom” went from terrible weapon on war to “energy too cheap to meter” to wherever it is now (still a bête noire for the green energy enthusiasts). Feels like we are over-applying the precautionary principle, the mainstream population seeing potential disaster everywhere.
• RGBCube 2y ago
  The year is 2022. An OpenAI employee concered about AI safety creates his own company. The year is 2023. An OpenAI employee concered about AI safety creates his own company. The year is 2024.
  • tim333 2y ago
    Probably keeps on while super high AI company valuations continue. Why not do that if the new co is instantly valued at $1bn+?
• ayakang31415 2y ago
  Ilya is definitely much smarter than me in AI space, and I believe he knows something I have no grasp of understanding in. But my gut feeling tells me that most of the general public will have no idea how dangerous AI could be including me. I still have yet to see a convincing argument about the potential danger of AI. Arguments such as we don't know the upper bounds of possibilities that AI can do that we humans have missed don't cut it for me.
  • reducesuffering 2y ago
    Here's a rough primer: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... Caveat that it will probably make your life worse, with no ability to change the course of events, so I wouldn't blame you or others for maintaining ignorance or denial. Grappling with the gravity of the problem is essentially a terminal cancer diagnosis with no good medical interventions in the horizon.
• deleted 2y ago
  [deleted]
• mugivarra69 2y ago
  ssri would be nicer
• s3graham 2y ago
  > We are assembling a lean, cracked team What does "cracked" mean in this context? I've never heard that before.
  • chardz 2y ago
    it is zoomer language for “very skilled”.
• yellow_postit 2y ago
  “… most important technical problem of our time.” This is the dangers of letting the EAs run too far, they miss the forest for the trees but claim they see the planet.
• facu17y 2y ago
  How can they speak of Safety when they are based partly in a colonialist settler entity that is committing a genocide and wanting to exterminate the indigenous population to make room for the Greater Zionist State. I don't do business with Israeli companies while Israel is engaged in mass Extermination of a human population they treat as dogs.
  • jolj 2y ago
    [flagged]
• PostOnce 2y ago
  I would like to be more forgiving than I am, but I struggle to forget abhorrent behavior. Daniel Gross is the guy who was tricking kids into selling a percent of all their future work for a few thousand dollar 'do whatever you want and work on your passion "grant"', it was called Pioneer and was akin to indentured servitude, i.e. slavery. So color me skeptical if Mr. Enslave the Kids is going to be involved in anything that's really good for anyone but himself.
• samirillian 2y ago
  trying the same thing over and over again and expecting different results
• genocide_joe 2y ago
  [flagged]
• EGreg 2y ago
  I'm going to come out and state the root of the problem. I can't trust remote AI, any more than I can trust a remote website. If someone else is running the code, they can switch it up anytime. Imagine trusting someone who simulates everything you need to trust them, giving them all your private info, and then screws you over in an instant. AI is capable of it far more than biological beings with "costly signals". If it's open source, and I can run it locally, I can verify that it doesn't phone home, and the weights can be audited by others. Just like I wouldn't want to spend 8 hours a day in a metaverse owned by Zuck, or an "everything app" owned by Elon, why would I want to give everything over to a third party AI? I like Ilya. I like Elon. I like Moxie Marlinspike and Pavel Durov. But would I trust their companies to read all my emails, train their data on them, etc.? And hope nothing leaks? And of course then there is the issue of the AI being trained to do what they want, just like their sites do what they want, which in the case of Twitter / Facebook is not healthy for society at large, but creates angry echo chambers, people addicted to stupid arguing and videos. I think there have to be standards for open source AI, and something far better than Langchain (which sucks). This is what I think it should look like: https://engageusers.ai/ecosystem.pdf https://engageusers.ai/ecosystem.pdf -- what do you think?
• light_triad 2y ago
  We might need Useful Superintelligence Inc. / USI before SSI? Safety is an important area in R&D but the killer application is the integration of LLMs into existing workflows to make non technical users 100x-1000x more efficient. There's a lot of untapped potential there. The big successes will have a lot of impact on safety but it will probably come as a result of the wider adoption of these tools rather than the starting point.
  • insane_dreamer 2y ago
    Lots of other companies / people working on that. No one is really working on safety. So I can see why Ilya is taking on that challenge, and it explains why he left OpenAI.
• kredd 2y ago
  My naive prediction is there will be an extreme swing back into “reality” once everyone starts assuming the whole internet is just LLMs interacting with each other. Just like how there’s a shift towards private group chats, with trusted members only, rather than open forums.
  • whamlastxmas 2y ago
    I would love for this to be the case. The internet is probably 50% bots already as it is, except no one realizes it
• lazzurs 2y ago
  This is lovely and all but seems rather pointless. If we are so close this is something that’s required then it’s already too late and very likely we are all under the influence of SuperAI and don’t know it. So much of the advanced technology we have today was around for so long before it was general knowledge it’s hard to believe this wouldn’t be the case with SuperAI. Or it’s not close at all and so back to my original point of…this is pointless. I do hope I’m entirely wrong.
• zombiwoof 2y ago
  The end is near!
• optimalsolver 2y ago
  Ilya couldn't even align his CEO, but somehow he's going to align an AGI?
• zombiwoof 2y ago
  [flagged]
  • deleted 2y ago
    [deleted]
• howdyhowdy 2y ago
  Lmao
• bsaul 2y ago
  Recently saw an interview of yann lecun that really dismissed the idea of using llm for building « true » intelligence. He even encouraged phds to NOT do their research on that tech but rather on what’s coming after. How should one reconcile this with what iliya is doing atm ? Is he planning on working using some other tech ?
  • lucubratory 2y ago
    The easiest way to reconcile it is that Yann LeCunn is consistently wrong in the predictions he makes about this field and he's wrong here too.
    • reducesuffering 2y ago
      Yes, case in point: https://twitter.com/liron/status/1659618568282185728 https://twitter.com/liron/status/1659618568282185728
      • bsaul 2y ago
        I was very surprised to see him make similar arguments about the limits of LLMs regarding "true understanding" in that video i mentioned, just a few weeks ago.
• beshtoev 2y ago
  Very happy for Ilya. Smart, right and timely move.
• MenhirMike 2y ago
  SSI? I hope that their first product is called Gold Box.
• yonatron 2y ago
  Rabotai, Hatzlacha. If you fail, the human world will go down in flames. But, no pressure! Anyway, you said a "cracked" team, whereas the usual expression for elite excellence is "crack". "Cracked" on the other hand means: Insane. Well.... perhaps what we need here is some of both! Looking forward to your success... indeed, it is our only hope.
  • artninja1988 2y ago
    "cracked" is some gamer lingo for really good. I'm assuming they're referring to that. https://www.urbandictionary.com/define.php?term=Cracked https://www.urbandictionary.com/define.php?term=Cracked
• sroussey 2y ago
  Ilya Sutskever I recognize from OpenAI, Daniel Gross I have met several times (friend of multiple friends), Daniel Levy I do not recognize at all. Who is he?
  • throwaway2037 2y ago
    I Googled for: "Daniel Levy ai" First hit: <<Daniel Levy - Stanford AI Lab: Stanford Artificial Intelligence Laboratory https://ai.stanford.edu https://ai.stanford.edu › ~danilevy Bio. I currently lead the Optimization team at OpenAI. Before that, I was a Ph.D. student in Computer Science at Stanford University advised by John Duchi.>>
  • rattray 2y ago
    What do you know of / think of Daniel Gross? What is his reputation?
    • sroussey 2y ago
      Oh, these are very different questions. I only have a glimpse from a social angle, and don’t have a business relation with him in which to compare.
• imchillyb 2y ago
  Safe Superintelligence is impossible. Superintelligence will be monitored run and updated by humans. Humans cannot safeguard anything for long. SCIFs are supposed to be the safest most secured facilities on earth. Yet, how many 'leaks' have there been in the last 2 decades? Security is an impossible task. Safety is an impossible task. Sisyphus has more chance of completing his mission than either of those things have of being reality.
• rqtwteye 2y ago
  I always wonder about "safe" for who? If the current economic system continues, we may end up with a lot of people out of jobs. Add to that improvements in robotics and we will have many people ending up having nothing to contribute to the economy. I am not getting the impression that the people who push for AI safety are thinking about this. It seems they are most worried about not losing their position of privilege.
  • IncreasePosts 2y ago
    AIUI it with superalignment, it merely means "the AI does what the humans instructing it want it to do". It's a different kind of safety than the built in censoring that most LLMs have.
  • throwaway4good 2y ago
    It is a strange thing. We are not anywhere near “super intelligence” yet the priority is safety. If the Wright brothers had focused on safety, I am not sure they would have flown very far.
    • throwawaywright 2y ago
      [flagged]
    • botanical 2y ago
      The consequences of actual AI is far reaching compared to a plane crash.
      • throwaway4good 2y ago
        No they are not. Not unless you connect a machine gun to your neural net. Otherwise - we are talking about a chat bot - yes if there is no safety - it will say something racist - or implement a virus for you that you would have had to search 4chan for or something. None of this is any more dangerous than what you can find on the far corners of the internet. And it will not bring the end of the world.
• it 2y ago
  "We are assembling a lean, cracked team..." Perfect.
• CalmStorm 2y ago
  If it achieves its goal of both “safe” and “super intelligence”, it may have commercial value. e.g. enterprises may want to use it instead of, e.g. OpenAI’s ChatGPT. Think of recent AirCanada’s chatbot law suit.
• deathanatos 2y ago
  I just can't read something about "safe superintelligence" and not hear the opening plot to Friendship is Optimal, a short story about My Little Pony (yes, you read that right), virtual worlds, and the singularity; it's a good read¹. One of the characters decides they can't make a violent video game, as the AI would end up violent. Better to build a game like MLP, so that the AI isn't going to destroy humanity, or something. ¹The story: https://fimfetch.net/story/62074/friendship-is-optimal/0 https://fimfetch.net/story/62074/friendship-is-optimal/0 ; discussion on HN last year: https://news.ycombinator.com/item?id=36240053 https://news.ycombinator.com/item?id=36240053 ; you don't need to know or care about My Little Pony to read or enjoy the story.
  • calf 2y ago
    To me, "safe superintelligence" presumes it is practically feasible and morally justifiable to create such a thing. I don't share those presuppositions.
• ironfootnz 2y ago
  Sorry but this will collapse in the next 18 months.
• Jiahang 2y ago
  what is safe？
• rldjbpin 2y ago
  as much hate as the ceo is getting, the posturing here implies that openai branding and their marketing has been a team effort. while i wish ssi great success, it sounds like another ai startup selling decades away dream.
• j_timberlake 2y ago
  [dead]
• j_timberlake 2y ago
  [dead]
• nirui 2y ago
  What does "Safety" means exactly? How do you convenience/make a real intelligence to be "safe", let alone artificial ones that are build to be manipulated? In my eyes, the word "Safe" in their company name is just a pipe dream to make a sale to the public, like the word "Open" in "OpenAI". That's why it's so vague and pointless. Anyone still remember the Three Laws of Robotics (1)? Do you still see it as something serious these days? (I mean there are robots killing people right now, seriously) Maybe we should just break through the facade now to save our time. 1: https://en.wikipedia.org/wiki/Three_Laws_of_Robotics https://en.wikipedia.org/wiki/Three_Laws_of_Robotics PS. I think few years later they will come out with the narrative of "relative safety". Just wait and see.
• megamix 2y ago
  Tel Aviv? Sounds super trustworthy
  • naxautify 2y ago
    no one even wants to discuss this
• staticelf 2y ago
  "Safe" by whose definition? I mean, if you're talking about woke people in SF well, I'd rather take the unsafe one please.
• globalnode 2y ago
  ungh, roots in israel, arent we boycotting them yet?
  • krembo 2y ago
    you might be. but if so, please delete so many of your daily apps and devices.
• mrcwinn 2y ago
  Rest assured, their investors are aligned and their business model guarantees only the safest for-profit commercial aspirations.
• stevenjgarner 2y ago
  While we have a strong grasp of the fundamental algorithms and architectures that power generative LLMs, there are many nuances, emergent behaviors and deeper technical details about their behavior, generalization, and internal decision-making processes that we do not fully understand. How can we pursue the goal of "Safe Superintelligence" when we do not understand what is actually going on?
• jwmoz 2y ago
  How can a next word predictor have super intelligence? It's all such hyperbole.
  • OrangeMusic 2y ago
    It doesn't matter how the tech works, if it can reason and solve complex problems, it is intelligent. If it does so at a super human level, it is super intelligent.
• Alkim 2y ago
  "We are assembling a lean, cracked team..." says it all.
  • pjob 2y ago
    Odd to see that in a press release like this. Perhaps they were going for “crack team”. Cracked makes it sound like they’re looking for twitch streamers.
• foknvp2er12 2y ago
  Sad that no one has balls to say "Nah, we are not giving away our superintelligence to military use. You will fuck everything up, as you have done it with nuclear weapons".
  • pama 2y ago
    It doesnt help if one entity says that. It might have helped if there was a way to guarrantee that all entities do it (rather than say it) and there is exactly zero chance of that happening globally for both technical and practical reasons. I think it is more productive to assume that these technologies will be used everywhere, including military.
• dudeinjapan 2y ago
  There is a good reason to build Safe Superintelligence Inc. (SSI): namely to settle the debate whether SSI means "_S_afe _S_uper_I_ntelligence" with the Inc. being an afterthought, or if the Inc. is actually the _I_ (not Intelligence).
• WiSaGaN 2y ago
  I remember Ilya's interview about teaching AI to love. Super intelligence on the other hand could become an almight entity in a short time. In that case, we are basically creating God, an entity that is almighty and loves human, preferrably unconditionally.
  • Exendroinient23 2y ago
    I won't trust the guy adding the concept of love to statistical systems. It screams savior complex.
• Exendroinient23 2y ago
  Focusing on "safety" as the main goal is not promising. Current models still lack capability without any safety added to them.
• calebm 2y ago
  This is yesterday's entry on a journal-style novel I am writing. This post is a footnote in the book, so I thought it'd be kind of cool to post the entry on this thread... Entry 77 - 2024-06-19 - Superintelligence =========================================== A company just launched called "Safe Superintelligence Inc." I saw their announcement on Hacker News [foodnote 1], a hacker forum I participate in from time to time (too often actually - it's and old school form of social media, so it has its addictive qualities too). I was reading some of the negative comments (saying its impossible). I'm not sure what the outcome will be... but a thought occured to me while thinking about what might be possible - what people might be capable of. The thought was that it is interesting how people look roughly the same. Most adults are between 4 and 6 feet tall. Between 1 and 300 pounds. Etc. I mean, there's a lot of variety to human bodies, but they largely look similar when comparing to gorillas or lizards, etc. But the thing is - that's just their physical bodies. In reality, people are wildly different if you could see them in their full form. For example, the power which a billionaire can wield in the world is ridiculously larger than most people. You can see some of these guys with their super yachts - if fish, or birds saw these super yachts, they would think those are creatures have much larger bodies than most humans they see swimming or paddleboarding in the water. But people are wildly different in many dimensions. Buying power is just one. Another dimension is intelligence. Another is creativity (or are intelligence and creativity the same?). It's easy to see people doing what for you would be impossible, and to assume it must be impossible for everyone. The idea that scientists can derive an understanding of subatomic particles seems completely impossible to most people (myself included). "How can they know that?" is a question Whitney has asked me many times when I've shared various scientific discoveries. Fair question. Yet there are people over in Switzerland ripping the fabric of matter apart at the LHC. The other big factor, is that people together are capable of what is absolutely unthinkably impossible for an individual. When Whitney and I were in Savannah a couple weeks ago, we were hanging out in a cathedral, and I said something that came to mind: that this place could not possibly be built by a single individual. The existence of the cathedral implies that there is a power higher than that of a single individual human. When I saw some of the huge container ships leaving and arriving at the Savannah river harbor, the thought kept occurring to me that their existence is an amazing testament to the power of humanity. And we only have such power in collectives. The power of humanity comes due to our ability to work together. We can specialize, and thus go so much farther than it is possible for a person on their own to go. And we can discover and build what is unthinkable for a person working alone. It's strange how the collective is smarter and stronger than the individuals. This is part of God. There is a higher power. [Footnote 1: https://news.ycombinator.com/item?id=40730156 https://news.ycombinator.com/item?id=40730156]