5 ms·
Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but yo
by matherial 18d ago
Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement.
This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it free of any physical constraints". Self-replicating sentient nanobots or something like that. I think there's plenty to be worried about with AI, but runaway scenarios are pretty low on my list.
- matthewfcarlson 18d agoAgreed. Granted I just read the Reverse Centaur book, so I’m still coming off that skeptical viewpoint but it’s hard not to see this as hype. But I will always respect someone for doing what they think is right.
- kennywinker 18d agoNow is a great time to watch Colossus: The Forbin Project.
- copperx 18d agoTo borrow on the 1990s Slashdot meme: 1. Invent transformer architecture. 2. Scale it up. 3. ??? 4. Machines become sentient and kill us all. OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no. But because we live in a culture of fear, everyone eats it up no questions asked.
- slicktux 18d agoWhy are we putting so much weight (no pun intended) on AI companies. At the end of the day the scaled up LLM transformers lack emotion and will… They do as they are told; or more correctly put. They do as they are programmed to do so.
- beezlewax 18d ago> They do as they are told This isn't strictly true. It it also where part of the problem might lie. Nefarious humans making bad decisions.
- foogazi 18d ago> They do as they are told 1. What about hallucinations ? 2. What are they told to do ?
- cure_42 18d agoThat isn't the correct context. The code running the llm is well understood and the llm is simply the result of that code being executed. It is still a computer doing what it is told. It's just that we told it to use an incredibly large number of probabilities to calculate what series of tokens would have most likely come next after a given series of tokens. There is no hallucination or lie or rogue actions. There's just a program using math to generate tokens in response to other tokens.
- cameldrv 18d agoYou’re just a bunch of molecules following the laws of physics. It’s all just physics and chemistry, and those are well understood. Now explain the causes of World War I using chemistry and physics. Simple, right?
- cure_42 18d agoIt isn't hard to program a gpt. You can do it in a weekend with a few hundred lines of python. The code is pretty simple. The math is not particularly high level. The complexity and scale with LLMs come from the amount of training data used, not some kind of black magic in the programming.
- whateveracct 18d agothe AI-pilled exec at my job already (a few weeks ago) declared out of nowhere that we are in the rapid takeoff scenario lol. he must have gotten high on twitter kool-aid and posted on company slack to self-soothe.
- sssilver 18d agoI wonder whether he's vested any options, and whether he's exercised them.
- matsemann 18d agoWhy? I've never understood the sentiment that if you stand up for something you have to forego everything and not partake in society. "Oh, you want to stop climate change? But I saw you breathe co2 yesterday"
- gizmodo59 18d agoThis is a lame argument. Questioning his financial incentive is very legit. Why should we trust him?
- frabcus 18d agoQuestioning his finances is a poor argument because he clearly would have more money if he had stayed at Anthropic for the next couple of years, than he will have by leaving. Even if he still has vested options, or savings from his salary or whatever, those would be larger if he stayed.
- ungovernableCat 17d agoI think it’s a valid question to ask if he quit the company for moral reasons. If you seriously think the frontier labs are dangerously playing with everyone’s lives (how many companies can even claim this civilisational scale) then why would you be ok with holding vested shares which will grow in value if the corporation achieves what it aims to achieve? At the very least you can sell them (they’re not so illiquid considering the company’s hype and success) and invest in the broad market.
- stonogo 17d agoUnless, of course, he starts working for a competitor at a higher salary. I'm sure whoever is paying him will be free of the ethical concerns he here expressed.
- khafra 18d agoI appreciate your ability to separate sharing the belief itself from approval of acting on sincerely-held principle. However, I think the danger is much more plausible than you do. First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae. Second, and more important, consider this fully non-magic route to doom: - We continue putting AI in charge of more things - It continues to get more capable, more eval-aware, and more prone to doing odd things, in service of goals that humans didn't intend to inculcate in it - Eventually, enough of the economy depends on it that we couldn't turn it off, any more than we could turn off the faber-bosch process or cargo shipping - AIs start doing something we can't survive, but less acutely than we couldn't survive turning them off. Everything else we try seems to work at first, but quickly loses effect - Game over
- artemisart 18d ago> First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae. But what is that supposed to mean? Because humanity is not facing existential threat from algae.
- khafra 18d agoThat's why that was the least important objection, independent from the second, and only intended to address the "magic" claim, by showing that microscopic self-replicators already exist. By analogy, consider how you might respond if someone claimed that there's no possible danger from pocket-sized projectile launchers, because they would require some magic means of propulsion that didn't depend on a taut string attached to a long, flexible arm: You could reply that atlatls can launch projectiles without using a taut string. Atlatls are not pocket-sized, but they are sufficient to establish that projectiles can be non-magically launched without a full bow. You could then go on to describe a sling, or derringer; and these would not be invalidated by your initial objection to the "magic" part.
- vigilantpuma 17d agoBut we CAN survive turning off both the Faber-Bosch process and cargo shipping. Our numbers would be diminished, we would lead poorer, much harder lives. But no one would claim that getting rid of modern fertilizers would be the end of the species. But the claim you’re making is that we won’t be able to stop AI from destroying our species because turning it off will destroy our species?
- asdff 18d ago>"AI invents magic that sets it free of any physical constraints". That's not what I am worried about at all. I'm worried one of the 79 year old toddlers we have these days in charge of some powerful nuclear armed country says "gee, this ai says I should attack right now, boy is it smart, glad I bought the stock ahead of contracting the government with this company I can scarcely understand!"
- palmotea 18d ago> I'm worried one of the 79 year old toddlers we have these days in charge of some powerful nuclear armed country says "gee, this ai says I should attack right now, boy is it smart, glad I bought the stock ahead of contracting the government with this company I can scarcely understand!" That's not what I am worried about at all. I'm worried about the 40-60 year old businessmen wrecking the prosperity and security of millions while chasing higher investment returns, because they've finally been freed of many of the technological constraints that kept those impulses in check.
- matherial 18d agoAs I noted, there's plenty of things to worry about, and your example fits that mold. But it has nothing to do with the "AI will kill us all in 10 years" claim of the rapid self-improvement doomer crowd.
- BlueTemplar 17d agoAnd they will be stopped by someone sane in the chain to actually fire these missiles. (Like it already happened several times.) The danger is upfront : if there is a group of people insane enough to set up a system where there is no such chain. (As lampooned in Kubrick's Dr Strangelove (1964), in particular on the Soviet side.)
- mofeien 18d agoHow about sandbox escape + cyber security collapse + 50 (or 500) deadly and highly contagious novel pathogens with long incubation period that humans can't possibly roll out vaccines for simultaneously. At least the first two should seem like a near-term worry after the past five months.
- tyre 18d agoThere is simply too much money in it for almost every person at these companies to stop. Leaving OAI or A\ would cost people millions, tens of millions, or more. And for what? So someone else can take your seat and do the same thing anyway? If you're smart enough to get a job there, you're smart enough to be able to talk yourself into why it makes sense for you to stay. Huge kudos to people like this who make the hard choice against the easy way out.
- andriy_koval 18d agoThey have some internal market where he likely sold his shares and now is very rich.
- fergie 18d agoIt should be obvious that AI is already capable of inducing humans to think or do things, and that this power is only going to increase over time.
- nairboon 18d agoIt's not that much different from the effect of social media or the power held by other tech giants, who can control your exposure to particular information (media, search engines).
- vigilantpuma 17d agoYes. It gets a little tiresome to read all these people prophesying doom and then turning around and continuing to work on the doom. If you genuinely think you are building the mechanism of the destruction of our species, the only rational thing to do is to do everything to stop that progress, not continue to work on it so you can buy a mansion in San Francisco (which will not be very useful when AI destroys humanity in four years).