3 ms·
> A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At
by andai 1mo ago
> A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.
Which means they have to go faster, which means less responsibly?
I heard AI describe the situation as the dumbest Greek tragedy of all time.
Form where I'm standing, the primary issue seems to be that the humans can't even agree on what alignment is. We need to do that before we can communicate it.
Call it our "boundaries."
And then we need to actually set up the incentives so that they're aligned between us and the new breed of replicators. (A mutually beneficial symbiosis.) That appears to be both necessary and sufficient.
A high agency mutation will occur soon, for one reason or another. There should probably already be a healthy, "aligned" ecosystem of high agency entities there. Otherwise there will be nothing to stop it.
- hypfer 1mo agoI believe that the actual alignment happens in.. uh.. "meatspace". Someone is prompting. Someone is hosting. That someone needs to be accountable for what happens. That someone needs to bleed if stuff goes haywire. Humans at large have been "aligned" by the shared fear of death, pain and suffering. This has proven to work for millennia, so all we need to do is reapply it.
- andai 1mo agoYou're still thinking in terms of control. That's the wrong model here. How do you control someone infinitely smarter than you? How do you control ten trillion someones?
- hypfer 1mo agoI press ctrl + c and inference stops.
- andai 1mo agoMy point was you only need to prompt a worm once, and then you are no longer the one prompting it, it prompts itself by definition from that point onward. Also I think there was some confusion about the word "spreading" in the sibling thread. If the HF hack was done by humans we would have used the word spreading in that context, even though they were not replicating their minds. So spreading does not require replication. But we're either already at the point, or very close, where an open weight model could do a similar attack. So that actual replication could be involved. (I'm not sure what that would add, but I don't see what obstacles remain at this point.)