2 ms·
no because we have no way to understand the full consequence chain of any stated goal. We see this time and time again in all complex system goals; unintended c
by apsurd 3mo ago
no because we have no way to understand the full consequence chain of any stated goal. We see this time and time again in all complex system goals; unintended consequences.
So RSI toward what prompted goal? Then, how can you be so sure that reaching that goal looks like the straight line you envision?
edit: more accurately, it's about incentives. We generally understand that encoding a goal in a complex system is about aligning incentives. So now it's the incentives that are proxies for intended outcomes. But as mentioned, we have a growing body of evidence for just how bad we are at aligning incentives with intended outcomes. And it makes sense because complex systems are not a straight line.
- jmoggr 3mo agoIs a coherent, human-interpretable, goal necessary for recursive improvement? > how can you be so sure that reaching that goal looks like the straight line you envision? Because the thing pursuing the goal would be able to improve it's ability to pursue the goal. Doesn't that follow from the premise of RSI? > we have a growing body of evidence for just how bad we are at aligning incentives with intended outcomes. We are terrible at it! I don't think that will stop us letting something rip in some unknowable direction (with unintended outcomes). I hope that aligned incentives are a prerequisite, but it seems increasingly likely that they are not.
- pron 3mo ago> Because the thing pursuing the goal would be able to improve it's ability to pursue the goal But not necessarily without needing a huge amount of resources. It is a mathematical certainty that no intelligence can solve computationally intractable problems (including forecasting the weather, or the economy) without access to resources we simply don't have.