All that matters is if they are effective terms or not. That's most of the point of the article; how to constructively inquire about whether giant LLMs might b

All that matters is if they are effective terms or not. That's most of the point of the article; how to constructively inquire about whether giant LLMs might be effective at solving novel problems, perhaps better than humans can, in ways that matter to humans. Since we don't have a formal model of how systems like humans work we can't just plug numbers into equations or generate proofs that LLMs and other ML systems will or will not be able to control other systems better than humans, so we revert to less-formal concepts as best as we can. The formal AI safety folks (MIRI et al) already tried the formal approach; by their own admission it's hopeless and they poured a lot of time and bright minds into it and we won't be able to formally answer questions about intelligence via decision theory or alignment before we have actually-intelligent systems interacting with us. What will happen at that point? We don't know and can only use imprecise concepts to make educated guesses. If we don't make honest attempts at this we run the risk of being blindsided by unexpectedly capable AI/ML systems. This is why a large number of AI alignment researchers recommend pausing AI/ML systems at their current ability before they have a chance at being superhuman on most tasks. I wouldn't trust "distributed indexing" or "stochastic generation" if it could beat me at any particular skill of my choosing like similar systems can beat any of us at Chess or Go or Poker.