> A fundamental problem with this entire class of machine learning is that it is based on a model / simulation of reality… all such simulations that attempt to

> A fundamental problem with this entire class of machine learning is that it is based on a model / simulation of reality… all such simulations that attempt to model any non-trivial system are imperfect Depends on what your goal is. If you are trying to solve the narrow problem of rocketry or whatever, sure. But maybe not if your goal is making models smarter. The broader context is that we need new oracles beyond math and programming in order to exercise CoT models on longer planning-horizon tasks. In this case, if working with a toy world model lets you learn generalizable strategies (I bet it does, as video games do too) then this sort of eval can be a useful addition.