We probably stay with the next token prediction task for training the foundation. It generalizes really well. However, transformers are not able to loop. In a

We probably stay with the next token prediction task for training the foundation. It generalizes really well. However, transformers are not able to loop. In a fixed amount of computing steps, they need to predict. For long-term planning, we probably want something more recursive/iterative (but that will be harder to train). Also today we can compensate that with wrappers (eg. Langchain), but ultimately the machine will learn end to end.