I wonder what the creative spark even is in the context of an autoregressive transformer. Perhaps it’s an ability to confabulate facts into the context window

I wonder what the creative spark even is in the context of an autoregressive transformer. Perhaps it’s an ability to confabulate facts into the context window which are not present in the training data but which are, in the context of maths, viable hypotheses? Every LLM can generate bullshit, but maybe we just need the right bullshit?