They don't all do that. There's many approaches being experimented on. Some start with random tokens, or with masks, others even start with random vector embed

They don't all do that. There's many approaches being experimented on. Some start with random tokens, or with masks, others even start with random vector embeddings.