As a mathematical model, it's almost completely unhelpful, like saying that all computers are technically state machines because they have a finite amount of me

As a mathematical model, it's almost completely unhelpful, like saying that all computers are technically state machines because they have a finite amount of memory. Treating every combination of 4k tokens as a separate state with independent probabilities is useless for making probability estimates. Better to say that it's a stateless function that computes probabilities for the next token and leave Markov out of it.