That approach is pretty much what I was describing. The example in the paper uses the previous 4 tokens to derive the bias for the next token, so some 5 token s

That approach is pretty much what I was describing. The example in the paper uses the previous 4 tokens to derive the bias for the next token, so some 5 token sequences become more likely and others becomes less likely compared to what the LLM would normally produce. The context and temperature of the LLM stays the same, but the sampling process is biased by something that depends on far shorter runs of text.