I do not agree with this. I work heavily with NLP models for production in the Legal domain (where my baseline is where a 8GB 1080 must predict more than 1000 w

I do not agree with this. I work heavily with NLP models for production in the Legal domain (where my baseline is where a 8GB 1080 must predict more than 1000 words/sec). What kind of network are you using? I can do near-SoTA multi-task syntax annotation [1] with ~4000 tokens/s (~225 sentences/s) on a CPU with 4 threads using a transformer. Predicting 1000 words/second on a reasonably modern a GPU is easy, even with a relatively deep transformer network. [1] 8 tasks, including dependency parsing.