Some of the points you mention are not unlike the ones that ASR faces: Lip reading is so dependant on context of the conversation. -> So is speech recognition.

Some of the points you mention are not unlike the ones that ASR faces: Lip reading is so dependant on context of the conversation. -> So is speech recognition. And actually, it's the same context. There are so many possible combination of words that could look the same from a lip reading perspective. -> That's the same in speech recognition, and it's the reason you don't just model the acoustics but also have a language model. everyone's lips are different -> Everyone's voice is different. but then what if they have an accent? -> That is possibly a bigger problem for ASR than for lip reading, but that's just my guess. However: facing the wrong direction -> This for me is the real trouble. In the video clips you see in the paper (as stills), you have nice full frontal view of the lips. But when do people actually hold that still outside a lab situation? In ASR, if people turn their head, your signal quality drops (and hence your recognition rate does too) but when you move the head > 80-90 degrees away from the camera, you have pretty much no signal left to work with.