Hotword recognition does already exist and works on fairly modest hardware (I've never tested it on a pi, so I don't know specifically how well that would work

Hotword recognition does already exist and works on fairly modest hardware (I've never tested it on a pi, so I don't know specifically how well that would work - or more accurately how fast). The trouble is that it's an enormous hassle to set up and while it is open source, it involves some company or other having it's grubby paws all over the recognition networks. Which may or may not rub you the wrong way.