I don't have an answer to your question, but have you considered scrapping voice control altogether and instead using motion detectors or other rules to govern

I don't have an answer to your question, but have you considered scrapping voice control altogether and instead using motion detectors or other rules to govern whether the lights are on?