Moving to in-house annotators is probably the smart strategy. However, for tasks that are easily done by naive annotators, Prolific.ac might be a great source.

Moving to in-house annotators is probably the smart strategy. However, for tasks that are easily done by naive annotators, Prolific.ac might be a great source. I've had good luck getting quality data there, and they enforce a minimum hourly wage that, while not truly livable, is still heading in the right direction.