Background subtraction works really well for these kinds of problems. You should give that a go, even better if you can make your background based on a rolling

Background subtraction works really well for these kinds of problems. You should give that a go, even better if you can make your background based on a rolling average of the image pixels. As for the speed issue, your can often get massive gains by restricting the colour classification and blob detection to areas of the image which have changed since the previous frame.