No, the dot product won't work in this case. Take the two edges of the triangle as vectors. You want the mouse vector to fall in between. The dot product can't

No, the dot product won't work in this case. Take the two edges of the triangle as vectors. You want the mouse vector to fall in between. The dot product can't tell you which side of the two edge vectors your mouse vector falls on, but the cross product can if you look at the component of the cross product that points in/out of the screen. Actually, you can make the dot product give you that answer, but you would need use the dot product of the mouse vector and two vectors normal to the edge vectors, which is really just a roundabout way of using the cross product.