Java wasn't opensource. It had a specific license for mobile phones. The reason that Android didn't use their VM was the $1/device licensing fee for using Java

Java wasn't opensource. It had a specific license for mobile phones. The reason that Android didn't use their VM was the $1/device licensing fee for using Java on mobile devices. They reimplemented everything in order to avoid that $1.