This is essentially where Sony was trying to go with their Cell architecture in the PS3. Only at the very end did they realize that they actually needed a GPU t

This is essentially where Sony was trying to go with their Cell architecture in the PS3. Only at the very end did they realize that they actually needed a GPU that can do rasterization in hardware. In fact, a lot of games actually did graphics workloads on the SPEs to help out the pretty weak GPU. The concept can definitely work, especially if the driver takes care of all the programmable bits and exposes a more classical graphics pipeline to the host.