That's easier said than done, AMD and Nvidia probably have licensed and patented code etc in their closed-source drivers, which would make it difficult to open

That's easier said than done, AMD and Nvidia probably have licensed and patented code etc in their closed-source drivers, which would make it difficult to open source, where as a project open source from the get go won't have these issues. Nvidia got around this on their kernel driver by moving most of it to the cards firmware.