IIRC, Microsoft tried to build something along those lines with the Singularity research operating system - it did not use virtual memory to isolate processes f

IIRC, Microsoft tried to build something along those lines with the Singularity research operating system - it did not use virtual memory to isolate processes from each other, but it strongly restricted how processes could exchange data, thus getting pretty much the same isolation level conventional operating systems get by using virtual memory. I even think the source code to Singularity has been released under an open source-ish license, if you want to look into it.