Wat I don't think you actually understand what's going on here. The untrusted programs in this case would be javascript and other exploitable things that fire

Wat I don't think you actually understand what's going on here. The untrusted programs in this case would be javascript and other exploitable things that firefox interprets every time you visit a webpage. Unless you run firefox itself under a different UID or inside of a VM you are effected by this, regardless of whether you are running other "untrusted" software or not. Maybe you are thinking that using X11 as an attack vector requires a "bad" program running in addition to firefox? If so, that is not the case, any program that is running on X11 that can execute commands or read and write files could be used for this purpose. A terminal emulator is a pretty good example of such a program.