Yes to all of the above. Although namespacing is not enough, the packages need to be built in a specific way so that they don't put files in any system location

Yes to all of the above. Although namespacing is not enough, the packages need to be built in a specific way so that they don't put files in any system locations. This isn't anything specific to Ubuntu, this is just how Debian works. If you want to do things "the right way" you're supposed to submit your packages upstream and be patient. This is not exactly a new problem, it's why Docker is popular, it's why Fedora Silverblue and NixOS exist. It's just not done on the OS level in Debian because in some ways it doesn't really need to be.