If you use the Minimal CD for installing ubuntu (https://help.ubuntu.com/community/Installation/MinimalCD https://help.ubuntu.com/community/Installation/Minimal

If you use the Minimal CD for installing ubuntu (https://help.ubuntu.com/community/Installation/MinimalCD https://help.ubuntu.com/community/Installation/MinimalCD) you can pick a metapackage along the way that may be more to your liking. I dont know what exactly you dont want from a standard xubuntu install but there is an xfce "minimal" metapackage in the ecosystem which may help you install only what you need from the start