It depends on how you are using your desktop. More memory used for the DE means less memory for other stuff, like running VMs or development servers, compiling

It depends on how you are using your desktop. More memory used for the DE means less memory for other stuff, like running VMs or development servers, compiling and dev tools, games or media applications, a browser, and even just caching disk files in memory.