You are correct in that all of the distributions only vary by the set of packages installed by default. The server edition, however, is designed to run headless

You are correct in that all of the distributions only vary by the set of packages installed by default. The server edition, however, is designed to run headless and is thus very functionally different (i.e. X is not running by default, there are no window managers installed, etc). Converting the "Desktop Edition" to the "Server Edition" is just as much about removing packages as it is about adding them.