Don't know about apt specifically, but using pacman (Arch Linux), you can list exactly what files on your filesystem were installed by what package and remove t

Don't know about apt specifically, but using pacman (Arch Linux), you can list exactly what files on your filesystem were installed by what package and remove them. You can't do this on Windows, as far as I know.