Are you building the Debian kernel with the same .config as when you build the kernel directly? If not, that's probably why. A Debian kernel build will build

Are you building the Debian kernel with the same .config as when you build the kernel directly? If not, that's probably why. A Debian kernel build will build basically everything, mostly as modules. A kernel tailored for your specific hardware and needs will be much faster to build. Also disabling building the _dbg package for the Debian build will significantly speed things up. Building that package takes a strangely long amount of time.