If the Linux kernel can be sliced and diced, what's the point in putting non-essential code, that won't be used by the majority of running systems, in the kerne

If the Linux kernel can be sliced and diced, what's the point in putting non-essential code, that won't be used by the majority of running systems, in the kernel? Why not offer the driver as additional package / download?