Difficulty adjustment is used to keep block times stable. Without this adjustment block times would continue to go down until you would eventually end up with a

Difficulty adjustment is used to keep block times stable. Without this adjustment block times would continue to go down until you would eventually end up with a chain that has much more frequent forks of greater lengths. You eventually end up with a state where the current head becomes indeterminate and all sorts of security issues become possible, including double spends. Bitcoin could have easily based block rewards on timestamps to keep the supply of money consistent, similar to how the current difficulty is calculated.