The difficulty would adjust down, but only when the next milestone is reached. The argument seems to be that if enough miners drop out, the next milestone might

The difficulty would adjust down, but only when the next milestone is reached. The argument seems to be that if enough miners drop out, the next milestone might never be reached. That seems unlikely to me.