I'm sure that if he were in any way relevant to the conversation, then his affair would come up in conversation. But he's not, so let's not wander down that

I'm sure that if he were in any way relevant to the conversation, then his affair would come up in conversation. But he's not, so let's not wander down that rabbit hole.