As I understand it, all of the allegations were investigated by Madame Justice Boyd who found that none of them were substantiated except for the affair, which

As I understand it, all of the allegations were investigated by Madame Justice Boyd who found that none of them were substantiated except for the affair, which I agree is far from advisable, but on its own seems unlikely to have ended his career so completely. There is no way to go back to 2015 and find out what would have happened if the affair had been the only accusation.