Sure, I agree to some extent. In the hypothetical case it may be reasonable to assume that X knows what they are talking about; one can reasonably assume that t

Sure, I agree to some extent. In the hypothetical case it may be reasonable to assume that X knows what they are talking about; one can reasonably assume that they are better informed regarding the subject than a layperson. But the reason for it being a fallacy is that this sort of argument doesn't provide evidence of any kind. Rather, it makes the assumption that it must be true, because the authority says so.