To the extent that you've diluted the market for new copies in some small but real way, then yes, you have theoretically harmed the author. However, the law do

To the extent that you've diluted the market for new copies in some small but real way, then yes, you have theoretically harmed the author. However, the law doesn't recognize that harm as significant enough to be legally actionable. To the contrary, it recognizes that always placing the interests of authors above those of the public would do the public intolerable harm, and that the author is going to have to live with a less-than perfectly controlled market if he wants any control at all. What the author of a desirable work CAN expect is that there will be a reasonable volume of exchanges in which unavoidable obligations to pay do arise. Based on these he (or, more likely, his publisher) can invest in production accordingly. If your copies and exchanges take place outside this range, then there's no way for the publisher to claim he's being unreasonably harmed by them, since the law gave him no reason to expect that he'd profit from these transactions in the first place. But if they take place within the parameters of demand that the law allowed him to bank against, then the illegal dilution of his legally recognized market constitutes very real harm.