It is trivial, the command ipfs dht findprovs <hash> will list all the peer ids that do have the content in their repo so unless you blacklist every one but you

It is trivial, the command ipfs dht findprovs <hash> will list all the peer ids that do have the content in their repo so unless you blacklist every one but your friend (and he does the same) then the fact that you have accessed <hash> is public knowledge. You could also gc your repo as soon as you have downloaded that content but both of these methods defeat the whole point of IPFS.