Right now signal's infrastructure could figure out (if it stored data that everyone claims it doesn't) which IP address is sending messages to which IP address.

Right now signal's infrastructure could figure out (if it stored data that everyone claims it doesn't) which IP address is sending messages to which IP address. If stories end up being implemented as a group conversation with all your contacts, everyone will be reasonably frequently sending a message to all their contacts, which changes the above into "which IPs are contacts of which IPs".