This kills the forward secrecy. IIRC Signal just has each group member send each group message to each recipient with the standard pair-wise encryption keys. I

This kills the forward secrecy. IIRC Signal just has each group member send each group message to each recipient with the standard pair-wise encryption keys. It's the message's headers that lets the recipient know it's intended for the group and not the 1:1 group.