Good question! In retrospect, I should have explained that section differently. I will probably revise that section of the text (that's what I love about arxiv

Good question! In retrospect, I should have explained that section differently. I will probably revise that section of the text (that's what I love about arxiv). We don't want the recurrent terms "h_{t-1}" to dominate the output of "z". But we don't want to get eliminate the recurrent terms because without "h_{t-1}" the output of "z" will not be sequence dependent. So the "tanh" term is included to allow the recurrent terms to change the sign of "z". I don't know if that helps. Thanks!