3 ms·
This is like the density ratio approach (https://arxiv.org/abs/2002.11268 https://arxiv.org/abs/2002.11268). The Hybrid Autoregressive Transducer also uses a si
by albertzeyer 6y ago
This is like the density ratio approach (https://arxiv.org/abs/2002.11268 https://arxiv.org/abs/2002.11268). The Hybrid Autoregressive Transducer also uses a similar idea (https://arxiv.org/abs/2003.07705 https://arxiv.org/abs/2003.07705).
Do you really need the renormalization, i.e. the outer softmax?
- sweezyjeezy 6y agoWe logit to start so we need to go back from logit space -> probability space. Linear transformations in logit space don't preserve the sum = 1 requirement of the corresponding probability so you can't just take the sigmoid of each element individually. This is just a slick way of writing it, you can write it all out without any logits / sigmoids, but it's easier to do the algebra in logit space.
- albertzeyer 6y agoSorry, maybe my question was misleading. Why do you need the sum=1? See the linked papers. They do the same but without the renormalization.
- sweezyjeezy 6y agoAh ok, I'm not overly familiar with the problem they're trying to solve really - wouldn't always be necessary depending on your application, but I was talking in broad ML terms, if you need a calibrated prediction at the end you do.