4 ms·
I don't the point they are trying to make about BERT not learning dependencies between masked words. Isn't the mask randomly chosen each time, so it has a chan
by jeremysalwen 7y ago
I don't the point they are trying to make about BERT not learning dependencies between masked words. Isn't the mask randomly chosen each time, so it has a chance to learn with all possible words unmasked?
- xxxpupugo 7y agoI think the argument behind is that XLNet retains all context regardless. Ultimately, it is the performance that speaks.
- anewhnaccount2 7y agoSee the section "Comparison with BERT" which has a nice worked example that is reasonably understandable without necessarily fully understanding the rest of the paper.
- jeremysalwen 7y agoThat section you are saying was reasonably understandable was the part that confused me. For example: > it is obvious that XLNet always learns more dependency pairs given the same target and contains “denser” effective training signals BERT is only masking 15% of the tokens, so isn't the amount of dependency pairs like 18% higher at most?
- yorwba 7y ago> BERT is only masking 15% of the tokens, so isn't the amount of dependency pairs like 18% higher at most? That's a small but significant difference, allowing them to improve performance by a small but significant amount.
- psb217 7y agoMost of the statements they make regarding orderless autoregression, including statements about the "independence assumption" made by BERT, are misleading at best.
- jeremysalwen 7y agoThank you, it seemed that way to me, but I thought maybe there was something I wasn't understanding.