2 ms·
To be specific, what is the probability that the next word is "sdh7777asljashd_"? Any word level model trained on any dataset will say it is zero. But that is
by nl 9y ago
To be specific, what is the probability that the next word is "sdh7777asljashd_"?
Any word level model trained on any dataset will say it is zero. But that is incorrect; humans know that there is always a probability that some arbitrary sequence of characters will suddenly appear in text (think a GUID in technical documentation). To account for this, the model's author has to make sure to smooth the output so it doesn't produce these zero probabilities.
No amount of data can ever fix this problem (although more data can put better bounds on the smoothing factor).
- JoeAltmaier 9y agoAnd I'm saying, depends on how the model is built. If it is a simple dictionary then yes. But there are better ways to build a model. Its a naïve view that all models are like some strawman.
- nl 9y agoI don't understand what you are arguing. >>> the maximum-likelihood model is one where the frequency of every word is the frequency that you observed that word, and the frequency of unseen words is 0 >> I think that's false. A model may never reduce any probability to zero. Novel data is a normal feature of modelling. > But there are better ways to build a model. Yes of course there are. That's exactly what the OP said ("You correct it by being an intelligent human who knows the maximum-likelihood model is wrong, and applying a correction on top of it") but you seemed to be arguing against it. (And to be clear, there are also alternatives to the maximum likelihood model)