3 ms·
Does anyone know if the average document length mentioned in the document length normalization is median? It seems like it would need to be to properly deweight
by MPSimmons 2y ago
Does anyone know if the average document length mentioned in the document length normalization is median? It seems like it would need to be to properly deweight excessively long documents, otherwise the excessively long documents would unfairly weight the average, right?
- softwaredoug 2y agoIt’s the mean. At least in Lucene. Using median would be an interesting experiment. Do you know of a search dataset with very large document length differences? MSMarco for example is pretty consistent in length.
- MPSimmons 2y agoWas just thinking about some of the docs we have at work, and how most are relatively short ( probably < 10 pages) and some are like... 200+ page government things