4 ms·
> Training a sequence-to-sequence language model (T5 derivative) on our custom dataset designed for factual generation yielded much better results with less hal
by aquajet 4y ago
> Training a sequence-to-sequence language model (T5 derivative) on our custom dataset designed for factual generation yielded much better results with less hallucination.
Could you elaborate more on this or point to a paper/benchmark results?
- rushingcreek 4y agoI'd be happy to talk a bit about how we evaluated the model. The task we're performing is fundamentally long-form question answering (LFQA), and recent papers (https://arxiv.org/pdf/2103.06332.pdf https://arxiv.org/pdf/2103.06332.pdf) have shown that metrics such as ROUGE (used for the KITE benchmark) aren't great at evaluating the quality & truthfulness of generated answers. On our dataset, our approach is to use a combination of human evaluation (which is still arguably the most reliable metric used by the NLP research community to evaluate generated answer quality) and an entailment score (checking if a generated answer is consistent with a "ground truth" context document).