you add a linear classifier at the top to predict start and end positions of the answer span. The augmented model is trained on a qa dataset like squad to actua

you add a linear classifier at the top to predict start and end positions of the answer span. The augmented model is trained on a qa dataset like squad to actually learn how to answer questions. hugging face has a simple implementation that augments bert in this manner and you can see the code there. their bertqa model get like an 84 F1 on squad 1.1 which really strong performance. you can augment the thier gpt2 implementation similarly.