I haven’t really grokked this work yet well enough to critique it, but to answer your question: Yes you could incorporate a semantic equivalence dataset into y

I haven’t really grokked this work yet well enough to critique it, but to answer your question: Yes you could incorporate a semantic equivalence dataset into your training but: 1) when you have a bunch of ‘clear-cut’ functions (“achieve good AUC on semantics”) and you mix them to compensate for the weaknesses of a complicated model with an unknown perceptual objective, things are still kinda weird. You don’t know if you’re mixing them well, to start, and you also don’t know if they introduce unpredictable consequences or hazards or biases in the learning. 2) on a kinda narrowly defined task like: “can you determine semantic equivalence”, you can build a good model with less risk of unknown unknowns (than when there are myriad unpredictable interactions with other goal scoring measures) 3) if you can apply that model in a relatively clear cut way, you also have fewer unknowns unknowns. Thus, carving a path to a particular reasonable heuristic using two slightly biased estimators can be MUCH safer and more general than mixing that data into a preexisting unholy brew and expecting its contribution to be predictable.