3 ms·
>Do they really know exactly what the raw text looks like? I mean yea, it's too big. That said, in a post ad hoc fashion they do. When the model spits out weir
by pixl97 3y ago
>Do they really know exactly what the raw text looks like?
I mean yea, it's too big. That said, in a post ad hoc fashion they do. When the model spits out weird crap at times, they can search the raw corpus and filter those strings out. There was an incident around this with Reddit counting forums and strange usernames that were added in tokenization, but later removed from weights leading to odd behavior when doing inference.