4 ms·
That should be proven. The two approaches - predicting tokens vs predicting "sentences" - should be compared to see how much their output differ in terms of qua
by mdp2021 2y ago
That should be proven. The two approaches - predicting tokens vs predicting "sentences" - should be compared to see how much their output differ in terms of quality.
Edit2: ...and both (and their variants) be compared to other ideas such as "multi-token prediction"...
Edit: or, appropriateness of the approach should be demonstrated after acquired "transparency" of how the LLMs effectively internally work. I am not aware of studies that make the inner workings of LLMs adequately clear.
Edit3: Substantially, the architecture should be as solid as possible (and results should reflect that).
- blackeyeblitzar 2y agoIsn’t “sentence prediction” roughly the same as multi token prediction of sufficient length? In the end are we just talking about a change to hyper parameters or maybe a new hyper parameter that controls the granularity of “prediction length”?
- mdp2021 2y ago> multi token prediction of sufficient length Is multi token prediction the same as predicting the embedding of a complex token (the articulation of those input tokens in a sentence)?
- blackeyeblitzar 2y agoTo be honest I don’t know. Maybe the only way to know is to build and measure all these variations.