3 ms·Isn’t the purpose of self attention exactly to recognize the relevance of some tokens over others?by dcrazy 8mo agoIsn’t the purpose of self attention exactly to recognize the relevance of some tokens over others?melecas 8mo ago[dead]kimixa 8mo agoThat may help with tokens being "ignored" while still being in the context window, but not context window size costs and limitations in the first place.