3 ms·
The primary source is the liked Twitter thread. I wonder how credible this source is. (I'm not familiar with the norm of ML community - They seem to be Twitter-
by flakiness 3y ago
The primary source is the liked Twitter thread. I wonder how credible this source is. (I'm not familiar with the norm of ML community - They seem to be Twitter-heavy than other part of tech.)
- ShamelessC 3y agoCan you clarify what you’re referring to?
- Lerc 3y agoI only gave it a quick skim but it seems to match what I have learned so far, but I'm also learning from things that people said online so there remains the possibility of common misconceptions. The ALiBi stuff just makes sense to me. I don't understand why the Positional Sinusoidal Encoding was used initially. I assume there were good reasons for it but I haven't seen an explanation, (pointers to one appreciated).
- jimsimmons 3y agoDL practitioner for a decade here: The OP doesn’t explain anything. It just vaguely talks about a few things that might break when scaling context. But that means nothing. Take for example sinusoidal embeddings they talk about. Of course it breaks for large contexts but in no one uses it. GPT uses learned positional embeddings so the entire section is irrelevant. Copy this for pretty much everything else. Being an expert in a field has never been this exhausting
- it_citizen 3y agoTry virologist 3 years ago.
- oneseven 3y agoIt seems like learned positional encodings would still prevent you from doing fine tuning on a larger context size, though, so maybe using alibi is still relevant (although I have not read that paper).
- jimsimmons 3y agoYou can collapse all positions beyond a length to a specific bucket like T5