I think the biggest issue is that those language models are still very limited by transformer window (2k tokens, each word usually consists of 3 tokens), and th

I think the biggest issue is that those language models are still very limited by transformer window (2k tokens, each word usually consists of 3 tokens), and there is no visible improvement over this. Your problem, code base don't fit into 700 words, you have no luck.