3 ms·
i was not able to find the 80 programing languages it supports using my phone. you would think this would be front and center somewhere.
by SomeHacker44 3y ago
i was not able to find the 80 programing languages it supports using my phone. you would think this would be front and center somewhere.
- enum 3y agoThere is a lot of subtlety here. The model is trained on 80+ languages, but the volume and quality of data varies significantly. We have results showing benchmark performance on 19 languages, which is a broader evaluation than most Code LLMs. But, I would hesitate to say that BigCode supports 80 PLs. Any LLM that claims to support 80 PLs is not presenting evidence that it does.
- EvgeniyZh 3y agoYou are correct that there is no evaluation of level of support, and it is hard to get evaluation set on 86 languages. On the other hand, we can try to extrapolate from what we have and try to guess in which languages we'd have reasonable performance. Note that out of 19 languages it was evaluated on only 17 were "officially" in the training set (language detection is not perfect and there may be some data for languages not included in training set) and it work reasonably well on the remaining two (Swift and D).