I've performed a couple different experiments around this with niche programming languages. I gave one of OAI's 128k models the entirety of official documentat

I've performed a couple different experiments around this with niche programming languages. I gave one of OAI's 128k models the entirety of official documentation on MOO[1], which is almost entirely absent in training because materials online for it are scarce. It was able to answer questions to a reasonable degree, and provide passable example code. The downside is prompting this cost $0.35 per prompt, but maybe a finetune could handle it. I then attempted to have it walk me through writing a Greyscript[2] program, which is an interesting challenge since it carries many similarities to Lua, but is distinctly not. While I didn't have nearly the same amount of context as [1], relying on explaining it myself, it frequently hallucinated Lua functionality and had to be corrected every step of the way on this. Would be interesting if someone found niche languages in different styles with similar levels of documentation to test with. [1] https://en.m.wikipedia.org/wiki/MOO https://en.m.wikipedia.org/wiki/MOO [2] https://greyhackgame.com/ https://greyhackgame.com/