3 ms·

You could do this for sure but its quite a lot of effort and a normal software stack is quite well known due to the massive amount of github projects and other

You could do this for sure but its quite a lot of effort and a normal software stack is quite well known due to the massive amount of github projects and other opensource code. You can also get a lot out of a harness in this case or your Agents.md or Claude.md file by just enhancing the context. You might even question yourself if you are doing something wrong if a modern LLM really struggles with your code. We do the finetuning only on small semantic data were it helps a lot. I'm still wondering when we see smaller models (faster and cheaper) for more specific stacks like spring boot + java + angular + english only or so. Interstingly enough, i assumed LLMs are really good in any language but it seems that non english languages do reduce the ooutput quality of an LLM. At least last years GTC there was a talk about it. There are companies though which ahve this exact problem with programming languages you normally don't see. ABAP for example is a very well known SAP language.

runeks 2mo ago
> We do the finetuning only on small semantic data were it helps a lot. This sounds interesting. Would you care to expand a bit on how you do this? What is this semantic data? > There are companies though which ahve this exact problem with programming languages you normally don't see. This applies to my company. We have a substantial code base in our own dialect of APL. You think post-training would help substantially here?
- Zylokloto 2mo ago
  So we have free text paragraphs which describe machines. Our model detects if the machine in question is of the main category we are looking for, then our finetuned model extracts from the free form text semantic information about the machine. Like color, features, horsepower etc. This would normally take quite a long time to do manually but we already had the semantic version of these texts because the company was doing this for a while. We now use gemma or Qwen (we regularly re-finetune the newest models to just see if they get better and they actually do) and then use these finetuned models to save us a lot of time. ---- If I had a coding language which isn't available much online and coding LLMs are bad on it, I would definitly try to finetune this but its defintily a lot more work than just doing what I explained above. Depending on what your usage of this APL Dialect is, it might be easier to fine tune it to migrate from your APL to something a lot more common. If this is not an option at all: You need to start creating data for the finetuning. You need a few hundred up to a few thousand of them in different formats like Q & A pairs. Documentation, syntax, a lot of diverse small examples. The intersting thing about this data generation: You can either leverage, to a certain degree , what you already have, or you start collecting them through your team/work collegues or you do them by hand. You can put the text into RAG and experiment with context engineering until a LLM is 'good enough' in it to be able to help you generating examples for and with you. Like you give an LLM all the relevant context for it, then you let it generate pairs: { "instruction": "Write an expression to find the maximum along the rows of a 2D array.", "input": "Array matrix: A", "dialect_notes": "Custom dialect uses ⌆ (max-reduce) and ⌥ for axis specification instead of /[1].", "output": "⌆ ⌥2 A" } (I have no clue about APL this is just a random example I asked an LLM to generate). You might have luck and finding communities with the same issue you face.

runeks 2mo ago

> We do the finetuning only on small semantic data were it helps a lot. This sounds interesting. Would you care to expand a bit on how you do this? What is this semantic data? > There are companies though which ahve this exact problem with programming languages you normally don't see. This applies to my company. We have a substantial code base in our own dialect of APL. You think post-training would help substantially here?

Zylokloto 2mo ago
So we have free text paragraphs which describe machines. Our model detects if the machine in question is of the main category we are looking for, then our finetuned model extracts from the free form text semantic information about the machine. Like color, features, horsepower etc. This would normally take quite a long time to do manually but we already had the semantic version of these texts because the company was doing this for a while. We now use gemma or Qwen (we regularly re-finetune the newest models to just see if they get better and they actually do) and then use these finetuned models to save us a lot of time. ---- If I had a coding language which isn't available much online and coding LLMs are bad on it, I would definitly try to finetune this but its defintily a lot more work than just doing what I explained above. Depending on what your usage of this APL Dialect is, it might be easier to fine tune it to migrate from your APL to something a lot more common. If this is not an option at all: You need to start creating data for the finetuning. You need a few hundred up to a few thousand of them in different formats like Q & A pairs. Documentation, syntax, a lot of diverse small examples. The intersting thing about this data generation: You can either leverage, to a certain degree , what you already have, or you start collecting them through your team/work collegues or you do them by hand. You can put the text into RAG and experiment with context engineering until a LLM is 'good enough' in it to be able to help you generating examples for and with you. Like you give an LLM all the relevant context for it, then you let it generate pairs: { "instruction": "Write an expression to find the maximum along the rows of a 2D array.", "input": "Array matrix: A", "dialect_notes": "Custom dialect uses ⌆ (max-reduce) and ⌥ for axis specification instead of /[1].", "output": "⌆ ⌥2 A" } (I have no clue about APL this is just a random example I asked an LLM to generate). You might have luck and finding communities with the same issue you face.