4 ms·
Perhaps this could make useful synthetic data for improving large language models improving code, because you could have two pairs of code, better code, and the
by luke-stanley 3y ago
Perhaps this could make useful synthetic data for improving large language models improving code, because you could have two pairs of code, better code, and the terrible code. This is probably already done internally at the top AI labs.
- detinho 3y agoBut to do that "shittier" would have to get a good implementation and turn on a bad implementation. Formatting might not be enough. Howerver the main idea is very interesting.
- luke-stanley 3y agoI would presume the code before was better than the code after, so that's how you could get two sets. It might not be much of a boost, it could depend, and I'm sure there are other data augmentation techniques. Good model improvement is usually an iterative loop, trying things out to optimise an objective.
- truculent 3y agoIt would be useful in conjunction with other data, but wouldn’t the model just learn the logic of the obfuscation code (unless perhaps you heavily regularised it?)
- luke-stanley 3y agoYes in conjunction. For causal models like GPT, this sort of approach might help augment data, making it go further when training, or fine-tuning and the model could be trained on "improving" the bad code, actually the existing code. So the before and after would be reversed. For encoder-decoder models they can can be trained, or fine-tuned on a more explicit translation task. So just a kind of https://en.wikipedia.org/wiki/Data_augmentation https://en.wikipedia.org/wiki/Data_augmentation