3 ms·

https://huggingface.co/spaces/bigcode/bigcode-playground https://huggingface.co/spaces/bigcode/bigcode-playground Can someone explain to me the "Model Formats"

https://huggingface.co/spaces/bigcode/bigcode-playground https://huggingface.co/spaces/bigcode/bigcode-playground Can someone explain to me the "Model Formats" section at the end of that page. It makes sense to me as a description of how to format the training data, but it also says "Use these templates to explore the model’s capacities" How would you make use of this prefix format in a prompt? <reponame>REPONAME<filename>FILENAME<gh_stars>STARS code<|endoftext|>

enum 3y ago
You may (but do not have to) use <reponame>, <filename>, etc. as special tokens to prompt the model with extra metadata. These help you use the model go beyond just code completion. Page 30 of the TR has a few examples: https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0kl4BYs/view https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0k... Note that they were produced with StarCoderBase. E.g., here is one: Input: <commit_before>def fibonacci(n):<commit_msg>add type hints to function<commit_after>def Output: fibonacci(n: int) -> int: (If you try this in the playground, dial down the repetition penalty to say 1.0 in Advanced Settings.)
espadrine 3y ago
StarCoder is not just a code predictor, it is an assistant. It doesn’t just predict code; it can also help you review code and solve issues using metadata, thanks to being trained with special tokens. The <reponame> token specifies the name of the repository, and the same goes for the filename. A high gh_stars count can make it mimick popular open-source libraries. Admittedly, there is a bit of information that can be retrieved from those tokens, but not as much as the others. The more interesting ones are the commit tokens, which you can use to ask it as an assistant to implement a commit whose high-level description you give in plain English, and the issues token, to make it solve an issue. The Fill-in-the-middle tokens are most useful for autocompletion: it will allow it to gain information from the code that is present after your cursor, instead of just the code before your cursor. Of course, in practice, those tokens are meant for code editor plugin writers. Normal users won’t know about them.

enum 3y ago

You may (but do not have to) use <reponame>, <filename>, etc. as special tokens to prompt the model with extra metadata. These help you use the model go beyond just code completion. Page 30 of the TR has a few examples: https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0kl4BYs/view https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0k... Note that they were produced with StarCoderBase. E.g., here is one: Input: <commit_before>def fibonacci(n):<commit_msg>add type hints to function<commit_after>def Output: fibonacci(n: int) -> int: (If you try this in the playground, dial down the repetition penalty to say 1.0 in Advanced Settings.)

espadrine 3y ago

StarCoder is not just a code predictor, it is an assistant. It doesn’t just predict code; it can also help you review code and solve issues using metadata, thanks to being trained with special tokens. The <reponame> token specifies the name of the repository, and the same goes for the filename. A high gh_stars count can make it mimick popular open-source libraries. Admittedly, there is a bit of information that can be retrieved from those tokens, but not as much as the others. The more interesting ones are the commit tokens, which you can use to ask it as an assistant to implement a commit whose high-level description you give in plain English, and the issues token, to make it solve an issue. The Fill-in-the-middle tokens are most useful for autocompletion: it will allow it to gain information from the code that is present after your cursor, instead of just the code before your cursor. Of course, in practice, those tokens are meant for code editor plugin writers. Normal users won’t know about them.