3 ms·
It would be nice if they shared some before and after result because I did some spot checks and I get the same output(using default Mistral) found in the pyth
by iamcreasy 3y ago
It would be nice if they shared some before and after result
because I did some spot checks and I get the same output(using default Mistral) found in the python_code_instructions_18k_alpaca dataset.
Setting that aside, is there a guide on how to create good training dataset for fine tuning models like Mistral?
- tarruda 3y agoI'd say the original Alpaca paper is a good source of inspiration on how to create datasets, they even shared the script used to generate data using OpenAI API. One thing that came to mind for creating datasets for other programming languages would be to start with this Python dataset and use GPT-4 to convert to equivalents in other languages. You can even automatically test each generates example and ask GPT-4 to correct any errors
- rahimnathwani 3y agoctrl+F for Baseline and you'll see a part where they show the two models' outputs with the same prompt. I didn't look closely to see whether this is deterministic with the right temperature and top_p or whatever.
- iamcreasy 3y agoUh, missed that. Thank you! What do you referring to by temperature?
- rahimnathwani 3y agoDue to the default settings of `top_k`, `top_p`, and `temperature`, which introduce a probabilistic element to token selection, two runs of the same prompt with the same model may yield different outputs. The first two parameters, `top_k` and `top_p`, define the subset of tokens eligible for selection, while `temperature` influences the probability distribution from which tokens are sampled.