5 ms·
FYI, I'm running lm-eval now w/ the tests Bellard uses (lambada_standard, hellaswag, winogrande, piqa,coqa) on the biggest 7B an 40GB A100 atm (non-quantized ve
by lhl 4y ago
FYI, I'm running lm-eval now w/ the tests Bellard uses (lambada_standard, hellaswag, winogrande, piqa,coqa) on the biggest 7B an 40GB A100 atm (non-quantized version, requires 31.4GB) so will be directly comparable to what various LLaMAs look like: https://bellard.org/ts_server/ https://bellard.org/ts_server/
(UPDATE: run took 1:36 to complete run, but failed at the end with a TypeError, so will need to poke and rerun).
I'll place results in my spreadsheet (which also has my text-davinci-003 results): https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit#gid=0 https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...
- lunixbochs 4y agoAre you using https://github.com/EleutherAI/lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness?
- lhl 4y agoYeah, although looks like it currently has some issues with coqa: https://github.com/EleutherAI/lm-evaluation-harness/issues/238 https://github.com/EleutherAI/lm-evaluation-harness/issues/2... There's also the bigscience fork, but I ran into even more problems (although I didn't try too hard) https://github.com/bigscience-workshop/lm-evaluation-harness https://github.com/bigscience-workshop/lm-evaluation-harness And there's https://github.com/EleutherAI/lm-eval2/ https://github.com/EleutherAI/lm-eval2/ (not sure if it's just starting over w/ a new repo or what?) but it has limited tests available
- guywithabowtie 4y agoDo you also have results of GPT4 somewhere? or text-davinci-003-turbo
- lhl 4y agoI'm still on the waitlist for GPT-4 API access. Note, that text-davinci-003 cost about $90 to benchmark at $0.02/1K tokens, so if you're able to use a GPT-4 model (for completion and not just instruction) that'll probably be $270-$540 in credits to benchmark...
- jimsimmons 4y agoJust think about benchmarking 32K GPT4 haha
- hhh 4y agoI have GPT-4 8k access and am willing to run the evals if someone wants to pay. Email in my acc info (the character is h) Just a note, I get errors semi-frequently when running queries against GPT-4 often (timeouts mostly…) so any code would need to handle that well.
- MacsHeadroom 4y agoYou should benchmark GPT-3 Curie (7B) for comparison since it is the same size as llama-7B and StableLM-7B. That will give us some indication of how much better these models are than GPT-3 at the same size.
- sebzim4500 4y agoHow possible is it that every other model suffers from dataset contamination and this model is being unfairly penalized for having properly sanitized training data?
- MacsHeadroom 4y agoNote that this is StableLM ALPHA (only 0.52 epochs into training). The fully trained version will surely be much better. Also, you should benchmark GPT-3 Babbage for a fair comparison since that is the same size as 7B.
- ALittleLight 4y agoHow many epochs will they run?
- lhl 4y agoLooks like my edit window closed, but my results ended up being very low so there must be something wrong (I've reached out to StabilityAI just in case). It does however seem to roughly match another user's 3B testing: https://twitter.com/abacaj/status/1648881680835387392 https://twitter.com/abacaj/status/1648881680835387392 The current scores I have place it between gpt2_774M_q8 and pythia_deduped_410M (yikes!). Based on training and specs you'd expect it to outperform Pythia 6.9B at least... this is running on a HEAD checkout of https://github.com/EleutherAI/lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness (releases don't support hf-casual) for those looking to replicate/debug. Note, another LLM currently being trained, GeoV 9B, already far outperforms this model at just 80B tokens trained: https://github.com/geov-ai/geov/blob/master/results.080B.md https://github.com/geov-ai/geov/blob/master/results.080B.md