4 ms·

From what I've seen, the 40B model leaves a lot to be desired outside of academic benchmarks - it got almost zero answers correctly in the PyLLMs benchmark.

• jmugan 3y ago
  Was it trained on code? On my quick glance at the training data it seemed to be a clean-up version of common crawl.
  • aperrien 3y ago
    I got it to make a tiny FORTH model in python. It failed pretty hard when trying to make a Lisp, though.