3 ms·
It'd be great to have a chronicle of all these efforts. I lost track of the variations quite a long time ago. It'd be quite a lift unless we're just willing to
by kristopolous 3y ago
It'd be great to have a chronicle of all these efforts. I lost track of the variations quite a long time ago.
It'd be quite a lift unless we're just willing to just accept the self reported metrics as golden. And even then, they're always qualified by hardware and usage scope. Making it good enough to be useful is the hard part. CI/CD pipeline with a bunch of machine configurations and benchmarks along with a reasonable way to communicate them...
If anyone's up for it you'd legitimately become indispensable.
- danielhanchen 3y agoHey there! Yes that's exactly what I thought! I was writing up a blog post at https://colab.research.google.com/drive/1AOuhMVILE06mD-Go7-RuN47Dg35t1wuK?usp=sharing https://colab.research.google.com/drive/1AOuhMVILE06mD-Go7-R... which step by step shows every change I did plus gave timings / mem savings. I'll post it once its completed if you're interested!
- kristopolous 3y agoThanks. There's been a few substantial and laudable efforts which are much appreciated but what I'm suggesting is an actual continuous infrastructure, like how those benchmarking sites have software for people to run on their machines that phone home so that people who make new benchmarks or new variations can submit them and refine the results. For instance, are any of your prompting tests in say, Korean? What about winograd schema challenges in languages other than English? Japanese for instance, comes with its own unique set of context ambiguities that do not appear in English. I'm sure dozens of languages are similar. It'd be nice to have user contributable tests to cover the breadth of use cases here. A great optimization that moves a score let's say from 95% -> 5% on "winograd-persian" may be fine or may be a show stopper, depends on what you care about. That's why it's gotta be normalized, future-proof, and crowdsourced.
- danielhanchen 3y agoOhhh interesting! So like ML Perf or Faster CPython https://github.com/faster-cpython/benchmarking-public https://github.com/faster-cpython/benchmarking-public?
- kristopolous 3y agoSure. This thing I propose might already exist in a nascent form. I haven't checked in a few months and the way things are moving now...
- danielhanchen 3y agoInteresting!! I'll talk with my brother about this - sounds very cool!
- ptd 3y agoHey Daniel, I would love to help out on this. I'm learning about LLMs and this benchmarking project sounds like a fun way to further my knowledge and skills. I sent you a message on LinkedIn. Cheers!
- danielhanchen 3y agoCool lets chat on Linkedin! :)
- deleted 3y ago[deleted]