4 ms·

Hey there! Yes that's exactly what I thought! I was writing up a blog post at https://colab.research.google.com/drive/1AOuhMVILE06mD-Go7-RuN47Dg35t1wuK?usp=shar

Hey there! Yes that's exactly what I thought! I was writing up a blog post at https://colab.research.google.com/drive/1AOuhMVILE06mD-Go7-RuN47Dg35t1wuK?usp=sharing https://colab.research.google.com/drive/1AOuhMVILE06mD-Go7-R... which step by step shows every change I did plus gave timings / mem savings. I'll post it once its completed if you're interested!

kristopolous 3y ago
Thanks. There's been a few substantial and laudable efforts which are much appreciated but what I'm suggesting is an actual continuous infrastructure, like how those benchmarking sites have software for people to run on their machines that phone home so that people who make new benchmarks or new variations can submit them and refine the results. For instance, are any of your prompting tests in say, Korean? What about winograd schema challenges in languages other than English? Japanese for instance, comes with its own unique set of context ambiguities that do not appear in English. I'm sure dozens of languages are similar. It'd be nice to have user contributable tests to cover the breadth of use cases here. A great optimization that moves a score let's say from 95% -> 5% on "winograd-persian" may be fine or may be a show stopper, depends on what you care about. That's why it's gotta be normalized, future-proof, and crowdsourced.
- danielhanchen 3y ago
  Ohhh interesting! So like ML Perf or Faster CPython https://github.com/faster-cpython/benchmarking-public https://github.com/faster-cpython/benchmarking-public?
  - kristopolous 3y ago
    Sure. This thing I propose might already exist in a nascent form. I haven't checked in a few months and the way things are moving now...
    - danielhanchen 3y ago
      Interesting!! I'll talk with my brother about this - sounds very cool!
      - ptd 3y ago
        Hey Daniel, I would love to help out on this. I'm learning about LLMs and this benchmarking project sounds like a fun way to further my knowledge and skills. I sent you a message on LinkedIn. Cheers!
        danielhanchen 3y ago
        Cool lets chat on Linkedin! :)

kristopolous 3y ago

Thanks. There's been a few substantial and laudable efforts which are much appreciated but what I'm suggesting is an actual continuous infrastructure, like how those benchmarking sites have software for people to run on their machines that phone home so that people who make new benchmarks or new variations can submit them and refine the results. For instance, are any of your prompting tests in say, Korean? What about winograd schema challenges in languages other than English? Japanese for instance, comes with its own unique set of context ambiguities that do not appear in English. I'm sure dozens of languages are similar. It'd be nice to have user contributable tests to cover the breadth of use cases here. A great optimization that moves a score let's say from 95% -> 5% on "winograd-persian" may be fine or may be a show stopper, depends on what you care about. That's why it's gotta be normalized, future-proof, and crowdsourced.

danielhanchen 3y ago
Ohhh interesting! So like ML Perf or Faster CPython https://github.com/faster-cpython/benchmarking-public https://github.com/faster-cpython/benchmarking-public?
- kristopolous 3y ago
  Sure. This thing I propose might already exist in a nascent form. I haven't checked in a few months and the way things are moving now...
  - danielhanchen 3y ago
    Interesting!! I'll talk with my brother about this - sounds very cool!
    - ptd 3y ago
      Hey Daniel, I would love to help out on this. I'm learning about LLMs and this benchmarking project sounds like a fun way to further my knowledge and skills. I sent you a message on LinkedIn. Cheers!
      - danielhanchen 3y ago
        Cool lets chat on Linkedin! :)