3 ms·

I implemented this for PyTorch too at https://github.com/Shopify/torch-grammar https://github.com/Shopify/torch-grammar. I have a hacked version of text-generat

by burke 3y ago

I implemented this for PyTorch too at https://github.com/Shopify/torch-grammar https://github.com/Shopify/torch-grammar. I have a hacked version of text-generation-inference that uses it—happy to share that if it’s useful to anyone.

Scene_Cast2 3y ago
Yes, please share. I've been meaning to play around with dumping the token probability vectors inside one of the LLM UIs. Having a diff starting point would help a bunch.
- burke 3y ago
  Sure, here it is. It's extremely rough but it works as a starting point: https://gist.github.com/burke/6d035758f7492612e2ad86bb7de2d5fb https://gist.github.com/burke/6d035758f7492612e2ad86bb7de2d5...