3 ms·
Their experiment seems to handle less than 100,000 tokens of text. I wonder if this method could scale to more than 1,000,000 or 10,000,000 tokens. It seems li
by ilaksh 3y ago
Their experiment seems to handle less than 100,000 tokens of text. I wonder if this method could scale to more than 1,000,000 or 10,000,000 tokens.
It seems like it's limited to context length? So with 512 token chunks at 4096 context length that is 2 million tokens. Which could be good for some things but is not going to help for a really large knowledgebase.
Edit: I see the source code is in the Files section on the Hugging Face page.
- mike_hearn 3y agoIt's open source. The PyTorch sources are linked from the blog post.
- ilaksh 3y agoThanks. I just assumed that it would be on their Github. But it's in the Files section on HuggingFace.
- bsenftner 3y agoIt's open source with a noncommercial use license.
- mike_hearn 3y agoHuh. I wonder if that meets the definition of open source. I didn't notice the special license.
- bsenftner 3y agoI additionally noticed an Apache license on their models at Huggingface, but the blog post states the work described by that blog post is open source for noncommercial use.
- selfhoster11 3y agoIf you go by Open Source Initiative's OSD, then no. Restriction on fields of endeavour is prohibited.