3 ms·
It's not using GPT-2 - the README is incorrect. It's using "from langchain.embeddings import OpenAIEmbeddings" - which is the OpenAI embeddings API, text-embed
by simonw 3y ago
It's not using GPT-2 - the README is incorrect.
It's using "from langchain.embeddings import OpenAIEmbeddings" - which is the OpenAI embeddings API, text-embedding-ada-002
The only aspect of GPT-2 this is using is GPT2TokenizerFast.from_pretrained("gpt2") - which it uses as the length function to count tokens for the RecursiveCharacterTextSplitter() langchain utility.
Which doesn't really make sense - why use the GPT-2 tokenizer for that? May as well just count characters or even count words based on .split(), it's not particularly important how the counting works here.