6 ms·

Can you list a few I'm interested in checking them out

TheCaptain4815 4y ago
Slightly outdated article, but still relevant imo to show the different types. https://medium.com/@nils_reimers/openai-gpt-3-text-embeddings-really-a-new-state-of-the-art-in-dense-text-embeddings-6571fe3ec9d9 https://medium.com/@nils_reimers/openai-gpt-3-text-embedding... I've also used https://huggingface.co/flax-sentence-embeddings/all_datasets_v3_mpnet-base https://huggingface.co/flax-sentence-embeddings/all_datasets...
- jimhi 4y ago
  I am a fairly technical guy (check out my submissions) and I read your links and have no idea how to use these to make responses the way I can with OpenAI. It says I can input a Source Sentence and compare it to other sentences. For example, how do I get it to reply to a question as if I am George from Seinfeld?
  - gamegoblin 4y ago
    Embeddings are not for that. Embeddings take text and encode it into a high dimensional vector space. Similar texts will be closer together in the vector space. The idea I was proposing was to use embeddings as a way to store and retrieve relevant "memories" so the AI could maintain coherence across time. I.e. whenever the user sends a message, we pull up the N most relevant memories (where relevance == closeness in the vector space) and include those in the prompt, so GPT3 can use the information when it forms its response.
    - visarga 4y ago
      I just implemented exactly this. In the corpus I put a few hundred papers I am interested in. Now I can ask a question, the search engine will find a few snippets and put them in the GPT-3 prompt.
      - samching 4y ago
        Any good guides for embedding generation?
        fareesh 4y ago
        Yes this would be useful - does anyone have a crash course or something similar?
  - jimhi 4y ago
    As I can't reply to the child - that makes sense it is for embeddings. So would GPT3 still need to be used combined with this then?
    - gamegoblin 4y ago
      HN prevents users from responding to responses to their own comments without some delay to prevent flame wars -- just wait a few minutes next time, or click on the link to the comment directly and you'll be able to reply. Yes you would still need GPT3 in this system. Right now, the incredibly simple system just wires gives GPT3 a window of the last 100 messages and has it output the next message to send. The following is an excerpt SMS conversation between two friends: Transcript: <splice in the last 100 messages here> Then you can have GPT3 output what it believes the most likely next message is, and you send it. But this system means it loses context if a message is outside the window. So you can augment this system by creating an embedding of the last few messages of the conversation, and creating a prompt like: The following is an excerpt SMS conversation between two friends, and relevant past memories that are related to the current conversation: Relevant past memories: <splice in the N past messages with the most similar embedding to the most recent messages> Transcript: <splice in the last 100 messages> So this gets you a kind of short term memory (the last 100 messages) and a long term memory (the embeddings).
      - nl 4y ago
        > Relevant past memories: > <splice in the N past messages with the most similar embedding to the most recent messages> This is a really good idea. Presumably you'd keep the memory per-person.
      - abraxas 4y ago
        Oh, that makes a ton of sense. Thank you!
- gamegoblin 4y ago
  Thanks for the links, will check this out. It does seem compelling.
macrolime 4y ago
Here's a bunch and their scores in the Massive Text Embedding Benchmark https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard
cpdomina 4y ago
You can call any of the huggingface models [1] using their API [2]. A few examples: - https://huggingface.co/EleutherAI/gpt-j-6B https://huggingface.co/EleutherAI/gpt-j-6B - https://huggingface.co/t5-base https://huggingface.co/t5-base - https://huggingface.co/facebook/opt-66b https://huggingface.co/facebook/opt-66b - https://huggingface.co/bigscience/bloomz-3b https://huggingface.co/bigscience/bloomz-3b There are also other companies offering large models as a service: - https://www.forefront.ai https://www.forefront.ai - https://nlpcloud.com https://nlpcloud.com - https://www.goose.ai https://www.goose.ai - https://cohere.ai/generate https://cohere.ai/generate [1] https://huggingface.co/models https://huggingface.co/models [2] https://huggingface.co/inference-api https://huggingface.co/inference-api
- visarga 4y ago
  I wouldn't have suggested those models. Just use a semantically fine-tuned BERT. > GPT-3 Embeddings by @OpenAI was announced this week. I was excited and tested them on 20 datasets. Sadly they are worse than open models that are 1000 x smaller https://twitter.com/Nils_Reimers/status/1487014195568775173 https://twitter.com/Nils_Reimers/status/1487014195568775173 Get models here: https://sbert.net/docs/pretrained_models.html https://sbert.net/docs/pretrained_models.html

Slightly outdated article, but still relevant imo to show the different types. https://medium.com/@nils_reimers/openai-gpt-3-text-embeddings-really-a-new-state-of-the-art-in-dense-text-embeddings-6571fe3ec9d9 https://medium.com/@nils_reimers/openai-gpt-3-text-embedding... I've also used https://huggingface.co/flax-sentence-embeddings/all_datasets_v3_mpnet-base https://huggingface.co/flax-sentence-embeddings/all_datasets...

jimhi 4y ago
I am a fairly technical guy (check out my submissions) and I read your links and have no idea how to use these to make responses the way I can with OpenAI. It says I can input a Source Sentence and compare it to other sentences. For example, how do I get it to reply to a question as if I am George from Seinfeld?
- gamegoblin 4y ago
  Embeddings are not for that. Embeddings take text and encode it into a high dimensional vector space. Similar texts will be closer together in the vector space. The idea I was proposing was to use embeddings as a way to store and retrieve relevant "memories" so the AI could maintain coherence across time. I.e. whenever the user sends a message, we pull up the N most relevant memories (where relevance == closeness in the vector space) and include those in the prompt, so GPT3 can use the information when it forms its response.
  - visarga 4y ago
    I just implemented exactly this. In the corpus I put a few hundred papers I am interested in. Now I can ask a question, the search engine will find a few snippets and put them in the GPT-3 prompt.
    - samching 4y ago
      Any good guides for embedding generation?
      - fareesh 4y ago
        Yes this would be useful - does anyone have a crash course or something similar?
- jimhi 4y ago
  As I can't reply to the child - that makes sense it is for embeddings. So would GPT3 still need to be used combined with this then?
  - gamegoblin 4y ago
    HN prevents users from responding to responses to their own comments without some delay to prevent flame wars -- just wait a few minutes next time, or click on the link to the comment directly and you'll be able to reply. Yes you would still need GPT3 in this system. Right now, the incredibly simple system just wires gives GPT3 a window of the last 100 messages and has it output the next message to send. The following is an excerpt SMS conversation between two friends: Transcript: <splice in the last 100 messages here> Then you can have GPT3 output what it believes the most likely next message is, and you send it. But this system means it loses context if a message is outside the window. So you can augment this system by creating an embedding of the last few messages of the conversation, and creating a prompt like: The following is an excerpt SMS conversation between two friends, and relevant past memories that are related to the current conversation: Relevant past memories: <splice in the N past messages with the most similar embedding to the most recent messages> Transcript: <splice in the last 100 messages> So this gets you a kind of short term memory (the last 100 messages) and a long term memory (the embeddings).
    - nl 4y ago
      > Relevant past memories: > <splice in the N past messages with the most similar embedding to the most recent messages> This is a really good idea. Presumably you'd keep the memory per-person.
    - abraxas 4y ago
      Oh, that makes a ton of sense. Thank you!
gamegoblin 4y ago
Thanks for the links, will check this out. It does seem compelling.

macrolime 4y ago

Here's a bunch and their scores in the Massive Text Embedding Benchmark https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard

cpdomina 4y ago

You can call any of the huggingface models [1] using their API [2]. A few examples: - https://huggingface.co/EleutherAI/gpt-j-6B https://huggingface.co/EleutherAI/gpt-j-6B - https://huggingface.co/t5-base https://huggingface.co/t5-base - https://huggingface.co/facebook/opt-66b https://huggingface.co/facebook/opt-66b - https://huggingface.co/bigscience/bloomz-3b https://huggingface.co/bigscience/bloomz-3b There are also other companies offering large models as a service: - https://www.forefront.ai https://www.forefront.ai - https://nlpcloud.com https://nlpcloud.com - https://www.goose.ai https://www.goose.ai - https://cohere.ai/generate https://cohere.ai/generate [1] https://huggingface.co/models https://huggingface.co/models [2] https://huggingface.co/inference-api https://huggingface.co/inference-api

visarga 4y ago
I wouldn't have suggested those models. Just use a semantically fine-tuned BERT. > GPT-3 Embeddings by @OpenAI was announced this week. I was excited and tested them on 20 datasets. Sadly they are worse than open models that are 1000 x smaller https://twitter.com/Nils_Reimers/status/1487014195568775173 https://twitter.com/Nils_Reimers/status/1487014195568775173 Get models here: https://sbert.net/docs/pretrained_models.html https://sbert.net/docs/pretrained_models.html