3 ms·

Depends what you mean by taking all your articles. If it's scraping no, but if you provide text content and urls as key/values pairs yes. SageMaker and VertexA

by m3at 3y ago

Depends what you mean by taking all your articles. If it's scraping no, but if you provide text content and urls as key/values pairs yes. SageMaker and VertexAI are the AI services of AWS and GCP respectively, and they both offer embedding generation and vector databases (the two key pieces necessary for embedding search). There are a bunch of smaller companies offering vector search as a service too, example pinecone to name just one: https://www.pinecone.io/ https://www.pinecone.io/

politelemon 3y ago
Thanks, will have a look. Yeah in this case I'd want to provide the text content. I also just found pgvector which seems like it could help? It has a similarity search.
- m3at 3y ago
  I would only recommend pgvector if you're already primarily relying on postgres and the scale is limited (<1M documents). It won't handle the part that generates the embeddings though. You could use cloud vendors if you're in one of their ecosystem, do it yourself [1] (but model serving can be tricky without prior experience in ML), or use some other service to generate embeddings [2]. Alternatively vespa cloud [3] offer both but… not the easiest to work with, it's tailored for businesses where search is a primary component. Feel free to shoot me an email (in profile) with your context if you have more questions, in case I can help [1] This model is a solid baseline if you're working with English text: https://huggingface.co/sentence-transformers/all-mpnet-base-v2 https://huggingface.co/sentence-transformers/all-mpnet-base-... [2] OpenAI's embeddings is probably the easiest to get started, and the API is straightforward. It's not the best performing embeddings for retrieval but good enough in some cases: https://platform.openai.com/docs/guides/embeddings/use-cases https://platform.openai.com/docs/guides/embeddings/use-cases [3] https://cloud.vespa.ai/ https://cloud.vespa.ai/

by m3at 3y ago

politelemon 3y ago

Thanks, will have a look. Yeah in this case I'd want to provide the text content. I also just found pgvector which seems like it could help? It has a similarity search.

m3at 3y ago
I would only recommend pgvector if you're already primarily relying on postgres and the scale is limited (<1M documents). It won't handle the part that generates the embeddings though. You could use cloud vendors if you're in one of their ecosystem, do it yourself [1] (but model serving can be tricky without prior experience in ML), or use some other service to generate embeddings [2]. Alternatively vespa cloud [3] offer both but… not the easiest to work with, it's tailored for businesses where search is a primary component. Feel free to shoot me an email (in profile) with your context if you have more questions, in case I can help [1] This model is a solid baseline if you're working with English text: https://huggingface.co/sentence-transformers/all-mpnet-base-v2 https://huggingface.co/sentence-transformers/all-mpnet-base-... [2] OpenAI's embeddings is probably the easiest to get started, and the API is straightforward. It's not the best performing embeddings for retrieval but good enough in some cases: https://platform.openai.com/docs/guides/embeddings/use-cases https://platform.openai.com/docs/guides/embeddings/use-cases [3] https://cloud.vespa.ai/ https://cloud.vespa.ai/