9 ms·

Show HN: I made a free course, 'NLP for Semantic Search'

239 points by jamesbriggs 5y ago

jamesbriggs 5y ago
Hi all, I'm James, past data scientist, ML engineer, and now dev advocate at Pinecone. I've been dedicating much of the past few months to building a course on what I've found to be one of the most fascinating emerging technologies in AI and ML, Semantic Search: https://www.pinecone.io/learn/nlp https://www.pinecone.io/learn/nlp The course is completely free, it includes written and visual explanations, code implementations, and video walkthroughs. It focuses on the NLP side of semantic search and the first seven chapters cover: 1. Dense Vectors 2. Sentence Embeddings and Transformers 3. Training Sentence Transformers with Softmax Loss 4. Training Sentence Transformers with MNR Loss 5. Multilingual Sentence Transformers 6. Question Answering 7. Unsupervised Training for Sentence Transformers There is more coming soon and I'll be around to answer any questions, let me know what you think. Thanks all!
SleekEagle 5y ago
Looks really informative and polished!
- jamesbriggs 5y ago
  Thanks!
notjulianjaynes 5y ago
What skills would you say are nessecary prerequisites for understanding this material? I'm a writer, and in the past I have had success making poems and shit via messing around with various NLP tools like speech to text software and Google Books ngram viewer, but I've run into situations where I wish I could do something that isn't supported by the application. Could this course be a good place to start? If not do you have any recommendations? I don't really know how to code but am comfortable using a command line interface.
- jamesbriggs 5y ago
  Ideally I think it's useful to be familiar with Python and some NLP. Parts of the course can get reasonably in-depth, but we have tried to simplify every as much as possible and I think much of it can be followed without too much prerequisite knowledge. That being said, as a writer it sounds like you'd be most interested in generative models that generate new text, we do cover that a little in the Question Answering chapter, but otherwise, not so much. Although I don't know your ideas or plans so I can't say for sure what is good/bad for you to focus on. I do think you should read into transformer models like BERT and GPT, there is a very good free course from HuggingFace who are the 'de-facto framework' for NLP at the moment, that could be useful for you too: https://huggingface.co/course/chapter1/1 https://huggingface.co/course/chapter1/1 (chapter one will introduce everything too, definitely go through that) With HuggingFace, you can put together a simple T5 or GPT-2 text generation script with a few lines of code
- amrrs 5y ago
  If you want to just try something, you can look at the latest tranformer models like GPT and try fine-tuning them for poems. You can see a tutorial for content creation here https://youtu.be/d_xRYyy2LFM https://youtu.be/d_xRYyy2LFM
MR4D 5y ago
Nice site! What tools did you use to build it?
- gk1 5y ago
  It's built with Hugo and deployed through Netlify.
nbeleski 5y ago
I am browsing the web in search for NLP resources as I consider a pivot into this new area (I already have a few years in data science and image processing) and finding this post on HN is a bless! I will be going through the course for sure!
- jamesbriggs 5y ago
  That's awesome, glad to hear it!
__vim__ 5y ago
Just want to say thank you for your efforts. I will be checking this out for sure.
deleted 5y ago
[deleted]
maestrae 5y ago
Fortuitous timing! I've actually been tasked to build something that will need semantic search at work. I'll be diving in - thanks!
thewarrior 5y ago
Can I use this to build an HN search where I can search for threads that discuss various topics, books or frameworks ?
- gk1 5y ago
  Definitely, combined with the HN API. We built a user lookup app: https://blog.streamlit.io/finding-your-look-alikes-with-semantic-search/ https://blog.streamlit.io/finding-your-look-alikes-with-sema...
NetOpWibby 5y ago
Bless you for sharing this knowledge.
danuker 5y ago
I very much appreciate the course you shared. But I want to mention for everyone taking it: please also keep in mind "dumb" logical operators like OR, AND, NOT, and quotes that skip the NLP and use the exact text. What frustrates me with Google Search is that it won't show me what I know precisely how to find. This is why I use Google less and go to specialized sites directly (Stack Exchange, Wikipedia, PubMed...).
- smt88 5y ago
  This was my thought exactly. I absolutely loathe semantic search and have never found it more useful than literal search.
  - rmbyrro 5y ago
    It's extremely useful when you don't know exactly what keywords to search for. You probably benefited from it several times already, but haven't realized, since it's subtle and works behind the scenes.
    - smt88 5y ago
      It most often results in being unable to find something I know exact keywords for. I have used search long enough to remember having only literal search, and I never had an issue with it.
Pandabob 5y ago
Oh wow, gk1 now works[0] as the VP or marketing at pinecone. That's news for me. The product seems interesting as well. [0]: https://www.gkogan.co/blog/pinecone/ https://www.gkogan.co/blog/pinecone/
- gk1 5y ago
  Hi! :)
rolisz 5y ago
Does this semantic search work for longer documents? I know BERT&Co models are limited to around 500-1000 tokens.
- crucialfelix 5y ago
  You can split the document by paragraph or sentence. Search results will then match a sentence in a document. You display with a link to the document with a reference to the span.
  - jamesbriggs 5y ago
    I don't remember the paper, but the authors working on this same problem suggested to do this + also include an overlap between spans.
    - crucialfelix 5y ago
      Would you include the overlap in the text for training, or just for the displayed result? I assume the later.
rmbyrro 5y ago
Looks very good, thanks for sharing! How does your course relate to the vector search approach discussed here? https://cloud.google.com/blog/topics/developers-practitioners/find-anything-blazingly-fast-googles-vector-search-technology https://cloud.google.com/blog/topics/developers-practitioner...
- gk1 5y ago
  NLP + Vector Search = Semantic Search. This course teaches how to use NLP to turn text data into dense vectors while capturing their meaning. Then you take those vectors and put them in a vector database — like Pinecone.io or Google Matching Engine — so you could do on-demand vector search. And voilà, you have a semantic search application!
ewuhic 5y ago
Does anyone know of any similar courses related to NLP?
- jamesbriggs 5y ago
  The HuggingFace course is very good for NLP: https://huggingface.co/course/chapter1/1 https://huggingface.co/course/chapter1/1

239 points by jamesbriggs 5y ago

Hi all, I'm James, past data scientist, ML engineer, and now dev advocate at Pinecone. I've been dedicating much of the past few months to building a course on what I've found to be one of the most fascinating emerging technologies in AI and ML, Semantic Search: https://www.pinecone.io/learn/nlp https://www.pinecone.io/learn/nlp The course is completely free, it includes written and visual explanations, code implementations, and video walkthroughs. It focuses on the NLP side of semantic search and the first seven chapters cover: 1. Dense Vectors 2. Sentence Embeddings and Transformers 3. Training Sentence Transformers with Softmax Loss 4. Training Sentence Transformers with MNR Loss 5. Multilingual Sentence Transformers 6. Question Answering 7. Unsupervised Training for Sentence Transformers There is more coming soon and I'll be around to answer any questions, let me know what you think. Thanks all!

SleekEagle 5y ago

Looks really informative and polished!

jamesbriggs 5y ago
Thanks!

notjulianjaynes 5y ago

What skills would you say are nessecary prerequisites for understanding this material? I'm a writer, and in the past I have had success making poems and shit via messing around with various NLP tools like speech to text software and Google Books ngram viewer, but I've run into situations where I wish I could do something that isn't supported by the application. Could this course be a good place to start? If not do you have any recommendations? I don't really know how to code but am comfortable using a command line interface.

jamesbriggs 5y ago
Ideally I think it's useful to be familiar with Python and some NLP. Parts of the course can get reasonably in-depth, but we have tried to simplify every as much as possible and I think much of it can be followed without too much prerequisite knowledge. That being said, as a writer it sounds like you'd be most interested in generative models that generate new text, we do cover that a little in the Question Answering chapter, but otherwise, not so much. Although I don't know your ideas or plans so I can't say for sure what is good/bad for you to focus on. I do think you should read into transformer models like BERT and GPT, there is a very good free course from HuggingFace who are the 'de-facto framework' for NLP at the moment, that could be useful for you too: https://huggingface.co/course/chapter1/1 https://huggingface.co/course/chapter1/1 (chapter one will introduce everything too, definitely go through that) With HuggingFace, you can put together a simple T5 or GPT-2 text generation script with a few lines of code
amrrs 5y ago
If you want to just try something, you can look at the latest tranformer models like GPT and try fine-tuning them for poems. You can see a tutorial for content creation here https://youtu.be/d_xRYyy2LFM https://youtu.be/d_xRYyy2LFM

MR4D 5y ago

Nice site! What tools did you use to build it?

gk1 5y ago
It's built with Hugo and deployed through Netlify.

nbeleski 5y ago

I am browsing the web in search for NLP resources as I consider a pivot into this new area (I already have a few years in data science and image processing) and finding this post on HN is a bless! I will be going through the course for sure!

jamesbriggs 5y ago
That's awesome, glad to hear it!

__vim__ 5y ago

Just want to say thank you for your efforts. I will be checking this out for sure.

deleted 5y ago

[deleted]

maestrae 5y ago

Fortuitous timing! I've actually been tasked to build something that will need semantic search at work. I'll be diving in - thanks!

thewarrior 5y ago

Can I use this to build an HN search where I can search for threads that discuss various topics, books or frameworks ?

gk1 5y ago
Definitely, combined with the HN API. We built a user lookup app: https://blog.streamlit.io/finding-your-look-alikes-with-semantic-search/ https://blog.streamlit.io/finding-your-look-alikes-with-sema...

NetOpWibby 5y ago

Bless you for sharing this knowledge.

danuker 5y ago

I very much appreciate the course you shared. But I want to mention for everyone taking it: please also keep in mind "dumb" logical operators like OR, AND, NOT, and quotes that skip the NLP and use the exact text. What frustrates me with Google Search is that it won't show me what I know precisely how to find. This is why I use Google less and go to specialized sites directly (Stack Exchange, Wikipedia, PubMed...).

smt88 5y ago
This was my thought exactly. I absolutely loathe semantic search and have never found it more useful than literal search.
- rmbyrro 5y ago
  It's extremely useful when you don't know exactly what keywords to search for. You probably benefited from it several times already, but haven't realized, since it's subtle and works behind the scenes.
  - smt88 5y ago
    It most often results in being unable to find something I know exact keywords for. I have used search long enough to remember having only literal search, and I never had an issue with it.

Pandabob 5y ago

Oh wow, gk1 now works[0] as the VP or marketing at pinecone. That's news for me. The product seems interesting as well. [0]: https://www.gkogan.co/blog/pinecone/ https://www.gkogan.co/blog/pinecone/

gk1 5y ago
Hi! :)

rolisz 5y ago

Does this semantic search work for longer documents? I know BERT&Co models are limited to around 500-1000 tokens.

crucialfelix 5y ago
You can split the document by paragraph or sentence. Search results will then match a sentence in a document. You display with a link to the document with a reference to the span.
- jamesbriggs 5y ago
  I don't remember the paper, but the authors working on this same problem suggested to do this + also include an overlap between spans.
  - crucialfelix 5y ago
    Would you include the overlap in the text for training, or just for the displayed result? I assume the later.

rmbyrro 5y ago

Looks very good, thanks for sharing! How does your course relate to the vector search approach discussed here? https://cloud.google.com/blog/topics/developers-practitioners/find-anything-blazingly-fast-googles-vector-search-technology https://cloud.google.com/blog/topics/developers-practitioner...

gk1 5y ago
NLP + Vector Search = Semantic Search. This course teaches how to use NLP to turn text data into dense vectors while capturing their meaning. Then you take those vectors and put them in a vector database — like Pinecone.io or Google Matching Engine — so you could do on-demand vector search. And voilà, you have a semantic search application!

ewuhic 5y ago

Does anyone know of any similar courses related to NLP?

jamesbriggs 5y ago
The HuggingFace course is very good for NLP: https://huggingface.co/course/chapter1/1 https://huggingface.co/course/chapter1/1