4 ms·

Embedding Archives: Millions of Wikipedia Article Embeddings in Many Languages

75 points by zerojames 3y ago

scrum-treats 3y ago
[dead]
jerrygenser 3y ago
Seems like it would lock in a user to cohere proprietary model in order to embed new search terms to the same space? Would love to see similar but embedded with a more open representation model or even sent2vec
- minimaxir 3y ago
  Unfortunately yes, as the corresponding model is not also open-source.
- jayalammar 3y ago
  There's a lot you can do with the vectors themselves without needing to embed any more text (e.g., clustering, exploration, visualization after dimensionality reduction...etc). Here's a previous embeddings exploration of top HN posts: https://txt.cohere.com/combing-for-insight-in-10-000-hacker-news-posts-with-text-clustering/ https://txt.cohere.com/combing-for-insight-in-10-000-hacker-... A lot of that code can be used here as well. If you want to query for a search term, you can use a trial API key which is free to use for prototyping. The embedding model itself is not open source, though. [co-author of the post here]
  - minimaxir 3y ago
    If that's the intent, IMO the release dataset should have more metadata (e.g. paragraph heading, article taxonomy)
    - jayalammar 3y ago
      How would you add that data? As new columns you mean? Or add the paragraph headings to the text of the paragraphs before embedding them?
      - minimaxir 3y ago
        New columns. For the headings, I mean the Wikipedia section headings (which isn't always a paragraphs, my mistake). In both cases the data can be used like to classify/visualize Show HNs in your linked post.
txtai 3y ago
If you want something similar to this that is fully downloadable and open source, check this out - https://huggingface.co/NeuML/txtai-wikipedia https://huggingface.co/NeuML/txtai-wikipedia It's an embeddings database of Wikipedia abstracts with page view data integrated to enable filtering pages based on popularity in addition to similarity.
deleted 3y ago
[deleted]
gaogao 3y ago
In a few of the examples given, WikiData would also be searchable enough to find the relevant articles to pull in. I've actually been really curious if you could use recent advancements in ML to augment the filling out of WikiData, since it's a bit sparse as is.
kristianp 3y ago
How recent is the Wikipedia snapshot used? Edit, going to [1] the datasets are labelled 2022-12. [1] https://huggingface.co/Cohere https://huggingface.co/Cohere

75 points by zerojames 3y ago

[dead]

Seems like it would lock in a user to cohere proprietary model in order to embed new search terms to the same space? Would love to see similar but embedded with a more open representation model or even sent2vec

minimaxir 3y ago
Unfortunately yes, as the corresponding model is not also open-source.
jayalammar 3y ago
There's a lot you can do with the vectors themselves without needing to embed any more text (e.g., clustering, exploration, visualization after dimensionality reduction...etc). Here's a previous embeddings exploration of top HN posts: https://txt.cohere.com/combing-for-insight-in-10-000-hacker-news-posts-with-text-clustering/ https://txt.cohere.com/combing-for-insight-in-10-000-hacker-... A lot of that code can be used here as well. If you want to query for a search term, you can use a trial API key which is free to use for prototyping. The embedding model itself is not open source, though. [co-author of the post here]
- minimaxir 3y ago
  If that's the intent, IMO the release dataset should have more metadata (e.g. paragraph heading, article taxonomy)
  - jayalammar 3y ago
    How would you add that data? As new columns you mean? Or add the paragraph headings to the text of the paragraphs before embedding them?
    - minimaxir 3y ago
      New columns. For the headings, I mean the Wikipedia section headings (which isn't always a paragraphs, my mistake). In both cases the data can be used like to classify/visualize Show HNs in your linked post.

txtai 3y ago

If you want something similar to this that is fully downloadable and open source, check this out - https://huggingface.co/NeuML/txtai-wikipedia https://huggingface.co/NeuML/txtai-wikipedia It's an embeddings database of Wikipedia abstracts with page view data integrated to enable filtering pages based on popularity in addition to similarity.

deleted 3y ago

[deleted]

gaogao 3y ago

In a few of the examples given, WikiData would also be searchable enough to find the relevant articles to pull in. I've actually been really curious if you could use recent advancements in ML to augment the filling out of WikiData, since it's a bit sparse as is.

kristianp 3y ago

How recent is the Wikipedia snapshot used? Edit, going to [1] the datasets are labelled 2022-12. [1] https://huggingface.co/Cohere https://huggingface.co/Cohere