3 ms·

Word vectorization + cosine distance, perhaps. See also this post by marginalia[1] (HN discussion[2]) which discusses the same thing. [1] https://memex.margin

Word vectorization + cosine distance, perhaps. See also this post by marginalia[1] (HN discussion[2]) which discusses the same thing. [1] https://memex.marginalia.nu/log/69-creepy-website-similarity.gmi https://memex.marginalia.nu/log/69-creepy-website-similarity... [2] https://news.ycombinator.com/item?id=34143101 https://news.ycombinator.com/item?id=34143101

stingraycharles 4y ago
I’ve done this by using word vectorisation, and using nearest neighbour. At least assuming the content is key, not the design and/or other visuals.
marginalia_nu 4y ago
Yeah, I use the incident link graph similarity though. It's been almost ridiculously effective. Not had the same success with word models, although it is probably largely a matter of finding an appropriate embedding. The data is pretty stale, but here is a demo: https://explore2.marginalia.nu/ https://explore2.marginalia.nu/ OP (or anyone else): Shoot me an email and I'll give you a copy of Marginalia's link graph to play with.
- supriyo-biswas 4y ago
  Playing around with the explore link, I see most of the popular news websites are excluded (such as wsj.com and nytimes.com)?
  - marginalia_nu 4y ago
    Yeah, I excluded them from the computation as a performance optimization. The juice didn't seem worth the squeeze given my work specializes in smaller websites.
m-i-l 4y ago
There are a couple of challenges with the word vectorization + cosine distance approach: 1. The current embedding models are for relatively small amounts of content rather than whole web pages. For example, most of the popular Huggingface sentence-transformers are for 256 tokens max, which is only sufficient for a small part of most web pages. There are some models for larger sizes, e.g. Longformer and BigBird, but they're still only 4096 tokens, and there aren't any ready-made implementations for cosine or dot product similarity at the moment so you'd need to roll your own. Some of the new LLM OpenAI embedding models do go to 8191 tokens and can be used for cosine similarity, and costs may be acceptable for the content embedding, but for the query embedding you'd have to be super-careful about costs because if you put any kind of search on the public internet you could find that you'll soon be overwhelmed by SEO spam bots (most of which even Cloudflare can't block) and that could become prohibitively expensive to service via a paid-for API solution like OpenAI. A popular approach to the token length issue is to chunk your pages into multiple blocks. 2. Even if you get the embedding and chunking approach working, that is still just for page comparison, and single web pages are not necessarily representative of whole web sites. For example, if comparing home pages, many blogs just have a link to the posts page on the home page without any of the actual content from the posts. Or some websites cover an eclectic range of topics, and if your goal is to find other sites that cover a similar eclectic range of topics then you you need an embedding for the whole site rather than individual pages. Not sure about the best way to address this one though. Apparently averaging embeddings may work surprisingly well but I haven't tried that out yet. Other possible options might include summarising (chunked) pages and summarising the summaries for the site, or using topic modelling (e.g. BERTopic) alongside some kind of website taxonomy, or something like that. Keen to read of other possible approaches here.
- deleted 4y ago
  [deleted]

stingraycharles 4y ago

I’ve done this by using word vectorisation, and using nearest neighbour. At least assuming the content is key, not the design and/or other visuals.

marginalia_nu 4y ago

Yeah, I use the incident link graph similarity though. It's been almost ridiculously effective. Not had the same success with word models, although it is probably largely a matter of finding an appropriate embedding. The data is pretty stale, but here is a demo: https://explore2.marginalia.nu/ https://explore2.marginalia.nu/ OP (or anyone else): Shoot me an email and I'll give you a copy of Marginalia's link graph to play with.

supriyo-biswas 4y ago
Playing around with the explore link, I see most of the popular news websites are excluded (such as wsj.com and nytimes.com)?
- marginalia_nu 4y ago
  Yeah, I excluded them from the computation as a performance optimization. The juice didn't seem worth the squeeze given my work specializes in smaller websites.

m-i-l 4y ago

There are a couple of challenges with the word vectorization + cosine distance approach: 1. The current embedding models are for relatively small amounts of content rather than whole web pages. For example, most of the popular Huggingface sentence-transformers are for 256 tokens max, which is only sufficient for a small part of most web pages. There are some models for larger sizes, e.g. Longformer and BigBird, but they're still only 4096 tokens, and there aren't any ready-made implementations for cosine or dot product similarity at the moment so you'd need to roll your own. Some of the new LLM OpenAI embedding models do go to 8191 tokens and can be used for cosine similarity, and costs may be acceptable for the content embedding, but for the query embedding you'd have to be super-careful about costs because if you put any kind of search on the public internet you could find that you'll soon be overwhelmed by SEO spam bots (most of which even Cloudflare can't block) and that could become prohibitively expensive to service via a paid-for API solution like OpenAI. A popular approach to the token length issue is to chunk your pages into multiple blocks. 2. Even if you get the embedding and chunking approach working, that is still just for page comparison, and single web pages are not necessarily representative of whole web sites. For example, if comparing home pages, many blogs just have a link to the posts page on the home page without any of the actual content from the posts. Or some websites cover an eclectic range of topics, and if your goal is to find other sites that cover a similar eclectic range of topics then you you need an embedding for the whole site rather than individual pages. Not sure about the best way to address this one though. Apparently averaging embeddings may work surprisingly well but I haven't tried that out yet. Other possible options might include summarising (chunked) pages and summarising the summaries for the site, or using topic modelling (e.g. BERTopic) alongside some kind of website taxonomy, or something like that. Keen to read of other possible approaches here.

deleted 4y ago
[deleted]