4 ms·
I bet language models could definitely help here, yeah. Perhaps something like (1) get content offeed items as they come in, (2) embed the content, (3) use thos
by nbbaier 3y ago
I bet language models could definitely help here, yeah. Perhaps something like (1) get content offeed items as they come in, (2) embed the content, (3) use those embeddings to group items. Probably not that difficult to be honest
- gwern 3y agoYep. That would be a classic sort of k-means problem. Just throw them all into a standard embedding, like the OA API embeddings, run k-means from sci-kit, then convert them into a list-of-lists: one RSS item (containing a list of title-URLs) per cluster.
- woodglyst 3y agoThe problem with this approach is determining the what k is for the k-means. But again, we could use the “elbow” technique to determine what’s the optimal k and then start grouping them together. I wonder if there are any automatic sophisticated clustering algorithms?
- memhole 3y agoHierarchical and DBSCAN don’t require upfront knowledge about the number of clusters.