5 ms·

BERTopic: The Future of Topic Modeling

104 points by gk1 4y ago

uniqueuid 4y ago
It is true that bertopic is a great tool. It's modern, it's modular, and it's pretty performant. That said, I want to caution against using topic modeling as a one-fits-all-solution. As the author stresses, this is one particular approach which uses a combination of embeddings (sentence, or other), umap and hdbscan. Both umap and hdbscan can be slow, so it might be worthwhile to check out the GPU enabled versions of both from the cuml package. In addition, topic models have a huge number of degrees of freedom, and the solution you will get depends on many (seemingly arbitrary) choices. In other words, these are not the topics, they are some topics. That said, it's awesome, really great work by Maarten Grootendorst and a great blog post by James Briggs. [edit] here is a link to the fast cuda version of bertopic by rapidsai: https://github.com/rapidsai/rapids-examples/tree/main/cuBERT_topic_modelling https://github.com/rapidsai/rapids-examples/tree/main/cuBERT...
- hack_ml 4y ago
  Its seamless to accelerate BERTOPIC on GPU's with cuML now with the latest release. (v0.10.0) Checkout the docs at: https://maartengr.github.io/BERTopic/faq.html#can-i-use-the-gpu-to-speed-up-the-model https://maartengr.github.io/BERTopic/faq.html#can-i-use-the-... All you need to do is below from bertopic import BERTopic from cuml.cluster import HDBSCAN from cuml.manifold import UMAP # Create instances of GPU-accelerated UMAP and HDBSCAN umap_model = UMAP(n_components=5, n_neighbors=15, min_dist=0.0) hdbscan_model = HDBSCAN(min_samples=10, gen_min_span_tree=True) # Pass the above models to be used in BERTopic topic_model = BERTopic(umap_model=umap_model, hdbscan_model=hdbscan_model) topics, probs = topic_model.fit_transform(docs)
- whakim 4y ago
  I agree that this is a cool. That being said, the results show that we have a long, long way to go. The topics are pretty incoherent: what are "would", "should", and "use" doing in there? The words have no context, so (for example) "self" clearly refers to the python reserved keyword, but you have no way of knowing that. Not to mention (as another comment brings up), the topics aren't named so it's pretty hard to actually figure out what they're about. If we think about real-world usage this output would be practically useless - it tells you that people talk about investing in r/investing and pytorch in r/pytorch. If you want meaningful, actionable information about what people are talking about in a large corpus of unstructured text data then for the forseeable future you'll need to involve humans in the loop even if ML assistance plays a big part.
victorianoi 4y ago
take a look at Graphext ( https://www.graphext.com https://www.graphext.com ) it automatically creates the clustering embeddings using BERT for you + great visualization libraries to interpret the clusters :D it took us 5 years to build the product
- CabSauce 4y ago
  It sure has to be much, much better than free. Especially if your pricing is 'contact sales'.
- m1sta_ 4y ago
  I'm not going to contact sales. I have no problem paying though. Part of your value proposition is saving people time, but your sales model is time expensive.
  - victorianoi 4y ago
    You are totally right! We are actually making the pricing public next week. We will keep the free version plus starting at $30/month for individual accounts and $150 for team accounts ;)
- lmeyerov 4y ago
  Latest pygraphistry has this flow for free and OSS, just `pip install graphistry[umap-learn]` or, for transformers, `pip install graphistry[ai]` :) And per the article, with pluggable sentence transformers -> UMAP automatically as part of the auto-featurization: graphistry.nodes(accounts_df).umap().plot() :) We haven't published tutorials yet, just been using with some fraud/cyber/misinfo/genomics/sales/gov/etc teams (including with RAPIDS gpu accel), so cool to see excitement here already! Till then, it should work out-of-the-box with no parameters, and then all sorts of fun things to tune: https://github.com/graphistry/pygraphistry/blob/21fad42412ccb5d007dfe4085d6616c065610a7c/graphistry/umap_utils.py#L252 https://github.com/graphistry/pygraphistry/blob/21fad42412cc...
- jasonys 4y ago
  I think you should open source the core part of the HTML visualization algos and if ppl like it, they may consider paying for the premium version. I don’t feel like ppl want to move their analysis workflow to yet another platform without trying it enough in their existing workflows (e.g Jupyter/Colab/Databricks notebooks)
Der_Einzige 4y ago
A huggingface space I wrote to let you play with BERTopic in your browser: https://huggingface.co/spaces/Hellisotherpeople/HF-BERTopic https://huggingface.co/spaces/Hellisotherpeople/HF-BERTopic
- leobg 4y ago
  Wow. This is great!
tbonza 4y ago
What happens on a slightly different task where domain experts have tried to create a set of topics, not all domain experts talk to each other, and so we instead need a way to merge existing topics? I continue to see benchmarks where human expertise significantly outperforms AI on common sense reasoning tasks (most recently https://arxiv.org/abs/2112.11446 https://arxiv.org/abs/2112.11446). What about an approach using directed acyclic graphs and entities?
- whakim 4y ago
  In traditional qualitative research, you'd usually have a bunch of experts get together and figure out a set of topics (or import and adapt a set of topics from similar work) before you go about classifying the bulk of your data.
toxik 4y ago
How does this compare to LDA? It doesn’t seem like there’s a huge difference here. For good reason perhaps, the BERT part is only to embed the sentences.
- berto4 4y ago
  yeah exactly my question. LDA is probabilistic and very performant if you clean up the documents well. The approach using Bert seems pretty powerful given that you can now cluster based on semantics, not just word occurrence/frequencies as in LDA (though ngrams help). However using a clustering approach would mean that each document is a part of a single topic, rather than being made up of multiple topics. But this is a cool idea nonetheless. [EDIT] quickly checked it out, seems like it uses some kind of soft clustering so documents can occur in many clusters (topics)
  - a-dub 4y ago
    would it make sense to preprocess with a transformer style model to produce per document semantic vectors which can then be piped into LDA to find topic mixtures of those vectors?
    - uoaei 4y ago
      Is that not exactly what's happening in TFA?
      - a-dub 4y ago
        if TFA means "the forememtioned article", i don't think so. i'm not convinced that the clusters found and the frequencies in those clusters would be the same as what LDA computes with gibbs sampling or the variational calculus method would find. but i must admit it's been a while since i've played with this stuff. if TFA is some other method, i am unfamiliar and would like to know more. in my experience, while it's true that it's hard to score and verify these sorts of models, the hierarchical multinomial nature of LDA topic models makes it easy to generate data and then verify behavior in the fitting process by recovering generative model parameters used by the test data generation process. obviously this makes no sense for the bert frontend, but a comparison of the differing backend clustering methods could be interesting.
visarga 4y ago
Next step: automatically naming clusters using few-shot GPT-3. Cluster naming is a non-trivial problem.
- malshe 4y ago
  Absolutely. This is the point where I think we shift from astronomy to astrology!
- vosper 4y ago
  I've run into this problem at a previous employer. Do you know if anyone's working on it?
  - visarga 4y ago
    There is a recent discussion in r/machinelearning about it https://www.reddit.com/r/MachineLearning/comments/umgdts/p_cluster_and_analyze_text_using_both_embedding/ https://www.reddit.com/r/MachineLearning/comments/umgdts/p_c...
  - jll29 4y ago
    "topic labeling" papers: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22topic+labeling%22+%22topic+modeling%22&btnG= https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22t...
ahoho 4y ago
Color me skeptical on BERTopic. Without human validation, I'm not convinced that it's an improvement over existing methods. I'm an author on a recent paper about automated topic model evaluation [1], and we found that current metrics do not line up with human judgements as well as previously thought. To my knowledge, BERTopic has only been evaluated on these automated metrics. For datasets of under a few hundred thousand documents, Mallet (LDA estimated with Gibbs sampling) can produce stable, high-quality outputs in minutes on a laptop [2]. Even larger datasets remain tractable, although depending on your use case you may be better off subsampling. It's possible that I've missed something, but I'm not clear on what benefits BERTopic has that existing methods do not. I don't mean to be overly negative---it has a nice API and the approach seems reasonable---I'm just wondering what's really new here. [1]: https://proceedings.neurips.cc/paper/2021/hash/0f83556a305d789b1d71815e8ea4f4b0-Abstract.html https://proceedings.neurips.cc/paper/2021/hash/0f83556a305d7... [2]: https://mimno.github.io/Mallet/ https://mimno.github.io/Mallet/ [3]: https://maartengr.github.io/BERTopic/faq.html#why-are-the-results-not-consistent-between-runs https://maartengr.github.io/BERTopic/faq.html#why-are-the-re...

104 points by gk1 4y ago

It is true that bertopic is a great tool. It's modern, it's modular, and it's pretty performant. That said, I want to caution against using topic modeling as a one-fits-all-solution. As the author stresses, this is one particular approach which uses a combination of embeddings (sentence, or other), umap and hdbscan. Both umap and hdbscan can be slow, so it might be worthwhile to check out the GPU enabled versions of both from the cuml package. In addition, topic models have a huge number of degrees of freedom, and the solution you will get depends on many (seemingly arbitrary) choices. In other words, these are not the topics, they are some topics. That said, it's awesome, really great work by Maarten Grootendorst and a great blog post by James Briggs. [edit] here is a link to the fast cuda version of bertopic by rapidsai: https://github.com/rapidsai/rapids-examples/tree/main/cuBERT_topic_modelling https://github.com/rapidsai/rapids-examples/tree/main/cuBERT...

hack_ml 4y ago
Its seamless to accelerate BERTOPIC on GPU's with cuML now with the latest release. (v0.10.0) Checkout the docs at: https://maartengr.github.io/BERTopic/faq.html#can-i-use-the-gpu-to-speed-up-the-model https://maartengr.github.io/BERTopic/faq.html#can-i-use-the-... All you need to do is below from bertopic import BERTopic from cuml.cluster import HDBSCAN from cuml.manifold import UMAP # Create instances of GPU-accelerated UMAP and HDBSCAN umap_model = UMAP(n_components=5, n_neighbors=15, min_dist=0.0) hdbscan_model = HDBSCAN(min_samples=10, gen_min_span_tree=True) # Pass the above models to be used in BERTopic topic_model = BERTopic(umap_model=umap_model, hdbscan_model=hdbscan_model) topics, probs = topic_model.fit_transform(docs)
whakim 4y ago
I agree that this is a cool. That being said, the results show that we have a long, long way to go. The topics are pretty incoherent: what are "would", "should", and "use" doing in there? The words have no context, so (for example) "self" clearly refers to the python reserved keyword, but you have no way of knowing that. Not to mention (as another comment brings up), the topics aren't named so it's pretty hard to actually figure out what they're about. If we think about real-world usage this output would be practically useless - it tells you that people talk about investing in r/investing and pytorch in r/pytorch. If you want meaningful, actionable information about what people are talking about in a large corpus of unstructured text data then for the forseeable future you'll need to involve humans in the loop even if ML assistance plays a big part.

victorianoi 4y ago

take a look at Graphext ( https://www.graphext.com https://www.graphext.com ) it automatically creates the clustering embeddings using BERT for you + great visualization libraries to interpret the clusters :D it took us 5 years to build the product

CabSauce 4y ago
It sure has to be much, much better than free. Especially if your pricing is 'contact sales'.
m1sta_ 4y ago
I'm not going to contact sales. I have no problem paying though. Part of your value proposition is saving people time, but your sales model is time expensive.
- victorianoi 4y ago
  You are totally right! We are actually making the pricing public next week. We will keep the free version plus starting at $30/month for individual accounts and $150 for team accounts ;)
lmeyerov 4y ago
Latest pygraphistry has this flow for free and OSS, just `pip install graphistry[umap-learn]` or, for transformers, `pip install graphistry[ai]` :) And per the article, with pluggable sentence transformers -> UMAP automatically as part of the auto-featurization: graphistry.nodes(accounts_df).umap().plot() :) We haven't published tutorials yet, just been using with some fraud/cyber/misinfo/genomics/sales/gov/etc teams (including with RAPIDS gpu accel), so cool to see excitement here already! Till then, it should work out-of-the-box with no parameters, and then all sorts of fun things to tune: https://github.com/graphistry/pygraphistry/blob/21fad42412ccb5d007dfe4085d6616c065610a7c/graphistry/umap_utils.py#L252 https://github.com/graphistry/pygraphistry/blob/21fad42412cc...
jasonys 4y ago
I think you should open source the core part of the HTML visualization algos and if ppl like it, they may consider paying for the premium version. I don’t feel like ppl want to move their analysis workflow to yet another platform without trying it enough in their existing workflows (e.g Jupyter/Colab/Databricks notebooks)

Der_Einzige 4y ago

A huggingface space I wrote to let you play with BERTopic in your browser: https://huggingface.co/spaces/Hellisotherpeople/HF-BERTopic https://huggingface.co/spaces/Hellisotherpeople/HF-BERTopic

leobg 4y ago
Wow. This is great!

tbonza 4y ago

What happens on a slightly different task where domain experts have tried to create a set of topics, not all domain experts talk to each other, and so we instead need a way to merge existing topics? I continue to see benchmarks where human expertise significantly outperforms AI on common sense reasoning tasks (most recently https://arxiv.org/abs/2112.11446 https://arxiv.org/abs/2112.11446). What about an approach using directed acyclic graphs and entities?

whakim 4y ago
In traditional qualitative research, you'd usually have a bunch of experts get together and figure out a set of topics (or import and adapt a set of topics from similar work) before you go about classifying the bulk of your data.

toxik 4y ago

How does this compare to LDA? It doesn’t seem like there’s a huge difference here. For good reason perhaps, the BERT part is only to embed the sentences.

berto4 4y ago
yeah exactly my question. LDA is probabilistic and very performant if you clean up the documents well. The approach using Bert seems pretty powerful given that you can now cluster based on semantics, not just word occurrence/frequencies as in LDA (though ngrams help). However using a clustering approach would mean that each document is a part of a single topic, rather than being made up of multiple topics. But this is a cool idea nonetheless. [EDIT] quickly checked it out, seems like it uses some kind of soft clustering so documents can occur in many clusters (topics)
- a-dub 4y ago
  would it make sense to preprocess with a transformer style model to produce per document semantic vectors which can then be piped into LDA to find topic mixtures of those vectors?
  - uoaei 4y ago
    Is that not exactly what's happening in TFA?
    - a-dub 4y ago
      if TFA means "the forememtioned article", i don't think so. i'm not convinced that the clusters found and the frequencies in those clusters would be the same as what LDA computes with gibbs sampling or the variational calculus method would find. but i must admit it's been a while since i've played with this stuff. if TFA is some other method, i am unfamiliar and would like to know more. in my experience, while it's true that it's hard to score and verify these sorts of models, the hierarchical multinomial nature of LDA topic models makes it easy to generate data and then verify behavior in the fitting process by recovering generative model parameters used by the test data generation process. obviously this makes no sense for the bert frontend, but a comparison of the differing backend clustering methods could be interesting.

visarga 4y ago

Next step: automatically naming clusters using few-shot GPT-3. Cluster naming is a non-trivial problem.

malshe 4y ago
Absolutely. This is the point where I think we shift from astronomy to astrology!
vosper 4y ago
I've run into this problem at a previous employer. Do you know if anyone's working on it?
- visarga 4y ago
  There is a recent discussion in r/machinelearning about it https://www.reddit.com/r/MachineLearning/comments/umgdts/p_cluster_and_analyze_text_using_both_embedding/ https://www.reddit.com/r/MachineLearning/comments/umgdts/p_c...
- jll29 4y ago
  "topic labeling" papers: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22topic+labeling%22+%22topic+modeling%22&btnG= https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=%22t...

ahoho 4y ago

Color me skeptical on BERTopic. Without human validation, I'm not convinced that it's an improvement over existing methods. I'm an author on a recent paper about automated topic model evaluation [1], and we found that current metrics do not line up with human judgements as well as previously thought. To my knowledge, BERTopic has only been evaluated on these automated metrics. For datasets of under a few hundred thousand documents, Mallet (LDA estimated with Gibbs sampling) can produce stable, high-quality outputs in minutes on a laptop [2]. Even larger datasets remain tractable, although depending on your use case you may be better off subsampling. It's possible that I've missed something, but I'm not clear on what benefits BERTopic has that existing methods do not. I don't mean to be overly negative---it has a nice API and the approach seems reasonable---I'm just wondering what's really new here. [1]: https://proceedings.neurips.cc/paper/2021/hash/0f83556a305d789b1d71815e8ea4f4b0-Abstract.html https://proceedings.neurips.cc/paper/2021/hash/0f83556a305d7... [2]: https://mimno.github.io/Mallet/ https://mimno.github.io/Mallet/ [3]: https://maartengr.github.io/BERTopic/faq.html#why-are-the-results-not-consistent-between-runs https://maartengr.github.io/BERTopic/faq.html#why-are-the-re...