3 ms·

I'm still in the stone ages trying to use topic modeling on a free text field. Can LLMs help me decipher a large dataset of human text and put it in buckets?

quickthrower2 3y ago
It might help you create the buckets at least if you take a sample of say 1000 and give that to the LLM.
causalmodels 3y ago
If you know your buckets you can pretty easily get most LLMs to generate structured outputs using something like zod-gpt [1]. [1] https://github.com/dzhng/zod-gpt https://github.com/dzhng/zod-gpt
jcims 3y ago
Absolutely. Give it a try. I pasted these comments into ChatGPT and asked it to score them according to slashdot’s system (which i didn’t actually define) and got this reply: In Slashdot's scoring system, comments are scored from -1 to 5, based on factors like insightfulness, informativeness, and whether they're interesting or funny. Here are the hypothetical scores for the comments from Hacker News: owlninja (42 minutes ago): This user is asking about the applicability of LLMs for text categorization in large datasets. It's a straightforward query, indicating a need for information. Score: 3 (Interesting) - As it opens up a discussion on a relevant and technical topic. causalmodels (18 minutes ago): This comment provides a direct solution with a resource (zod-gpt). Score: 4 (Informative) - It not only addresses the query but also provides a specific tool to help achieve the goal, adding value to the discussion. quickthrower2 (41 minutes ago): This comment suggests an approach for creating buckets using a sample and an LLM. Score: 3 (Interesting) - It proposes a practical method, contributing constructively to the original query. These scores are subjective and would depend on the perspectives of the individual moderators or the community's voting on Slashdot.
ta988 3y ago
absolutely here is an example with openai. First pass you explain the task and let it reply to a hundredish examples. Fix the errors, fine tune. And it should be much better. You can repeat that any time you need until you reach satisfactory results but they come quickly when I worked on large number of short texts i wanted to classify and extract info from.
soultrees 3y ago
I just did this today using openAI’s function calling. I have a bunch of elements in a scene and trying to classify them between various ‘buckets’ has been the challenge. The way I set it up is that in the schema expected, you take that top level free text, wrap it in quotes and it becomes the parent object with the elementCategory as a string required inside it, with a list of all category types in the description. Then loop through and create a dynamic schema based on how you chunk your data, then add a validation step at the end to ensure that chatgpt doesn’t forget, add or change keys. I found that if you create a numberedKeywordString for your chunk, and wrap each in quotes, then stuff that numberedKeywordString into the system prompt then it’s solid and will ensure the LLM catches every key and gets the associated elementCategory value. It works quite well for dynamic classification in my purposes but I’m sure there is a better way.
- bravura 3y ago
  Try this: https://github.com/refuel-ai/autolabel https://github.com/refuel-ai/autolabel Then the main challenge just becomes prompt design, which can sometimes be nebulous for NLP annotation.
anon-feaata2311 3y ago
Have a look at Using text embeddings and approximate nearest neighbour search to explore ... https://medium.com/gft-engineering/using-text-embeddings-and-approximate-nearest-neighbour-search-to-explore-shakespeares-plays-29e6bde05a16 https://medium.com/gft-engineering/using-text-embeddings-and... You can reduce sentences to vectors and then create similarity scores to build a graph over the corpus. If you choose to create clusters then use a llm to summarise them to create labels.
- anon-feaata2311 3y ago
  Interestingly I don't see much progress on the core of vector DBs... Does anyone have so innovative references?
kolinko 3y ago
With no problem. The easiest way is to go with GPT4 api for a start (you just tell it how you want it organised / what metadata to extract) - it will be costly, but you can prove that it works, and generate examples to fine tune a simpler model to do the same.

quickthrower2 3y ago

It might help you create the buckets at least if you take a sample of say 1000 and give that to the LLM.

causalmodels 3y ago

If you know your buckets you can pretty easily get most LLMs to generate structured outputs using something like zod-gpt [1]. [1] https://github.com/dzhng/zod-gpt https://github.com/dzhng/zod-gpt

jcims 3y ago

Absolutely. Give it a try. I pasted these comments into ChatGPT and asked it to score them according to slashdot’s system (which i didn’t actually define) and got this reply: In Slashdot's scoring system, comments are scored from -1 to 5, based on factors like insightfulness, informativeness, and whether they're interesting or funny. Here are the hypothetical scores for the comments from Hacker News: owlninja (42 minutes ago): This user is asking about the applicability of LLMs for text categorization in large datasets. It's a straightforward query, indicating a need for information. Score: 3 (Interesting) - As it opens up a discussion on a relevant and technical topic. causalmodels (18 minutes ago): This comment provides a direct solution with a resource (zod-gpt). Score: 4 (Informative) - It not only addresses the query but also provides a specific tool to help achieve the goal, adding value to the discussion. quickthrower2 (41 minutes ago): This comment suggests an approach for creating buckets using a sample and an LLM. Score: 3 (Interesting) - It proposes a practical method, contributing constructively to the original query. These scores are subjective and would depend on the perspectives of the individual moderators or the community's voting on Slashdot.

ta988 3y ago

absolutely here is an example with openai. First pass you explain the task and let it reply to a hundredish examples. Fix the errors, fine tune. And it should be much better. You can repeat that any time you need until you reach satisfactory results but they come quickly when I worked on large number of short texts i wanted to classify and extract info from.

soultrees 3y ago

I just did this today using openAI’s function calling. I have a bunch of elements in a scene and trying to classify them between various ‘buckets’ has been the challenge. The way I set it up is that in the schema expected, you take that top level free text, wrap it in quotes and it becomes the parent object with the elementCategory as a string required inside it, with a list of all category types in the description. Then loop through and create a dynamic schema based on how you chunk your data, then add a validation step at the end to ensure that chatgpt doesn’t forget, add or change keys. I found that if you create a numberedKeywordString for your chunk, and wrap each in quotes, then stuff that numberedKeywordString into the system prompt then it’s solid and will ensure the LLM catches every key and gets the associated elementCategory value. It works quite well for dynamic classification in my purposes but I’m sure there is a better way.

bravura 3y ago
Try this: https://github.com/refuel-ai/autolabel https://github.com/refuel-ai/autolabel Then the main challenge just becomes prompt design, which can sometimes be nebulous for NLP annotation.

anon-feaata2311 3y ago

Have a look at Using text embeddings and approximate nearest neighbour search to explore ... https://medium.com/gft-engineering/using-text-embeddings-and-approximate-nearest-neighbour-search-to-explore-shakespeares-plays-29e6bde05a16 https://medium.com/gft-engineering/using-text-embeddings-and... You can reduce sentences to vectors and then create similarity scores to build a graph over the corpus. If you choose to create clusters then use a llm to summarise them to create labels.

anon-feaata2311 3y ago
Interestingly I don't see much progress on the core of vector DBs... Does anyone have so innovative references?

kolinko 3y ago

With no problem. The easiest way is to go with GPT4 api for a start (you just tell it how you want it organised / what metadata to extract) - it will be costly, but you can prove that it works, and generate examples to fine tune a simpler model to do the same.