4 ms·
This should be quantifiable. Use an LLM to help identify this undercurrent you’re referring to in maybe 10,000 conversation threads using the API from the foot
by cheschire 2y ago
This should be quantifiable. Use an LLM to help identify this undercurrent you’re referring to in maybe 10,000 conversation threads using the API from the footer. A thread being a reply to a topic, and a singular chain of replies.
Then use that to train an RNN for sentiment analysis based on the values you’ve just described.
Then run the entire HN corpus through your RNN and chart the values over time.
For bonus points, incorporate the score of replies in the weighting of values.
- alephnerd 2y agoYou don't need an LLM for sentiment analysis. I also don't want to pay for scraping and data ingest costs out of my personal pocket. Feel free to do it if you want - it would make a nice blogpost.
- cheschire 2y agoI wasn't suggesting you need the LLM to perform sentiment analysis. The problem the LLM overcomes is generating the training data you need without having to trudge through massive amounts of material to mark a comment / thread with a 1 or a 0. There's no scraping cost, the API is free. Your message, for example, is here: https://hacker-news.firebaseio.com/v0/item/40801233.json?print=pretty https://hacker-news.firebaseio.com/v0/item/40801233.json?pri... Processing 10,000 threads is probably not more than a few dollars on OpenAI's GPT API. The reason I don't do it is I don't see the trend you see.