3 ms·

Evaluating a Decade of Hacker News Predictions: An Open-Source Approach

27 points by eamag 2y ago

bogwog 2y ago
I'm disappointed that the guy with insider knowledge who predicted that Stadia would not be shutdown isn't there. Edit: also, awesome project.
8organicbits 2y ago
Anyone bothered by the AI generated cover photo trend? They are always low quality, like this one with the weird font and creepy robot, and never add anything to the article. I'm guessing the goal it to boost engagement by taking up more space in any news feed that displays cover images in line? I see it as a negative quality signal.
- xyst 2y ago
  I have seen so much trash in the AI field so far that I am no longer bothered. It’s become standard operating procedure in this bubble.
- deleted 2y ago
  [deleted]
- noman-land 2y ago
  Photos on your #content are a well documented #engagement #enhancer.
xyst 2y ago
Interesting project but I hate how I have to manually confirm whether or not the analysis by LLM is “hallucinated” or not. For example, just looking at the first result in the “Top Authors” category. The author “asah” purportedly has a “88.10% success rate, 21 predictions”. Clicking the link just points to the comment then I (the reader) have to manually look through which ones were “successful” Were the ones that I verified as “successful” the same ones as your analysis with LLM? I can’t exactly verify that since there are no sources provided. No history of the “conversation” preserved. Instead of trusting the information the reader is left with more questions than answers. Did it source it from a reliable first party source or maybe it got the information from a PR firm overstating certain numbers/figures. Yet another reason why LLM/AI tech is overblown in my opinion. It’s catering to the same shit as the clickbait farms. Junk information with no backing/sources. Ask these algos to cross verify sources and probably get hallucinated trash.
- spencerflem 2y ago
  Agreed, if the author spent an hour our two doing it themselves this could have been a very fun article
- eamag 2y ago
  You can go to the table [1] page and filter by username, then you can see what prediction LLM is evaluating and why. I can agree with you on hallucination part, but for the easy predictions it's correct most of the time (like Microsoft downfall) [1]https://hn-predictions.eamag.me/table https://hn-predictions.eamag.me/table
smusamashah 2y ago
Interestingly, technology predictions have lower than random success rate 49% (in 1086 predictions) according to these results.

27 points by eamag 2y ago

I'm disappointed that the guy with insider knowledge who predicted that Stadia would not be shutdown isn't there. Edit: also, awesome project.

8organicbits 2y ago

Anyone bothered by the AI generated cover photo trend? They are always low quality, like this one with the weird font and creepy robot, and never add anything to the article. I'm guessing the goal it to boost engagement by taking up more space in any news feed that displays cover images in line? I see it as a negative quality signal.

xyst 2y ago
I have seen so much trash in the AI field so far that I am no longer bothered. It’s become standard operating procedure in this bubble.
deleted 2y ago
[deleted]
noman-land 2y ago
Photos on your #content are a well documented #engagement #enhancer.

xyst 2y ago

Interesting project but I hate how I have to manually confirm whether or not the analysis by LLM is “hallucinated” or not. For example, just looking at the first result in the “Top Authors” category. The author “asah” purportedly has a “88.10% success rate, 21 predictions”. Clicking the link just points to the comment then I (the reader) have to manually look through which ones were “successful” Were the ones that I verified as “successful” the same ones as your analysis with LLM? I can’t exactly verify that since there are no sources provided. No history of the “conversation” preserved. Instead of trusting the information the reader is left with more questions than answers. Did it source it from a reliable first party source or maybe it got the information from a PR firm overstating certain numbers/figures. Yet another reason why LLM/AI tech is overblown in my opinion. It’s catering to the same shit as the clickbait farms. Junk information with no backing/sources. Ask these algos to cross verify sources and probably get hallucinated trash.

spencerflem 2y ago
Agreed, if the author spent an hour our two doing it themselves this could have been a very fun article
eamag 2y ago
You can go to the table [1] page and filter by username, then you can see what prediction LLM is evaluating and why. I can agree with you on hallucination part, but for the easy predictions it's correct most of the time (like Microsoft downfall) [1]https://hn-predictions.eamag.me/table https://hn-predictions.eamag.me/table

smusamashah 2y ago

Interestingly, technology predictions have lower than random success rate 49% (in 1086 predictions) according to these results.