3 ms·

I believe that probably in the future that would be possible. However, today, nope. The website itself is a system of crawling, parsing, analysis, ML and LLM—mo

• PaulHoule 2y ago
  It is not months of work. I wrote an HN crawler that got all the posts in less than two hours of coding. (There is the tricky question of polling schedules though: if you want to get every story the minute it comes out it will not have any votes or comments and you'll have to come back later if you want to get representative numbers. I think discussions close in 2 weeks so a double scan is good enough for some purposes but not the up-to-the-minute analysis they claim to be doing) The rest of the analysis is all straightforward except for the topic classification: If I was running on all cylinders for nights and weekend hacking I'd expect to spend a week or so on gathering training data for that (maybe suck down category assignments from Tildes) and probably a week coding up a classifier w/ a BERT-family embedding and classifiers from scikit-learn. You could attempt the same with prompt engineering in a day or two of work and get something expensive to run working pretty quickly that will probably not perform as well as my classifier. (Their classifier gives the empty string as a category which is a bad sign) If you want to improve on it you need the same training set I do.
  • behindai 2y ago
    You have enough interesting points to do your project. Would love to see the results! Cheers!