10 ms·

Do you think blogs with most recent posts can be bumped up (instead of a static ordering)?

by alpb 3y ago

MuffinFlavored 3y ago
Like... an RSS feed keeping tabs on all of these blogs/their "last updated" field value?
jcynix 3y ago
Remember "folksonomy" ...? So I would like to see some kind of automated keyword extraction for each blog to allow selection of topical blogs. OK, keywords would need to normalied (e.g.plural to singular, plus maybe a bit of abstraction) and this would be a first (tiny) step to make it a blog-centered search engine ... ;-0
- marginalia_nu 3y ago
  I've been experimenting with techniques for figuring out the topic of a blog, and never quite gotten anything to work well that didn't involve extreme computational costs that just do not work on a search-engine scale without a serious hardware budget (zero shot classifiers did a decent job IIRC). An unga bunga technique like TF-IDF works well for a given document, but less well for a website. Had some half-decent success finding similar blogs given a reference though.
  - jcynix 3y ago
    Yes, I expect full text extraction and normalization to be computationally expensive. I did some simple minded experiments on Usenet feeds back in the 90ies and it was quite some work for the VAX it ran on.
    - marginalia_nu 3y ago
      You can get it to run fairly fast on modern hardware. Like run a text extraction, tokenization and POS-tagging workflow on a quarter billion documents on PC hardware, takes like 24-36 hours. That's doable and affordable. But ML-adjacent methods are not. Requires far too much GPU compute, have no A100s :-/
surprisetalk 3y ago
I originally implemented it by roughly sorting via HN karma, which you can see in my comment here: [0] https://github.com/surprisetalk/blogs.hn/blob/22a471d56504d5471905c1b7ceecab332a175a8f/build.js#L13 https://github.com/surprisetalk/blogs.hn/blob/22a471d56504d5... To maximize serendipity, I ended up going with random ordering. I think there are other RSS-focus solutions (like https://ooh.directory https://ooh.directory) that are probably a better fit for sorting by recent.

Do you think blogs with most recent posts can be bumped up (instead of a static ordering)?

by alpb 3y ago

Do you think blogs with most recent posts can be bumped up (instead of a static ordering)?

MuffinFlavored 3y ago

Like... an RSS feed keeping tabs on all of these blogs/their "last updated" field value?

jcynix 3y ago

Remember "folksonomy" ...? So I would like to see some kind of automated keyword extraction for each blog to allow selection of topical blogs. OK, keywords would need to normalied (e.g.plural to singular, plus maybe a bit of abstraction) and this would be a first (tiny) step to make it a blog-centered search engine ... ;-0

marginalia_nu 3y ago
I've been experimenting with techniques for figuring out the topic of a blog, and never quite gotten anything to work well that didn't involve extreme computational costs that just do not work on a search-engine scale without a serious hardware budget (zero shot classifiers did a decent job IIRC). An unga bunga technique like TF-IDF works well for a given document, but less well for a website. Had some half-decent success finding similar blogs given a reference though.
- jcynix 3y ago
  Yes, I expect full text extraction and normalization to be computationally expensive. I did some simple minded experiments on Usenet feeds back in the 90ies and it was quite some work for the VAX it ran on.
  - marginalia_nu 3y ago
    You can get it to run fairly fast on modern hardware. Like run a text extraction, tokenization and POS-tagging workflow on a quarter billion documents on PC hardware, takes like 24-36 hours. That's doable and affordable. But ML-adjacent methods are not. Requires far too much GPU compute, have no A100s :-/

surprisetalk 3y ago

I originally implemented it by roughly sorting via HN karma, which you can see in my comment here: [0] https://github.com/surprisetalk/blogs.hn/blob/22a471d56504d5471905c1b7ceecab332a175a8f/build.js#L13 https://github.com/surprisetalk/blogs.hn/blob/22a471d56504d5... To maximize serendipity, I ended up going with random ordering. I think there are other RSS-focus solutions (like https://ooh.directory https://ooh.directory) that are probably a better fit for sorting by recent.