5 ms·

I'd been thinking recently about how to write a "train your own recommendation algorithm" feed reader and fediverse reader, but I was unsure how those sorts of

I'd been thinking recently about how to write a "train your own recommendation algorithm" feed reader and fediverse reader, but I was unsure how those sorts of algorithms work. It looks like cosine similarity is a good place to start. I wonder if you could get anything useful from just upvoting/downvoting entries in your own feed for a few days/weeks this way, or if much more signal would be needed for a real algorithm.

marginalia_nu 4y ago
This sort of algorithm requires quite a lot of data to work. If you had a large collection of many people's votes/bookmarks, that would work. It's not enough to just use one person's preferences as there is nothing to compare them to. Although I guess you could use link data (or some other source) to base suggestion off.
- culi 4y ago
  I've thought about it a lot and I really thinking having these sorts of "interest" or "friend" groups are the way to go. A major problem with major search engines like Google is they're really trying to do it all in one. We can keep a generalized utility search as the default, but I wish I could somehow tell a search engine "show me stuff you'd show to my friends from this community". I.e. what would HackerNews' community search group look like? What would this friendgroup that for some reason includes a lot of people who work in botany but that isn't the defining feature of the group. Maybe they all also happen to like board games, are allergic to social media, and are weirdly well-read on continental philosophic traditions. I think the key is that people have exit. My main gripe with things like TikTok and twitter is the algorithm tries to optimize results for what you'd like most. But people don't really work that way. People grow over time. People join different communities, put on different hats, try different identities. Sometimes it happens over time or sometimes people are in phases of their life where they're constantly in between multiple communities. I would feel more comfortable with the "profile" these algorithms made for me if I had some sort of say in it and could switch between at will or do an "incognito"/fresh slate mode whenever I want
- KirillPanov 4y ago
  Hey, I love the idea behind marginalia. I love the fact that it doesn't require javascript, and in fact lets you search only for sites that work without javascript! So incredibly awesome! Unfortunately Marginalia itself requires javascript. Could you please do something about this? I would use Marginalia a whole lot more, except that if I do more than two or three searches an hour, all I get is this: search.marginalia.nu Checking if the site connection is secure Enable JavaScript and cookies to continue search.marginalia.nu needs to review the security of your connection before proceeding. ... and then I just change my search engine back to Yandex in frustration. After a few days I tend to forget about Marginalia. At least until it pops up here on HN.
  - marginalia_nu 4y ago
    You're probably unlucky enough to share an ASN with a spammer. I wish I didn't have to degrade service like this for some people, but the alternative is not serving any traffic at all because the site is constantly DOSed by bots.
    - KirillPanov 4y ago
      Wait, isn't marginalia's crawler a bot? Also, what does any of this have to do with email spam? We're talking about HTTP(S) here, not SMTP, right?
      - rzzzt 4y ago
        Cloudflare and other providers add this in front of websites for DDOS protection purposes, and it is triggered based on different factors, one of which is location. I regularly get this on an Australian forum that discusses SOHO routers (and Javascript-based validation always fails so I have to solve a captcha instead).
      - akavel 4y ago
        I imagine "spammer" here is not an email spammer, but either e.g. a comments spammer (a bot trying to post links in comment forms to game google), or a generalized thought shortcut meaning something like "a bot/botnet crawling a website for nefarious purposes in a high-traffic way with abusive disregard for the site's computational and bandwidth resources".

marginalia_nu 4y ago

This sort of algorithm requires quite a lot of data to work. If you had a large collection of many people's votes/bookmarks, that would work. It's not enough to just use one person's preferences as there is nothing to compare them to. Although I guess you could use link data (or some other source) to base suggestion off.

culi 4y ago
I've thought about it a lot and I really thinking having these sorts of "interest" or "friend" groups are the way to go. A major problem with major search engines like Google is they're really trying to do it all in one. We can keep a generalized utility search as the default, but I wish I could somehow tell a search engine "show me stuff you'd show to my friends from this community". I.e. what would HackerNews' community search group look like? What would this friendgroup that for some reason includes a lot of people who work in botany but that isn't the defining feature of the group. Maybe they all also happen to like board games, are allergic to social media, and are weirdly well-read on continental philosophic traditions. I think the key is that people have exit. My main gripe with things like TikTok and twitter is the algorithm tries to optimize results for what you'd like most. But people don't really work that way. People grow over time. People join different communities, put on different hats, try different identities. Sometimes it happens over time or sometimes people are in phases of their life where they're constantly in between multiple communities. I would feel more comfortable with the "profile" these algorithms made for me if I had some sort of say in it and could switch between at will or do an "incognito"/fresh slate mode whenever I want
KirillPanov 4y ago
Hey, I love the idea behind marginalia. I love the fact that it doesn't require javascript, and in fact lets you search only for sites that work without javascript! So incredibly awesome! Unfortunately Marginalia itself requires javascript. Could you please do something about this? I would use Marginalia a whole lot more, except that if I do more than two or three searches an hour, all I get is this: search.marginalia.nu Checking if the site connection is secure Enable JavaScript and cookies to continue search.marginalia.nu needs to review the security of your connection before proceeding. ... and then I just change my search engine back to Yandex in frustration. After a few days I tend to forget about Marginalia. At least until it pops up here on HN.
- marginalia_nu 4y ago
  You're probably unlucky enough to share an ASN with a spammer. I wish I didn't have to degrade service like this for some people, but the alternative is not serving any traffic at all because the site is constantly DOSed by bots.
  - KirillPanov 4y ago
    Wait, isn't marginalia's crawler a bot? Also, what does any of this have to do with email spam? We're talking about HTTP(S) here, not SMTP, right?
    - rzzzt 4y ago
      Cloudflare and other providers add this in front of websites for DDOS protection purposes, and it is triggered based on different factors, one of which is location. I regularly get this on an Australian forum that discusses SOHO routers (and Javascript-based validation always fails so I have to solve a captcha instead).
    - akavel 4y ago
      I imagine "spammer" here is not an email spammer, but either e.g. a comments spammer (a bot trying to post links in comment forms to game google), or a generalized thought shortcut meaning something like "a bot/botnet crawling a website for nefarious purposes in a high-traffic way with abusive disregard for the site's computational and bandwidth resources".