5 ms·

Hi, I think discovering blogs and articles by other engineers is quite hard. Particularly non-commercial content. I made blogosphere last Summer in order to im

Hi, I think discovering blogs and articles by other engineers is quite hard. Particularly non-commercial content. I made blogosphere last Summer in order to improve discovery. It’s seeded with personal blogs that have at some point appeared on the HN front-page. The intent of the site is to recommend blogs and articles to you, with the objective that it’ll get better the more you use it. The site uses a co-occurrence matrix to recommend articles and bloggers to you based on the articles you ‘star’ and topics you follow. If you’re not logged in, the site will show you popular articles. It doesn’t track what you read, and doesn’t have google analytics. I have some ideas for improving recommendations (i.e. people like you also like.., and improving the auto tagging of articles), but I thought I’d get feedback on whether people want this first. The site is at an MVP stage, there will be bugs and missing features, but feedback on whether this is interesting to you would be useful.

m_a_g 6y ago
I wonder, do you handpick every blog post or did you write some sort of a crawler?
- BillFranklin 6y ago
  I started with the contents of my own RSS feed, and then crawled Hacker News for popular blogs, and then crawled the feeds of those blogs (in order to get posts that didn't get upvoted on HN). The time consuming part was manually auditing blogs for quality (and to ensure it's a personal blog, not e.g. a BBC feed). I came up with some heuristics (i.e. blogs with github.io subdomains that got upvoted were often good so I blanket approved those) to speed things along, but it did take a while.
  - taoothereporter 6y ago
    Have you tried to apply some ML stuff?
    - BillFranklin 6y ago
      Back in August I had a look at using Latent Dirichlet Allocation (http://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-allocation/ http://blog.echen.me/2011/08/22/introduction-to-latent-diric...) for improving how I auto-tag articles. It looked promising but I haven't deployed it yet. There are various recommendation approaches I'd like to try. If the site had more users (beyond the ~4 friends I showed it to) I'd like to try out a collaborative filtering approach (https://en.wikipedia.org/wiki/Collaborative_filtering https://en.wikipedia.org/wiki/Collaborative_filtering).
      - swsieber 6y ago
        I took a course about bayesian stats and ML in college. We covered LDA. It's pretty awesome! I hope you can get it out and that it goes well.
  - djhn 6y ago
    This would make for an interesting... technical blog post.
jarbus 6y ago
Nice work, I really like this idea!
godelski 6y ago
> It doesn’t track what you read, and doesn’t have google analytics. Just wanted to say thanks for minimizing tracking of users.
soldehierro 6y ago
Wow, this is brilliant. I know my way around a computer and have programmed before, well enough to understand the discussions around here, but I am decidely not a "tech person" by any means. I think this concept would be really great if expanded to fields outside tech.
polote 6y ago
https://findka.com/ https://findka.com/ is a similar service > it’s seeded with personal blogs that have at some point appeared on the HN front-page. also this https://hnblogs.substack.com/ https://hnblogs.substack.com/ newsletter (than I manage) send you everyday with non commercial blogpost of HN of the day, if that can help seeds your aglo
m-i-l 6y ago
Looks good. I wonder if there'd be any interest in combining with my side project, which is a search engine for user-submitted personal and independent websites (and also search as a service). I could do with a nicer user interface and more structured data like yours, and perhaps you could benefit from features like user-submitted sites and scheduled (plus on-demand for verified sites) reindexing. Details in my profile, with links to the (open source) code on the site.

m_a_g 6y ago

I wonder, do you handpick every blog post or did you write some sort of a crawler?

BillFranklin 6y ago
I started with the contents of my own RSS feed, and then crawled Hacker News for popular blogs, and then crawled the feeds of those blogs (in order to get posts that didn't get upvoted on HN). The time consuming part was manually auditing blogs for quality (and to ensure it's a personal blog, not e.g. a BBC feed). I came up with some heuristics (i.e. blogs with github.io subdomains that got upvoted were often good so I blanket approved those) to speed things along, but it did take a while.
- taoothereporter 6y ago
  Have you tried to apply some ML stuff?
  - BillFranklin 6y ago
    Back in August I had a look at using Latent Dirichlet Allocation (http://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-allocation/ http://blog.echen.me/2011/08/22/introduction-to-latent-diric...) for improving how I auto-tag articles. It looked promising but I haven't deployed it yet. There are various recommendation approaches I'd like to try. If the site had more users (beyond the ~4 friends I showed it to) I'd like to try out a collaborative filtering approach (https://en.wikipedia.org/wiki/Collaborative_filtering https://en.wikipedia.org/wiki/Collaborative_filtering).
    - swsieber 6y ago
      I took a course about bayesian stats and ML in college. We covered LDA. It's pretty awesome! I hope you can get it out and that it goes well.
- djhn 6y ago
  This would make for an interesting... technical blog post.

jarbus 6y ago

Nice work, I really like this idea!

godelski 6y ago

> It doesn’t track what you read, and doesn’t have google analytics. Just wanted to say thanks for minimizing tracking of users.

soldehierro 6y ago

Wow, this is brilliant. I know my way around a computer and have programmed before, well enough to understand the discussions around here, but I am decidely not a "tech person" by any means. I think this concept would be really great if expanded to fields outside tech.

polote 6y ago

https://findka.com/ https://findka.com/ is a similar service > it’s seeded with personal blogs that have at some point appeared on the HN front-page. also this https://hnblogs.substack.com/ https://hnblogs.substack.com/ newsletter (than I manage) send you everyday with non commercial blogpost of HN of the day, if that can help seeds your aglo

m-i-l 6y ago

Looks good. I wonder if there'd be any interest in combining with my side project, which is a search engine for user-submitted personal and independent websites (and also search as a service). I could do with a nicer user interface and more structured data like yours, and perhaps you could benefit from features like user-submitted sites and scheduled (plus on-demand for verified sites) reindexing. Details in my profile, with links to the (open source) code on the site.