10 ms·

Show HN: Search Engine for Blogs

• pomokhtari 5y ago
  Google has been giving me a very hard time for a while. It's time for SEO to die. We need stuff like this.
• codeconscious 5y ago
  Lovely! I agree with others that this is promising. Thanks for sharing. One point of feedback: Searching for "C#" seems to bring up C articles and no C# ones, so I suspect perhaps the "#" isn't being included.
• Reith 5y ago
  I love the idea and I cannot stress how many times I regret finding some undistracting results from those knowledgeable people who put time and energy to produce high-quality materials instead of search-engine-optimized dull content! That said, a mere full-text search lacks understanding context. I tried getting some results about gardening and plants, but returned results where anything from bitcoin to power plants, or even DNA, and not vegetation.
• diogenesjunior 5y ago
  this is very cool, thank you
• taubek 5y ago
  This is great. What is the policy on accepting blogs to be indexed?
• superasn 5y ago
  This is definitely very cool as I've been looking for something like this since technorati (which was originally a blog search engine). Would love to hear details about how you created the database, the infrastructure, etc if it's not a trade secret. Kudos on the launch!
  • dbrereton 5y ago
    > This is definitely very cool as I've been looking for something like this since technorati (which was originally a blog search engine). Technorati was one of the inspirations here so that's great to hear. > Would love to hear details about how you created the database, the infrastructure, etc if it's not a trade secret. Kudos on the launch! Sure, it's actually fairly simple! The search backend itself is running on Typesense [0], which was very quick and easy to setup. Due to the way ranking is calculated, I can actually avoid doing any real web crawling (though, I may add that in soon to help increase the index size). Ranking is based on submission to online communities, so all I really need is those submissions. Using the Reddit, HN and Twitter APIs, I search for any submissions related to any blogs in the database, then those submissions give me the post URLs. Once I have the post URLs, I just need to request those specific URLs to get the post data. Then there's scripts for things like content extraction, inflation calculation, currency conversion etc. All of those scripts are in python. The frontend is a simple React app built with Next. All pages are statically generated. Let me know if there's any more questions! [0] https://typesense.org/ https://typesense.org/
    • qchris 5y ago
      Any plans on open-sourcing the code? I'm not sure if your intention is to build a business using it (or, if you were, using AGPLv3 might help prevent third-parties from unfairly competing with you), but I'm sure a number of people would be interested in trying to run this on their own hardware, building their own personal index, hacking on it to add features they find interesting for themselves, or otherwise just learning something by taking a look under the hood (I'm probably in this category myself).
      • 5y ago
• NAR8789 5y ago
  I dig this! Since you're full-text indexing blogs with an eye towards content discovery, can I tempt you towards building out a reverse link index function to enable me to browse by thread? My use case: often times blogs will respond to other blogs, linking to the original post in the process. The nature of linking means it's very easy to follow threads backwards in time, but given the original post it's often hard to discover the responses and ongoing conversation to follow things downstream. I'd like that downstream browsing to be easier. My hope would be that such a tool could unlock higher-quality discourse. As a reader, this would let me hijack my natural tendency to follow comment threads, and redirect that attention towards slower-paced, more nuanced, more focused writing. Edit: hmmm... though looking further, maybe this goes against your MarketRank philosophy.
  • dbrereton 5y ago
    I totally agree with you on this. Being able to follow the links easily in either direction would make it easier to fall into interesting rabbit holes. > Edit: hmmm... though looking further, maybe this goes against your MarketRank philosophy. It doesn't at all, but curious as to why you'd think that. We're not talking about using backlinks to rank pages after all, just as a discovery tool, which I think is great.
    • NAR8789 5y ago
      Ah, glad to hear it's not in conflict after all! Thanks for clarifying. As to the source of my confusion... I think I was skimming and about to turn back to work and not thinking too deeply. At the time I wrote that edit I had just gotten as far as "pagerank values links whereas marketrank values upvotes" (heavily paraphrased interpretation of 2.1 here: https://dkb.io/post/market-rank https://dkb.io/post/market-rank). My reaction was "oh, maybe that means links are bad... that's an interesting perspective but I don't have time to digest it right this moment. I'll put a pin in it and return to this article later to better understand, and hey I should add an edit to my comment to signal reading comprehension." That last thought seems pretty ironic in retrospect.
• ropeladder 5y ago
  This is a great idea! It's very tech centric, though (at least judging from your directory).
• polote 5y ago
  This is cool and the quality of content is great too especially to get the most known blogs of a topic (and I feel like the quality of content is better than all the blogs search engine I have seen). But I don't feel like manual curation by one person is easily compatible with search engine. To me the content of your website is more suited to a weekly newsletter or something like that. Because after trying a few search "getting a job in vc", "best computer chair", "learning erlang" I'm not confident this answer better results than Google. You've got a content size problem as you are manually curating, and this will lead to people not use your search as a default, and probably not use it as a search engine, but instead as a discovery system. You can also try to get more blogs on your search engine, and create a community around it, if you want more more, you can follow this newsletter [1] and you will get probably 5 new blogs per day. Congratz on the job, this is very cool [1] https://hnblogs.substack.com https://hnblogs.substack.com
  • dbrereton 5y ago
    > But I don't feel like manual curation by one person is easily compatible with search engine. I see the manual curation as more of a temporary measure in the beginning. There are various ways blog detection can be automated and scaled, but manual curation for now gives me a better understanding of the data, and ensures I don't run into random edge cases. > Because after trying a few search "getting a job in vc", "best computer chair", "learning erlang" I'm not confident this answer better results than Google. Right now it's more useful for very broad queries like "inflation" or "covid". The index is pretty small at the moment, but the more posts that get added to the index, the more specific queries we'll be able to find good results for. > You've got a content size problem as you are manually curating, and this will lead to people not use your search as a default, and probably not use it as a search engine, but instead as a discovery system. That's actually what I want! This is not a search engine to replace Google, it's a discovery tool for blog posts. Thanks for all the feedback here, and will definitely check out the newsletter.
• recuter 5y ago
  Would you mind sharing some stats on the index? Are you populating it with manual curation?
  • dgivney 5y ago
    There are currently 900 blogs in the index. Every blog is manually reviewed, so this number will grow slowly and steadily over time, until I maybe automate things. https://blogsurf.io/about https://blogsurf.io/about
    • NoMAD76 5y ago
      I will advice against any automation for the foreseen future of your project. Having a few guys (volunteers) helping with manual review shouldn't be so hard to find.
      • egberts1 5y ago
        Do not automate, not in this early stage, yet.
• codazoda 5y ago
  I love the idea, but I couldn’t find any results that didn’t look mostly random. I thought I’d look for posts about shooting or editing video. Couldn’t find anything close, no matter how I ordered my query.
  • dbrereton 5y ago
    Ah yeah, there's probably not any blogs that talk about video editing. The index is fairly small right now and does better for tech/business/politics queries at the moment. Will work on increasing the index.
• ChrisArchitect 5y ago
  What year is it? Don't hate blogs and happy for resurgence, but repeating an uphill battle with indexing like it's 2007. Also, random interesting posts on front page are like from 2009, 2011, 2015...... What? That's the freshest more relevant content?
  • kingryan 5y ago
    yeah, reminds me of working at technorati in 2005
  • Minor49er 5y ago
    > Also, random interesting posts on front page are like from 2009, 2011, 2015...... What? That's the freshest more relevant content? Why would you expect a section titled "Random Interesting Posts" to have the freshest more relevant(?) content?
  • dbrereton 5y ago
    Right now the "random interesting posts" are a random selection from the top 1000 of all time. However, if you want fresher content, you can use the date range selector and set it to "Past Week" for the best posts of the week.
  • marginalia_nu 5y ago
    Dunno, I typically find fresh and interesting mutually exclusive.
  • ChrisArchitect 5y ago
    Freshest as in more recent and not outdated, relevant content. An index of blogs being built now doesn't need to be sharing old essays from 2009, if they're even indexed for whatever weird reason. Newer, updated thoughts (that may reference prior materials, sure), fresher. Instead of same old posts being shared over and over from forever ago(esp in tech when things change so much/evolve)
• valdect 5y ago
  That's an awesome implementation and works really neat. I have been thinking to add this capability to https://refined.blog/ https://refined.blog/ . also if you need tagged blog sites you can use our bloglist. also i previously posted in hn so there are some good blogs in here ( https://news.ycombinator.com/item?id=27973836 https://news.ycombinator.com/item?id=27973836 )
  • dbrereton 5y ago
    Thanks! And that's awesome, I could definitely use more blogs, will check it out.
• applgo443 5y ago
  Interesting. How do you figure out which are blogs and which arent'?
• u2077 5y ago
  Cool Idea, I love search engines with content made by real people. I’m not sure how many of these you have, but you might be able to pull some more blogs from https://bloggingfordevs.com/trends/ https://bloggingfordevs.com/trends/ or https://blogdb.org/blogs https://blogdb.org/blogs
  • dbrereton 5y ago
    Some good sources, will check them out!
• SecurityLagoon 5y ago
  I love this. I am always on the lookout for material written by individuals; but, it's surprisingly hard on the modern web. Tbh I'll probably use the random bit more than search but definitely going to keep checking back to pad my RSS feeds with interesting content.
  • dbrereton 5y ago
    > Tbh I'll probably use the random bit more than search That's interesting to hear, and fits well with the goals of the site. I want it to be more of a "discovery engine" than a "search engine". Search is one path to discovery, random posts are another, there are probably more. One thing I'm thinking of adding is the ability to easily see the blog posts that any given post links to. If you see an interesting post, you could pull up everything that may be related. > definitely going to keep checking back to pad my RSS feeds with interesting content. Sadly not every blog has RSS, and many RSS feeds are incomplete. Another thing I would like to build is auto-generated RSS feeds for all blogs, which would also make it easy for people to programmatically parse any blog and do interesting things.
• pete_nic 5y ago
  This is great. As a frequent Google user, I breathed a sigh of relief seeing ad free search results from individuals. Curious - how do you know whether a site is a blog versus something else?
  • stanislavb 5y ago
    He’s sheared in the blog post - manual review of all blogs
• Minor49er 5y ago
  The tags are pretty limited and have some duplicate entries. Submissions also require a tag for them to be posted, so if they don't fit what's already there, a wrong one will have to be supplied.
  • dbrereton 5y ago
    Yeah tagging is definitely a weak point as the current focus is on search. I just removed the tag requirement. If you're submitting a blog and we don't already have a tag that fits, you can add it to the "Notes" section.
    • Minor49er 5y ago
      Thank you for making that change so quickly
• shortformblog 5y ago
  Dmitri showed this to me a couple of weeks ago, and I was super-impressed, enough so that even though he sent me a note about it at the end of the night, I stayed up to respond to him. This makes me feel like the spirit of Technorati has a chance of making a comeback someday.
• rmason 5y ago
  Potentially quite useful. But I ran into one snag. I searched on my friend and frequent blogger Ben Nadel. But at the top all the posts were about Angular. What I wanted were all his posts that weren't about Angular. So I tried adding -angular which works in Google. It pulled up one non-angular post and all the rest were the original ones that are there when you load the page. Add that one feature and I will probably use it a lot.
  • dbrereton 5y ago
    Currently don't support any query operators, but yeah it would be very useful. Will add that functionality soon.
• rambambram 5y ago
  Really like it, good job! Nice color scheme, font choice, and elegant layout. One little thing though: changing a search phrase or word and doing a new search, I notice the results do change, but there's no way to know if it really happened. Changing a Google search, the whole page flashes empty, that way I see/sense there's something new. In your case, a change is subtle, very subtle, too subtle. In one instance I had to look carefully to see the change in results. Maybe adding a "you searched for X" is good enough, but I guess you can come up with a better way.
  • dbrereton 5y ago
    Thank you! > changing a search phrase or word and doing a new search, I notice the results do change, but there's no way to know if it really happened. Changing a Google search, the whole page flashes empty, that way I see/sense there's something new. In your case, a change is subtle, very subtle, too subtle. In one instance I had to look carefully to see the change in results. That is a good point. I did try to remove any loading indicators because I thought it would be smoother, but maybe it's a bit too smooth for people to realize their search went through. Will think more on how to fix this.
• derekzhouzhen 5y ago
  I agree with everything you said, except the popularity ranking. The value of a content shall be in the content itself; popularity is only a flawed measurement. Worse, popularity has very strong positive feedback that contributes to the great polarization of opinions.
  • dbrereton 5y ago
    > The value of a content shall be in the content itself; popularity is only a flawed measurement. Popularity is certainly a flawed measurement, but it's hard to come up with a scalable way to determine quality that isn't flawed in some way. Instead of being flawed by encouraging people to get tons of backlinks, this is flawed by encouraging people to do stuff that gets lots of upvotes. Very open to more ideas on how to measure quality.
    • derekzhouzhen 5y ago
      "a scalable way to determine quality" is the billion dollar question so I have no idea. I'd just use plain old text index with no algorithmic ranking.
• azhenley 5y ago
  This is awesome! I see some blog posts of mine already on here but using an outdated URL. Are there any plans to check for redirects and update the URL or to recrawl?
• COil 5y ago
  Excellent idea, I was thinking of creating something similar. My new homepage, for sure.
• drBonkers 5y ago
  Many search-engine posts recently. When will someone make the Search Engine for Search Engines?
  • marginalia_nu 5y ago
    I do think some form of collaboration between small search engines would be very beneficial. I've been thinking about how to make that happen. So far I've added a public API to my search engine, and published some data. Not sure what is a good way of creating a space for collaboration...
    • 1vuio0pswjnm7 5y ago
      I created a metasearch for myself based on the idea of "continuation searches". One obvious point of collaboration bwtween search engines could be a uniform API and SERP format. Currently, there are slight variations between search engines in terms of the submission URL syntax/paramaters and the HTML used to display results, not to mention HTTP method, limits on number of results and sometimes additional, optional URL parameters. The differences are generally small^1 and this makes it relatively easy to create a personal metasearch. However it could be much less cumbersome if these differences were eliminated. 1. Exceptions are, e.g., ones that require two HTTP requests per query, such as Gigablast or ones that have strange limitations, e.g., Startpage, which has become unusable for me without Javascript. Contacting their "customer support" yielded no response. Even better would be if search engines all shared their indexes and made these available for download. This would faciltate people building new search engines without needing to have their own index. In theory it would also bring a stop to the problem of people who submit large numbers of queries since all the bulk data they need would be available for download. www indexes that comprise public information could be freely shared as public data.
      • marginalia_nu 5y ago
        An index is, lowballing it, hundreds of gigabytes of dense binary soup; probably in some custom format specific to that search engine (sometimes there's some form of hash table going on, sometimes a B-tree), almost certainly with its own idiosyncrasies concerning keyword extraction. I think reconciling API differences is probably a lot easier than making use of index data.
• mr90210 5y ago
  Great idea mate. Keep up the good work.
• stringlytyped 5y ago
  You might want consider using OpenSearch [1] to make it easier to add Blog Surf to browsers as a search engine that can be accessed from the location bar. I added it manually in Firefox but it would have been handy to just be able to right-click the search field and choose "Add a Keyword for this Search". [1] https://developer.mozilla.org/en-US/docs/Web/OpenSearch https://developer.mozilla.org/en-US/docs/Web/OpenSearch
  • dbrereton 5y ago
    Thanks for the tip, just added an OpenSearch xml.
• akselmo 5y ago
  I love the flaming comic sans Directory header, lol
• stanislavb 5y ago
  I tried searching for "saashub", and non of the results had a single mention of that term. Do you know what is the reason? Some stemming?
• ahmadrosid 5y ago
  I love this, it would be cool if we can submit blog to be indexed. Of course you can review first to avoid spam.
  • eatonphil 5y ago
    There's a link to exactly that in the header of the site: https://blogsurf.io/submit https://blogsurf.io/submit.
• throwawaylala1 5y ago
  Love the idea, but I can't search for exact matches using quotes :(
  • dbrereton 5y ago
    query operators coming soon
• jwood27 5y ago
  This is great! Tangentially related - any maybe an interesting way to view the network you have built up - I put together a quick d3.js force visualization of the "blogrolls" for ~300 linked blogs visible here: https://jacobwood27.github.io/035_blog_graph/ https://jacobwood27.github.io/035_blog_graph/
  • dbrereton 5y ago
    This is really cool, it would be awesome to make a graph of the blogosphere like this. I'd love to chat about this more, will send you an email.
• davepeck 5y ago
  I love seeing new search engines and this appears to be very well done. The web needs better visibility into its blogs. Bravo! I wonder about the use of MarketRank. For instance, search for "COVID" and your top hit will be from Alex Berenson, a well-known purveyor of outright COVID misinformation. Is this post "interesting"? Yup. Is it trustworthy? Absolutely not.
• escapedmoose 5y ago
  Yes! Death to SEO! Really great tool; thank you for sharing.
• bspear 5y ago
  This is cool! how are you thinking about attracting users (besides HN)
• asicsp 5y ago
  Great to have full-text search option! I used to browse latest posts using https://blogsurf.io/posts https://blogsurf.io/posts link, is that functionality not available anymore?
  • dbrereton 5y ago
    Oops, yeah I meant to re-add that. Will get a recent posts page up soon.
• jedwhite 5y ago
  This is really awesome! I love that you're curating the sites included, and it shows in the quality of the results. The world needs more specialized searches like this, and you've done a brilliant job with implementation. I also really love that you have a directory. I'm going to play with the API, and that's awesome you've made that available. [Disclaimer: also working on a new search engine, and would love to include results from this!]
• markdown 5y ago
  I searched using a keyword (kava), and got a list of random blog posts with absolutely nothing to do with kava. https://blogsurf.io/?query=kava https://blogsurf.io/?query=kava
  • dbrereton 5y ago
    The simple and unfortunate explanation is that the index is just not that big right now (only 900 blogs). Working on increasing it significantly, but will take some time. Try it again in a month and you may find it more useful. Right now it's mostly filled with tech, business, and politics.
    • markdown 5y ago
      Thanks. I figured that, but perhaps a message saying that no results were found would be more useful than showing random blog posts. Or maybe show that message, along with "... here are some interesting posts from the past month..."
• mgarfias 5y ago
  We, sphere.com, did this starting in 2006. After a year or so, we realized the only people using the service were looking to stroke their egos. Ice rocket, and something else (I can’t remember the name) tried it at the same time and failed. We pivoted, which ended up leading to some unspeakable horrors. At any rate, good luck, hope it works better for you.
  • semitones 5y ago
    The blogosphere was certainly very different in 2006 than it is now - perhaps it's worth another go?
    • dbrereton 5y ago
      My understanding is that the early days of blogging were mostly people sharing personal things, and now that has moved to social media. A lot of the online writing happening now is essays on some topic, or people trying to share notes on things they learn. And I think this type of writing is more conducive to a project like this.
• Kye 5y ago
  This is neat, but it's quietly doing some kind of fuzzy matching with my query. https://blogsurf.io/?query=furries https://blogsurf.io/?query=furries I actually have no idea what it's really searching for. Only one of the first ten results contains my query. Compare to HN's search which highlights the matched words so it's at least clear when it's going off-script. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=furries&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... Quotes provide an exact match here, but not on your search engine.
  • dbrereton 5y ago
    Yeah I think the bigger issue here is not highlighting the matched text, which makes it look like it's doing something totally random. There is some typo tolerance, so it's hard to tell exactly what is matching on those pages. I'm working on adding query operators so you can specify things like exact matching, and in general working on improving the results. This type of feedback is very helpful, so thank you!
• marban 5y ago
  For business news articles: https://yup.is https://yup.is
• ICodeSometimes 5y ago
  This is dope! Wondering how you determine the number of blog points a certain post gets? Is there a blog quality score that's programmatically determined? How so? It seems you are planning to introduce automation instead of manually reviewing every submission, would be interesting if you could crawl via links from blog to blog. I also feel you could likely just type in a url and postfix /blog to it to get a niche blog on some topic. Not sure if that's too simple but seems like it might work as a v0 for your automation.
• heywoodlh 5y ago
  This is a super cool product! If there was some way on blogsurf to have RSS feeds per category I'm sure that would make my RSS feed curation much easier, random and interesting. I.E. subscribe to all the blogs labeled in cybersecurity, linux, etc. Or maybe this functionality is already present and I didn't see it (I saw the RSS feeds per blog). Unrelated: it was interesting to see my blog listed on the site. Kind of surreal that someone finds my content useful and/or interesting. Very motivating and humbling.
• ilrwbwrkhv 5y ago
  Bookmarked. Going to use it. Now if someone would just make a better Reddit search. And then another one for high value properties like mayo clinic, wikipedia, GitHub etc, I will not need to use Google anymore.
• dustinmoris 5y ago
  The search algorithm seems to be extremely poor. I searched for the exact blog post title of some very good blog posts which in other search engines end up in the top 5 if you search for their topic and on blogsurf it didn’t look like it appeared in the results at all. That’s very very weak, especially the first 10 results had nothing to do with the topic I searched for. It was just super high profile websites which had remotely mentioned a few of the keywords.
  • camel-cdr 5y ago
    I think the problem might be that the blogs aren't added yet. I also couldn't find some quite popular blogs, but I submitted them, so hopefully this will change in the future.
• camel-cdr 5y ago
  I encourage all of you to submit the blogs you are following. Only one of the ten blogs I'm following was supported. Even quite well known once like oldnewthings or lemir's blog.
• pcthrowaway 5y ago
  Very excited to see this! I noticed some of my favourite bloggers don't appear to be indexed for whatever reason: Josh Comeau, Julia Evans, Amelia Wattenberger. Any idea why these aren't indexed and if you plan to add them? I wonder if you could get a list of some of the most popular blogs on HN (perhaps the maintainer of upvotetracker can help) to add to the index.
• hidden-spyder 5y ago
  Where can I see an index of all posts?
• davidgerard 5y ago
  Google used to have a good blog search. Biggest problem was SEO spam blogs - but even these used to fall down the rankings. Sadly it died of Google deprecating an API. https://en.wikipedia.org/wiki/Google_Blog_Search https://en.wikipedia.org/wiki/Google_Blog_Search