10 ms·

Show HN: Hndex.org – a full-text search engine of articles submitted to HN

442 points by mcovalt 6y ago

mcovalt 6y ago
I spend a lot of time trying to find articles I once read on here via Google with... site:news.ycombinator.com some term ...but a many times "some term" is not in the title of the HN post, but in the body of the article. I didn't see any other tools that did this for HN.
- cxr 6y ago
  Consider expanding the scope to index links posted in user comments, too.
- walterbell 6y ago
  There's also Algolia, https://hn.algolia.com https://hn.algolia.com
  - tomhoward 6y ago
    It doesn't index the article contents though; only the story titles and comments, same as Google.
    - city41 6y ago
      It does seem to have a much larger index though, which for me at least makes it more useful. Hopefully hndex's index will grow.
    - 1vuio0pswjnm7 6y ago
      How did you come to this conclusion with respect to hndex? Most articles have no comments. Yet one can search on hndex for terms only found in the article, not in the title.
      - tomhoward 6y ago
        I wasn't really saying anything about hndex, apart from that it indexes the contents of external articles submitted to HN, as you said. The correction I was making was about HN Algolia search (which is linked from the search box at the bottom of each HN page), which only indexes content on news.ycombinator.com itself – i.e., article titles, comments and text-only posts like Show/Ask HN – but not the external content in submitted articles.
- owenversteeg 6y ago
  Yes, me too! This is really great, thanks! I'll definitely be using it.
- danielskogly 6y ago
  This is really useful! You might want to consider hiding the “More”-button if the current page isn’t filled up, so as to not just have an empty page when clicking it. For some reason I couldn’t find my own blog post[0], even when searching for the embarrassing typo I made in the title - acommodating. [0] https://news.ycombinator.com/item?id=23426951 https://news.ycombinator.com/item?id=23426951
- browsergap 6y ago
  what are you using to do the full text search?
- lhoff 6y ago
  Very nice. Works as expected. I stumbled about two things that could be improved: - Add a search Button for convenient mobile use or if a user copy pasts things into to the search field using the mouse - Add a comment counter on the result page, Since you index every article a lot of them have none or very few comments. Oh and just a warning, depending on the jurisdiction providing the cache could be problematic under copyright laws since its basically a copy of the article.
- run2arun 6y ago
  For that specific use case, you can try the intitle keyword site:news.ycombinator.com intitle:"some term"
BlackLotus89 6y ago
Ohhhh I first thought this was a search engine for hacker news and I was wondering why the hell anyone would reimplement this, but this is fricking cool. Tbh I wanted to implement something similar for more than hackernews, but this is a nice thing to have. I especially love that you implemented a cache :) Seems nice. Sort and filter functionality would probably add to this, but I will bookmark this for sure and try it out as a search engine for tech topics in general. Thanks for creating this!
edjrage 6y ago
What does this offer over https://hn.algolia.com https://hn.algolia.com? (Algolia was backed by YC btw)
- thsowers 6y ago
  Looks like this search engine searches the _articles_ and links that have been shared, whereas hn.algolia.com only searches title, author, and the text of a post if it is a text post.
1123581321 6y ago
Neat project. It’s a bit depressing how many of the article links don’t work anymore. For example, only the official signal vs noise article works on the first page of results for Basecamp: https://hndex.org/?q=Basecamp https://hndex.org/?q=Basecamp How does ranking work? Apologies if I’ve missed the explanation.
- 1f60c 6y ago
  It would be great if each result also had [archive] and [web] links like HN. > How does ranking work? Great question, it's something I'd love to know more about as well.
  - aspenmayer 6y ago
    It would be great if that was native HN functionality. Why isn't it? It seems like it would only help, and how hard could it be?
    - 1f60c 6y ago
      They easily could (I think it would take the average HN user no more than a couple of hours to implement this functionality), but they won't, because they're very reluctant to make changes. This is also why HN still looks like a site from the 90s, instead of New Reddit (thank God).
      - aspenmayer 6y ago
        It’s not all bad. We still have HN as it is now. Status quo is its own kind of success. HN is essentially feature complete, so I don’t blame them for not trying to fix what ain’t broke.
- codetrotter 6y ago
  Articles which have been posted to HN often end up in the archives of Archive.org and Archive.is in my experience. The OP site also has a “cached” link for each article, don’t know if you saw that. Also, to the maker of HNdex, consider adding links to Archive.org and Archive.is next to the cached link you have, so that readers can click through and check if they have a version of it in case there’s images etc
benatkin 6y ago
That's awesome! I searched for MuleSoft, not expecting anything in particular, and found a random fact (at the end) that certainly wasn't in the title of the article. Bug report: in Safari on dark mode, the text in the search box is almost the same color as the background (white on white). From "GitHub was also talking to Google about a deal, but went with Microsoft instead" https://www.cnbc.com/2018/06/05/github-interest-from-google-and-others-revenue-about-300-million.html?__source=twitter%7Cmain https://www.cnbc.com/2018/06/05/github-interest-from-google-... - Salesforce got MuleSoft for $7.5 billion, Microsoft got GitHub for $7.5 billion.
tyingq 6y ago
Well done. One nit...if you enter something it doesn't like, the error that pops up isn't helpful: "Please match the requested format"
disqard 6y ago
Beautiful! Please consider adding a dark mode option.
- 1f60c 6y ago
  Hm, dark mode is working for me (iPhone, iOS 13.6)
- airstrike 6y ago
  I don't know why but to me the background is #000 black with #fff text by default... It's such an intense "dark mode" that I couldn't read the first full article of interest I found because the contrast was killing me, and then coming back to regular HN left me nearly blinded for a few secs
1f60c 6y ago
This is neat! - What is your stack? - How often is your database updated? - And will it be open source in the future?
crazypython 6y ago
I'm worried making it public will encourage companies to "SEO"ify HN.
ALittleLight 6y ago
I love this. The search box had an option to autocomplete terms I had recently searched, I clicked on one (a term I was interested in, obviously) and immediately found an interesting HackerNews post. Nice! I think HackerNews and other link aggregators (e.g. reddit) have a kind of recency problem, where there is a lot of great content, but people only see the recent stuff. This seems like a great way to uncover some of the latent value of old HackerNews content. Is there a way to suggest content too? e.g. If you liked that, you'd probably also like X, Y, and Z.
- MattGaiser 6y ago
  >I think HackerNews and other link aggregators (e.g. reddit) have a kind of recency problem, where there is a lot of great content, but people only see the recent stuff. With Reddit and HackerNews, I want a relative ranking index. I can search by the top content, but something 5th today could have more votes than the top submission of 2015 because of forum/subreddit growth. I want them ranked by something like the ratio of views to upvotes or upvotes compared to total upvotes for that day.
  - rhaksw 6y ago
    > I want them ranked by something like the ratio of views to upvotes or upvotes compared to total upvotes for that day. Yeah, this is definitely possible with public data. I did something similar on reveddit [1] for removed reddit content. Hovering over the graph shows the item with the highest vote ratio [3], and clicking skips to that point in time. Code is here [2] for anyone interested and I apologize in advance.. [1] https://i.imgur.com/p3Bi5IS.png https://i.imgur.com/p3Bi5IS.png [2] https://github.com/reveddit/ragger/blob/master/revddit_aggregator.py https://github.com/reveddit/ragger/blob/master/revddit_aggre... [3] https://www.reveddit.com/r/worldnews/?showFilters=true https://www.reveddit.com/r/worldnews/?showFilters=true
  - nwienert 6y ago
    I want a map-reduce over their content that turns them into a canonical “master guide” of their topics.
  - 6y ago
nelsonenzo 6y ago
can i sort by date? i searched for kubernetes and the first page results were from 2016.
- nmstoker 6y ago
  Yes, with so much context missing from the results it's pretty confusing. Ideally for this to be useful I'd want date on HN, date of article (although I see that'd perhaps be hard to extract from unstructured pages) and also the HN points, as those are a major proxy to quality usually. As others have said it has a nice crisp UI and i like that it's so quick.
breakfastduck 6y ago
This produces such a unique set of results for things. Fantastic project and well worth creating! As another commentor has noted - this totally disobeys recency bias and throws up interesting articles for a topic. Edit - it is disappointing how many of these links 404. But even if that's the case the headline and intro is a set of time capsules of sorts nonetheless.
kody 6y ago
So the first term I searched was "cognitive". Search for it yourself, and click the first result. Go back and click "cached". That was a cool experience.
- freediver 6y ago
  I wonder how cached results are created and is there some kind of copyright violation because they store the content.
  - JoshuaRLi 6y ago
    I suspect it might be something mentioned in https://github.com/masukomi/arc90-readability https://github.com/masukomi/arc90-readability. I believe it's what powers "readability" view in modern browsers (firefox https://github.com/mozilla/readability https://github.com/mozilla/readability, safari).
wh-uws 6y ago
this is great whats the stack / architecture you built this with? Would love some insights. I often find myself searching HN for opinions on different technologies over time and this will be invaluable for that use case.
noseratio 6y ago
Very cool! How frequently does it index the HN content? I've just tried searching for DevComrade which I submitted 14hrs ago and nothing shows up.
city41 6y ago
What is the criteria for an article getting into the index? To test it out I tried several of my submissions and could only find one in the index.
z3t4 6y ago
It's fast and light weight! What stack are you using? My only wish is that it should be possible to sort chronologically.
asdf333 6y ago
awesome! would be great to have a submissiondate/votes/num comments/link to the hn submission in the results!
ksangeelee 6y ago
I wrote something similar to this a year ago (though shabby looking by comparison!): - http://kakapo.susa.net:8080/cfs/ http://kakapo.susa.net:8080/cfs/ I abandoned it when Google deprecated the WebRequest API, but the code's still available on GitLab. https://gitlab.com/ksangeelee/cfs_build https://gitlab.com/ksangeelee/cfs_build It allows article score and uBlock Origin 'hits' as search criteria.
- freediver 6y ago
  Very cool implementation. I am wondering have you considered using something like https://github.com/cliqz-oss/adblocker https://github.com/cliqz-oss/adblocker which can sit on top of headless browser and do not require bridging to an extension.
tucif 6y ago
So many articles found that are gone from the original sites, but still readable from the cached version, that's pretty useful!
gkhartman 6y ago
I would love to see the source and/or a write up about the stack used here. I've thought about making something similar, but got distracted while in the planning phase.
reactchain 6y ago
This is fantastic. Way better than the Algolia search which seems to be the most "official" one. I love that you cached the article content too.
IncludeSecurity 6y ago
Tried to find an article from earlier this week, no dice. https://hndex.org/?q=open%20source%20security https://hndex.org/?q=open%20source%20security https://hndex.org/?q=openssf https://hndex.org/?q=openssf Great idea, I appreciate how fast it returns results! Just needs more more control over the search parameters and figuring out why articles like the example I posted above aren't working and you got yourself a nice HN search.
aabbcc1241 6y ago
How does the "full-text search" work? I tried to search my previous post[1] but no luck, even tried the exact title. [1] https://news.ycombinator.com/item?id=22830472 https://news.ycombinator.com/item?id=22830472
sriram_malhar 6y ago
Love it, love it! Thank you. Is it possible to give a little inside peek of the techniques, algorithms and tech stack etc. you have used?
arawde 6y ago
I was just telling a friend about a post I had read about buying cheap tools to start out. I couldn't find it at all googling, and the HN search didn't work. I saw this post, searched "cheap tools quality", and immediately found it. Tnank you! The post in question: https://www.johndcook.com/blog/2020/07/25/worst-tool-for-the-job/ https://www.johndcook.com/blog/2020/07/25/worst-tool-for-the...
- haddr 6y ago
  where actually the best happens in the comments under hn submission to this article (in short: the claim seems not valid for certain domains)
NOGDP 6y ago
Nice tool, though I tried a few searches, and a few searches of <same text> + ' hackernews' on google, and I gotta say I like the google results better. Search options for comments/titles/users, date range, ask/show/jobs, sorting by date/votes (weighted?), etc would be nice additions.
zerop 6y ago
Thanks for making it. great resource. Two things: 1. Show dates in results. 2. Are results better than searching on Google with "site:news.ycombinator.com"
- rustamm 6y ago
  I think it indexes the content of articles referenced on hachernews which makes it different from Google with "site:news.ycombinator.com".
mindhash 6y ago
I would love to search books, video recommendations that people post on HN. It's one of the cool things I like here
- smoe 6y ago
  There is this site / newsletter for books mentioned in hn comments https://hackernewsbooks.com/ https://hackernewsbooks.com/ There are much fewer books in the last six or so weeks editions. Don't know why this is. Whether it is some scraper not working, or actually no books being mentioned.
arc-in-space 6y ago
Holy crap this is fast! At first I was mildly disappointed it wasn't a live search like hn.algolia.com, but I was seriously not expecting the results page to load so quickly. I guess goog et al. trained me to expect garbage latency in internet search forms. This seems to be a good complement to algolia, given that it searches through the linked pages instead of comments. Minor nitpick: would it be possible to make it give a 'past' link, to search for all discussions on a result? Some of the 'comments' take you to duplicate posts with no comments instead of the more popular cases.
abhayhegde 6y ago
How does a site which can index text inside a document usually work? Is it similar to Linux grep?
omarchowdhury 6y ago
I searched for "red light therapy" and none of the articles matched, not even the most recent article on red light therapy that was on the homepage. Same for "red light".
visarga 6y ago
May I suggest you add time as well to the search results? It's super relevant in the context of news and technical articles.
dennisy 6y ago
Would you be willing to provide an API for this service?
chvid 6y ago
What stack did you use to make this? Ps: show the date / age of the posts on the results page
Kelamir 6y ago
I love the option to view cached versions of websites. This is a nice project. Thank you, Mcovalt.
Watchnerd 6y ago
this is really cool, now I can filter by topics I really care about, well done!
edent 6y ago
Found a bug! Searched for my name and found a bunch of unrelated BoingBoing links. https://hndex.org/?q=Terence+Eden https://hndex.org/?q=Terence+Eden Why? Because those old posts have a "what's new" set of links. One of which contains my name. I'd suggest only searching the `<main>` element, perhaps?
smhmd 6y ago
Could you add post points and how much time since/when was the article posted? It would help as some topics do not age as well as others. Having sorting (most popular, most recent) and range search (last year, custom range) would also help those who want to narrow their search.
baccredited 6y ago
Wow - would love an RSS feed feature to subscribe to a search
antihero 6y ago
This is absolutely fantastic, is there any way to display the dates on the cached version and/or result?
throwawaysea 6y ago
Does this work with articles behind a paywall?
maddyboo 6y ago
Here we are in 2020, and web search is not what I’d hoped it would be. I would love to see more niche, small search tools like this rather than better general-purpose search. Building upon that, maybe an aggregator of search results from many small niche search engines. Actually, just writing that sentence reminded me of Searx [0], which I have to admit I haven’t tried in earnest but really should. 0: https://searx.me/ https://searx.me/
sarosh 6y ago
Hi Matt, can you provide any details on the code/backend?
adaszko 6y ago
There's also Falcon[1] Chrome extension which does full text indexing on your browser history so if you read something and can't either Google it or find it in the browser's history, Falcon will broaden the search scope. [1] https://chrome.google.com/webstore/detail/falcon/mmifbbohghecjloeklpbinkjpbplfalb?hl=en https://chrome.google.com/webstore/detail/falcon/mmifbbohghe...

442 points by mcovalt 6y ago

I spend a lot of time trying to find articles I once read on here via Google with... site:news.ycombinator.com some term ...but a many times "some term" is not in the title of the HN post, but in the body of the article. I didn't see any other tools that did this for HN.

cxr 6y ago
Consider expanding the scope to index links posted in user comments, too.
walterbell 6y ago
There's also Algolia, https://hn.algolia.com https://hn.algolia.com
- tomhoward 6y ago
  It doesn't index the article contents though; only the story titles and comments, same as Google.
  - city41 6y ago
    It does seem to have a much larger index though, which for me at least makes it more useful. Hopefully hndex's index will grow.
  - 1vuio0pswjnm7 6y ago
    How did you come to this conclusion with respect to hndex? Most articles have no comments. Yet one can search on hndex for terms only found in the article, not in the title.
    - tomhoward 6y ago
      I wasn't really saying anything about hndex, apart from that it indexes the contents of external articles submitted to HN, as you said. The correction I was making was about HN Algolia search (which is linked from the search box at the bottom of each HN page), which only indexes content on news.ycombinator.com itself – i.e., article titles, comments and text-only posts like Show/Ask HN – but not the external content in submitted articles.
owenversteeg 6y ago
Yes, me too! This is really great, thanks! I'll definitely be using it.
danielskogly 6y ago
This is really useful! You might want to consider hiding the “More”-button if the current page isn’t filled up, so as to not just have an empty page when clicking it. For some reason I couldn’t find my own blog post[0], even when searching for the embarrassing typo I made in the title - acommodating. [0] https://news.ycombinator.com/item?id=23426951 https://news.ycombinator.com/item?id=23426951
browsergap 6y ago
what are you using to do the full text search?
lhoff 6y ago
Very nice. Works as expected. I stumbled about two things that could be improved: - Add a search Button for convenient mobile use or if a user copy pasts things into to the search field using the mouse - Add a comment counter on the result page, Since you index every article a lot of them have none or very few comments. Oh and just a warning, depending on the jurisdiction providing the cache could be problematic under copyright laws since its basically a copy of the article.
run2arun 6y ago
For that specific use case, you can try the intitle keyword site:news.ycombinator.com intitle:"some term"

BlackLotus89 6y ago

Ohhhh I first thought this was a search engine for hacker news and I was wondering why the hell anyone would reimplement this, but this is fricking cool. Tbh I wanted to implement something similar for more than hackernews, but this is a nice thing to have. I especially love that you implemented a cache :) Seems nice. Sort and filter functionality would probably add to this, but I will bookmark this for sure and try it out as a search engine for tech topics in general. Thanks for creating this!

edjrage 6y ago

What does this offer over https://hn.algolia.com https://hn.algolia.com? (Algolia was backed by YC btw)

thsowers 6y ago
Looks like this search engine searches the _articles_ and links that have been shared, whereas hn.algolia.com only searches title, author, and the text of a post if it is a text post.

1123581321 6y ago

Neat project. It’s a bit depressing how many of the article links don’t work anymore. For example, only the official signal vs noise article works on the first page of results for Basecamp: https://hndex.org/?q=Basecamp https://hndex.org/?q=Basecamp How does ranking work? Apologies if I’ve missed the explanation.

1f60c 6y ago
It would be great if each result also had [archive] and [web] links like HN. > How does ranking work? Great question, it's something I'd love to know more about as well.
- aspenmayer 6y ago
  It would be great if that was native HN functionality. Why isn't it? It seems like it would only help, and how hard could it be?
  - 1f60c 6y ago
    They easily could (I think it would take the average HN user no more than a couple of hours to implement this functionality), but they won't, because they're very reluctant to make changes. This is also why HN still looks like a site from the 90s, instead of New Reddit (thank God).
    - aspenmayer 6y ago
      It’s not all bad. We still have HN as it is now. Status quo is its own kind of success. HN is essentially feature complete, so I don’t blame them for not trying to fix what ain’t broke.
codetrotter 6y ago
Articles which have been posted to HN often end up in the archives of Archive.org and Archive.is in my experience. The OP site also has a “cached” link for each article, don’t know if you saw that. Also, to the maker of HNdex, consider adding links to Archive.org and Archive.is next to the cached link you have, so that readers can click through and check if they have a version of it in case there’s images etc

benatkin 6y ago

That's awesome! I searched for MuleSoft, not expecting anything in particular, and found a random fact (at the end) that certainly wasn't in the title of the article. Bug report: in Safari on dark mode, the text in the search box is almost the same color as the background (white on white). From "GitHub was also talking to Google about a deal, but went with Microsoft instead" https://www.cnbc.com/2018/06/05/github-interest-from-google-and-others-revenue-about-300-million.html?__source=twitter%7Cmain https://www.cnbc.com/2018/06/05/github-interest-from-google-... - Salesforce got MuleSoft for $7.5 billion, Microsoft got GitHub for $7.5 billion.

tyingq 6y ago

Well done. One nit...if you enter something it doesn't like, the error that pops up isn't helpful: "Please match the requested format"

disqard 6y ago

Beautiful! Please consider adding a dark mode option.

1f60c 6y ago
Hm, dark mode is working for me (iPhone, iOS 13.6)
airstrike 6y ago
I don't know why but to me the background is #000 black with #fff text by default... It's such an intense "dark mode" that I couldn't read the first full article of interest I found because the contrast was killing me, and then coming back to regular HN left me nearly blinded for a few secs

1f60c 6y ago

This is neat! - What is your stack? - How often is your database updated? - And will it be open source in the future?

crazypython 6y ago

I'm worried making it public will encourage companies to "SEO"ify HN.

ALittleLight 6y ago

I love this. The search box had an option to autocomplete terms I had recently searched, I clicked on one (a term I was interested in, obviously) and immediately found an interesting HackerNews post. Nice! I think HackerNews and other link aggregators (e.g. reddit) have a kind of recency problem, where there is a lot of great content, but people only see the recent stuff. This seems like a great way to uncover some of the latent value of old HackerNews content. Is there a way to suggest content too? e.g. If you liked that, you'd probably also like X, Y, and Z.

MattGaiser 6y ago
>I think HackerNews and other link aggregators (e.g. reddit) have a kind of recency problem, where there is a lot of great content, but people only see the recent stuff. With Reddit and HackerNews, I want a relative ranking index. I can search by the top content, but something 5th today could have more votes than the top submission of 2015 because of forum/subreddit growth. I want them ranked by something like the ratio of views to upvotes or upvotes compared to total upvotes for that day.
- rhaksw 6y ago
  > I want them ranked by something like the ratio of views to upvotes or upvotes compared to total upvotes for that day. Yeah, this is definitely possible with public data. I did something similar on reveddit [1] for removed reddit content. Hovering over the graph shows the item with the highest vote ratio [3], and clicking skips to that point in time. Code is here [2] for anyone interested and I apologize in advance.. [1] https://i.imgur.com/p3Bi5IS.png https://i.imgur.com/p3Bi5IS.png [2] https://github.com/reveddit/ragger/blob/master/revddit_aggregator.py https://github.com/reveddit/ragger/blob/master/revddit_aggre... [3] https://www.reveddit.com/r/worldnews/?showFilters=true https://www.reveddit.com/r/worldnews/?showFilters=true
- nwienert 6y ago
  I want a map-reduce over their content that turns them into a canonical “master guide” of their topics.
- 6y ago

nelsonenzo 6y ago

can i sort by date? i searched for kubernetes and the first page results were from 2016.

nmstoker 6y ago
Yes, with so much context missing from the results it's pretty confusing. Ideally for this to be useful I'd want date on HN, date of article (although I see that'd perhaps be hard to extract from unstructured pages) and also the HN points, as those are a major proxy to quality usually. As others have said it has a nice crisp UI and i like that it's so quick.

breakfastduck 6y ago

This produces such a unique set of results for things. Fantastic project and well worth creating! As another commentor has noted - this totally disobeys recency bias and throws up interesting articles for a topic. Edit - it is disappointing how many of these links 404. But even if that's the case the headline and intro is a set of time capsules of sorts nonetheless.

kody 6y ago

So the first term I searched was "cognitive". Search for it yourself, and click the first result. Go back and click "cached". That was a cool experience.

freediver 6y ago
I wonder how cached results are created and is there some kind of copyright violation because they store the content.
- JoshuaRLi 6y ago
  I suspect it might be something mentioned in https://github.com/masukomi/arc90-readability https://github.com/masukomi/arc90-readability. I believe it's what powers "readability" view in modern browsers (firefox https://github.com/mozilla/readability https://github.com/mozilla/readability, safari).

wh-uws 6y ago

this is great whats the stack / architecture you built this with? Would love some insights. I often find myself searching HN for opinions on different technologies over time and this will be invaluable for that use case.

noseratio 6y ago

Very cool! How frequently does it index the HN content? I've just tried searching for DevComrade which I submitted 14hrs ago and nothing shows up.

city41 6y ago

What is the criteria for an article getting into the index? To test it out I tried several of my submissions and could only find one in the index.

z3t4 6y ago

It's fast and light weight! What stack are you using? My only wish is that it should be possible to sort chronologically.

asdf333 6y ago

awesome! would be great to have a submissiondate/votes/num comments/link to the hn submission in the results!

ksangeelee 6y ago

I wrote something similar to this a year ago (though shabby looking by comparison!): - http://kakapo.susa.net:8080/cfs/ http://kakapo.susa.net:8080/cfs/ I abandoned it when Google deprecated the WebRequest API, but the code's still available on GitLab. https://gitlab.com/ksangeelee/cfs_build https://gitlab.com/ksangeelee/cfs_build It allows article score and uBlock Origin 'hits' as search criteria.

freediver 6y ago
Very cool implementation. I am wondering have you considered using something like https://github.com/cliqz-oss/adblocker https://github.com/cliqz-oss/adblocker which can sit on top of headless browser and do not require bridging to an extension.

tucif 6y ago

So many articles found that are gone from the original sites, but still readable from the cached version, that's pretty useful!

gkhartman 6y ago

I would love to see the source and/or a write up about the stack used here. I've thought about making something similar, but got distracted while in the planning phase.

reactchain 6y ago

This is fantastic. Way better than the Algolia search which seems to be the most "official" one. I love that you cached the article content too.

IncludeSecurity 6y ago

Tried to find an article from earlier this week, no dice. https://hndex.org/?q=open%20source%20security https://hndex.org/?q=open%20source%20security https://hndex.org/?q=openssf https://hndex.org/?q=openssf Great idea, I appreciate how fast it returns results! Just needs more more control over the search parameters and figuring out why articles like the example I posted above aren't working and you got yourself a nice HN search.

aabbcc1241 6y ago

How does the "full-text search" work? I tried to search my previous post[1] but no luck, even tried the exact title. [1] https://news.ycombinator.com/item?id=22830472 https://news.ycombinator.com/item?id=22830472

sriram_malhar 6y ago

Love it, love it! Thank you. Is it possible to give a little inside peek of the techniques, algorithms and tech stack etc. you have used?

arawde 6y ago

I was just telling a friend about a post I had read about buying cheap tools to start out. I couldn't find it at all googling, and the HN search didn't work. I saw this post, searched "cheap tools quality", and immediately found it. Tnank you! The post in question: https://www.johndcook.com/blog/2020/07/25/worst-tool-for-the-job/ https://www.johndcook.com/blog/2020/07/25/worst-tool-for-the...

haddr 6y ago
where actually the best happens in the comments under hn submission to this article (in short: the claim seems not valid for certain domains)

NOGDP 6y ago

Nice tool, though I tried a few searches, and a few searches of <same text> + ' hackernews' on google, and I gotta say I like the google results better. Search options for comments/titles/users, date range, ask/show/jobs, sorting by date/votes (weighted?), etc would be nice additions.

zerop 6y ago

Thanks for making it. great resource. Two things: 1. Show dates in results. 2. Are results better than searching on Google with "site:news.ycombinator.com"

rustamm 6y ago
I think it indexes the content of articles referenced on hachernews which makes it different from Google with "site:news.ycombinator.com".

mindhash 6y ago

I would love to search books, video recommendations that people post on HN. It's one of the cool things I like here

smoe 6y ago
There is this site / newsletter for books mentioned in hn comments https://hackernewsbooks.com/ https://hackernewsbooks.com/ There are much fewer books in the last six or so weeks editions. Don't know why this is. Whether it is some scraper not working, or actually no books being mentioned.

arc-in-space 6y ago

Holy crap this is fast! At first I was mildly disappointed it wasn't a live search like hn.algolia.com, but I was seriously not expecting the results page to load so quickly. I guess goog et al. trained me to expect garbage latency in internet search forms. This seems to be a good complement to algolia, given that it searches through the linked pages instead of comments. Minor nitpick: would it be possible to make it give a 'past' link, to search for all discussions on a result? Some of the 'comments' take you to duplicate posts with no comments instead of the more popular cases.

abhayhegde 6y ago

How does a site which can index text inside a document usually work? Is it similar to Linux grep?

omarchowdhury 6y ago

I searched for "red light therapy" and none of the articles matched, not even the most recent article on red light therapy that was on the homepage. Same for "red light".

visarga 6y ago

May I suggest you add time as well to the search results? It's super relevant in the context of news and technical articles.

dennisy 6y ago

Would you be willing to provide an API for this service?

chvid 6y ago

What stack did you use to make this? Ps: show the date / age of the posts on the results page

Kelamir 6y ago

I love the option to view cached versions of websites. This is a nice project. Thank you, Mcovalt.

Watchnerd 6y ago

this is really cool, now I can filter by topics I really care about, well done!

edent 6y ago

Found a bug! Searched for my name and found a bunch of unrelated BoingBoing links. https://hndex.org/?q=Terence+Eden https://hndex.org/?q=Terence+Eden Why? Because those old posts have a "what's new" set of links. One of which contains my name. I'd suggest only searching the `<main>` element, perhaps?

smhmd 6y ago

Could you add post points and how much time since/when was the article posted? It would help as some topics do not age as well as others. Having sorting (most popular, most recent) and range search (last year, custom range) would also help those who want to narrow their search.

baccredited 6y ago

Wow - would love an RSS feed feature to subscribe to a search

antihero 6y ago

This is absolutely fantastic, is there any way to display the dates on the cached version and/or result?

throwawaysea 6y ago

Does this work with articles behind a paywall?

maddyboo 6y ago

Here we are in 2020, and web search is not what I’d hoped it would be. I would love to see more niche, small search tools like this rather than better general-purpose search. Building upon that, maybe an aggregator of search results from many small niche search engines. Actually, just writing that sentence reminded me of Searx [0], which I have to admit I haven’t tried in earnest but really should. 0: https://searx.me/ https://searx.me/

sarosh 6y ago

Hi Matt, can you provide any details on the code/backend?

adaszko 6y ago

There's also Falcon[1] Chrome extension which does full text indexing on your browser history so if you read something and can't either Google it or find it in the browser's history, Falcon will broaden the search scope. [1] https://chrome.google.com/webstore/detail/falcon/mmifbbohghecjloeklpbinkjpbplfalb?hl=en https://chrome.google.com/webstore/detail/falcon/mmifbbohghe...