8 ms·
Show HN: I made a search engine for Hacker News
I love HN but always felt the search with algolia is okay but does have some limitations. Since I work at Vectara I decided to try and create a better search for HN. It's based on data from roughly the last 6 months of HN stories and comments.
Would love to hear feedback and how useful this is relative to the existing search.
- wjb3 2y agoExcellent!
- metadat 2y agoCool project, but I'm struggling to understand what is better about the Vectara solution? Compared to Algolia.hn, this gives 0 filter controls (time window, stories vs. comments, `author:metadat', sort order, and so on), and no ability to search for exact matches. It failed to turn up anything interesting or even relevant for the 4 or 5 queries I ran. You've still made it further than I in the HN search engine adventures, which is commendable.
- mikepurvis 2y agoIf it's searching article content as well as just titles/comments, that's already quite interesting and useful. I've long wanted something like that for my browser history, like a "help me find this article I know I read sometime last week about X."
- metadat 2y agoYes, I've only recently rediscovered using the browser history for this exact purpose and it hasn't failed me yet. It would be remarkable and interesting to have a super deep search capability that indexes all first-order links on this site.
- nashashmi 2y agobrowser history seems to only save the last 3 months of history. It used to be many years of data but that also slowed the browser down.
- yaj54 2y agoAnd you think that hasn't been made yet? :-D Show HN: DeepHN - https://news.ycombinator.com/item?id=26791582 https://news.ycombinator.com/item?id=26791582 Of note, I was able to find that item (which I recalled existed but not by what name) with hn.algolia, but I was not able to find it with the OP search engine, or with the DeepHN search itself. So in my book, Algolia is winning. But projects like this are super fun and educational to build so props to OP.
- metadat 2y agoOP said they only indexed the past 6 months.
- seaal 2y agoPerhaps someone will be able to help me find a website I’ve been trying to Google for with no luck: the main focus of the website is a music player with 3D pipes and arrows that the camera would follow along, live rendering the entire scene while music was playing, but it was also a personal blog if you scrolled down. Domain was relatively short and I think it started with K.
- ofermend 2y agoCurrently it's indexing the content of stories and comments. Are you suggesting also to index the content of the main story link (which is outside HN)?
- fluential 2y agoThis already exist: "searching over the titles, text and URLs of sites you've visited before" -- https://www.browserparrot.com/ https://www.browserparrot.com/
- kid64 2y agoMac-only
- yaj54 2y agoDitto. I somewhat recently learned that Safari only keeps the history log for one year unless you explicitly set "Remove history items" to "manually." Which I've done so that I at least have a list that I could crawl in the future to build a full text index on.
- squigz 2y agoI'm fairly confident every single major browser (and probably most minor) have a history limit of some sort. Sometimes it's age, sometimes it's # of entries.
- metadat 2y agoUsually I'm only looking back the past week or three. Otherwise I snapshot the page to evernote when I think it might be interesting later. Hopefully they don't completely end the free tier.
- tjlav5 2y agoI think Microsoft is working on that right now :p
- rovr138 2y agoThat’s exactly what it sounded like to me. Of course, local/offline only would be great.
- shortrounddev2 2y agoLol you're describing Microsoft's recall feature that everyone here is saying is the death of privacy
- kelnos 2y agoIf Recall was open source and auditable, and didn't send any data at all about usage to the internet, I guess that would be fine. Otherwise, no thanks.
- tamimio 2y ago>algolia.hn Is that a valid link? I get an error when opening it.
- miloignis 2y agoIt's the other way around: https://hn.algolia.com/ https://hn.algolia.com/
- gnabgib 2y agoNo that's not valid (it's attached to the search box at the bottom of the page) https://hn.algolia.com/?q= https://hn.algolia.com/?q=
- sattoshi 2y agoThey meant https://hn.algolia.com/ https://hn.algolia.com/
- potatoman22 2y agoYou can also use !hn on duckduckgo
- owenpalmer 2y agoCool, I like it! I found a bug. Under the "When will GPT-5 be released?" search results, there are double duplicate results. On one of the duplicates, the "username (date)" says "undefined (undefined)"
- mewpmewp2 2y agoI tried this query as well, I was a bit frightened when I saw my own comment in the results.
- ofermend 2y agoGood find. let me check why that occurs.
- ofermend 2y agoThis should be fixed now. Thanks for the find.
- owenpalmer 2y agoHappy to help
- pedalpete 2y agoNice work. I wonder if there may be a better application for the Vectara capabilities than search? Algolia has already done the search thing, can the Vectara search be 10x better? What I do find missing from HN is the ability for me to see things that may be of interest to me, but that I may have missed. I like how I get everything in the main feed which is pure popularity, but I don't have the time to go through all posts, and definitely likely miss things I would probably have been interested in. Though this can be done with collaborative filtering, or other non-AI methods, might this be a decent use case for your AI?
- whiplash451 2y agoUnclear that the HN feed is pure popularity. There's got to be something else to it [1], otherwise it would look like all the medium-like crap you'll find elsewhere. [1] my hunch is that some human expert curation is involved.
- pedalpete 2y agoIt is not pure popularity, I suspect it is a combination of clicks/upvotes/comments over time. I know time is a key component in the algorithm. If you don't rise fast, you don't rise. Who is commenting/upvoting is also likely an easy metric to add in. If I have more points, my votes are probably worth more than someone without. Human curation also exists, but I think that is aimed at removing spam and uplifting YC company posts.
- flir 2y agoI've been wondering if segmentation might be the way to go. Have the chatbot look at all the items, cluster them into a few buckets of its choosing, then throw each new item into the most appropriate bucket. (I've been thinking about this not just in terms of HN, but treating all my RSS feeds as one undifferentiated stream and just having a chatbot sort incoming items into whatever bucket it deems most appropriate). What's stopping me is that it might work, and I doubt making the internet even stickier is good for me long term.
- 2y ago
- smusamashah 2y agoIt doesn't always work correctly. For example "Text to diagram tool" is returning very few results and some of the results are not even correct. While this topic has been discussed a lot here. I was mainly looking for the list of tools I keep sharing whenever this topic comes up, or when I share a related tool in a thread.
- aaron695 2y ago[dead]
- yoouareperfect 2y agoAmazing! Congrats on launching!
- marcodiego 2y agoPlease, add also the possibility to search in links posted/commented in hacker news. I bet it would be competitive against google for the hn crowd.
- ofermend 2y agoFor sure. Will test that.
- handfuloflight 2y ago[dead]
- ceving 2y agoSearching for "iptables" returns first: > Arm says it wants all Snapdragon X Elite laptops destroyed Not so useful.
- codetrotter 2y agoIt’s matching two comments in the thread. It bolds the part that talks about iptables in each. So it’s not like it’s irrelevant, even though it is certainly not actually the most relevant one either. It seems to give better results if you are more specific. For example, try the following search: how to use iptables effectively And have a look at the first five or so results. Also, note that OP said it’s searching about six months worth of data. So if anything specific about iptables that you were looking for is older than that then their search tool doesn’t know about it.
- ofermend 2y agoYes, it's just about 6 months back. If requested by folks here, we can certainly crawl back more years - this was just the first crawl I did.
- codetrotter 2y agoAre you crawling the HN via the normal website? It’s better to use the API. https://github.com/HackerNews/API https://github.com/HackerNews/API
- dcoder2311 2y agoCool work!
- call-me-al 2y agoreally like this one from [Show HN: Hacker Search – A semantic search engine for Hacker News](https://news.ycombinator.com/item?id=40238509 https://news.ycombinator.com/item?id=40238509). The URL is: https://hackersearch.net/ask https://hackersearch.net/ask
- ravishing0223 2y agoNice. https://hackernews.demo.vectara.com/?query=I+made+a+search+engine+for+Hacker+News&filter= https://hackernews.demo.vectara.com/?query=I+made+a+search+e...
- yanko 2y agoHow to full text search on hn given user favorites only... If no such such option I feel disappointed
- Jiahang 2y agoi use Google ：｛｝+ news.ycombinator.com
- n4r9 2y agoLike some other comments here I find HN search useful and powerful and am a little unsure what the added value is here. Possibly/probably it's for people that search in a different way to me. One of the most frequent searches I do is to look for a specific comment that I know a user made recently. For example, I might want to look for my own comment here: https://news.ycombinator.com/item?id=40801389 https://news.ycombinator.com/item?id=40801389 (sorry, this is a slightly political one but I just picked it randomly for test purposes). Searching Vectara for "n4r9 NHS" produces no results: https://hackernews.demo.vectara.com/?query=n4r9+NHS&filter= https://hackernews.demo.vectara.com/?query=n4r9+NHS&filter= HN's own search however produces the goods in the top result: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=n4r9%20NHS&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... [ EDIT except for this very post :p ] Maybe 6 days ago is outside the dataset that this is based on? Some other thoughts/suggestions: - Ability to click through to the comment itself? At the moment it looks like the link goes just to the main comments page and then I have to find the relevant comment on the page. - Filter comments vs posts? - Order by datetime? - Filter within a date range?
- ofermend 2y agoThank you - these are great suggestions. Will work to add these...
- omneity 2y agoI searched for "Supabase" and none of top the results in your demo contained an actual post about supabase. Following the example queries I then tried "What is supabase?" and the results were equally irrelevant. My personal opinion is that I'll keep using the HN search for the foreseeable time.
- zmccormick7 2y agoI had the same issue when searching for specific companies/products. It feels like a pretty basic vector search with no hybrid search component or reranking.
- bschwarz 2y agoStruggled to find a specific comment earlier today via Algolia, found it as the second result on Vectara.
- d4rkp4ttern 2y agoI really like this HackerSearch app: https://hackersearch.net/ https://hackersearch.net/
- d4rkp4ttern 2y agoPreviously discussed here: https://news.ycombinator.com/item?id=40238509 https://news.ycombinator.com/item?id=40238509
- christophilus 2y ago[flagged]
- sharpshadow 2y agoLike Algolia it doesn’t work without JS.
- bckr 2y agoNext someone should make a hackernews meta search engine :)
- ptsd_dalmatian 2y agoAm I the only one that hasn't noticed the search input in the footer? :O
- sph 2y agoThese days I use the search feature much more than commenting or reading posts. The frontpage is the usual recent news addiction treadmill, while for research into niche topics you can find a treasure trove of interesting comments in the archives. Want more posts about Lisp, Smalltalk and reverse engineering, for example, rather than the usual front page drivel? Search for them. On one hand I wish Algolia didn't give very old posts a lot of weight (it often prefers to show posts > 8+ years ago), on the other hand old content tends to be before the Eternal September of tech-adjacent people coming to this forum to discuss tech-adjacent light content, so it's actually a feature. The real value of HN is its archives IMO.
- satvikpendem 2y agoI also made a bookmarklet to show me HN posts from random dates, it is quite interesting to see what was interesting to people a decade ago, for example. Lots of comments in HN's heyday were pretty eye opening as well. javascript:(function() {function randomDate(start, end) {var date = new Date(+start + Math.random() \* (end - start));var day = ("0" + date.getDate()).slice(-2);var month = ("0" + (date.getMonth() + 1)).slice(-2);var year = date.getFullYear();return year + '-' + month + '-' + day;}var startDate = new Date(2007, 9, 9);var endDate = new Date();var randomDateStr = randomDate(startDate, endDate);var newUrl = 'https://news.ycombinator.com/front?day=' + randomDateStr;window.location.href = newUrl;})();
- djeastm 2y agoThat's really neat. Thanks for providing it
- dang 2y agoAlso recommended is https://news.ycombinator.com/front https://news.ycombinator.com/front, which shows you the frontpage stories from any day—a little bit like archive.org would do, except that it's not a snapshot, but a composite of all the front pages from a 24 hour period. For example, here's HN from a year ago: https://news.ycombinator.com/front?day=2023-07-02 https://news.ycombinator.com/front?day=2023-07-02. https://news.ycombinator.com/highlights https://news.ycombinator.com/highlights is another good resource (and if anyone notices a great HN comment, past or present, they're welcome to nominate it for the highlights list! just email hn@ycombinator.com).
- jiehong 2y agoCongratulations! Although, something I value a lot from algolia is the very fast live search as you type[0]. Vectara seems to be smarter, but much slower. My needs are satisfied with algolia 99% of the time as a technical user. [0]: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=hello&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
- jpl56 2y agoGreat! My first thought was "funniest XKCD". Loved the Self Driving one [0]. Thanks! [0]: https://xkcd.com/1897/ https://xkcd.com/1897/
- okhuman 2y agoI'm thrilled to see Vectara here – they have one of the top Product Managers I've ever had the chance to work with and be mentored by.
- uwemaurer 2y agoGreat work! I am currently playing with the Algolia hackernews search API myself and experimenting with spaCy Named Entity Recognition and llama3 to come up with some interesting data. Work in progress version here: https://news.facts.dev/topic https://news.facts.dev/topic
- KomoD 2y agoOP mentions Algolia having limitations but this seems more limited? It doesn't seem like it has any filtering or sorting like the Algolia one has, like comments/stories by a specific user, during certain dates, sorting by upvotes/recency, searching by just title/content/comments. Say I wanted to search for comments by the OP, ofermend, it doesn't seem like I can... Entering just their name returns results that aren't made by them nor mention their username, I tried other queries too without any luck.
- douglaskayama 2y agoI tried searching for lootitooti, Vectara found nothing, Algolia found two results, one in a post and one in a comment in another post. PS: no, lootitooti is not my project. I decided to finally watch Game of Thrones with my wife and I remembered that site when I was watching the opening. I remembered seeing it here on HN, searched and found it.
- kuzej 2y agoGreat project! But I'm curious about how the frontend was implemented. What's the tech stack? For example, is it Next.js + Tailwind CSS?
- rgaidot 2y agoThat's interesting to hear you've developed a new search interface for Hacker News. I can relate, as I've worked on another search interface as a personal project myself (validated in "Cool Apps" at algolia), though mine has no ambitions beyond personal exploration. Knowing there are various other options already in existence, I'm genuinely curious to see how your approach distinguishes itself. Finally site hn.algolia.com is no longer maintained. https://hnpaper-labs.gaidot.net https://hnpaper-labs.gaidot.net
- rgaidot 2y agoEven if I understood correctly that you were using AI.
- ofermend 2y agoThank you everyone for the feedback. For those interested in asking questions about articles, here's another nice demo (still in beta): an Agentic RAG chatbot demo (hosted on Huggingface, using streamlit for UI): https://huggingface.co/spaces/vectara/hacker-news-chat https://huggingface.co/spaces/vectara/hacker-news-chat Feel free to ask it some things and let me know how it works.