5 ms·

How I hacked HN Search at the fHACKtory hackathon last weekend

21 points by redox_ 13y ago

mauricesvay 13y ago
I was expecting an article explaining how the search engine was built.
- hobs 13y ago
  Same, searching HN is no "hack" unless it was done in a novel or interesting way. This might even be a cool hack, but I dont know!
  - wslh 13y ago
    I hacked HNSearch in the formal way in this article: http://blog.databigbang.com/ideas-and-execution-magic-chart/ http://blog.databigbang.com/ideas-and-execution-magic-chart/ python code is also here: https://gist.github.com/srw/1360455 https://gist.github.com/srw/1360455 Basically I could search beyond the HNSearch limits. The article is from 2011 so I don't know if it's working now.
    - redox_ 13y ago
      Nice post, I'll release the rails4-based source code before the end of the week. Hopefully with a blog post explaining the underlying ranking algorithm.
- redox_ 13y ago
  I'll write a blog post about it in a few days. Thx for the feedback
maximesalomon 13y ago
Good stuff !
jared314 13y ago
This looks like it is just crawling HN, and using Algolia to provide a search over that data. What exactly are you hacking?
- redox_ 13y ago
  If by "hack" you mean "break/exploit/...", I totally agree. But hackatons are more codefests, calling "hack" every piece of code written in a short amount of time. That's only my 4-hours project implementing a custom crawler, work-arounding black-listing, and indexing it with Algolia. BTW, I'll write a blog post explaining the underlying ranking algorithm, which is the real point.
  - deleted 13y ago
    [deleted]
  - jared314 13y ago
    It looks like it's only a couple of days of data. Are you crawling the pages directly or using the existing api [1]? [1] https://www.hnsearch.com/api https://www.hnsearch.com/api
ProAm 13y ago
I search for "fracking" and only get back "tracking", "hacking" and "f__king" as results "meteor" only brings back 2 results.
- redox_ 13y ago
  Since the crawl started 3 days ago only, assuming that I fetch only the first 10 pages, lots of terms are not yet in the indexed corpus. But querying "fracking", you can at least see the typo-tolerant instant search in action ;)
  - ProAm 13y ago
    Ahh, makes sense. I thought you had 'primed' the results database with history first.
olive_ 13y ago
Why does not "Ask HN" return results while "Show HN" does?

21 points by redox_ 13y ago

mauricesvay 13y ago

I was expecting an article explaining how the search engine was built.

hobs 13y ago
Same, searching HN is no "hack" unless it was done in a novel or interesting way. This might even be a cool hack, but I dont know!
- wslh 13y ago
  I hacked HNSearch in the formal way in this article: http://blog.databigbang.com/ideas-and-execution-magic-chart/ http://blog.databigbang.com/ideas-and-execution-magic-chart/ python code is also here: https://gist.github.com/srw/1360455 https://gist.github.com/srw/1360455 Basically I could search beyond the HNSearch limits. The article is from 2011 so I don't know if it's working now.
  - redox_ 13y ago
    Nice post, I'll release the rails4-based source code before the end of the week. Hopefully with a blog post explaining the underlying ranking algorithm.
redox_ 13y ago
I'll write a blog post about it in a few days. Thx for the feedback

maximesalomon 13y ago

Good stuff !

jared314 13y ago

This looks like it is just crawling HN, and using Algolia to provide a search over that data. What exactly are you hacking?

redox_ 13y ago
If by "hack" you mean "break/exploit/...", I totally agree. But hackatons are more codefests, calling "hack" every piece of code written in a short amount of time. That's only my 4-hours project implementing a custom crawler, work-arounding black-listing, and indexing it with Algolia. BTW, I'll write a blog post explaining the underlying ranking algorithm, which is the real point.
- deleted 13y ago
  [deleted]
- jared314 13y ago
  It looks like it's only a couple of days of data. Are you crawling the pages directly or using the existing api [1]? [1] https://www.hnsearch.com/api https://www.hnsearch.com/api

ProAm 13y ago

I search for "fracking" and only get back "tracking", "hacking" and "f__king" as results "meteor" only brings back 2 results.

redox_ 13y ago
Since the crawl started 3 days ago only, assuming that I fetch only the first 10 pages, lots of terms are not yet in the indexed corpus. But querying "fracking", you can at least see the typo-tolerant instant search in action ;)
- ProAm 13y ago
  Ahh, makes sense. I thought you had 'primed' the results database with history first.

olive_ 13y ago

Why does not "Ask HN" return results while "Show HN" does?