21 ms·

Show HN: DeepHN – Full-text search of 30M Hacker News posts and linked webpages

324 points by wolfgarbe 5y ago

jschveibinz 5y ago
I like the UI. Very nice.
syntaxing 5y ago
Neat but the back button hijacking is not cool.
- dastx 5y ago
  This kind of thing is enough for me to never want to use a site. If it's a bug, high recommend you fix it, if it's not, highly recommend you reconsider your position.
  - wolfgarbe 5y ago
    Your are right. It IS a bug, and we will fix it asap.
  - SamBam 5y ago
    It is so common in SPA that the long-press of the back button to see the list of recent sites should be muscle-memory by now for everyone. Yes, it's an annoyance. Yes, sites should fix it. But not ever using a site because of it seems silly when it's literally a half-second longer click. (Also, for this site I don't actually see the bug. So either they fixed it very rapidly, or GP was just referring to individual searchers being in the history, which is common for any search engine.)
    - dastx 5y ago
      > It is so common in SPA that the long-press of the back button to see the list of recent sites should be muscle-memory by now for everyone. This is not a good excuse for laziness. > But not ever using a site because of it seems silly when it's literally a half-second longer click. I highly disagree. With this site at least it is actually possible to leave. Many websites I've come across with this issue, it is entirely impossible to leave without physically holding down the back button in the browser to get a list of history items, and then clicking a site from earlier. > (Also, for this site I don't actually see the bug. So either they fixed it very rapidly, or GP was just referring to individual searchers being in the history, which is common for any search engine.) Looks like they've fixed it.
- deleted 5y ago
  [deleted]
- wolfgarbe 5y ago
  Thanks for pointing this out. That was not intentionally, probably a side effect from the navigation inside our search results within a dynamic page (using the back button to return to previous search queries). We will fix this.
- wolfgarbe 5y ago
  We have just fixed the back button bug (make sure to clear the browser cache).
- pelagicAustral 5y ago
  I'm a bit curious about this 'back button hijacking', since the bug has now been fixed I'm afraid I'm not able to test. What was the unintended behaviour?
  - wolfgarbe 5y ago
    If you were entering deephn.org in your browser address bar, you couldn't use the browser back button to return to the previous page. That was a bug caused by the navigation code within our dynamic HTML page. It is fixed now.
    - pelagicAustral 5y ago
      Interesting. Thanks for your reply.
exdsq 5y ago
It’s fun to see Bitcoin posts from 2008-2010. I wonder how many bought and are now retired because of them. Seeing recommendations to use MtGox is pretty funny too https://news.ycombinator.com/item?id=1998144 https://news.ycombinator.com/item?id=1998144
- myth_drannon 5y ago
  Earlier post 2009: https://news.ycombinator.com/item?id=599852 https://news.ycombinator.com/item?id=599852 "Well this is an exceptionally cute idea, but there is absolutely no way that anyone is going to have any faith in this currency." Written by co-founder/CEO of Pachyderm and first employee of RethinkDB none the less.
  - LittlePeter 5y ago
    Reminds me of the infamous dropbox comment https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
  - smt88 5y ago
    I still don't think many people have faith in it. The vast majority of people who own it are speculators who never actually use it as a currency.
  - dang 5y ago
    People love to pick out the comments of the past that seem most wrong now, because—let's be honest—it makes us feel superior. But if you go back and look at that original 3-comment Bitcoin thread, what you find is that those 3 comments span an entire spectrum: elated, skeptical, and bewildered: https://news.ycombinator.com/item?id=599852 https://news.ycombinator.com/item?id=599852. The first cryptocurrency thread on HN—about 9 months earlier—was even more interesting: https://news.ycombinator.com/item?id=253963 https://news.ycombinator.com/item?id=253963. It contains another skeptical comment: "The bigger problem is that everyone has a incentive to run their computers day and night cranking out solutions, which burns up lots of natural resources and processor time for a zero-sum result." (https://news.ycombinator.com/item?id=253999 https://news.ycombinator.com/item?id=253999), which seems astonishingly prescient now, even though the objection is still hotly (no pun intended) debated. Cherry-picking the wrongest-seeming skeptical comment from that collection of data points is a case study in survivorship bias. (I don't mean to pick on you personally! This is common of course.) The infamous Dropbox comment from 2008 is similar in that it has gotten repeated out of context in a way that is unfair to the original commenter (https://news.ycombinator.com/item?id=23229275 https://news.ycombinator.com/item?id=23229275). When we repeat these things, it says more about us than it does about the thing we're repeating.
yamrzou 5y ago
This is nice! How are the stories tagged?
- wolfgarbe 5y ago
  We are deriving the tags from the terms and bigrams in title, text and parsed html of linked web pages. Top frequent terms per post, but only if terms are within top 65k tags per index. Stopwords are excluded.
  - deleted 5y ago
    [deleted]
yewenjie 5y ago
Nice to see full-text of the articles linked and really fast. Do you provide an API like Algolia does for HN?
- wolfgarbe 5y ago
  DeeepHN is powered by our SeekStorm search-as-a-service, which has a full-featured API: https://seekstorm.com/docs https://seekstorm.com/docs SeekStorm is intended that the user can index and search their own private data. But if there is demand we could also allow to search in a central public index for web data or other public data.
  - morrbo 5y ago
    How does seekstorm perform on large documents? Say documents 1MB-10MB in size, are you able to search entire docs or just paragraphs within the doc?
    - wolfgarbe 5y ago
      Currently the maximum size of a single document is limited to 1 MByte. That is an artificial limitation resulting from the fact, that SeekStorm does not only indexes the content of the document, but also stores the original document. Somewhere there has to be a limit.
matt_wilson_206 5y ago
Seems odd to publish a benchmark against Lucene where your results are "Lucene crashes at 4 concurrent users". Since there are thousands of people using Lucene and products built on Lucene every day, I think the issue may be in your benchmarking code.
- wolfgarbe 5y ago
  There is no secret about the benchmark, its all open source: https://github.com/wolfgarbe/LuceneBench https://github.com/wolfgarbe/LuceneBench
  - arafalov 5y ago
    It seems you tried MMapDirectory and then commented it out. https://github.com/wolfgarbe/LuceneBench/blob/master/LuceneBench/src/LuceneCore.java#L52-L53 https://github.com/wolfgarbe/LuceneBench/blob/master/LuceneB... So, you may be hitting SimpleFSDirectory instead, which does have issues with too many searches. Could you share the reasons, MMapDirectory did not work for you?
    - wolfgarbe 5y ago
      "NIOFSDirectory and MMapDirectory implementations face file-channel issues in Windows and memory release problems respectively. To overcome such environment peculiarities Lucene provides the FSDirectory.open() method. When invoked, it tries to choose the best implementation depending on the environment." https://www.baeldung.com/lucene-file-search https://www.baeldung.com/lucene-file-search
      - arafalov 5y ago
        Right. I found you are running on Windows after posting the comment. Makes sense, though there were some fixes in the latest Lucene I believe. The Lucene's own benchmarks are at: https://home.apache.org/~mikemccand/lucenebench/ https://home.apache.org/~mikemccand/lucenebench/ , though I admit to not know enough about benchmarking to form a strong opinion. Either way, good luck with the project/service. More competition is always great. The open-source components look interesting too.
nitramm 5y ago
It is really great tool too see what used to be very popular posts in the past. I have one observation. Posts linking to twitter - https://deephn.org/?sort=score&filter=%7B%22score%22%3A%7B%22from%22%3A1000%2C%22to%22%3Anull%7D%2C%22domain%22%3A%5B%22twitter.com%22%5D%7D https://deephn.org/?sort=score&filter=%7B%22score%22%3A%7B%2... - have most of the preview occupied with navigation and other not interested parts. So maybe it is possible to use some hacky solution to show the content part for the most popular websites, or if multiple pages are downloaded from the same domain the content part could be detected automatically.
- wolfgarbe 5y ago
  You are right. The preview is far from perfect. We just wanted to ship early. We will continue to fix and improve.
sturza 5y ago
How can we learn more about this indexing service?
- wolfgarbe 5y ago
  https://seekstorm.com/about-deephn https://seekstorm.com/about-deephn https://seekstorm.com/ https://seekstorm.com/ https://seekstorm.com/search-api-features https://seekstorm.com/search-api-features https://seekstorm.com/search-as-a-service-architecture https://seekstorm.com/search-as-a-service-architecture
daibo 5y ago
I assume you are using Symspell for fuzzy search and prioritize candidate frequency over edit distance. My input "algolai" returns results for "angola" even though "algolia" was just 1 transposition away. Anyway just dropping in to say that stuff on your github helped me figure out a lot while building a search engine from scratch. Good luck with SeekStorm!
- wolfgarbe 5y ago
  We are combining auto correction (SymSpell) and auto completion (PruningRadixTrie). That is sometimes tricky. We are using a static spelling correction dictionary which does not contains the term algolia, but angola. SymSpell is very fast, but requires a lot of memory. Therefore we are using a static dictionary, and not a dynamic dictionary derived from each individual index of each customer. Auto completion on the other hand we are dynamically generating for each individual index/customer.
  - daibo 5y ago
    I am combining symmetric deletes with a trie for fuzzy autocompletion and it is working pretty well.
de6u99er 5y ago
I like it. I built something similar recently for indexing scientific publiations on r/COVID19 based on Elasticsearch and Kibana.
- finikytou 5y ago
  can you share a link?
0xmohit 5y ago
Curious to know how does it differ from HN Search [1]? HN Search API documentation [2]. [1] https://hn.algolia.com https://hn.algolia.com [2] https://hn.algolia.com/api https://hn.algolia.com/api
- jankotek 5y ago
  It searches content of referenced webpages. Also has filtering by year intervals (for example 2012-2016), domains, tags etc.. And I find usability better. I think it is great product.
crazypython 5y ago
I was hoping it would do full-text search of the text inside the linked pages, like hndex.org but with ranking.
- johtso 5y ago
  I think it is supposed to be doing that?
  - crazypython 5y ago
    If I search "friendship" all the article titles have the word friendship in them. In my experience many HN titles don't contain the topic word.
- wolfgarbe 5y ago
  Yes, it does exactly that. If you check "Web" on the left sidebar, and uncheck "Stories" it will find the posts where "friendship" occurs in the linked web pages. But we still show the title of the original hacker news story in the search result and highlight the search term if it ALSO occurs in the title. If you scroll long enough within the search results you will finally reach results where the search term is not in the title, but only within the linked web page. It is more obvious if you search for queries which are less popular than "friendship"
- Hitton 5y ago
  You have to check box "Web" for it to search in linked articles too. Without that it highlights the words in articles that are found only because of title. And the indexing of contents isn't perfect, sometimes it seems to work for main article content but not for article author name even when it's shown.
reivalc 5y ago
Neat! I wonder if you quantified the performance of your search somehow? I guess labelling data for that would be laborious.
- wolfgarbe 5y ago
  You mean performance in terms of latency or relevancy?
  - reivalc 5y ago
    Relevancy - I'd guess it's difficult to measure, due to the large dataset, the large number of possible search terms, and large number of possible results.
throwaway823882 5y ago
Interesting, it doesn't seem to show the dead pages. I have been indexing dead HN pages for several years now, with the idea that eventually I'd run some statistics on it to better understand both the auto-dead filters and the people who are submitting dead content.
berto4 5y ago
Can you search by location? lets say I have house listings and I want to search all the listings by location, does your api allows for this?
b1tshift 5y ago
Love it! I would kill for a reverse order button to see oldest posts first
- wolfgarbe 5y ago
  We can add that button, if you promise not to kill ;)
  - b1tshift 5y ago
    Okay :D
    - wolfgarbe 5y ago
      Done. Now you can sort by newest/oldest date.
freediver 5y ago
The effort definetely gets my upvote. What did you use to crawl the pages and how long did it take? Curious about your experience doing it and crawler integration with Seekstorm. Is is possible to easilly expand the index with embeddings (vectors) and perform semantic search in parallel? Your pricing indicates that hosting similar index as this demo would cost $500/month. Wondering what kind of infrastructure is supporting the demo? Thanks! ps. Small quirk: https://deephn.org/?q=how+to+be+productive&filter=%7B%22hashtags%22%3A%5B%22CPU%22%5D%7D https://deephn.org/?q=how+to+be+productive&filter=%7B%22hash... First three tags seem not to be relevant to the post itself.
- wolfgarbe 5y ago
  Crawling speed is between 100...1000 pages per second. We crawled about 4 million linked unique web pages The pricing for an Index like DeepHN would be $99/month: While we are indexing 30 million Hacker news posts, for DeepHN we are combining a single HN story with all its comments and its linked webpage into a single SeekSorm document. So that we index just 4 million SeekStorm documents. Yes, it would be possible to expand index with embeddings (vectors) and perform semantic search. This would we an auxiliary step between crawling and indexing.
  - freediver 5y ago
    What is being used as a crawler and is it integrated with Seekstorm? The same article referenced here https://deephn.org/?q=how+to+be+productive&filter=%7B%22hashtags%22%3A%5B%22CPU%22%5D%7D https://deephn.org/?q=how+to+be+productive&filter=%7B%22hash... contains the phrase 'well-defined' Any idea why doesn't the article surface when searching for this ?
    - wolfgarbe 5y ago
      The crawler is a part of SeekStorm. "Well-defined": Just a guess: We are doing key text extraction, i.e. we try not to index boilerplate stuff an and menu items. As "well-defined" is within a short list item, it might be accidentally skipped. So, that is not yet perfect, and considering the diversity in web page structure it probably never will. But we will try to improve.
abcc8 5y ago
I'm curious how this site is compliant with copyright law. I've thought of working on similar projects, but the possibility of being sued has dissuaded me.
- deleted 5y ago
  [deleted]
- doh 5y ago
  Search engines are generally considered fair use [0], at least in US. So up until HN doesn't send cease and desist, it's in the clear. Once it does, then it's all about what the claims are. Google litigated the crap out of this [1] over the years, so there is a positive precedent minefield all over the place. [0] https://www.everycrsreport.com/reports/RL33810.html https://www.everycrsreport.com/reports/RL33810.html [1] https://en.wikipedia.org/wiki/Fair_use#Text_and_data_mining https://en.wikipedia.org/wiki/Fair_use#Text_and_data_mining
- ehmmmmmmmm 5y ago
  It's probably not compliant. As with all good things, if you're in the Land of the Free (TM), eventually people will hunt you down if you try to do good things. Alternatively, base your site out of a country which gives you the freedom to do things like this. Build it in China and you'll be applauded instead of sued. Copyright suits for petty things like this would laughed off when it's something actually useful.
  - NicoJuicy 5y ago
    China's monopolies are far worse than in other countries. Founders there have no choice but to sell or get copied by the platform that you're building on. It's baba or tenscent.
    - dheera 5y ago
      Yes, that's also true, but on the flip side, technological advancement of society as a whole is not stifled by idiotic IP law. I agree, some anti-monopoly regulation is in order in China, and I'm not a fan of the Alibaba/Tencent monopolies on the ecosystem. However, I think there are other ways to go about fostering and encouraging small businesses and protecting creators and their intentions than American-style IP law, which allows you to sit on an invention or work and hinder society from having access to the fruits of science, which I'm vehemently against. IP isn't even real property, IMHO.
youngNed 5y ago
So, can you explain what is going on with 'covid' articles from <2019 - are these where the original page that was linked to has now become a spam link?
- wolfgarbe 5y ago
  Not necessarily a spam link. Linked pages from older post have sometimes updated their content. Sometimes this is legitimate, sometimes spam.
karterk 5y ago
Really cool demo. I'm curious to know what kind of hardware this is this hosted on. Full disclosure: I work on a similar fast, typo tolerant, fuzzy search engine search engine called Typesense (https://github.com/typesense/typesense https://github.com/typesense/typesense).
- goodmachine 5y ago
  Yeah, was also interested. It would be nice to see Typesense take up the same challenge for comparison ;)
  - karterk 5y ago
    Happy to benchmark if the dataset is shared!
- wolfgarbe 5y ago
  SeekStorm uses dedicated root servers with NVMe SSDs, hosted by IONOS https://www.ionos.com/servers/intel-servers https://www.ionos.com/servers/intel-servers
zoomablemind 5y ago
Amazingly fast! I'm used to seeing a Firefox's load progress bar for most of requests on my mobile, yet here the results "just show up". More observations: - When trying to go back after clicking the original HN link, there's no response. - Clicking on what I understand are tags (hashtags) in a given post has no effect. - Sorting by date is working, though it'd be nice to see the date. Also a question: how do you generate the tags (hashtags) from the original contents?
- wolfgarbe 5y ago
  >> When trying to go back after clicking the original HN link, there's no response. The original HN link is opened in a new browser page/tab. The search results should remain unchanged in the previous tab. So you don't use the back button, but the previous browser tab. >> Clicking on what I understand are tags (hashtags) in a given post has no effect. Search for "google", go to the first result. there is a hashtag "oracle", click and the search results are filtered by that hashtags. Please post an example where its doesn't work, so that we can fix it. The date is shown in the preview panel on the right hand side. But wa are thinking adding the time also to the result list. We are deriving the tags from the terms and bigrams in title, text and parsed html of linked web pages. Top frequent terms per post, but only if terms are within top 65k tags per index. Stopwords are excluded.
  - zoomablemind 5y ago
    > Please post an example where its doesn't work, so that we can fix it. Note: I'm trying this in a mobile browser (Firefox). - Search for a word 'Decentralized' - pick the second result: 'A decentralized web would give power back to the people online' - On the result's (preview?) page, click the tag 'Decentralization' I get no response, nothing updates, no new tab opens. Not sure what is the intended action (I'd assume it should be a new set of results corresp. to the hashtag). Same issue with 'google' example that you described.
    - wolfgarbe 5y ago
      If you click a tag on the result/preview page (in mobile Firefox), the result list is filtered by the tag (you get a new/shorter result list with an updated result number). But because you are still on the preview page you don't immediately see this as it's done in the background. But once you close the preview page (with the big arrow on the top/left of the preview page) you will see the updated result list and number. Of course, this is a design flaw, we should automatically close the preview window once you select a tag to immediately show you the updated result list. In the desktop version, this is no issue, as there result list and preview window are always simultaneously visible.
localhost 5y ago
Have you tried against a corpus of text like email? How well does that work? I've been frustrated with the search speed (and results) from email search in general and was wondering if I could do better with an offline search?
- wolfgarbe 5y ago
  That would definitely work. We are working on local client that could index your local PDF and Word documents. So importing emails would be a nice addition.
  - localhost 5y ago
    I'd be interested in seeing how well it performs vs. integrated search. Outlook search (which is what I use) suffers from performance and relevance issues, yet when I tried this experiment with another full-text search provider (Azure Search) it performed worse on relevancy. I use email as kind of an activity stream - generally I can remember roughly the people involved and the time period. Would be lovely to have search that performs well in this regard.
artembugara 5y ago
Nice! How did you fetch the content of those articles?
- wolfgarbe 5y ago
  With the crawler which is part of the SeekStorm search-as-a-service.
LinuxBender 5y ago
We're sorry but seekstorm-deephn doesn't work properly without JavaScript enabled. Please enable it to continue. Could the folks with JS disabled get a small semi-static page with the top interesting HN stats, and a message that say something like "Here are some simple stats. For the full interactive experience, please enable javascript"?
- wolfgarbe 5y ago
  Good idea!
- eloff 5y ago
  For the less than 1% of such people? I doubt that it's worth it. If somebody can't be bothered to enable JavaScript to use the site, they're not a desirable customer anyway.
superasn 5y ago
Very cool and useful. I was trying to find all comments on HN which contains the word youtube.com (basically trying to find some recently linked to videos from comments) but it returns 0 results when i set the search filter to comments(1). Any idea what I'm doing wrong? https://deephn.org/?q=youtube.com&sort=time&in=Comments https://deephn.org/?q=youtube.com&sort=time&in=Comments
- wolfgarbe 5y ago
  We will look into this.
smusamashah 5y ago
https://deephn.org/?q=procedural+planet&filter=%7B%22domain%22%3A%5B%22jsfiddle.net%22%5D%7D https://deephn.org/?q=procedural+planet&filter=%7B%22domain%... A bug probably. Huge amount of text in title here.
- wolfgarbe 5y ago
  Thank you. We will look into this and fix it asap.
smusamashah 5y ago
Can we exclude certain terms in search with a "-" e.g. "-oculus"?
- wolfgarbe 5y ago
  It should, but it currently doesn't in DeepHN. We will fix this asap.
- wolfgarbe 5y ago
  The '-' operator has been fixed. 'vr -oculus' should work correctly now.
djhn 5y ago
A page to bookmark for sure and an interesting product. The UI takes getting used to, but then again so do many actually useful tools.
wolfgarbe 5y ago
Thank you all for your great feedback. We will fix all the bugs and add some of the suggested features and release a next iteration of DeepHN within the next few days!
- codeproject 5y ago
  that is a great site. To be really honest, Those bugs are not much trouble compared with your service to all. that is really remarkable useful website. Thank you so much from the bottom of our hearts.
FairDune 5y ago
Apologies for being nitpicky about the already fixed back button While viewing a specific drilled-down result, each time the back button is pressed, it seems to navigate to the previous character in the incremental search. In other words, if the search term is 'C#', navigating back seems to highlight 'C' in the result detail page.
- wolfgarbe 5y ago
  This is most likely a timing problem. If you type c# fast, then the back button brings you back to the previous query. If you pause between typing 'c' and '#'. then it brings you back to the previous character only. That is related to our instant search, where you don't need to hit the return key to search. This causes the problem, that we need to identify when a query ends and the next query starts, to create the correct entries for the back button. We do this via timing - if the pause between key presses is too long, a new, separate entry is assumed. As the DeepHN is a single dynamic HTML page, we need to implement and manage the browser navigation logic ourselves.

324 points by wolfgarbe 5y ago

jschveibinz 5y ago

I like the UI. Very nice.

syntaxing 5y ago

Neat but the back button hijacking is not cool.

dastx 5y ago
This kind of thing is enough for me to never want to use a site. If it's a bug, high recommend you fix it, if it's not, highly recommend you reconsider your position.
- wolfgarbe 5y ago
  Your are right. It IS a bug, and we will fix it asap.
- SamBam 5y ago
  It is so common in SPA that the long-press of the back button to see the list of recent sites should be muscle-memory by now for everyone. Yes, it's an annoyance. Yes, sites should fix it. But not ever using a site because of it seems silly when it's literally a half-second longer click. (Also, for this site I don't actually see the bug. So either they fixed it very rapidly, or GP was just referring to individual searchers being in the history, which is common for any search engine.)
  - dastx 5y ago
    > It is so common in SPA that the long-press of the back button to see the list of recent sites should be muscle-memory by now for everyone. This is not a good excuse for laziness. > But not ever using a site because of it seems silly when it's literally a half-second longer click. I highly disagree. With this site at least it is actually possible to leave. Many websites I've come across with this issue, it is entirely impossible to leave without physically holding down the back button in the browser to get a list of history items, and then clicking a site from earlier. > (Also, for this site I don't actually see the bug. So either they fixed it very rapidly, or GP was just referring to individual searchers being in the history, which is common for any search engine.) Looks like they've fixed it.
deleted 5y ago
[deleted]
wolfgarbe 5y ago
Thanks for pointing this out. That was not intentionally, probably a side effect from the navigation inside our search results within a dynamic page (using the back button to return to previous search queries). We will fix this.
wolfgarbe 5y ago
We have just fixed the back button bug (make sure to clear the browser cache).
pelagicAustral 5y ago
I'm a bit curious about this 'back button hijacking', since the bug has now been fixed I'm afraid I'm not able to test. What was the unintended behaviour?
- wolfgarbe 5y ago
  If you were entering deephn.org in your browser address bar, you couldn't use the browser back button to return to the previous page. That was a bug caused by the navigation code within our dynamic HTML page. It is fixed now.
  - pelagicAustral 5y ago
    Interesting. Thanks for your reply.

exdsq 5y ago

It’s fun to see Bitcoin posts from 2008-2010. I wonder how many bought and are now retired because of them. Seeing recommendations to use MtGox is pretty funny too https://news.ycombinator.com/item?id=1998144 https://news.ycombinator.com/item?id=1998144

myth_drannon 5y ago
Earlier post 2009: https://news.ycombinator.com/item?id=599852 https://news.ycombinator.com/item?id=599852 "Well this is an exceptionally cute idea, but there is absolutely no way that anyone is going to have any faith in this currency." Written by co-founder/CEO of Pachyderm and first employee of RethinkDB none the less.
- LittlePeter 5y ago
  Reminds me of the infamous dropbox comment https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
- smt88 5y ago
  I still don't think many people have faith in it. The vast majority of people who own it are speculators who never actually use it as a currency.
- dang 5y ago
  People love to pick out the comments of the past that seem most wrong now, because—let's be honest—it makes us feel superior. But if you go back and look at that original 3-comment Bitcoin thread, what you find is that those 3 comments span an entire spectrum: elated, skeptical, and bewildered: https://news.ycombinator.com/item?id=599852 https://news.ycombinator.com/item?id=599852. The first cryptocurrency thread on HN—about 9 months earlier—was even more interesting: https://news.ycombinator.com/item?id=253963 https://news.ycombinator.com/item?id=253963. It contains another skeptical comment: "The bigger problem is that everyone has a incentive to run their computers day and night cranking out solutions, which burns up lots of natural resources and processor time for a zero-sum result." (https://news.ycombinator.com/item?id=253999 https://news.ycombinator.com/item?id=253999), which seems astonishingly prescient now, even though the objection is still hotly (no pun intended) debated. Cherry-picking the wrongest-seeming skeptical comment from that collection of data points is a case study in survivorship bias. (I don't mean to pick on you personally! This is common of course.) The infamous Dropbox comment from 2008 is similar in that it has gotten repeated out of context in a way that is unfair to the original commenter (https://news.ycombinator.com/item?id=23229275 https://news.ycombinator.com/item?id=23229275). When we repeat these things, it says more about us than it does about the thing we're repeating.

yamrzou 5y ago

This is nice! How are the stories tagged?

wolfgarbe 5y ago
We are deriving the tags from the terms and bigrams in title, text and parsed html of linked web pages. Top frequent terms per post, but only if terms are within top 65k tags per index. Stopwords are excluded.
- deleted 5y ago
  [deleted]

yewenjie 5y ago

Nice to see full-text of the articles linked and really fast. Do you provide an API like Algolia does for HN?

wolfgarbe 5y ago
DeeepHN is powered by our SeekStorm search-as-a-service, which has a full-featured API: https://seekstorm.com/docs https://seekstorm.com/docs SeekStorm is intended that the user can index and search their own private data. But if there is demand we could also allow to search in a central public index for web data or other public data.
- morrbo 5y ago
  How does seekstorm perform on large documents? Say documents 1MB-10MB in size, are you able to search entire docs or just paragraphs within the doc?
  - wolfgarbe 5y ago
    Currently the maximum size of a single document is limited to 1 MByte. That is an artificial limitation resulting from the fact, that SeekStorm does not only indexes the content of the document, but also stores the original document. Somewhere there has to be a limit.

matt_wilson_206 5y ago

Seems odd to publish a benchmark against Lucene where your results are "Lucene crashes at 4 concurrent users". Since there are thousands of people using Lucene and products built on Lucene every day, I think the issue may be in your benchmarking code.

wolfgarbe 5y ago
There is no secret about the benchmark, its all open source: https://github.com/wolfgarbe/LuceneBench https://github.com/wolfgarbe/LuceneBench
- arafalov 5y ago
  It seems you tried MMapDirectory and then commented it out. https://github.com/wolfgarbe/LuceneBench/blob/master/LuceneBench/src/LuceneCore.java#L52-L53 https://github.com/wolfgarbe/LuceneBench/blob/master/LuceneB... So, you may be hitting SimpleFSDirectory instead, which does have issues with too many searches. Could you share the reasons, MMapDirectory did not work for you?
  - wolfgarbe 5y ago
    "NIOFSDirectory and MMapDirectory implementations face file-channel issues in Windows and memory release problems respectively. To overcome such environment peculiarities Lucene provides the FSDirectory.open() method. When invoked, it tries to choose the best implementation depending on the environment." https://www.baeldung.com/lucene-file-search https://www.baeldung.com/lucene-file-search
    - arafalov 5y ago
      Right. I found you are running on Windows after posting the comment. Makes sense, though there were some fixes in the latest Lucene I believe. The Lucene's own benchmarks are at: https://home.apache.org/~mikemccand/lucenebench/ https://home.apache.org/~mikemccand/lucenebench/ , though I admit to not know enough about benchmarking to form a strong opinion. Either way, good luck with the project/service. More competition is always great. The open-source components look interesting too.

nitramm 5y ago

It is really great tool too see what used to be very popular posts in the past. I have one observation. Posts linking to twitter - https://deephn.org/?sort=score&filter=%7B%22score%22%3A%7B%22from%22%3A1000%2C%22to%22%3Anull%7D%2C%22domain%22%3A%5B%22twitter.com%22%5D%7D https://deephn.org/?sort=score&filter=%7B%22score%22%3A%7B%2... - have most of the preview occupied with navigation and other not interested parts. So maybe it is possible to use some hacky solution to show the content part for the most popular websites, or if multiple pages are downloaded from the same domain the content part could be detected automatically.

wolfgarbe 5y ago
You are right. The preview is far from perfect. We just wanted to ship early. We will continue to fix and improve.

sturza 5y ago

How can we learn more about this indexing service?

wolfgarbe 5y ago
https://seekstorm.com/about-deephn https://seekstorm.com/about-deephn https://seekstorm.com/ https://seekstorm.com/ https://seekstorm.com/search-api-features https://seekstorm.com/search-api-features https://seekstorm.com/search-as-a-service-architecture https://seekstorm.com/search-as-a-service-architecture

daibo 5y ago

I assume you are using Symspell for fuzzy search and prioritize candidate frequency over edit distance. My input "algolai" returns results for "angola" even though "algolia" was just 1 transposition away. Anyway just dropping in to say that stuff on your github helped me figure out a lot while building a search engine from scratch. Good luck with SeekStorm!

wolfgarbe 5y ago
We are combining auto correction (SymSpell) and auto completion (PruningRadixTrie). That is sometimes tricky. We are using a static spelling correction dictionary which does not contains the term algolia, but angola. SymSpell is very fast, but requires a lot of memory. Therefore we are using a static dictionary, and not a dynamic dictionary derived from each individual index of each customer. Auto completion on the other hand we are dynamically generating for each individual index/customer.
- daibo 5y ago
  I am combining symmetric deletes with a trie for fuzzy autocompletion and it is working pretty well.

de6u99er 5y ago

I like it. I built something similar recently for indexing scientific publiations on r/COVID19 based on Elasticsearch and Kibana.

finikytou 5y ago
can you share a link?

0xmohit 5y ago

Curious to know how does it differ from HN Search [1]? HN Search API documentation [2]. [1] https://hn.algolia.com https://hn.algolia.com [2] https://hn.algolia.com/api https://hn.algolia.com/api

jankotek 5y ago
It searches content of referenced webpages. Also has filtering by year intervals (for example 2012-2016), domains, tags etc.. And I find usability better. I think it is great product.

crazypython 5y ago

I was hoping it would do full-text search of the text inside the linked pages, like hndex.org but with ranking.

johtso 5y ago
I think it is supposed to be doing that?
- crazypython 5y ago
  If I search "friendship" all the article titles have the word friendship in them. In my experience many HN titles don't contain the topic word.
wolfgarbe 5y ago
Yes, it does exactly that. If you check "Web" on the left sidebar, and uncheck "Stories" it will find the posts where "friendship" occurs in the linked web pages. But we still show the title of the original hacker news story in the search result and highlight the search term if it ALSO occurs in the title. If you scroll long enough within the search results you will finally reach results where the search term is not in the title, but only within the linked web page. It is more obvious if you search for queries which are less popular than "friendship"
Hitton 5y ago
You have to check box "Web" for it to search in linked articles too. Without that it highlights the words in articles that are found only because of title. And the indexing of contents isn't perfect, sometimes it seems to work for main article content but not for article author name even when it's shown.

reivalc 5y ago

Neat! I wonder if you quantified the performance of your search somehow? I guess labelling data for that would be laborious.

wolfgarbe 5y ago
You mean performance in terms of latency or relevancy?
- reivalc 5y ago
  Relevancy - I'd guess it's difficult to measure, due to the large dataset, the large number of possible search terms, and large number of possible results.

throwaway823882 5y ago

Interesting, it doesn't seem to show the dead pages. I have been indexing dead HN pages for several years now, with the idea that eventually I'd run some statistics on it to better understand both the auto-dead filters and the people who are submitting dead content.

berto4 5y ago

Can you search by location? lets say I have house listings and I want to search all the listings by location, does your api allows for this?

b1tshift 5y ago

Love it! I would kill for a reverse order button to see oldest posts first

wolfgarbe 5y ago
We can add that button, if you promise not to kill ;)
- b1tshift 5y ago
  Okay :D
  - wolfgarbe 5y ago
    Done. Now you can sort by newest/oldest date.

freediver 5y ago

The effort definetely gets my upvote. What did you use to crawl the pages and how long did it take? Curious about your experience doing it and crawler integration with Seekstorm. Is is possible to easilly expand the index with embeddings (vectors) and perform semantic search in parallel? Your pricing indicates that hosting similar index as this demo would cost $500/month. Wondering what kind of infrastructure is supporting the demo? Thanks! ps. Small quirk: https://deephn.org/?q=how+to+be+productive&filter=%7B%22hashtags%22%3A%5B%22CPU%22%5D%7D https://deephn.org/?q=how+to+be+productive&filter=%7B%22hash... First three tags seem not to be relevant to the post itself.

wolfgarbe 5y ago
Crawling speed is between 100...1000 pages per second. We crawled about 4 million linked unique web pages The pricing for an Index like DeepHN would be $99/month: While we are indexing 30 million Hacker news posts, for DeepHN we are combining a single HN story with all its comments and its linked webpage into a single SeekSorm document. So that we index just 4 million SeekStorm documents. Yes, it would be possible to expand index with embeddings (vectors) and perform semantic search. This would we an auxiliary step between crawling and indexing.
- freediver 5y ago
  What is being used as a crawler and is it integrated with Seekstorm? The same article referenced here https://deephn.org/?q=how+to+be+productive&filter=%7B%22hashtags%22%3A%5B%22CPU%22%5D%7D https://deephn.org/?q=how+to+be+productive&filter=%7B%22hash... contains the phrase 'well-defined' Any idea why doesn't the article surface when searching for this ?
  - wolfgarbe 5y ago
    The crawler is a part of SeekStorm. "Well-defined": Just a guess: We are doing key text extraction, i.e. we try not to index boilerplate stuff an and menu items. As "well-defined" is within a short list item, it might be accidentally skipped. So, that is not yet perfect, and considering the diversity in web page structure it probably never will. But we will try to improve.

abcc8 5y ago

I'm curious how this site is compliant with copyright law. I've thought of working on similar projects, but the possibility of being sued has dissuaded me.

deleted 5y ago
[deleted]
doh 5y ago
Search engines are generally considered fair use [0], at least in US. So up until HN doesn't send cease and desist, it's in the clear. Once it does, then it's all about what the claims are. Google litigated the crap out of this [1] over the years, so there is a positive precedent minefield all over the place. [0] https://www.everycrsreport.com/reports/RL33810.html https://www.everycrsreport.com/reports/RL33810.html [1] https://en.wikipedia.org/wiki/Fair_use#Text_and_data_mining https://en.wikipedia.org/wiki/Fair_use#Text_and_data_mining
ehmmmmmmmm 5y ago
It's probably not compliant. As with all good things, if you're in the Land of the Free (TM), eventually people will hunt you down if you try to do good things. Alternatively, base your site out of a country which gives you the freedom to do things like this. Build it in China and you'll be applauded instead of sued. Copyright suits for petty things like this would laughed off when it's something actually useful.
- NicoJuicy 5y ago
  China's monopolies are far worse than in other countries. Founders there have no choice but to sell or get copied by the platform that you're building on. It's baba or tenscent.
  - dheera 5y ago
    Yes, that's also true, but on the flip side, technological advancement of society as a whole is not stifled by idiotic IP law. I agree, some anti-monopoly regulation is in order in China, and I'm not a fan of the Alibaba/Tencent monopolies on the ecosystem. However, I think there are other ways to go about fostering and encouraging small businesses and protecting creators and their intentions than American-style IP law, which allows you to sit on an invention or work and hinder society from having access to the fruits of science, which I'm vehemently against. IP isn't even real property, IMHO.

youngNed 5y ago

So, can you explain what is going on with 'covid' articles from <2019 - are these where the original page that was linked to has now become a spam link?

wolfgarbe 5y ago
Not necessarily a spam link. Linked pages from older post have sometimes updated their content. Sometimes this is legitimate, sometimes spam.

karterk 5y ago

Really cool demo. I'm curious to know what kind of hardware this is this hosted on. Full disclosure: I work on a similar fast, typo tolerant, fuzzy search engine search engine called Typesense (https://github.com/typesense/typesense https://github.com/typesense/typesense).

goodmachine 5y ago
Yeah, was also interested. It would be nice to see Typesense take up the same challenge for comparison ;)
- karterk 5y ago
  Happy to benchmark if the dataset is shared!
wolfgarbe 5y ago
SeekStorm uses dedicated root servers with NVMe SSDs, hosted by IONOS https://www.ionos.com/servers/intel-servers https://www.ionos.com/servers/intel-servers

zoomablemind 5y ago

Amazingly fast! I'm used to seeing a Firefox's load progress bar for most of requests on my mobile, yet here the results "just show up". More observations: - When trying to go back after clicking the original HN link, there's no response. - Clicking on what I understand are tags (hashtags) in a given post has no effect. - Sorting by date is working, though it'd be nice to see the date. Also a question: how do you generate the tags (hashtags) from the original contents?

wolfgarbe 5y ago
>> When trying to go back after clicking the original HN link, there's no response. The original HN link is opened in a new browser page/tab. The search results should remain unchanged in the previous tab. So you don't use the back button, but the previous browser tab. >> Clicking on what I understand are tags (hashtags) in a given post has no effect. Search for "google", go to the first result. there is a hashtag "oracle", click and the search results are filtered by that hashtags. Please post an example where its doesn't work, so that we can fix it. The date is shown in the preview panel on the right hand side. But wa are thinking adding the time also to the result list. We are deriving the tags from the terms and bigrams in title, text and parsed html of linked web pages. Top frequent terms per post, but only if terms are within top 65k tags per index. Stopwords are excluded.
- zoomablemind 5y ago
  > Please post an example where its doesn't work, so that we can fix it. Note: I'm trying this in a mobile browser (Firefox). - Search for a word 'Decentralized' - pick the second result: 'A decentralized web would give power back to the people online' - On the result's (preview?) page, click the tag 'Decentralization' I get no response, nothing updates, no new tab opens. Not sure what is the intended action (I'd assume it should be a new set of results corresp. to the hashtag). Same issue with 'google' example that you described.
  - wolfgarbe 5y ago
    If you click a tag on the result/preview page (in mobile Firefox), the result list is filtered by the tag (you get a new/shorter result list with an updated result number). But because you are still on the preview page you don't immediately see this as it's done in the background. But once you close the preview page (with the big arrow on the top/left of the preview page) you will see the updated result list and number. Of course, this is a design flaw, we should automatically close the preview window once you select a tag to immediately show you the updated result list. In the desktop version, this is no issue, as there result list and preview window are always simultaneously visible.

localhost 5y ago

Have you tried against a corpus of text like email? How well does that work? I've been frustrated with the search speed (and results) from email search in general and was wondering if I could do better with an offline search?

wolfgarbe 5y ago
That would definitely work. We are working on local client that could index your local PDF and Word documents. So importing emails would be a nice addition.
- localhost 5y ago
  I'd be interested in seeing how well it performs vs. integrated search. Outlook search (which is what I use) suffers from performance and relevance issues, yet when I tried this experiment with another full-text search provider (Azure Search) it performed worse on relevancy. I use email as kind of an activity stream - generally I can remember roughly the people involved and the time period. Would be lovely to have search that performs well in this regard.

artembugara 5y ago

Nice! How did you fetch the content of those articles?

wolfgarbe 5y ago
With the crawler which is part of the SeekStorm search-as-a-service.

LinuxBender 5y ago

We're sorry but seekstorm-deephn doesn't work properly without JavaScript enabled. Please enable it to continue. Could the folks with JS disabled get a small semi-static page with the top interesting HN stats, and a message that say something like "Here are some simple stats. For the full interactive experience, please enable javascript"?

wolfgarbe 5y ago
Good idea!
eloff 5y ago
For the less than 1% of such people? I doubt that it's worth it. If somebody can't be bothered to enable JavaScript to use the site, they're not a desirable customer anyway.

superasn 5y ago

Very cool and useful. I was trying to find all comments on HN which contains the word youtube.com (basically trying to find some recently linked to videos from comments) but it returns 0 results when i set the search filter to comments(1). Any idea what I'm doing wrong? https://deephn.org/?q=youtube.com&sort=time&in=Comments https://deephn.org/?q=youtube.com&sort=time&in=Comments

wolfgarbe 5y ago
We will look into this.

smusamashah 5y ago

https://deephn.org/?q=procedural+planet&filter=%7B%22domain%22%3A%5B%22jsfiddle.net%22%5D%7D https://deephn.org/?q=procedural+planet&filter=%7B%22domain%... A bug probably. Huge amount of text in title here.

wolfgarbe 5y ago
Thank you. We will look into this and fix it asap.

smusamashah 5y ago

Can we exclude certain terms in search with a "-" e.g. "-oculus"?

wolfgarbe 5y ago
It should, but it currently doesn't in DeepHN. We will fix this asap.
wolfgarbe 5y ago
The '-' operator has been fixed. 'vr -oculus' should work correctly now.

djhn 5y ago

A page to bookmark for sure and an interesting product. The UI takes getting used to, but then again so do many actually useful tools.

wolfgarbe 5y ago

Thank you all for your great feedback. We will fix all the bugs and add some of the suggested features and release a next iteration of DeepHN within the next few days!

codeproject 5y ago
that is a great site. To be really honest, Those bugs are not much trouble compared with your service to all. that is really remarkable useful website. Thank you so much from the bottom of our hearts.

FairDune 5y ago

Apologies for being nitpicky about the already fixed back button While viewing a specific drilled-down result, each time the back button is pressed, it seems to navigate to the previous character in the incremental search. In other words, if the search term is 'C#', navigating back seems to highlight 'C' in the result detail page.

wolfgarbe 5y ago
This is most likely a timing problem. If you type c# fast, then the back button brings you back to the previous query. If you pause between typing 'c' and '#'. then it brings you back to the previous character only. That is related to our instant search, where you don't need to hit the return key to search. This causes the problem, that we need to identify when a query ends and the next query starts, to create the correct entries for the back button. We do this via timing - if the pause between key presses is too long, a new, separate entry is assumed. As the DeepHN is a single dynamic HTML page, we need to implement and manage the browser navigation logic ourselves.