4 ms·

I've tried it out. It's quite obvious the limited number of crawled sites when searching for anything obscure or one step outside of programming. Even 'javascr

I've tried it out. It's quite obvious the limited number of crawled sites when searching for anything obscure or one step outside of programming. Even 'javascript reverse string', which I expected some docs or stack overflow pages seems to give me a HN thread, someone's github repo and a not very related SO thread. Is MDN, MDSN, more dev docs documentation on the roadmap? It's definitely an interesting technique. Do you have anything in place to detect garbage, substenceless articles like which has started popping up on Google? I've seen the occasional one using github repositories or pages. Looking at the current list you're broadly reliant on moderators and communities, and as the search engine you moderate which sites are indexed.

alangibson 5y ago
> one step outside of programming. Indeed, it's explicitly for programming only (for now). > Even 'javascript reverse string', which I expected some docs or stack overflow pages seems to give me a HN thread, Next up is indexing language documentation. At this point I'm relying heavily on Q&A and community sites since they have their own built in quality rankings. > Is MDN, MDSN, more dev docs documentation on the roadmap? Most definitely. Feel free to dump a list of urls of your favorite doc sites. I'm building a whitelist now. > Do you have anything in place to detect garbage, substenceless articles like which has started popping up on Google? My strategy is to not index spam in the first place. That's why I started by extracting links from with community sites that have their own moderation in place. The next step is to whitelist high quality sites. That is potentially a huge list to maintain, which is why I'm am narrowly focused on software development. Everything old is new again...
- ejp 5y ago
  I use this documentation aggregator/search in the browser to access most language docs. It might serve as a whitelist starting point! https://devdocs.io/ https://devdocs.io/
  - alangibson 5y ago
    Nice. Thanks!
- pbhjpbhj 5y ago
  >At this point I'm relying heavily on Q&A and community sites since they have their own built in quality rankings. // How are you using the "built in quality rankings", could you give some examples? On Reddit, say, except in a few groups like AskHistorians you can still get very high ranking for a meme post and very low ranking for a list that has high informational value. StackOverflow is extraordinarily prone to killing off reasonably good contributions and giving very high ranking to out of date answers (the latter is the biggest problem with SO sites at present IMO).
- thewebcount 5y ago
  If you haven't already, please crawl <https://developer.apple.com https://developer.apple.com> and <https://swift.org https://swift.org>.
- vulcan01 5y ago
  Ok, site request (aside from MDN): pkg.go.dev Many of these are linked to GitHub/GitLab repos, so I'm not sure how you'll deduplicate that.

alangibson 5y ago

> one step outside of programming. Indeed, it's explicitly for programming only (for now). > Even 'javascript reverse string', which I expected some docs or stack overflow pages seems to give me a HN thread, Next up is indexing language documentation. At this point I'm relying heavily on Q&A and community sites since they have their own built in quality rankings. > Is MDN, MDSN, more dev docs documentation on the roadmap? Most definitely. Feel free to dump a list of urls of your favorite doc sites. I'm building a whitelist now. > Do you have anything in place to detect garbage, substenceless articles like which has started popping up on Google? My strategy is to not index spam in the first place. That's why I started by extracting links from with community sites that have their own moderation in place. The next step is to whitelist high quality sites. That is potentially a huge list to maintain, which is why I'm am narrowly focused on software development. Everything old is new again...

ejp 5y ago
I use this documentation aggregator/search in the browser to access most language docs. It might serve as a whitelist starting point! https://devdocs.io/ https://devdocs.io/
- alangibson 5y ago
  Nice. Thanks!
pbhjpbhj 5y ago
>At this point I'm relying heavily on Q&A and community sites since they have their own built in quality rankings. // How are you using the "built in quality rankings", could you give some examples? On Reddit, say, except in a few groups like AskHistorians you can still get very high ranking for a meme post and very low ranking for a list that has high informational value. StackOverflow is extraordinarily prone to killing off reasonably good contributions and giving very high ranking to out of date answers (the latter is the biggest problem with SO sites at present IMO).
thewebcount 5y ago
If you haven't already, please crawl <https://developer.apple.com https://developer.apple.com> and <https://swift.org https://swift.org>.
vulcan01 5y ago
Ok, site request (aside from MDN): pkg.go.dev Many of these are linked to GitHub/GitLab repos, so I'm not sure how you'll deduplicate that.