5 ms·

I guess cppreference.com isn't even a part of the list? I tried a couple test queries: > lambda decay to function pointer c++ I get some FSF pages and the wi

by verall 4y ago

I guess cppreference.com isn't even a part of the list? I tried a couple test queries: > lambda decay to function pointer c++ I get some FSF pages and the wikipedia for Helium? > std function I get... tons of Rust docs? > std function c++ All rust docs? The wikipedia page for C++?? Interesting idea, but this seems like it would be the primary failure mode for an idea like this: as soon as you are researching outside of the curator's specializations, it doesn't have what you're looking for. Yet these results would both be fixed simply by adding cppreference.com to the index. Let's try and give it a real challenge: > How to define systemverilog interface And as I might expect, I get wikipedia pages. For "Verilog", for "System on a Chip" and for "Mixin". 1st google result: > An Interface is a way to encapsulate signals into a block... Working as expected

reidrac 4y ago
I guess missing content can be explained by... > Notice! This product is experimental and incomplete. User beware! But in reality, if you know where is the answer that you are looking for, why would you use that search engine. I use DDG and if I want to search on the Scala docs, I use "!scala whatever I'm searching" instead of just search with DDG.
meibo 4y ago
GitHub doesn't seem to be either. I get that it's a competitor but not being able to search GitHub is probably a deal breaker for most devs that aren't Drew.
- ddevault 4y ago
  I'm not opposed to indexing GitHub, but the signal to noise ratio on GitHub is poor. Nearly all GitHub repositories are useless, so we'd have to filter most of it out. I think instead I'll have to set it up where people can request that specific interesting repositories are added to the index, and maybe crawl /explore to fill in a decent base set.
  - interactivecode 4y ago
    Perhaps all repo's that have a published package is a good heuristic. Then you'll at least get all the repos of npm, python and other packages.
    - BlackLotus89 4y ago
      Some interesting repos have no published packages. A combination of number of commits, stars and forks would be probably more relevant.
      - yellowapple 4y ago
        And likewise, some uninteresting repos do have published packages.
  - marginalia_nu 4y ago
    GitHub is hella tricky to crawl too due to its sheer size and single entry point (meaning slow crawl speed). I've been looking at the problem as well, and so far just ignored it as un-crawlable, but I might do something like crawl only the about pages for repos that are linked to externally some time in the future.
    - mdaniel 4y ago
      There's an asterisk to that: they serve the underlying content through two different APIs so one can side-step the HTML wrapper around the bytes: the discovery phase has a formal API (both REST and GraphQL) for finding repos, and then the in-repo content can be git-cloned and one can locally index every branch, commit, and blob, without issuing hundreds of thousands of http requests to GH. One would still need to hit GH for the issues, if that's in scope, but it'd be way less http requests unless your repo is named kubernetes or terraform.
      - marginalia_nu 4y ago
        We're still talking about git clone:ing a hundred thousand github repos. Git Repos get big very fast. That's a lot of data when you're realistically only interested in is a few markdown files per repo.
ddevault 4y ago
I added cppreference.com now and kicked off a crawl. It'll be a while. The list of domains is pretty small right now -- it was intended to be bigger before the announcement was made. Will also add RFCs and man pages soon. There will (soon) be a form to request new domains are added to the index, so if there are any sites you want indexed which are outside of my personal expertise, then you'll be able to request them.
- ZWoz 4y ago
  You probably are already thought about it, but just in case feature idea: adding moderation support for collaboration. Somewhat trusted persons vetting niche subjects.

by verall 4y ago

reidrac 4y ago

I guess missing content can be explained by... > Notice! This product is experimental and incomplete. User beware! But in reality, if you know where is the answer that you are looking for, why would you use that search engine. I use DDG and if I want to search on the Scala docs, I use "!scala whatever I'm searching" instead of just search with DDG.

meibo 4y ago

GitHub doesn't seem to be either. I get that it's a competitor but not being able to search GitHub is probably a deal breaker for most devs that aren't Drew.

ddevault 4y ago
I'm not opposed to indexing GitHub, but the signal to noise ratio on GitHub is poor. Nearly all GitHub repositories are useless, so we'd have to filter most of it out. I think instead I'll have to set it up where people can request that specific interesting repositories are added to the index, and maybe crawl /explore to fill in a decent base set.
- interactivecode 4y ago
  Perhaps all repo's that have a published package is a good heuristic. Then you'll at least get all the repos of npm, python and other packages.
  - BlackLotus89 4y ago
    Some interesting repos have no published packages. A combination of number of commits, stars and forks would be probably more relevant.
    - yellowapple 4y ago
      And likewise, some uninteresting repos do have published packages.
- marginalia_nu 4y ago
  GitHub is hella tricky to crawl too due to its sheer size and single entry point (meaning slow crawl speed). I've been looking at the problem as well, and so far just ignored it as un-crawlable, but I might do something like crawl only the about pages for repos that are linked to externally some time in the future.
  - mdaniel 4y ago
    There's an asterisk to that: they serve the underlying content through two different APIs so one can side-step the HTML wrapper around the bytes: the discovery phase has a formal API (both REST and GraphQL) for finding repos, and then the in-repo content can be git-cloned and one can locally index every branch, commit, and blob, without issuing hundreds of thousands of http requests to GH. One would still need to hit GH for the issues, if that's in scope, but it'd be way less http requests unless your repo is named kubernetes or terraform.
    - marginalia_nu 4y ago
      We're still talking about git clone:ing a hundred thousand github repos. Git Repos get big very fast. That's a lot of data when you're realistically only interested in is a few markdown files per repo.

ddevault 4y ago

I added cppreference.com now and kicked off a crawl. It'll be a while. The list of domains is pretty small right now -- it was intended to be bigger before the announcement was made. Will also add RFCs and man pages soon. There will (soon) be a form to request new domains are added to the index, so if there are any sites you want indexed which are outside of my personal expertise, then you'll be able to request them.

ZWoz 4y ago
You probably are already thought about it, but just in case feature idea: adding moderation support for collaboration. Somewhat trusted persons vetting niche subjects.