12 ms·

Show HN: DontBeEvil.rip: Search, for developers (API, expressions, CLI)

269 points by alangibson 5y ago

I'd like to invite everyone to try out DontBeEvil.rip, an experimental search engine for developers. tl;dr $ alias rip="curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search https://dontbeevil.rip/search --data-urlencode " $ rip 'q=Heartbleed bug' DontBeEvil.rip is a year long experiment to see if a small team can build a developer-focused search engine that is self-sustaining on $10 monthly subscriptions. It works by only indexing high-quality resources that are relevant to developers. You won't get useless listicles because we'll never crawl them. Relevant urls are harvested from HN, StackOverflow, programmer Reddit, and a few others. Page content comes mostly from the Common Crawl project. The limited, but awesome, features in this first release are: - Expressions! Experience the power of Elasticsearch’s Simple Query Strings. - REST API. Just change 'text/plain' to `application/json` in the above alias. - CLI. Just use curl in the terminal. Simple as. HackerNews, StackOverflow, Arxiv abstracts, 2M Github repos, and programmer Reddit (up to 2020) are being indexed right now. There's much more to come in the next few months. I'd love to hear your questions, comments and suggestions in the comments below.

przemub 5y ago
I really like it and it already gives some useful results. A rise of curated search engines as yours would be lovely. It would be nice if the main page linked to your blog or anything really, because I would like to know where can I follow this project!
- alangibson 5y ago
  Thanks! I'm giving this project a year to build up momentum. If it looks promising, I plan on having other STEM verticals. Maybe even fix recipe searches one day :) A real homepage is coming. Feel free to subscribe to my blogs RSS feed for now: https://landshark.io/feed.xml https://landshark.io/feed.xml
eatbitseveryday 5y ago
I would recommend adding technical blogs. Not by hand, but if you can automate identifying some. Many are small but have good content. Edit: also some corporate technical documentation like Mozilla, Microsoft, IBM, etc have many such developer pages.
- alangibson 5y ago
  I automate it by pulling urls out of HN, programmer Reddit, etc. Right now my only source of page content is the Common Crawl, which is why there are relatively few web pages indexed. That will change. A next step is to index entire sites, not just individual pages, based on the positive votes their links get.
amar-laksh 5y ago
I saw your blog post a couple of days ago, This looks really promising!
- alangibson 5y ago
  Thanks! I'll be updating that post today. I changed quite a few things getting ready for this Show HN, so it's now out of date.
husainfazel 5y ago
Going to "https://dontbeevil.rip/ https://dontbeevil.rip/" results in a JSON error in the browser: {"message":"Missing Authentication Token"}
- kiettv 5y ago
  You need to access the sub path /search?q= :)
- przemub 5y ago
  Apparently developers need no homepages, just APIs :)
  - alangibson 5y ago
    Jup. No time for fancy things like HTML yet :)
  - genewitch 5y ago
    i run a few dozen internet and web services with FQDN and only 1 of them has something if you type http://example.com http://example.com - no homepages, but there's a webserver listening on ~80% of the domains. Stop attacking me!
- alangibson 5y ago
  There's nothing there yet. I'm 100% focused on building the index and tuning the master search query. There is however a blog post that goes into more detail on how to do things like pagination soon. tl;dr: https://dontbeevil.rip/search?q=monads&from=10 https://dontbeevil.rip/search?q=monads&from=10
kiettv 5y ago
It's powered by ElasticSearch, is it? So I can use all of its query parameters?
- alangibson 5y ago
  Indeed it is. You can you use simple query strings for the q parameter. See https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... I'm considering opening up full ES query support for paying customers, but it's too dangerous to expose it to the Internet unrestricted.
  - rmbyrro 5y ago
    I think it's dangerous to expect a malicious actor would not pay $10 to screw your service.
    - cj 5y ago
      "Risk management" is often not the same as "risk elimination"
    - alangibson 5y ago
      Indeed it is. Presumably I would have had time to build up some safeguards and run beefier servers by then though.
      - marginalia_nu 5y ago
        As a word of warning, when HN discovered my search engine, I was hit hard by a botnet within a few days. Saw about 30-40k queries/hour from some 10k IP addresses. I'm self hosted so the worst that happened is my search engine was a bit slow, but if I was cloud hosted I'd have a very sizable bill to pay. If you do not already have a global rate limit, implement one ASAP. Better to have one and not need it, than to need it and not have it.
        5y ago
dorianmariefr 5y ago
I'm getting: > {"message": "Internal server error"}
- alangibson 5y ago
  What query are you running?
- alangibson 5y ago
  Give it another try. I fixed a flaw in the json to text translation.
  - dorianmariefr 5y ago
    Thanks, it works now
feanaro 5y ago
Getting internal server error for many ordinary requests. I'm not able to discern a pattern. An example is `rip 'q=zelda'`.
- alangibson 5y ago
  Thanks for the report. I'll get this fixed. In the mean time you can use application/json: curl -G -H 'Accept: application/json' https://dontbeevil.rip/search?q=zelda https://dontbeevil.rip/search?q=zelda
- alangibson 5y ago
  It should be fixed now.
erklik 5y ago
> programmer Reddit Is it just /r/programmer? or many other programming related subreddits? In general though, this is great. I would similarly love a solution where we could submit sites to be indexed. A way to have a search engine for all the websites I want specifically would be awesome. You could probably add some sort of popular filter on top of it so that only sites popular enough get indexed. I don't know. Just an idea. I love the fact that it's accessible from the terminal. That's fantastic. Although, would be nice to have a very simple HTML front-end. Think very early Google or go very brutalistic. Anyway, excited to hear about it. Edit: Doing the following gives me an Internal Server Error for some reason. curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search https://dontbeevil.rip/search --data-urlencode 'q=Notes' {"message": "Internal server error"}
- alangibson 5y ago
  > Is it just /r/programmer? or many other programming related subreddits? There's about 30 programming focused subreddits. > I would similarly love a solution where we could submit sites to be indexed. This is on the plan. I want to allow common interest groups to maintain their own search verticals. I also want to allow individual users to add everything from bookmarks to notes (privately only of course) to act as a sort of external memory. That's very long term though. > Doing the following gives me an Internal Server Error for some reason. Should be fixed now
ykevinator2 5y ago
Can we drop the q= and the quotes from the shell cmd somehow, that would make it so much nicer, and rip is a great command line.
- alangibson 5y ago
  There's a real command line coming. If you're on a Debian Linux and feel like testing it out, just do apt install curl jq pip3 install jtbl curl -O https://raw.githubusercontent.com/alangibson/dontbeevil.rip/main/src/rip https://raw.githubusercontent.com/alangibson/dontbeevil.rip/... chmod u+x rip ./rip 'what is a monad'
  - l0b0 5y ago
    With long options, JSON output, and no extra Python dependencies: rip() { curl \ --data-urlencode "q=${1}" \ --get \ --header 'Content-Type: application/json' --header 'Accept: application/json' \ --silent \ 'https://dontbeevil.rip/search' \ | jq '[ .hits.hits[] | { title: .fields.title[0], url: .fields.url[0], highlight: .highlight.text[0] } ]' }
- alexrsagen 5y ago
  This function should work for you: $ rip() { curl -G -H "Accept: text/plain" --url https://dontbeevil.rip/search https://dontbeevil.rip/search --data-urlencode "q=$*"; } $ rip Heartbleed bug edit: alangibson's solution in this thread is better :)
mynameismon 5y ago
> StackOverflow Does this include the entire StackExchange network, or only StackOverflow? Because some SE sites (in particular, UnixSE and ServerFault) also produce highly relevant results.
- alangibson 5y ago
  > Does this include the entire StackExchange network Not yet. I'm focused on explicitly developer-oriented resources right now. Those you mentioned are on the TODO list though.
sitkack 5y ago
I love this idea. This is mostly just raw data, it isn't that useful (yet). The security issues with using curl directly to my terminal feel a bit dangerous. I'd rather use my browser and be able to see the results over a json tree. Providing raw access to ES has a high risk to reward. The search results for https://dontbeevil.rip/search?q=python%20context%20manager%20contextlib https://dontbeevil.rip/search?q=python%20context%20manager%2... are non-topical hits on SO records. I even put the name of the python package (from stdlib no less) in the query string. I was able to find what I needed on devdocs.io in less than 10 seconds. https://devdocs.io/python~3.10/library/contextlib#contextlib.contextmanager https://devdocs.io/python~3.10/library/contextlib#contextlib... In no way am I trying to discourage you, but until the basics are in place, search over arxiv abstracts is way less useful than just SO and docs (language and libraries). I would recommend returning text/plain by default and .json if someone asks for it (in the url), no everyone can set headers. I'd also put an about page at the root of your site, plain text is fine.
- alangibson 5y ago
  Thanks for the feedback. I'm considering making searching for language docs a special use case, maybe even with its own index. > Providing raw access to ES has a high risk to reward. I'm not quite that foolhardy :). The only thing that gets passed to ES is a sanitized simple_query_string.query. Should be relatively safe. I'm sure someone will prove me wrong though.
- tekromancr 5y ago
  What security implications does running curl have that wouldn't be present in a browser?
  - artursapek 5y ago
    I’m wondering the same. You’re not piping them into a shell.
  - feanaro 5y ago
    There have been instances of terminal vulnerabilities via terminal escape codes, as bad as an RCE in iterm2: https://blog.mozilla.org/security/2019/10/09/iterm2-critical-issue-moss-audit/ https://blog.mozilla.org/security/2019/10/09/iterm2-critical.... I suppose the OP is thinking of something like that.
bryanrasmussen 5y ago
how's ranking done, I searched for xslt and I saw a lot of HN results in the first part, seemed weird that HN would rank highly for that.
- alangibson 5y ago
  Results are heavily weighted to HN and Stackoverflow right now because they are the easiest resources to access and rank. Since posts have a score on both platforms, it's easy to pull out some 'authority' signal. There's many more web pages coming. They are much more difficult to get ahold of and rank though because I need to run my own crawler to fill in what Common Crawl doesn't have and then calculate my own site authority rankings.
wodenokoto 5y ago
My shell-fu isn't the greatest. I thought I could be clever and do >alias rip="curl -G -H 'Accept: text/plain' \ --url https://dontbeevil.rip/search --data-urlencode q=" >rip hello {"message": "Missing required request parameters: [q]"} curl: (6) Could not resolve host: hello
- harryvederci 5y ago
  You could also put a shell script on your `PATH` instead of creating an alias: #! /bin/sh query_string="q=$@" curl --get \ --header 'Accept: text/plain' \ --url https://dontbeevil.rip/search \ --data-urlencode "${query_string}"
  - mrlemke 5y ago
    I'd do it almost the same but without the variable. Note: the long shebang is for using on Termux, PC users should change it to something like #!/use/bin/env sh. #!/data/data/com.termux/files/use/bin/env sh curl -G -H "Accept: text/plain" \ --url "https://dontbeevil.rip/search" \ --data-urlenconde "q=$*"
    - harryvederci 5y ago
      Didn't know about "$*", thanks! Edit: typo in your version: "urlenconde"
      - laumars 5y ago
        Also every instance of ‘usr’ has been autocorrected to ‘use’. Autocorrect does make me laugh some days :D
        mrlemke 5y ago
        That's what I get for phone posting. Thanks for pointing it out you two!
- 5y ago
niek_pas 5y ago
This looks really cool! It would be neat to have a proper CLI with a more fully-flushed out UI, with things like shortcuts to quickly open links. Is there any way I can be kept up to date with the state of this project? Also, am I correct in assuming it's not open source?
- kordlessagain 5y ago
  Like an ultra powerful goosh.org UI, with AI command synthesis, image uploads, crawling, search, opening pages, etc.
  - alangibson 5y ago
    CLI in the browser. I love it.
- alangibson 5y ago
  The repo is over here: https://github.com/alangibson/dontbeevil.rip https://github.com/alangibson/dontbeevil.rip You'll be disappointed though as most of the important stuff only lives as BigQuery queries. I will be updating it in the near future though.
dotancohen 5y ago
This is great, even if I'm not getting as good results as from google for now. Can you expose the filtering features of ES? I'd love to query e.g. "+python lists" and get results only related to Python, no e.g. Lisp results. For Stack Overflow you could use the question's tags as filter keys, and for other sites you'd add them manually (so e.g. the PHP docs get the PHP key). If you're thinking of monetizing this, I'll tell you what I tell all the small, useful services that I'd like to pay for. There are too many small, useful services that I'd like to pay for. I'll gladly pay $1 for such a service, but you'll have a hard time convincing me to pay more.
operator-name 5y ago
I've tried it out. It's quite obvious the limited number of crawled sites when searching for anything obscure or one step outside of programming. Even 'javascript reverse string', which I expected some docs or stack overflow pages seems to give me a HN thread, someone's github repo and a not very related SO thread. Is MDN, MDSN, more dev docs documentation on the roadmap? It's definitely an interesting technique. Do you have anything in place to detect garbage, substenceless articles like which has started popping up on Google? I've seen the occasional one using github repositories or pages. Looking at the current list you're broadly reliant on moderators and communities, and as the search engine you moderate which sites are indexed.
- alangibson 5y ago
  > one step outside of programming. Indeed, it's explicitly for programming only (for now). > Even 'javascript reverse string', which I expected some docs or stack overflow pages seems to give me a HN thread, Next up is indexing language documentation. At this point I'm relying heavily on Q&A and community sites since they have their own built in quality rankings. > Is MDN, MDSN, more dev docs documentation on the roadmap? Most definitely. Feel free to dump a list of urls of your favorite doc sites. I'm building a whitelist now. > Do you have anything in place to detect garbage, substenceless articles like which has started popping up on Google? My strategy is to not index spam in the first place. That's why I started by extracting links from with community sites that have their own moderation in place. The next step is to whitelist high quality sites. That is potentially a huge list to maintain, which is why I'm am narrowly focused on software development. Everything old is new again...
  - ejp 5y ago
    I use this documentation aggregator/search in the browser to access most language docs. It might serve as a whitelist starting point! https://devdocs.io/ https://devdocs.io/
    - alangibson 5y ago
      Nice. Thanks!
martinald 5y ago
Why not make a website for this? Why just limit it to the terminal (hard to use on mobile for example)? Edit: obviously you can query it from a browser, but it would take like a couple hours to have a view that parsed the json and put it in a google-style layout with a search bar.
- alangibson 5y ago
  > Why not make a website for this? It's coming. I decided to get it out there as soon as I had an index that could theoretically be useful. The feedback I'm getting will drive the next chunk of work that gets done. For instance, I'll probably bring in language docs next as a lot of people have asked for them already.
baggachipz 5y ago
1. Stand up ElasticSearch instance 2. Have it index SO and HN 3. Charge $10 per month 4. Profit!!
- alangibson 5y ago
  Maybe you should read the other posts about future plans and how this is extremely alpha. Or maybe go anywhere else and do literally anything else but be obnoxious in this thread. I'm spending over $200 per week just to stand up the service as it is. $10 for a full functional search engine will likely not be even close to PROFIT!!!!
  - sdesol 5y ago
    > I'm spending over $200 per week just to stand up the service as it is What are using if you don't mind me asking? Not trying to criticize or anything. I have a Heztner box that gives me 1TB SSD in RAID 0 mode and 64 GB of RAM for about 80 CAD a month.
    - alangibson 5y ago
      I have 3 sizeable EC2 instances running an ElasticSearch cluster, plus a beefy box for data preprocessing and crawling. A big chunk actually goes to BigQuery. There are publicly available datasets for HN, Stackoverflow and a few others there. I've also loaded up the Common Crawl index. The query and storage fees really add up. I'm hopefully done with huge BigQuery queries, so that $200 will probably drop for a while.
      - rmbyrro 5y ago
        I'm probably wrong about my assumptions, but presume you are open to any kind of constructive feedback, so here it goes... Maybe you're overkilling with the infra stack. I would simplify until having a mature product, especially if I'm bootstrapping, which I think is your case. Right now, you're still a bit far from MVP, from my point of view. Those $200 can probably be reduced by 50%-75% if you compromise on stuff only important to non-alpha services (i.e. 99.99% availability). A single EC2 box should be enough. Maybe look into Postgres or another FOSS instead of BigQuery. These $100-$150 savings per week can go into promoting your service, getting as much attention as possible to maximize feedback. Good luck!
arminiusreturns 5y ago
Best of luck to you, I think there is a targettable niche that could utilize this. Having thought quite a bit about the search space, I think a whitelist approach is going to be the next big search thing, because advertising and bs sites have corrupted SEO too far. I'm reminded of the site indexer websites in the early days of the internet. Curation if done properly (based on quality of content and not certain other factors that currently play too heavy a role in the seo algo black boxes) seems to be how we adapt to the current information tsunami we are all dealing with. I think a long time ago I decided I would even pay for such a service, just like I am willing to pay for a good new source (FT for me, not cheap, but worth it). Im not positive the 10$ mark is low enough but I hope for the general landscape it is. Just dont forget to keep dontbeevil more than a catchphrase. In particlar, please be transparent with what user data you collect and how you use it.
- alangibson 5y ago
  > I think a whitelist approach is going to be the next big search thing It almost as to be. Spammers, growth hackers, et al. are just too numerous and too good. > Im not positive the 10$ mark is low enough but I hope for the general landscape it is. I saw enough people mention $10 that I decided to go for it. To be honest, $10 is already probably too low to be sustainable because of the huge amount of resources search consumes and the high cost of development. My gut feeling is that it's economically impossible to build a good search engine that isn't loaded with ads and spyware. But I spent so long complaining about G that I decided to prove to myself one way or another.
  - wolfgang42 5y ago
    > because of the huge amount of resources search consumes I’ve been intermittently working on much the same idea as the OP, and I suspect this is actually a lot less of a problem than it seems, since they’re focusing on a niche. Indexing everything the way Google does requires a lot of resources, but indexing the majority of useful material in a specific domain takes a lot less. (My ElasticSearch index for the entirety of StackOverflow is a mere 40 GB, for example.) By far the more expensive part is likely to be paying market rates for a developer (you need a decent number of users paying $10/mo to hit a mid-market salary), but in theory this scales relatively independently of userbase. Edit: I’ve just noticed I’m replying to the OP, who’s mentioned downthread that they’re using BigQuery and spending $200/week. I’ve gone the marginalia.nu route and run everything on a computer in my living room, which changes the calculus somewhat—it’s a lot cheaper, but probably involves more development time. For me it’s mainly about the learning experience but I’d be interested to hear your thoughts on the tradeoff.
pcthrowaway 5y ago
Are you not putting github (+issues and PRs) in the indexed set?
- alangibson 5y ago
  Not yet. That's an astonishing about of data, and I want to make sure that people genuinely want it first. I'm considering an index specifically for this actually. I'll put you down as a +1
dmix 5y ago
If you search for "Reddit" the first result is "Google Search is Dying" on Hacker News. https://dontbeevil.rip/search?q=reddit https://dontbeevil.rip/search?q=reddit
- alangibson 5y ago
  The reason is because there was a long discussion about Reddit as a search engine in that thread. reddit.com will likely never be indexed. Many of the subreddits already are, but I haven't exposed the ability to do something like Google's `site:reddit.com/r/*` yet. That's coming though.
gargarplex 5y ago
I guess blogs that are linked-to in non-killed HN comments should probably be crawled a bit. Have you considered using social user karma (this could be a 1-10 score uniquely calculated for users of each of HN, Twitter, Reddit as long as it's built in a modular way) as a weight in a PageRank style schema? Here's how I am going to evaluate your search engine. Yesterday I searched Google for "get dynamodb table row count" and found this URL, https://bobbyhadz.com/blog/aws-dynamodb-count-items https://bobbyhadz.com/blog/aws-dynamodb-count-items, which provides a terrible recommendation involving a full table scan. With DontBeEvil, I didn't find the correct answer, to use the describe-table API. If you really plan to dedicate a year to this, I would strongly encourage you to re-post again as soon as you have a strong update. Right now this has potential to provide value but really does not. So update us when you have confidence that you might be providing value! But we think you're on to a great opportunity.
- alangibson 5y ago
  > I guess blogs that are linked-to in non-killed HN comments should probably be crawled a bit They are, but there are relatively a few of them because my only page content source is the Common Crawl. The hit rate vs the total urls I'm interested in is not great. I expect to fix this soon. I'm also not indexing entire sites, only specific upvoted urls. This will change as well. > Have you considered using social user karma (this could be a 1-10 score uniquely calculated for users of each of HN, Twitter, Reddit as long as it's built in a modular way) as a weight in a PageRank style schema? Definitely. I've already started in on calculating a rank coefficient for submitters, but it's not completely clear now to best use it yet. > Here's how I am going to evaluate your search engine Feel free to dump more of these. Some solid test cases would be very helpful.
sanmartin65 5y ago
Traster 5y ago
This is an interesting approach to the general problem. The general problem being that whatever data source you use is inevitably going to be polluted by players who wish to be top of the rankings in a search engine if your engine is used. Maybe this solution - serving a very small niche will work, but I'd be really interested to know if you guys have spent any time trying to SEO your own search engine? Hire an intern whose sole task is to get a page to the top of a fairly common search query like replacing some common python package with your own one?
- rmbyrro 5y ago
  Seems he's relying on the communities/owners behind the sources to moderate and keep bad content away. I think it's a good bet. Would expect to be a LOT harder for a "growth hacker" to make his way up in HN points as opposed to Google rank.
leke 5y ago
Nice. Some very different results returned.
sjbrown 5y ago
Now do recipes
- alangibson 5y ago
  I want to. So much.
- all2 5y ago
  StackOverflow but for food. RecipeOverflow. StackFood.
hansott 5y ago
Couple of thoughts: Make it available through a web page instead of a raw search dump? Hide the internals of your search engine? In case you want to switch to meilisearch, algolia, ... (for cost reasons) Preferably use your own search DSL to avoid users to learn about Elasticsearch queries (goes hand in hand with hiding internals of search engine) Good luck! :)
- alangibson 5y ago
  That'll all happen (though ES simple search expressions are quite OK). The reason it is the way it is today so to enable me to get it out into the world as fast as possible. It puts the M in MVP.
- randomsilence 5y ago
  Why reinvent the wheel? It's a selling point that Elasticsearch queries can be used. If he changes the engine he might as well implement the Elasticsearch language then for the new engine.
wolfgang42 5y ago
Congrats on the launch! Over the past 6 months or so I’ve been intermittently working on building pretty much exactly the same thing, but with a lot of procrastinating on fiddling with the internals rather than just putting something out there. Your API-first approach is a clever way to get around the desire to keep fiddling around with the page design!
- alangibson 5y ago
  I find that plain text is a very effective anti-procrastination tool. That's why the "API" was actually text-first. Limiting your options can be very liberating.
waynecochran 5y ago
Are you using something similar to the original pagerank algorithm that uses eigen-analysis of the link graph?
- alangibson 5y ago
  It's not even that sophisticated yet. I'm ranking urls based on their normalized score on the various community sites I find them on. My next TODO is to roll up those ranks to get a rank for the site, then index the whole site. I will also be using the PageRank calculated by Common Crawl as soon as they release the next data set.
0x20cowboy 5y ago
Maybe use Gopher? Lynx supports it and there are a few other newish clients out there.
deleted 5y ago
[deleted]
ghawk1ns 5y ago
You can try with a function to simplify the cli: $ rip() { curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search https://dontbeevil.rip/search --data-urlencode 'q=$@'} $ rip heartbleed bug
- mananaysiempre 5y ago
  The single quotes probably need to be double ones in the last argument to permit parameter expansion, and the $@ (separately quote every argument if quoted) probably wants to be a $* (quote the entire space-separated argument array if quoted)? There’s also the grammar quirk where the last command inside braces (but not parens) needs a semicolon or newline to separate it from the brace itself. Thus: rip() { curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search --data-urlencode "q=$*"; } (tested). I still support the point that there is no reason for this to be a (grammar-defying) alias rather than a (tame) shell function or even a separate script.
cbreynoldson 5y ago
did this idea spark from PG's old talk on new ideas https://youtu.be/R9ITLdmfdLI https://youtu.be/R9ITLdmfdLI? One of them is literally "search engine for developers/hackers"
rdiddly 5y ago
Love the URL most of all but will be trying this out!
Natfan 5y ago
Here's a simple PowerShell wrapper for your lovely tool: function rip { param ( [Parameter(Mandatory, ValueFromRemainingArguments)] [String] $Query ) $RequestParameters = @{ URI = "https://dontbeevil.rip/search?q=$Query" Headers = @{ Accept = "text/plain" } } $Request = Invoke-RestMethod @RequestParameters return $Request } Usage: > rip heartbleed bug Heartbleed Bug <https://heartbleed.com/> Heartbleed Bug The Heartbleed Bug The Heartbleed Bug is a serious vulnerability in the popular OpenSSL cryptographic software library....
l0b0 5y ago
Nice! Do any more advanced query strings work right now? Like looking for recent pages or only searching titles?
AnyTimeTraveler 5y ago
I just tried a few queries related to rust and it's library rocket. I got only useless results on the first page and didn't check further. I'm guessing that's because it doesn't index docs.rs and the rust forum. Both incredibly important for Rust development. So as long as this engine doesn't also index most programming related forums, I won't be able to use it effectively, even though I really would like to. The concept of limiting the scope to just a few websites sounds really interesting, though. I think I will take this idea and build a little thing on top of google to implement that site filtering on my queries.
- alangibson 5y ago
  Thanks for the feedback. Language docs sites are the current weak point. Theyll be the next big addition to the index.

269 points by alangibson 5y ago

przemub 5y ago

I really like it and it already gives some useful results. A rise of curated search engines as yours would be lovely. It would be nice if the main page linked to your blog or anything really, because I would like to know where can I follow this project!

alangibson 5y ago
Thanks! I'm giving this project a year to build up momentum. If it looks promising, I plan on having other STEM verticals. Maybe even fix recipe searches one day :) A real homepage is coming. Feel free to subscribe to my blogs RSS feed for now: https://landshark.io/feed.xml https://landshark.io/feed.xml

eatbitseveryday 5y ago

I would recommend adding technical blogs. Not by hand, but if you can automate identifying some. Many are small but have good content. Edit: also some corporate technical documentation like Mozilla, Microsoft, IBM, etc have many such developer pages.

alangibson 5y ago
I automate it by pulling urls out of HN, programmer Reddit, etc. Right now my only source of page content is the Common Crawl, which is why there are relatively few web pages indexed. That will change. A next step is to index entire sites, not just individual pages, based on the positive votes their links get.

amar-laksh 5y ago

I saw your blog post a couple of days ago, This looks really promising!

alangibson 5y ago
Thanks! I'll be updating that post today. I changed quite a few things getting ready for this Show HN, so it's now out of date.

husainfazel 5y ago

Going to "https://dontbeevil.rip/ https://dontbeevil.rip/" results in a JSON error in the browser: {"message":"Missing Authentication Token"}

kiettv 5y ago
You need to access the sub path /search?q= :)
przemub 5y ago
Apparently developers need no homepages, just APIs :)
- alangibson 5y ago
  Jup. No time for fancy things like HTML yet :)
- genewitch 5y ago
  i run a few dozen internet and web services with FQDN and only 1 of them has something if you type http://example.com http://example.com - no homepages, but there's a webserver listening on ~80% of the domains. Stop attacking me!
alangibson 5y ago
There's nothing there yet. I'm 100% focused on building the index and tuning the master search query. There is however a blog post that goes into more detail on how to do things like pagination soon. tl;dr: https://dontbeevil.rip/search?q=monads&from=10 https://dontbeevil.rip/search?q=monads&from=10

kiettv 5y ago

It's powered by ElasticSearch, is it? So I can use all of its query parameters?

alangibson 5y ago
Indeed it is. You can you use simple query strings for the q parameter. See https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... I'm considering opening up full ES query support for paying customers, but it's too dangerous to expose it to the Internet unrestricted.
- rmbyrro 5y ago
  I think it's dangerous to expect a malicious actor would not pay $10 to screw your service.
  - cj 5y ago
    "Risk management" is often not the same as "risk elimination"
  - alangibson 5y ago
    Indeed it is. Presumably I would have had time to build up some safeguards and run beefier servers by then though.
    - marginalia_nu 5y ago
      As a word of warning, when HN discovered my search engine, I was hit hard by a botnet within a few days. Saw about 30-40k queries/hour from some 10k IP addresses. I'm self hosted so the worst that happened is my search engine was a bit slow, but if I was cloud hosted I'd have a very sizable bill to pay. If you do not already have a global rate limit, implement one ASAP. Better to have one and not need it, than to need it and not have it.
      - 5y ago

dorianmariefr 5y ago

I'm getting: > {"message": "Internal server error"}

alangibson 5y ago
What query are you running?
alangibson 5y ago
Give it another try. I fixed a flaw in the json to text translation.
- dorianmariefr 5y ago
  Thanks, it works now

feanaro 5y ago

Getting internal server error for many ordinary requests. I'm not able to discern a pattern. An example is `rip 'q=zelda'`.

alangibson 5y ago
Thanks for the report. I'll get this fixed. In the mean time you can use application/json: curl -G -H 'Accept: application/json' https://dontbeevil.rip/search?q=zelda https://dontbeevil.rip/search?q=zelda
alangibson 5y ago
It should be fixed now.

erklik 5y ago

> programmer Reddit Is it just /r/programmer? or many other programming related subreddits? In general though, this is great. I would similarly love a solution where we could submit sites to be indexed. A way to have a search engine for all the websites I want specifically would be awesome. You could probably add some sort of popular filter on top of it so that only sites popular enough get indexed. I don't know. Just an idea. I love the fact that it's accessible from the terminal. That's fantastic. Although, would be nice to have a very simple HTML front-end. Think very early Google or go very brutalistic. Anyway, excited to hear about it. Edit: Doing the following gives me an Internal Server Error for some reason. curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search https://dontbeevil.rip/search --data-urlencode 'q=Notes' {"message": "Internal server error"}

alangibson 5y ago
> Is it just /r/programmer? or many other programming related subreddits? There's about 30 programming focused subreddits. > I would similarly love a solution where we could submit sites to be indexed. This is on the plan. I want to allow common interest groups to maintain their own search verticals. I also want to allow individual users to add everything from bookmarks to notes (privately only of course) to act as a sort of external memory. That's very long term though. > Doing the following gives me an Internal Server Error for some reason. Should be fixed now

ykevinator2 5y ago

Can we drop the q= and the quotes from the shell cmd somehow, that would make it so much nicer, and rip is a great command line.

alangibson 5y ago
There's a real command line coming. If you're on a Debian Linux and feel like testing it out, just do apt install curl jq pip3 install jtbl curl -O https://raw.githubusercontent.com/alangibson/dontbeevil.rip/main/src/rip https://raw.githubusercontent.com/alangibson/dontbeevil.rip/... chmod u+x rip ./rip 'what is a monad'
- l0b0 5y ago
  With long options, JSON output, and no extra Python dependencies: rip() { curl \ --data-urlencode "q=${1}" \ --get \ --header 'Content-Type: application/json' --header 'Accept: application/json' \ --silent \ 'https://dontbeevil.rip/search' \ | jq '[ .hits.hits[] | { title: .fields.title[0], url: .fields.url[0], highlight: .highlight.text[0] } ]' }
alexrsagen 5y ago
This function should work for you: $ rip() { curl -G -H "Accept: text/plain" --url https://dontbeevil.rip/search https://dontbeevil.rip/search --data-urlencode "q=$*"; } $ rip Heartbleed bug edit: alangibson's solution in this thread is better :)

mynameismon 5y ago

> StackOverflow Does this include the entire StackExchange network, or only StackOverflow? Because some SE sites (in particular, UnixSE and ServerFault) also produce highly relevant results.

alangibson 5y ago
> Does this include the entire StackExchange network Not yet. I'm focused on explicitly developer-oriented resources right now. Those you mentioned are on the TODO list though.

sitkack 5y ago

I love this idea. This is mostly just raw data, it isn't that useful (yet). The security issues with using curl directly to my terminal feel a bit dangerous. I'd rather use my browser and be able to see the results over a json tree. Providing raw access to ES has a high risk to reward. The search results for https://dontbeevil.rip/search?q=python%20context%20manager%20contextlib https://dontbeevil.rip/search?q=python%20context%20manager%2... are non-topical hits on SO records. I even put the name of the python package (from stdlib no less) in the query string. I was able to find what I needed on devdocs.io in less than 10 seconds. https://devdocs.io/python~3.10/library/contextlib#contextlib.contextmanager https://devdocs.io/python~3.10/library/contextlib#contextlib... In no way am I trying to discourage you, but until the basics are in place, search over arxiv abstracts is way less useful than just SO and docs (language and libraries). I would recommend returning text/plain by default and .json if someone asks for it (in the url), no everyone can set headers. I'd also put an about page at the root of your site, plain text is fine.

alangibson 5y ago
Thanks for the feedback. I'm considering making searching for language docs a special use case, maybe even with its own index. > Providing raw access to ES has a high risk to reward. I'm not quite that foolhardy :). The only thing that gets passed to ES is a sanitized simple_query_string.query. Should be relatively safe. I'm sure someone will prove me wrong though.
tekromancr 5y ago
What security implications does running curl have that wouldn't be present in a browser?
- artursapek 5y ago
  I’m wondering the same. You’re not piping them into a shell.
- feanaro 5y ago
  There have been instances of terminal vulnerabilities via terminal escape codes, as bad as an RCE in iterm2: https://blog.mozilla.org/security/2019/10/09/iterm2-critical-issue-moss-audit/ https://blog.mozilla.org/security/2019/10/09/iterm2-critical.... I suppose the OP is thinking of something like that.

bryanrasmussen 5y ago

how's ranking done, I searched for xslt and I saw a lot of HN results in the first part, seemed weird that HN would rank highly for that.

alangibson 5y ago
Results are heavily weighted to HN and Stackoverflow right now because they are the easiest resources to access and rank. Since posts have a score on both platforms, it's easy to pull out some 'authority' signal. There's many more web pages coming. They are much more difficult to get ahold of and rank though because I need to run my own crawler to fill in what Common Crawl doesn't have and then calculate my own site authority rankings.

wodenokoto 5y ago

My shell-fu isn't the greatest. I thought I could be clever and do >alias rip="curl -G -H 'Accept: text/plain' \ --url https://dontbeevil.rip/search --data-urlencode q=" >rip hello {"message": "Missing required request parameters: [q]"} curl: (6) Could not resolve host: hello

harryvederci 5y ago
You could also put a shell script on your `PATH` instead of creating an alias: #! /bin/sh query_string="q=$@" curl --get \ --header 'Accept: text/plain' \ --url https://dontbeevil.rip/search \ --data-urlencode "${query_string}"
- mrlemke 5y ago
  I'd do it almost the same but without the variable. Note: the long shebang is for using on Termux, PC users should change it to something like #!/use/bin/env sh. #!/data/data/com.termux/files/use/bin/env sh curl -G -H "Accept: text/plain" \ --url "https://dontbeevil.rip/search" \ --data-urlenconde "q=$*"
  - harryvederci 5y ago
    Didn't know about "$*", thanks! Edit: typo in your version: "urlenconde"
    - laumars 5y ago
      Also every instance of ‘usr’ has been autocorrected to ‘use’. Autocorrect does make me laugh some days :D
      - mrlemke 5y ago
        That's what I get for phone posting. Thanks for pointing it out you two!
5y ago

niek_pas 5y ago

This looks really cool! It would be neat to have a proper CLI with a more fully-flushed out UI, with things like shortcuts to quickly open links. Is there any way I can be kept up to date with the state of this project? Also, am I correct in assuming it's not open source?

kordlessagain 5y ago
Like an ultra powerful goosh.org UI, with AI command synthesis, image uploads, crawling, search, opening pages, etc.
- alangibson 5y ago
  CLI in the browser. I love it.
alangibson 5y ago
The repo is over here: https://github.com/alangibson/dontbeevil.rip https://github.com/alangibson/dontbeevil.rip You'll be disappointed though as most of the important stuff only lives as BigQuery queries. I will be updating it in the near future though.

dotancohen 5y ago

This is great, even if I'm not getting as good results as from google for now. Can you expose the filtering features of ES? I'd love to query e.g. "+python lists" and get results only related to Python, no e.g. Lisp results. For Stack Overflow you could use the question's tags as filter keys, and for other sites you'd add them manually (so e.g. the PHP docs get the PHP key). If you're thinking of monetizing this, I'll tell you what I tell all the small, useful services that I'd like to pay for. There are too many small, useful services that I'd like to pay for. I'll gladly pay $1 for such a service, but you'll have a hard time convincing me to pay more.

operator-name 5y ago

I've tried it out. It's quite obvious the limited number of crawled sites when searching for anything obscure or one step outside of programming. Even 'javascript reverse string', which I expected some docs or stack overflow pages seems to give me a HN thread, someone's github repo and a not very related SO thread. Is MDN, MDSN, more dev docs documentation on the roadmap? It's definitely an interesting technique. Do you have anything in place to detect garbage, substenceless articles like which has started popping up on Google? I've seen the occasional one using github repositories or pages. Looking at the current list you're broadly reliant on moderators and communities, and as the search engine you moderate which sites are indexed.

alangibson 5y ago
> one step outside of programming. Indeed, it's explicitly for programming only (for now). > Even 'javascript reverse string', which I expected some docs or stack overflow pages seems to give me a HN thread, Next up is indexing language documentation. At this point I'm relying heavily on Q&A and community sites since they have their own built in quality rankings. > Is MDN, MDSN, more dev docs documentation on the roadmap? Most definitely. Feel free to dump a list of urls of your favorite doc sites. I'm building a whitelist now. > Do you have anything in place to detect garbage, substenceless articles like which has started popping up on Google? My strategy is to not index spam in the first place. That's why I started by extracting links from with community sites that have their own moderation in place. The next step is to whitelist high quality sites. That is potentially a huge list to maintain, which is why I'm am narrowly focused on software development. Everything old is new again...
- ejp 5y ago
  I use this documentation aggregator/search in the browser to access most language docs. It might serve as a whitelist starting point! https://devdocs.io/ https://devdocs.io/
  - alangibson 5y ago
    Nice. Thanks!

martinald 5y ago

Why not make a website for this? Why just limit it to the terminal (hard to use on mobile for example)? Edit: obviously you can query it from a browser, but it would take like a couple hours to have a view that parsed the json and put it in a google-style layout with a search bar.

alangibson 5y ago
> Why not make a website for this? It's coming. I decided to get it out there as soon as I had an index that could theoretically be useful. The feedback I'm getting will drive the next chunk of work that gets done. For instance, I'll probably bring in language docs next as a lot of people have asked for them already.

baggachipz 5y ago

1. Stand up ElasticSearch instance 2. Have it index SO and HN 3. Charge $10 per month 4. Profit!!

alangibson 5y ago
Maybe you should read the other posts about future plans and how this is extremely alpha. Or maybe go anywhere else and do literally anything else but be obnoxious in this thread. I'm spending over $200 per week just to stand up the service as it is. $10 for a full functional search engine will likely not be even close to PROFIT!!!!
- sdesol 5y ago
  > I'm spending over $200 per week just to stand up the service as it is What are using if you don't mind me asking? Not trying to criticize or anything. I have a Heztner box that gives me 1TB SSD in RAID 0 mode and 64 GB of RAM for about 80 CAD a month.
  - alangibson 5y ago
    I have 3 sizeable EC2 instances running an ElasticSearch cluster, plus a beefy box for data preprocessing and crawling. A big chunk actually goes to BigQuery. There are publicly available datasets for HN, Stackoverflow and a few others there. I've also loaded up the Common Crawl index. The query and storage fees really add up. I'm hopefully done with huge BigQuery queries, so that $200 will probably drop for a while.
    - rmbyrro 5y ago
      I'm probably wrong about my assumptions, but presume you are open to any kind of constructive feedback, so here it goes... Maybe you're overkilling with the infra stack. I would simplify until having a mature product, especially if I'm bootstrapping, which I think is your case. Right now, you're still a bit far from MVP, from my point of view. Those $200 can probably be reduced by 50%-75% if you compromise on stuff only important to non-alpha services (i.e. 99.99% availability). A single EC2 box should be enough. Maybe look into Postgres or another FOSS instead of BigQuery. These $100-$150 savings per week can go into promoting your service, getting as much attention as possible to maximize feedback. Good luck!

arminiusreturns 5y ago

Best of luck to you, I think there is a targettable niche that could utilize this. Having thought quite a bit about the search space, I think a whitelist approach is going to be the next big search thing, because advertising and bs sites have corrupted SEO too far. I'm reminded of the site indexer websites in the early days of the internet. Curation if done properly (based on quality of content and not certain other factors that currently play too heavy a role in the seo algo black boxes) seems to be how we adapt to the current information tsunami we are all dealing with. I think a long time ago I decided I would even pay for such a service, just like I am willing to pay for a good new source (FT for me, not cheap, but worth it). Im not positive the 10$ mark is low enough but I hope for the general landscape it is. Just dont forget to keep dontbeevil more than a catchphrase. In particlar, please be transparent with what user data you collect and how you use it.

alangibson 5y ago
> I think a whitelist approach is going to be the next big search thing It almost as to be. Spammers, growth hackers, et al. are just too numerous and too good. > Im not positive the 10$ mark is low enough but I hope for the general landscape it is. I saw enough people mention $10 that I decided to go for it. To be honest, $10 is already probably too low to be sustainable because of the huge amount of resources search consumes and the high cost of development. My gut feeling is that it's economically impossible to build a good search engine that isn't loaded with ads and spyware. But I spent so long complaining about G that I decided to prove to myself one way or another.
- wolfgang42 5y ago
  > because of the huge amount of resources search consumes I’ve been intermittently working on much the same idea as the OP, and I suspect this is actually a lot less of a problem than it seems, since they’re focusing on a niche. Indexing everything the way Google does requires a lot of resources, but indexing the majority of useful material in a specific domain takes a lot less. (My ElasticSearch index for the entirety of StackOverflow is a mere 40 GB, for example.) By far the more expensive part is likely to be paying market rates for a developer (you need a decent number of users paying $10/mo to hit a mid-market salary), but in theory this scales relatively independently of userbase. Edit: I’ve just noticed I’m replying to the OP, who’s mentioned downthread that they’re using BigQuery and spending $200/week. I’ve gone the marginalia.nu route and run everything on a computer in my living room, which changes the calculus somewhat—it’s a lot cheaper, but probably involves more development time. For me it’s mainly about the learning experience but I’d be interested to hear your thoughts on the tradeoff.

pcthrowaway 5y ago

Are you not putting github (+issues and PRs) in the indexed set?

alangibson 5y ago
Not yet. That's an astonishing about of data, and I want to make sure that people genuinely want it first. I'm considering an index specifically for this actually. I'll put you down as a +1

dmix 5y ago

If you search for "Reddit" the first result is "Google Search is Dying" on Hacker News. https://dontbeevil.rip/search?q=reddit https://dontbeevil.rip/search?q=reddit

alangibson 5y ago
The reason is because there was a long discussion about Reddit as a search engine in that thread. reddit.com will likely never be indexed. Many of the subreddits already are, but I haven't exposed the ability to do something like Google's `site:reddit.com/r/*` yet. That's coming though.

gargarplex 5y ago

I guess blogs that are linked-to in non-killed HN comments should probably be crawled a bit. Have you considered using social user karma (this could be a 1-10 score uniquely calculated for users of each of HN, Twitter, Reddit as long as it's built in a modular way) as a weight in a PageRank style schema? Here's how I am going to evaluate your search engine. Yesterday I searched Google for "get dynamodb table row count" and found this URL, https://bobbyhadz.com/blog/aws-dynamodb-count-items https://bobbyhadz.com/blog/aws-dynamodb-count-items, which provides a terrible recommendation involving a full table scan. With DontBeEvil, I didn't find the correct answer, to use the describe-table API. If you really plan to dedicate a year to this, I would strongly encourage you to re-post again as soon as you have a strong update. Right now this has potential to provide value but really does not. So update us when you have confidence that you might be providing value! But we think you're on to a great opportunity.

alangibson 5y ago
> I guess blogs that are linked-to in non-killed HN comments should probably be crawled a bit They are, but there are relatively a few of them because my only page content source is the Common Crawl. The hit rate vs the total urls I'm interested in is not great. I expect to fix this soon. I'm also not indexing entire sites, only specific upvoted urls. This will change as well. > Have you considered using social user karma (this could be a 1-10 score uniquely calculated for users of each of HN, Twitter, Reddit as long as it's built in a modular way) as a weight in a PageRank style schema? Definitely. I've already started in on calculating a rank coefficient for submitters, but it's not completely clear now to best use it yet. > Here's how I am going to evaluate your search engine Feel free to dump more of these. Some solid test cases would be very helpful.

sanmartin65 5y ago

Traster 5y ago

This is an interesting approach to the general problem. The general problem being that whatever data source you use is inevitably going to be polluted by players who wish to be top of the rankings in a search engine if your engine is used. Maybe this solution - serving a very small niche will work, but I'd be really interested to know if you guys have spent any time trying to SEO your own search engine? Hire an intern whose sole task is to get a page to the top of a fairly common search query like replacing some common python package with your own one?

rmbyrro 5y ago
Seems he's relying on the communities/owners behind the sources to moderate and keep bad content away. I think it's a good bet. Would expect to be a LOT harder for a "growth hacker" to make his way up in HN points as opposed to Google rank.

leke 5y ago

Nice. Some very different results returned.

sjbrown 5y ago

Now do recipes

alangibson 5y ago
I want to. So much.
all2 5y ago
StackOverflow but for food. RecipeOverflow. StackFood.

hansott 5y ago

Couple of thoughts: Make it available through a web page instead of a raw search dump? Hide the internals of your search engine? In case you want to switch to meilisearch, algolia, ... (for cost reasons) Preferably use your own search DSL to avoid users to learn about Elasticsearch queries (goes hand in hand with hiding internals of search engine) Good luck! :)

alangibson 5y ago
That'll all happen (though ES simple search expressions are quite OK). The reason it is the way it is today so to enable me to get it out into the world as fast as possible. It puts the M in MVP.
randomsilence 5y ago
Why reinvent the wheel? It's a selling point that Elasticsearch queries can be used. If he changes the engine he might as well implement the Elasticsearch language then for the new engine.

wolfgang42 5y ago

Congrats on the launch! Over the past 6 months or so I’ve been intermittently working on building pretty much exactly the same thing, but with a lot of procrastinating on fiddling with the internals rather than just putting something out there. Your API-first approach is a clever way to get around the desire to keep fiddling around with the page design!

alangibson 5y ago
I find that plain text is a very effective anti-procrastination tool. That's why the "API" was actually text-first. Limiting your options can be very liberating.

waynecochran 5y ago

Are you using something similar to the original pagerank algorithm that uses eigen-analysis of the link graph?

alangibson 5y ago
It's not even that sophisticated yet. I'm ranking urls based on their normalized score on the various community sites I find them on. My next TODO is to roll up those ranks to get a rank for the site, then index the whole site. I will also be using the PageRank calculated by Common Crawl as soon as they release the next data set.

0x20cowboy 5y ago

Maybe use Gopher? Lynx supports it and there are a few other newish clients out there.

deleted 5y ago

[deleted]

ghawk1ns 5y ago

You can try with a function to simplify the cli: $ rip() { curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search https://dontbeevil.rip/search --data-urlencode 'q=$@'} $ rip heartbleed bug

mananaysiempre 5y ago
The single quotes probably need to be double ones in the last argument to permit parameter expansion, and the $@ (separately quote every argument if quoted) probably wants to be a $* (quote the entire space-separated argument array if quoted)? There’s also the grammar quirk where the last command inside braces (but not parens) needs a semicolon or newline to separate it from the brace itself. Thus: rip() { curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search --data-urlencode "q=$*"; } (tested). I still support the point that there is no reason for this to be a (grammar-defying) alias rather than a (tame) shell function or even a separate script.

cbreynoldson 5y ago

did this idea spark from PG's old talk on new ideas https://youtu.be/R9ITLdmfdLI https://youtu.be/R9ITLdmfdLI? One of them is literally "search engine for developers/hackers"

rdiddly 5y ago

Love the URL most of all but will be trying this out!

Natfan 5y ago

Here's a simple PowerShell wrapper for your lovely tool: function rip { param ( [Parameter(Mandatory, ValueFromRemainingArguments)] [String] $Query ) $RequestParameters = @{ URI = "https://dontbeevil.rip/search?q=$Query" Headers = @{ Accept = "text/plain" } } $Request = Invoke-RestMethod @RequestParameters return $Request } Usage: > rip heartbleed bug Heartbleed Bug <https://heartbleed.com/> Heartbleed Bug The Heartbleed Bug The Heartbleed Bug is a serious vulnerability in the popular OpenSSL cryptographic software library....

l0b0 5y ago

Nice! Do any more advanced query strings work right now? Like looking for recent pages or only searching titles?

AnyTimeTraveler 5y ago

I just tried a few queries related to rust and it's library rocket. I got only useless results on the first page and didn't check further. I'm guessing that's because it doesn't index docs.rs and the rust forum. Both incredibly important for Rust development. So as long as this engine doesn't also index most programming related forums, I won't be able to use it effectively, even though I really would like to. The concept of limiting the scope to just a few websites sounds really interesting, though. I think I will take this idea and build a little thing on top of google to implement that site filtering on my queries.

alangibson 5y ago
Thanks for the feedback. Language docs sites are the current weak point. Theyll be the next big addition to the index.