8 ms·
Whoogle Search: A self-hosted, ad-free, privacy-respecting metasearch engine
- izzytcp 5y agoThis sounds too good to be true in practice. If I deploy it on a server and continuously query it from all of my devices, will Google ban that serve rIP?
- supah 5y agoscraping yes. normal usage no.
- tyingq 5y agoIsn't normal usage scraping in this case? Looking at whoogle-search/request.py it's "scraping" google urls via the python requests module. I'm reasonably sure google fingerprints requests and assigns different weights for "probably scraping". I wouldn't be surprised if this has a lower threshold for triggering their captchas and/or blocking.
- andai 5y agoI often trigger Google's captchas during normal usage. It seems to suspect the more "advanced" features like "intitle:" or "inurl:", or if I search too rapidly. I take being mistaken for a machine as a compliment!
- ricardo81 5y agoThat's understandable, a lot of exploit seekers use those features to find exploits e.g. "powered by [cms with known exploit]", Google (and Bing) are definitely are more prone to showing you a captcha for those searches, especially if you're looking beyond the first page.
- spinax 5y agoI'm on a shared IP with millions of people using the same public IP (T-Mobile CGNAT), there is one IP (many of them, actually) doing that right now from every T-Mobile customer. Your one server will be a blip on their radar if it even registers.
- izzytcp 5y agoDude, thanks for letting me know that everyone uses Google. I'm talking about static IPs like servers in the cloud, not your home. That stuff is automated and I am sure static IPs get banned but here must be a quota of something.
- spinax 5y agoDude, CGNAT is handled at the ISP layer, I do not have an IPv4 address at all locally, it's a 464XLAT done on T-Mobile's side. All users come from a shared IPv4 on their network, not mine. Dude.
- treesknees 5y agoDude - Each of those users behind the NAT will have a different set of cookies, user agents, screen sizes, among other fingerprints that qualify them as unique. ISPs also routinely place their CGNAT addresses on specific whitelists so that services don't block them for abuse (you can look through the NANOG email list to find examples of this.) IP addresses are also classified as residential, cloud/server, etc. If Google sees rapid requests from the same IP classified as a server that's sending a Python Requests user-agent, they can absolutely block it.
- 123pie123 5y agohow long should google block the IP? when the ISP reassigns it to a new home they would be blocked also blocked from google
- treesknees 5y ago
- packet_nerd 5y agoI set this up for my family and I a few months ago and we set all our browsers and devices to use it as the default search engine. We've been really happy with it. I also set up a small script on a cron job that queries random search strings every few minutes and opens the first few hits in selenium. My theory is that if I can't completely stop them from tracking us, I can at least dilute their data with bogus searches. We haven't had any issues from Google.
- DavideNL 5y agoI guess it will be just like Searx: https://searx.me/ https://searx.me/ If an IP generates too much search queries, Google will block / throttle it…
- johnchristopher 5y agoSo, I set it up this week-end on a vps and subdomain and today it's blocked :(. About this page Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot. Why did this happen? IP address: 137.x.x.x Time: 2021-08-30T16:56:45Z URL: https://www.google.com/search?gbv=1&q=broken&lr=&hl=en&safe=off
- keb_ 5y agoI deployed an instance of this for my own use. Been using it as my daily search engine for a little over a month now. No complaints. One overlooked feature is that Whoogle supports DDG-style bangs!
- obiwanpallav1 5y agoSide question: If I can just instruct the browser to delete the sessions, cookies, localstorage, etc. after I close a Google search tab, then would it require us to self host Whoogle? This considers that I'll never login to Google using that browser and 3rd party cookies are disabled. Or, can Google still recognize me?
- option_greek 5y agoIf you consistently do that, you will start seeing captcha wall of hell. Google gets its pound of flesh one way or the other.
- obiwanpallav1 5y agoI did not understand the last sentence. If I solve the captchas, get the search results and then do the cleanup, it'll just continuously ask for the captchas and that's the only added pain, no? Will they be able to conclude if all the requests are coming from the same user?
- sodality2 5y agoIP and other fingerprinting techniques are enough to identify you
- option_greek 5y agoYes. But its going to keep asking you the captchas till the user changes behavior (sample of 1) :) (and of course the ip address too like the other poster mentioned - you can try it by searching for insurance/anything with good money on your phone and switch to desktop - assuming both are connected through the same router).
- type0 5y agoIt might even temporarily block you and put out the message that they are seeing suspicious "bot" activity from your device.
- sildur 5y ago
- nathan_phoenix 5y agoWhat's the difference with SearX[1]? SearX seems both more popular and supports other search engines besides Google. [1]: https://github.com/searx/searx https://github.com/searx/searx
- bigethan 5y agoIt's mentioned in their FAQ[1], less config easier to set up [1]:https://github.com/benbusby/whoogle-search#faq https://github.com/benbusby/whoogle-search#faq
- anonymousisme 5y agoIs Whoogle a Lougle wannabe? https://whatculture.com/film/10-famous-things-invented-movie-time-travellers?page=2 https://whatculture.com/film/10-famous-things-invented-movie...
- ivrrimum 5y agoI have been hosting(and using as a full replacement) a self-hosted youtube alt(called piped). Might switch to this engine as well. Goal is to be fully self hosted! :)
- freediver 5y agoThe available public instances [1] seem to take 5-6 seconds for search results page. Is this to be expected with self-hosted too? [1] https://github.com/benbusby/whoogle-search#public-instances https://github.com/benbusby/whoogle-search#public-instances
- honie 5y agoI think that may be a bit of an unfair statement. They do note just above in the public instances the following: > Note: Use public instances at your own discretion. Maintainers of Whoogle do not personally validate the integrity of these instances, and popular public instances are more likely to be rate-limited or blocked. Having tested with various search strings, the time to first search-result paint for the the first public instance [1] feels just as fast a Google. [1] https://whoogle.sdf.org/ https://whoogle.sdf.org/
- keb_ 5y agoNo. My self-hosted instance is about as fast as using Google directly. Maybe a half a second slower, but barely noticeable if it is. I can't even tell.
- flatiron 5y agosame, just threw it up on my home lab box and switched my default search and don't notice a difference
- scottydelta 5y agoI just deployed a version for personal user, lets see how it works out!
- dave333 5y agoLooks like the public instance at https://whoogle.sdf.org https://whoogle.sdf.org is hacked/corrupt since it returns different results to the other public instances.
- johnchristopher 5y agoIt's missing an "I am feeling lucky" button :). And shareable links for search results or did I miss that ?
- fsflover 5y agoIf you don't like the "meta" in the title, have a look at https://yacy.net https://yacy.net.