7 ms·
Internet search tips
- 38 3y agomy problem is some operators just dont work any more, either on purpose or because of crappy quality control. for example, you used to be able to do: allintitle:Neil Diamond If You Go Away on YouTube, and get exactly what you would think, results with all those words in the title. but now, you dont: https://youtube.com/results?search_query=allintitle:Neil+Diamond+If+You+Go+Away https://youtube.com/results?search_query=allintitle:Neil+Dia... now, I get crap like this: Neil Diamond & Shirley Bassey - Play Me - "high quality" Barbra Streisand - If You Go Away (Ne Me Quitte Pas) how is that what I searched for? also, what is this: > A search for [site:nytimes.com] will work, but [site:nytimes.com] won't. https://support.google.com/websearch/answer/2466433 https://support.google.com/websearch/answer/2466433 did I just have a stroke? those two searches are exactly the same. I try to be understanding, but I am constantly tripping over big companies glaring software and/or documentation issues, it gets old.
- Daub 3y agoAgree. This even applies to simple Booleans, which are sometimes completely ignored.
- asteroidz 3y agoI sometimes wonder about PMs who sign off on decisions like these, and the seeming lack of protest the developers put up.
- bdn_ 3y agoHas anyone else experienced DuckDuckGo ignoring the exclusion operator? For example, searching `kiwi -fruit`, with no space between the hyphen and second word, used to bring up results that did not include the word "fruit". This no longer seems to be the case.
- lloydatkinson 3y agoSomeone needs to build a meta search engine…
- ghaff 3y agoAt least one existed in the dot com era. Forget the name.
- fuzztester 3y agometacrawler, IIRC.
- dev_hugepages 3y agosearx/searxng?
- mdp2021 3y ago> duckduckgo ... exclusion operator Removed a few weeks ago. Somebody posted in these pages the github diff showing the removal of the options.
- chkal 3y ago> did I just have a stroke? those two searches are exactly the same. I just had the exact same though while reading the page.
- nomilk 3y agoThe fact Google's own documentation is this poorly kept is a astonishing. Makes me a firm believer companies should have their documentation on GitHub (or similar) so anyone can make a PR to tidy these things up.
- hddqsb 3y ago> https://support.google.com/websearch/answer/2466433 https://support.google.com/websearch/answer/2466433 > those two searches are exactly the same Yes, that appears to be a recently-introduced typo -- the archived version from April does have a space: https://web.archive.org/web/20230412181331/https://support.google.com/websearch/answer/2466433 https://web.archive.org/web/20230412181331/https://support.g... I submitted a feedback comment, hopefully they'll fix the typo. (For future reference, here is a snapshot of the current version without a space: https://web.archive.org/web/20230722085337/https://support.google.com/websearch/answer/2466433 https://web.archive.org/web/20230722085337/https://support.g...)
- pdanpdan 3y agoThe one not working has a space after the colon. It's even in the section taking about this :)
- Daub 3y agoIf I had my way, I would introduce internet search techniques as a core module in all university programs. Too often I see in my students work evidence of lazy searching. It is as if they expect Google to be able to read their minds or even foresee their future intentions. Lack of variety of search terms is a key shortcoming. Also, lack of exploration of terms which are tangentially related to their search topic. An internet search should be playful and exploratory. Above all, it should be understood that the internet is beyond simple linear indexing.
- userbinator 3y agoIt is as if they expect Google to be able to read their minds or even foresee their future intentions. That's exactly what Google and other interests want --- gullible, uncritical sheeple that can be exploited to extract $$$ and worse.
- aardshark 3y agoBut search techniques change over time. You can see that in the responses in this thread. What used to work doesn't anymore.
- hattmall 3y agoSo I actually did have this as a class, or at least part of one. I think there is / was an official standard or name, I can't remember it though. It never worked with Google really, but it worked on the search engines for our university and other academic sites.
- imchillyb 3y agoSoon, there will be an LLM that can utilize indexes and large scale distributed databases, and that will replace every search engine on earth. Google should be quaking, and pissing themselves. It won't be their bardum, and they won't have the financial power to purchase said LLM. Poof. Goodbye useless SEO spam, advertisement hell. We hated your guts.
- gnicholas 3y ago> Goodbye useless SEO spam You think LLMs won't also be used to generate more SEO spam? I think this will be an arms race, or a game of LLM-cat and LLM-mouse.
- computator 3y ago> Goodbye useless SEO spam, advertisement hell. We hated your guts. That sounds great, but are you sure that there won't be legions of LLMs generating SEO spam and advertisement hell in warfare with the LLMs that do search? Just as an aside: Sci-fi seems to have one superintelligence (one Skynet) fighting humankind. If superintelligence emerges, I think it's likely that we'll have thousands, millions, or billions of superintelligences fighting each other, each with its own agenda.
- andirk 3y agoThe searching of data doesn't need to be controlled by insanely powerful insanely rich entities. Ex: Wikipedia, Craigslist, polio vaccine. If this LLM arms race leads to an organic solution that anyone can grow their own, it might not have as many "This 1 Trick Your Grocer Doesn't Want You To Know" ads in it. And for superintelligence, let's clarify that this "Artificial Intelligence" moniker has almost nothing to do with actual intelligence, so we're probably a few years off from that.
- Yemeth 3y agoI'm not sure how LLMs are going to be immune to SEO spam and advertising. As if human nature would magically transform and people would stop buying stupid stuff. LLMs are already being used to make the web even less useful, by shitting out vast ammounts of meaningless and even outright wrong text for SEO purposes. And don't forget the systems are being trained on the web in the first place… using a LLM that is able to utilize a web search already thinks listicles are useful information and not just a way to place affiliate links. When the hype is over and the VC money dried out, companies will find ways to make the LLM interfaces and outputs an 'advertising friendly' affair.
- 8bitsrule 3y agoMany good scarce tips in there (once past Google Scholar, unless you're monopoly-oriented). 'Dealing with paywalls' for example. He did mention IA searches, but I didn't spot a mention of their https://scholar.archive.org/ https://scholar.archive.org/ with "over 25 million research articles and other scholarly documents preserved in the Internet Archive." For book metadata, I find https://openlibrary.org/ https://openlibrary.org/ search has a lot of 'MARC' type data. Esp useful for books with many editions. Some services are getting more restrictive. e.g. WorldCat recently got harder to use, rejecting many searches. But if you can find a book's OCLC ###, then https://www.worldcat.org/oclc/### https://www.worldcat.org/oclc/### works every time. Useful for finding local hardcopy if you've got your location turned on. (With an ISBN , WPedia will also do this for you at: https://en.wikipedia.org/wiki/Special:BookSources/ https://en.wikipedia.org/wiki/Special:BookSources/ )
- jcmeyrignac 3y agoI also recommend https://www.semanticscholar.org/ https://www.semanticscholar.org/ because you can easily retrieve some of the PDFs.
- Jakob 3y ago+1 for his tip of "Search The Internet Archive". The other day I searched for the Tom & Jerry full episodes on the web to no avail (streaming platforms and video platforms like youtube only have rubbish cuts). The internet archive has every episode starting from the first one in 1940, in an easily accessible player without any ads or recommendations: https://archive.org/details/tom-and-jerry-all-114-episodes https://archive.org/details/tom-and-jerry-all-114-episodes
- andrepd 3y agoYour comment just reminded me of how great the Internet Archive is, so I guess it's time to donate another 100 bucks to IA again.
- bookofjoe 3y agoI give from time to time to Wikipedia, when I get an appeal from Jimmy Wales. Even more important than IA to me, which is saying A LOT.
- thakoppno 3y agoit’s within the realm of a few apple stores to serve the internet archive?
- WaffleIronMaker 3y agoJust as a quick link for others, it's possible to set up a single or monthly donation to the internet archive at this link: https://archive.org/donate/ https://archive.org/donate/
- nomilk 3y agoTIL: IA is more than Wayback Machine. Something I'd love to learn to do better is search WBM. I use WBM only a couple of times per month, but when I need it it's the only tool that can do the job and is therefore very valuable. Trouble is, I don't really know how to search it unless I have a record of the exact URL I want, which isn't always possible.
- ghghgfdfgh 3y ago
- Thoeu388 3y agoI found it useful to screenrecord all my computer activity. It is OCR searchable. This way if I vaguely remember some bits, I can reconstruct my sources. It also preserves original text, in case sources get edited.
- orliesaurus 3y agoWhat tool do you use?
- cole-k 3y agoI've thought about this or something similar (e.g. ArchiveBox). It seems like 95-99% of the content you archive will be junk that you never look at again, but that 1-4% really might be worth it. Especially if it gets taken down. What do you do for storage, though? I haven't invested in a NAS or personal storage over 1TB. I think I understand now why some of these archive utilities offer you the option to upload to the internet archive in addition to storing locally. You can build up a local cache and then start removing the oldest page snapshots (but keep the link trail) once you start to run out of space.
- BMc2020 3y agoI tried something similar, I had a text file called mylog.txt and whenever I hit alt + m it would append whatever was in the clipboard to the text file along with the date & time. But it turned out I was terrible at predicting what I would want to find again and it turned out to be not very useful to me.
- dang 3y agoRelated: Internet Search Tips - https://news.ycombinator.com/item?id=26847596 https://news.ycombinator.com/item?id=26847596 - April 2021 (77 comments) Internet Search Tips - https://news.ycombinator.com/item?id=18666574 https://news.ycombinator.com/item?id=18666574 - Dec 2018 (28 comments)
- Thoeu388 3y agoFor complement I would recommend Russian yandex, and vk.com (russian facebook). It is like snapshot of internet from 2018, before everything was hit by massive censorship. It has some very niche communities that are no longer on internet. For example Ancient Egypt history, western net is filled with alien theories, Russians are doing experimental archaeology. Also repair, electronics... Also Telegram is like second internet, nobody talks about!
- mxmlnkn 3y agoTelegram? How so? I'm still only using it for chatting. How would you even find anything on Telegram? The simple global search? I wouldn't like to join random shady groups found through there.
- mrweasel 3y agoNot sure either, but trying to follow the war in Ukraine has taught me both Ukrainians and Russians get a surprising amount of their news from various Telegrams channels.
- vGPU 3y agoThe search sucks, but you can find quite a lot of interesting stuff if you have a jumping off point by checking out the groups they forward from. I’ve discovered a few great groups like that. Most of the stuff that comes up on search is spam or garbage though, agreed. I’ve found the majority of news groups I follow through twitter.
- jo_beef 3y ago> Also Telegram is like second internet, nobody talks about! Do you have any groups/channels that you can recommend?
- vGPU 3y agoFor news? https://t.me/CIG_telegram https://t.me/CIG_telegram https://t.me/BellumActaNews https://t.me/BellumActaNews Both provide decent coverage of global happenings.
- coldblues 3y agoGwern is the person I'd become if I could properly monetize my ADHD and rabbit hole seeking behavior. I absolutely love his website and everything he publishes, and I am always delighted to look over his new stuff. Wish there are more people like him to read from.
- jterrys 3y agoAre you perchance able to provide some kind of RSS feed to his website? I'm having a hard time finding his newest stuff. You can subscribe to his newsletters but he stopped doing those two years ago
- Tenoke 3y agoThere is the firehose from his patreon though that might not be exactly what you are looking for. Personally, I just check 'newest' on his frontpage every now and then. https://www.patreon.com/gwern https://www.patreon.com/gwern
- RGBCube 3y agoHis website is the best designed website I have genuinely ever visited, so fast, easy to navigate and very little blank space which makes it information dense. I also love the inline link opening, I definitely will be implementing that when I make my own website in the future.
- wintermutestwin 3y ago>His website is the best designed website The forced R and L margins suck for people with oldster eyes who have to increase the text size. Kids these days!
- RGBCube 3y agoI have not seen the page on PC, all I can say is that it works very well on mobile.
- 3y ago
- self_awareness 3y agoYears ago the topic of Internet search was one of the core philosophies of Search Lores: https://web.archive.org/web/20191201105759/http://search.lores.eu/ https://web.archive.org/web/20191201105759/http://search.lor... Gwern's site somehow reminds me of this Fravia's site.
- 1vuio0pswjnm7 3y agoWho remembers +fravia's searchlores.org https://web.archive.org/web/20000620171446if_/http://www.searchlores.org:80/ https://web.archive.org/web/20000620171446if_/http://www.sea...
- marttt 3y agoa mirror without archive.org: http://biostatisticien.eu/www.searchlores.org/indexo.htm http://biostatisticien.eu/www.searchlores.org/indexo.htm Thanks very much for this; I'm very much interested in searching techniques, but wasn't aware of this site.
- deleted 3y ago[deleted]
- gwern 3y agoI do! RIP. But even while fravia was alive, his search tips were largely outdated, and that left a gap which needed to be filled. No one else had written up a search guide for the modern Internet which touched on even a tenth of what I routinely used in my own writing to find PDFs or repair dead links, so...
- orphea 3y agoI feel like Google became absolutely unusable as a search engine. You search for "keyword1 keyword2", and at least half of the results are either "Missing: keyword1" or "Missing: keyword2".
- discobean 3y agoMake your tool so idiots can use it and only idiots will use it
- mrweasel 3y agoGoogle is especially bad that this. Often what will be remove is the most important keyword, I assume because that yields more result.
- ImaCake 3y agoUntil a year ago using DuckDuckGo instead of google felt like only an equal or inferior option. But at some recent point I have found that DDG has slightly improved and Google has gotten much, much worse. With DDG and Bing Chat, google's future is looking very Internet Explorer 6.
- flyinghamster 3y agoDDG is following Google's path, with removal of the exclusion operator. Nothing like searching for "foo -quux" and finding "quux" in ALL of my search results, on DDG and Google alike.
- cma 3y agoThey moved over to really dumb word vector stuff, they mention it in the TPUv4 paper and it is pretty surprising but probably monetized better somehow.
- tzfld 3y agoWondering why we don't have web search engineers
- cal85 3y agoWe do have 'researchers' though
- ImaCake 3y agoMostly people don't think it's possible to scam your way into a job with that title. But that is a great reposte for "prompt engineer"; my coworker and I joke about being prompt engineers, but really we are technical people putting new tools to use.
- saltysalt 3y agoGreat article, but can't help feel it just highlights how broken Google search is from a UX perspective.
- skimdesk 3y agoAnother useful Google search trick not mentioned in the article is numeric range queries. You can use two numbers separated by two dots to represent all numbers in the range. For example, a search for taki 100..200 gives me results for taki 183. This is useful when you can't remember an exact year or number.
- icinganalysis 3y agoLearned a lot of new tips today. I have found a few obscure publications at the [Defense Technical Information Center](https://discover.dtic.mil https://discover.dtic.mil).
- costco 3y agoReally good list. Here are some others I've discovered: Some libgen clone sites like z-lib have fulltext search on books with support for exact matches: https://zlibrary-asia.se/fulltext/?q=%22frank+sinatra%22&type=phrase&e=1 https://zlibrary-asia.se/fulltext/?q=%22frank+sinatra%22&typ... Even if you are going to purchase book on a subject, this finds so much stuff that is not in Google because of copyright delisting and is sometimes useful in knowing which books to consider. Yandex results especially the non English ones can be good if you are willing to use a translator.
- krmblg 3y agoThere are a couple of comments on how search engine x dropped feature y on date z (i time periods after competitor j did). Can anyone recommend a site that tracks those changes? I find myself getting annoyed by serps of a given vendor and I might even be adopting to changes but ultimately jumping ship towards the next best thing if I can’t figure out a very low-effort way of influencing the results in my favor using flags that suddenly stop working - all I notice is a significant degradation of result quality.