5 ms·
Creating a search engine for fun and because Google sucks
- deleted 2y ago[deleted]
- roschdal 2y agoGoogle is a fine search engine.
- meowface 2y agoIt's way worse than before (partly due to the data set simply changing for the worse, to game them and for other reasons) but still the best.
- kleiba 2y agoIt's fashionable to bash Google and there's probably good reasons to do so. But I agree, their search is still good (enough) for me. Over the years, I've tried others, including DDG, but ultimately returned to Google every time. So, until a better one comes along, Google's search is fine for me.
- the_duke 2y agoGoogle is mostly fine if you know what you are looking for and if it's not an SEO / ad target topic. If those two don't apply Google is often pretty much useless, just giving you low quality/ AI spam blog posts, useless ad-ridden product comparison sites and marketing pages. There's a reason why adding "reddit" to Google searches has almost become a meme now.
- randomman1131 2y agoShow me an actual query then. Why do people always jump on 'umm Google bad' trends? It's all well and good to say it, but as someone who hasn't ever experienced this (apparently common) problem before, I would like to see to believe.
- Iulioh 2y agoGoogle is fine unless you need something really specific, chatgpt is better at this point for a lot of queries because Google give you the most simple answer no matter how specific you got with your keywords. I still miss the ability to use "".
- danpalmer 2y agoQuotes still work when there are good results. As far as I can tell all that has changed is a) dropping obviously bad duplicate results like Stack Overflow scrape dumps, and b) adding in more un-quoted results when there's a lack of good results for quoted. I personally haven't had any issues with quoting.
- deleted 2y ago[deleted]
- OtomotO 2y agoNo, it's a fine ad engine. Don't get me wrong, I still find anything I search for, on page 2-N I switched to duckduckgo about 5 years ago and look back occasionally. DDG simply works better for me. I find what I search for usually on page 1. Just to put things into context: I don't care about google, meaning I have no beef with them. I used google search for many years until I switched the default engine, simply because the search results for how I search got worse over time.
- ramigb 2y agoSo your "search engine" uses Google and DuckDuckGo to get results? and you are creating it because "Google sucks"? Am I missing something here?
- wood_spirit 2y agoYeah I think they are trying to get a search-results-only view without the ads etc. Perhaps they would like startpage.com?
- chipdart 2y ago> Yeah I think they are trying to get a search-results-only view without the ads etc. If that was the case then their description is completely off, as what they are doing is not creating a search engine but filtering results from other search engines. I mean, just because you pipe your output to grep it does not mean you created an entirely new app.
- bionsystem 2y agostartpage does have ads, it's just somebody else than google tracking you.
- scrollop 2y agoYou have just claimed that startpage is lying in their privacy policy. Please provide evidence for such a claim. https://www.startpage.com/en/privacy-policy?t=device https://www.startpage.com/en/privacy-policy?t=device "Our search result pages may include a small number of clearly labeled "sponsored links," which generate revenue and cover our operational costs. Those links are retrieved from platforms such as Google AdSense. In order to enable the prevention of click fraud, some non-identifying system information is shared, but because we never share personal information or information that could uniquely identify you, the ads we display are not connected to any individual user."
- bionsystem 2y ago
- sneak 2y agoI have been thinking about this. I’d like to index the blogosphere and some knowledge base sites and stuff and exclude most of the Top1000 websites. I imagine I could do it on consumer hardware for less than $10-20k. Perhaps common crawl has done much of the heavy lifting already and I just have an indexing task.
- jddj 2y agoPlugging https://search.marginalia.nu/ https://search.marginalia.nu/
- marginalia_nu 2y agoYeah I ran that on like $5k worth of consumer hardware initially. The only caveat is that you want a lot of RAM, and since consumer machines are mostly limited to 128 GB, that's a bit of a hard limit on how much you can do. In terms of storage space and compute, it's relatively fine. I'm on a single ~$17k server now, probably utilizing about 20% of it's capacity.
- timeon 2y agoNaive question: would lower-level language help here?
- marginalia_nu 2y agoThe problem domain is sprawling and varied, so you ideally want a bit of everything. You definitely need to do a bunch of custom low level programming, as a lot of standard tools can't cope with the data volumes involved. Like I'm not even storing known URLs in a DBMS anymore, the table would be too slow to update. High level language with the ability to make downcalls to a low level language is probably a sane compromise.
- PeterStuer 2y agoRobust crawling with all the gratuitous anti-bot infra out there (requesting the rss feed that is specifically meant to be consumed by machine for a public gov site is 'protected' by Cloudflare's anti-botting. Seriously?) takes way more work than you think.
- TheFragenTaken 2y agoThe phonebook capabilities of pretty much all Google alternatives suck. Google is currently the only search engine that actually works for local queries (i.e. phone repair in <local town>). Alternatives while probably fine in America, suck in the Nordics. I think people forget just how much search traffic happens in this category.
- marginalia_nu 2y agoThis is arguably a large part of why Google's tough to compete with. They have an absurd number of seamless integrations. Even if you do search well, you'll have users going back to Google for a bunch of other stuff all the time.
- r_singh 2y agoI've been using SearXNG[1] via Perplexica[2] and I couldn't be happier. It replaced Google and ChatGPT/Perplexity type search engines for me and it's the first tool I use for question answer type searches. [1] - https://docs.searxng.org https://docs.searxng.org [2] - https://github.com/ItzCrazyKns/Perplexica https://github.com/ItzCrazyKns/Perplexica
- jsnell 2y agoCreating a thin, non-working wrapper around DuckDuckGo as a "hire me" ad, claiming it's a search engine and mentioning Google in the headline because it'll get more clicks.
- growt 2y agoYes, I don’t know how that reached the frontpage. A failed 15 minute experiment from someone who has no clue what they are doing
- ctippett 2y agoI think it plays to the topics that are so often popularised here: the decline of Google Search and Rust™.
- ryandrake 2y agoTo be fair, a lot of resumes say "I created X" and "I developed Y" when, if you look even a little past the surface, the claim boils down to "Made a thin wrapper around one or more existing services." I interviewed a guy who was the sole developer of a web app that converted image files from one format to another. OK, cool project, bro, let's look at it. A little probing revealed it was just a small generic page that called into ImageMagick to do everything. I'm convinced that's a lot of what passes for software development in the 2020s: Make a splash screen and display some branding on top of older, robust, actually-complex projects.
- 0cf8612b2e1e 2y agoLeaning on IM seems the exact right way to handle the problem? If I needed web accessible image handing tricks, I could get away with not much code, because it is ultimately going to be passing some POSTed bytes into an image handing library.
- nprateem 2y agoDuckduckgo is now particularly riddled by AI spam. The top 5 summaries for lots of searches now begin with some variation of "In the fast-moving world of...". Utter shit. It'd be so easy to filter I wonder why they/Microsoft don't bother. Oh, wait...
- delanyoyoko 2y agohttps://vincents.dev/blog/creating-a-search-engine/ https://vincents.dev/blog/creating-a-search-engine/? I visited the page on mobile and I have to scroll horizontally back and forth before I can read? That is not fun at all.
- randomman1131 2y agoAs someone who cares about their online searches actually being good, fast and private, I cannot recommend SearXNG more. https://github.com/searxng/searxng/ https://github.com/searxng/searxng/ It's a metasearch engine that can query multiple search providers at once, including google, so you're not missing out on the good results you expect. Pick an instance at https://searx.space/ https://searx.space/ and tell your friends!
- sam29681749 2y agoThis article doesn't scale to fit the window on Firefox on android.
- deleted 2y ago[deleted]