4 ms·

I wonder if it would be possible to filter out the noise of the “modern web” by black listing big tech the same way we block ads. Might be an interesting projec

by wkjagt 3y ago

I wonder if it would be possible to filter out the noise of the “modern web” by black listing big tech the same way we block ads. Might be an interesting project.

saiya-jin 3y ago
I mean put adblocks into crazy mode, blacklist anything related to Google and Meta-crap by default, turn off JS and probably CSS too and you are almost there, maybe apart from blinking text and weirdly stretched bitmap backgrounds
jonpurdy 3y ago
Kagi (the search engine) has Small Web※, which (I assume) shows only results from the long tail and excludes popular sites (they use TinyGem search index for this). ※https://blog.kagi.com/small-web https://blog.kagi.com/small-web
- wkjagt 3y ago
  Oh this looks really nice. Thanks for sharing!
martinpw 3y ago
Wasn't there a site that showed search results but skipped the top N results, so you ended up seeing a lot more interesting smaller sites? Can't remember details and a quick search (hah) didn't find it. Pretty sure it was linked on HN a year or two ago... Edit - I think this was it, but looks like it needs an account: https://millionshort.com/ https://millionshort.com/
- 15457345234 3y ago
  > so you ended up seeing a lot more interesting smaller sites? A lot of smaller sites just aren't indexed any more, it's a common complaint you read here... "Smaller site, we don't use adwords, we're getting crawled 500x/day which is 40% of our bandwidth, still not indexed after 6 months"
flpm 3y ago
If a search engine were to penalize ads, I bet a lot of those old sites would surface. The old web was an "amateur" web, people created sites to share content they were excited about, not to monetize it (or to allow a platform to monetize it). If search engines also make most of their money from advertising, they have a big conflict of interest. We need an open source search engine owned by a non-profit organization following the model of Signal.
- wkjagt 3y ago
  > If a search engine were to penalize ads, I bet a lot of those old sites would surface. Maybe using an ad blocker to build a list of "websites that serve ads" (in addition to domains those ads are served from), and using that list to filter out results from an existing search engine API (like Duck Duck Go) would get you part of the way. But you'd probably need to search really far before you start finding sites that don't serve ads.
  - ornornor 3y ago
    Kagi partially does that. I lt shows how many trackers a result has on its results page and lets you boost or downgrade domains in your search results.
sirsinsalot 3y ago
I would use a search engine that only indexed sites that have zero ads, tracking or referral links. Not sure how useful it'd be, but I would use it! It would probably be like Google circa 1999-2000
- wkjagt 3y ago
  I would use that too, for sure! I kind of feel like experimenting with this now. I have no idea how doable it is to just "start scraping the web" though. uBlock Origin is open source, so I guess that could be used to help filter out sites that serve ads.
wconrm 3y ago
I find this CSE quite usefull for finding old websites https://www.oldestsearch.com/ https://www.oldestsearch.com/
robinsonb5 3y ago
All you need is a reliable way to detect cookie popups.

by wkjagt 3y ago

I wonder if it would be possible to filter out the noise of the “modern web” by black listing big tech the same way we block ads. Might be an interesting project.

saiya-jin 3y ago

I mean put adblocks into crazy mode, blacklist anything related to Google and Meta-crap by default, turn off JS and probably CSS too and you are almost there, maybe apart from blinking text and weirdly stretched bitmap backgrounds

jonpurdy 3y ago

Kagi (the search engine) has Small Web※, which (I assume) shows only results from the long tail and excludes popular sites (they use TinyGem search index for this). ※https://blog.kagi.com/small-web https://blog.kagi.com/small-web

wkjagt 3y ago
Oh this looks really nice. Thanks for sharing!

martinpw 3y ago

Wasn't there a site that showed search results but skipped the top N results, so you ended up seeing a lot more interesting smaller sites? Can't remember details and a quick search (hah) didn't find it. Pretty sure it was linked on HN a year or two ago... Edit - I think this was it, but looks like it needs an account: https://millionshort.com/ https://millionshort.com/

15457345234 3y ago
> so you ended up seeing a lot more interesting smaller sites? A lot of smaller sites just aren't indexed any more, it's a common complaint you read here... "Smaller site, we don't use adwords, we're getting crawled 500x/day which is 40% of our bandwidth, still not indexed after 6 months"

flpm 3y ago

If a search engine were to penalize ads, I bet a lot of those old sites would surface. The old web was an "amateur" web, people created sites to share content they were excited about, not to monetize it (or to allow a platform to monetize it). If search engines also make most of their money from advertising, they have a big conflict of interest. We need an open source search engine owned by a non-profit organization following the model of Signal.

wkjagt 3y ago
> If a search engine were to penalize ads, I bet a lot of those old sites would surface. Maybe using an ad blocker to build a list of "websites that serve ads" (in addition to domains those ads are served from), and using that list to filter out results from an existing search engine API (like Duck Duck Go) would get you part of the way. But you'd probably need to search really far before you start finding sites that don't serve ads.
- ornornor 3y ago
  Kagi partially does that. I lt shows how many trackers a result has on its results page and lets you boost or downgrade domains in your search results.

sirsinsalot 3y ago

I would use a search engine that only indexed sites that have zero ads, tracking or referral links. Not sure how useful it'd be, but I would use it! It would probably be like Google circa 1999-2000

wkjagt 3y ago
I would use that too, for sure! I kind of feel like experimenting with this now. I have no idea how doable it is to just "start scraping the web" though. uBlock Origin is open source, so I guess that could be used to help filter out sites that serve ads.

wconrm 3y ago

I find this CSE quite usefull for finding old websites https://www.oldestsearch.com/ https://www.oldestsearch.com/

robinsonb5 3y ago

All you need is a reliable way to detect cookie popups.