4 ms·

I wanted to suggest site:old.reddit.com since I use that version with automatic redirect, but this: https://old.reddit.com/robots.txt https://old.reddit.com/ro

by comboy 5y ago

I wanted to suggest site:old.reddit.com since I use that version with automatic redirect, but this: https://old.reddit.com/robots.txt https://old.reddit.com/robots.txt is very different from this: https://reddit.com/robots.txt https://reddit.com/robots.txt I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results for some queries since nobody is fighting about the position until it's widely known. I'm not sure how far are we from being able to do full text internet search. Or rather even quote search, preferably some fuzziness options. That would be cool, Google's quotation marks were really neat back when they were working.

oefrha 5y ago
> I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results for some queries since nobody is fighting about the position until it's widely known. That’s not going to happen before Cloudflare is dethroned. See this recent thread for some perspective: https://news.ycombinator.com/item?id=27153603 https://news.ycombinator.com/item?id=27153603 And even if there’s no Cloudflare, large sites that people want to search will always find ways to block bad bots. The only thing I can think of that might work is using crowd-sourced data, with all the problems that come with crowdsourcing.
- comboy 5y ago
  Sadness. There is a solution for all this mess and I'm blocking HN and a few different domains until I implement at least the first step after which I can share it here.
utf_8x 5y ago
Wonder what the story is behind these two... User-Agent: bender Disallow: /my_shiny_metal_ass User-Agent: Gort Disallow: /earth
- iso1210 5y ago
  Easter eggs
- pluc 5y ago
  It's neckbeard humor.
  - jedberg 5y ago
    I'll have you know our beards were neatly trimmed when we added those.
    - pluc 5y ago
      I know you, and I find that hard to believe ;)
- gostsamo 5y ago
  It might be related to the time few years ago when Google added exclusions for user agent t1300 in regard to its founders. Gort seems to be a robot from old scifi and bender might be something similar.
  - h2odragon 5y ago
    Bender is from Futurama, Gort is old classic scifi: https://en.wikipedia.org/wiki/Gort_(The_Day_the_Earth_Stood_Still) https://en.wikipedia.org/wiki/Gort_(The_Day_the_Earth_Stood_...
- deleted 5y ago
  [deleted]
- toastal 5y ago
  I reod some people use false slugs in the robots.txt as a honey pot of sorts. IPs that actually read the robots.txt, ignore the disallow, and still access the uri are outright banned.
  - 5y ago
bruce343434 5y ago
try editing your hosts file to redirect reddit to old.reddit /etc/hosts reddit.com old.reddit.com www.reddit.com old.reddit.com np.reddit.com old.reddit.com

by comboy 5y ago

oefrha 5y ago

> I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results for some queries since nobody is fighting about the position until it's widely known. That’s not going to happen before Cloudflare is dethroned. See this recent thread for some perspective: https://news.ycombinator.com/item?id=27153603 https://news.ycombinator.com/item?id=27153603 And even if there’s no Cloudflare, large sites that people want to search will always find ways to block bad bots. The only thing I can think of that might work is using crowd-sourced data, with all the problems that come with crowdsourcing.

comboy 5y ago
Sadness. There is a solution for all this mess and I'm blocking HN and a few different domains until I implement at least the first step after which I can share it here.

utf_8x 5y ago

Wonder what the story is behind these two... User-Agent: bender Disallow: /my_shiny_metal_ass User-Agent: Gort Disallow: /earth

iso1210 5y ago
Easter eggs
pluc 5y ago
It's neckbeard humor.
- jedberg 5y ago
  I'll have you know our beards were neatly trimmed when we added those.
  - pluc 5y ago
    I know you, and I find that hard to believe ;)
gostsamo 5y ago
It might be related to the time few years ago when Google added exclusions for user agent t1300 in regard to its founders. Gort seems to be a robot from old scifi and bender might be something similar.
- h2odragon 5y ago
  Bender is from Futurama, Gort is old classic scifi: https://en.wikipedia.org/wiki/Gort_(The_Day_the_Earth_Stood_Still) https://en.wikipedia.org/wiki/Gort_(The_Day_the_Earth_Stood_...
deleted 5y ago
[deleted]
toastal 5y ago
I reod some people use false slugs in the robots.txt as a honey pot of sorts. IPs that actually read the robots.txt, ignore the disallow, and still access the uri are outright banned.
- 5y ago

bruce343434 5y ago

try editing your hosts file to redirect reddit to old.reddit /etc/hosts reddit.com old.reddit.com www.reddit.com old.reddit.com np.reddit.com old.reddit.com