12 ms·

The web is now hostile. If you're starting a search engine, everyone else has written a robots.txt that bans you from starting a search engine. You either ignor

by immibis 10mo ago

The web is now hostile. If you're starting a search engine, everyone else has written a robots.txt that bans you from starting a search engine. You either ignore that, or you abandon your plan to make a search engine.

skybrian 10mo ago
Maybe only ethical choice is not to play? Or to do it the hard way. Scrape what people allow and try to make deals to get more data.
- immibis 10mo ago
  "Making a search engine is unethical" is certainly one of the takes of all time. I'm sure Google is glad you believe it.