4 ms·

You're taking someone else's labor and profiting off it, without any credit or compensation. To add insult to injury, the person you're scraping pays money to s

• 9dev 2y ago
  You can’t generalise that. Maybe I crawl to provide an annotated preview of their website, to make users of my application more likely to click the link and visit it? There are lots of ways in which crawling benefits everyone, it just requires some mutual respect.
• deisteve 2y ago
  We don't live in the 90s anymore. The bandwidth CPU cost is moot unless you are spinning up thousands of GPUs to render an HTML page. Also the claims of someone else's labor and profiteering is exactly what Google does
  • __loam 2y ago
    What Google does is mutually beneficial. Stealing content to reheat and serve in an LLM is very different.
    • deisteve 2y ago
      Google that creates monopoly on top of the scraped data and uses third party content to do so? LLM just places that ability at the ends of the user (local LLM strictly speaking)
      • __loam 2y ago
        No it doesn't. The end state of a Google search is a human interacting directly with a site.