3 ms·

A lot of negativity about scraped data, in comments…but both OpenAI and Google rely on scraping as their foundation for…everything. If smaller indie hackers ar

A lot of negativity about scraped data, in comments…but both OpenAI and Google rely on scraping as their foundation for…everything. If smaller indie hackers are going to build useful competitors, they’re going to need to scrape. I understand that a lot of scraping is from spammers, but not all.

bdcravens 3y ago
> If smaller indie hackers are going to build useful competitors Any useful competitor will have to scale well beyond the point of indie hacker, where someone is (eventually) getting rich. That doesn't make for a great argument for bypassing consent.
- troyvit 3y ago
  Depends on how you describe "useful." Kagi is a profitable search company that satisfies its customers while competing directly with Google. They aren't useful in the sense that they're taking marketshare from Google, but they are useful to their user base. I hope they continue being able to scrape sites with the same freedom as other search engines. Edit: As mmcclure points out Kagi actually doesn't index much. So that's a bad example that might invalidate my point.
  - mmcclure 3y ago
    I'm a happy Kagi user, but they might be somewhat a bad example. I suspect their actual indexing work is pretty limited, and they only seem to claim that it's for indexing "niche" stuff. > Our data includes anonymized API calls to traditional search indexes like Google, Yandex, Mojeek and Brave, specialized search engines like Marginalia, and sources of vertical information like Apple, Wikipedia, Open Meteo, Yelp, TripAdvisor and other APIs. Typically every search query on Kagi will call a number of different sources at the same time, all with the purpose of bringing the best possible search results to the user. source: https://help.kagi.com/kagi/search-details/search-sources.html https://help.kagi.com/kagi/search-details/search-sources.htm...
    - troyvit 3y ago
      Thanks I updated my comment. Also, damn that's cool.
mmcclure 3y ago
Scraping itself isn't a problem. Scraping content where people are actively trying to stop it from being scraped is where things get questionable. both OpenAI and Google rely on scraping as their foundation for…everything. OpenAI and Google also (claim to) respect the most basic bot deterrent, robots.txt, with their crawlers. If someone doesn't want their content scraped to be included by search indexes or LLMs/models, then that certainly includes "smaller indie hackers."

bdcravens 3y ago

> If smaller indie hackers are going to build useful competitors Any useful competitor will have to scale well beyond the point of indie hacker, where someone is (eventually) getting rich. That doesn't make for a great argument for bypassing consent.

troyvit 3y ago
Depends on how you describe "useful." Kagi is a profitable search company that satisfies its customers while competing directly with Google. They aren't useful in the sense that they're taking marketshare from Google, but they are useful to their user base. I hope they continue being able to scrape sites with the same freedom as other search engines. Edit: As mmcclure points out Kagi actually doesn't index much. So that's a bad example that might invalidate my point.
- mmcclure 3y ago
  I'm a happy Kagi user, but they might be somewhat a bad example. I suspect their actual indexing work is pretty limited, and they only seem to claim that it's for indexing "niche" stuff. > Our data includes anonymized API calls to traditional search indexes like Google, Yandex, Mojeek and Brave, specialized search engines like Marginalia, and sources of vertical information like Apple, Wikipedia, Open Meteo, Yelp, TripAdvisor and other APIs. Typically every search query on Kagi will call a number of different sources at the same time, all with the purpose of bringing the best possible search results to the user. source: https://help.kagi.com/kagi/search-details/search-sources.html https://help.kagi.com/kagi/search-details/search-sources.htm...
  - troyvit 3y ago
    Thanks I updated my comment. Also, damn that's cool.

mmcclure 3y ago

Scraping itself isn't a problem. Scraping content where people are actively trying to stop it from being scraped is where things get questionable. both OpenAI and Google rely on scraping as their foundation for…everything. OpenAI and Google also (claim to) respect the most basic bot deterrent, robots.txt, with their crawlers. If someone doesn't want their content scraped to be included by search indexes or LLMs/models, then that certainly includes "smaller indie hackers."