4 ms·

Part of this article is someone upset that a company is ignoring the robots.txt file and scraping their site. To that I say boo-hoo. Many other companies alre

Part of this article is someone upset that a company is ignoring the robots.txt file and scraping their site. To that I say boo-hoo. Many other companies already do this (not just Perplexity), Your moat shouldn't be hoping that actors follow your arbitrary business rules, and having data available to all (humans/robots) is a net benefit to all.

brigadier132 2y ago
> Your moat shouldn't be hoping that actors follow your arbitrary business rules, All these "should" statements always seem to be baseless opinion masquerading as fact. The unfortunate future is going to be all data hidden behind a login screen. Creators being able to sustain themselves is more important than consumers being able to get free and more convenient things. If consumers don't want to pay for something they should just create it themselves. See what I did there?
- bnchrch 2y ago
  We can both agree on the unfortunate future. But I think we can also both agree hoping that people will abide by your robots.txt file wont save us from that unfortunate future.
  - brigadier132 2y ago
    I think the difference is that now that the sites doing the scraping don't have a mutually beneficial relationship with the creator.
    - bnchrch 2y ago
      No, the difference is that theres nothing forcing you to abide by a robots.txt As long as that content is valuable it will be scraped and used. Theres no solution to this. So the structure of content sites will have to adapt. How? I dont know, likely will vary by publication and content type. Is it fair? Probably not Is it stoppable? No. Not without heavy, global legislation. (read impossible)
      - jaredwiener 2y ago
        This is such a weird take. If a house is robbed, you don't give the burglars a pass and say "well their security system wasn't very good." Yes, the defensive measures probably need improvement. But that doesn't mean that the act they are defending against is wrong.
        bnchrch 2y ago
        My point is that it isnt a security system. Its a sign on the door that says "Please don't rob me" while the door is unlocked and no body is home.
        jaredwiener 2y ago
        Right. And my point is that it would still be wrong to rob that person.
        bnchrch 2y ago
        We don't disagree there. In such a large ungoverned system as the internet morals don't dictate what people do / dont do. So it doesnt matter whats wrong / right. What does matter is what system participants will / wont do And they will scrape your site if you don't put up any real defences.
- jaredwiener 2y ago
  "Moat" implies preventing competition. This isn't competition, Perplexity isn't doing their own reporting and seeing who gets more readers/revenue. This is wholesale theft. CVS's moat isn't the security guard standing at the door. But like parent comment said, this is going to create the digital equivalent of those plastic lock covers over the decongestants.

brigadier132 2y ago

> Your moat shouldn't be hoping that actors follow your arbitrary business rules, All these "should" statements always seem to be baseless opinion masquerading as fact. The unfortunate future is going to be all data hidden behind a login screen. Creators being able to sustain themselves is more important than consumers being able to get free and more convenient things. If consumers don't want to pay for something they should just create it themselves. See what I did there?

bnchrch 2y ago
We can both agree on the unfortunate future. But I think we can also both agree hoping that people will abide by your robots.txt file wont save us from that unfortunate future.
- brigadier132 2y ago
  I think the difference is that now that the sites doing the scraping don't have a mutually beneficial relationship with the creator.
  - bnchrch 2y ago
    No, the difference is that theres nothing forcing you to abide by a robots.txt As long as that content is valuable it will be scraped and used. Theres no solution to this. So the structure of content sites will have to adapt. How? I dont know, likely will vary by publication and content type. Is it fair? Probably not Is it stoppable? No. Not without heavy, global legislation. (read impossible)
    - jaredwiener 2y ago
      This is such a weird take. If a house is robbed, you don't give the burglars a pass and say "well their security system wasn't very good." Yes, the defensive measures probably need improvement. But that doesn't mean that the act they are defending against is wrong.
      - bnchrch 2y ago
        My point is that it isnt a security system. Its a sign on the door that says "Please don't rob me" while the door is unlocked and no body is home.
        jaredwiener 2y ago
        Right. And my point is that it would still be wrong to rob that person.
        bnchrch 2y ago
        We don't disagree there. In such a large ungoverned system as the internet morals don't dictate what people do / dont do. So it doesnt matter whats wrong / right. What does matter is what system participants will / wont do And they will scrape your site if you don't put up any real defences.
jaredwiener 2y ago
"Moat" implies preventing competition. This isn't competition, Perplexity isn't doing their own reporting and seeing who gets more readers/revenue. This is wholesale theft. CVS's moat isn't the security guard standing at the door. But like parent comment said, this is going to create the digital equivalent of those plastic lock covers over the decongestants.