3 ms·

I'd kind of like to see that claim substantiated a little more. Is it all crawlers that switch to a non-bot UA, or how are they determining it's the same bot?

by loeg 2y ago

I'd kind of like to see that claim substantiated a little more. Is it all crawlers that switch to a non-bot UA, or how are they determining it's the same bot? What non-bot UA do they claim?

denschub 2y ago
> Is it all crawlers that switch to a non-bot UA I've observed only one of them do this with high confidence. > how are they determining it's the same bot? it's fairly easy to determine that it's the same bot, because as soon as I blocked the "official" one, a bunch of AWS IPs started crawling the same URL patterns - in this case, mediawiki's diff view (`/wiki/index.php?title=[page]&diff=[new-id]&oldid=[old-id]`), that absolutely no bot ever crawled before. > What non-bot UA do they claim? Latest Chrome on Windows.
- loeg 2y ago
  Thanks.
untitaker_ 2y ago
Presumably they switch UA to Mozilla/something but tell on themselves by still using the same IP range or ASN. Unfortunately this has become common practice for feed readers as well.
alphan0n 2y ago
I would take anything the author said with a grain of salt. They straight up lied about the configuration of the robots.txt file. https://news.ycombinator.com/item?id=42551628 https://news.ycombinator.com/item?id=42551628
- mplewis 2y ago
  What is causing you to be so unnecessarily aggressive?
  - alphan0n 2y ago
    Liars should be called out, necessarily. Intellectual dishonesty is cancer. I could be more aggressive if it were something that really mattered.
    - nkrisc 2y ago
      Lying requires intent to deceive. How have you determined their intent?
      - n144q 2y ago
        > Lying requires intent to deceive Since when do we ask people to guess other people's intent when they have better things to show, which is called evidence? Surely we should talk about things with substantiated matter?
        nkrisc 2y ago
        Because there’s a meaningful difference between being wrong and lying. There’s evidence the statement was false, no evidence it was a lie.
      - alphan0n 2y ago
        When someone says: > Oh, and of course, they don't just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don't give a single flying fuck about robots.txt, because why should they. Their self righteous indignation and specificity of the pretend subject of that indignation precludes any doubt about intent. This guy made a whole public statement that is verifiably false. And then tried to toddler logic it away when he got called out.
        nkrisc 2y ago
        That may all be true. That still doesn’t mean they intentionally lied.
- ribadeo 2y ago
  How do you know what the contextual configuration of their robots.txt is/was? Your accusation was directly addressed by the author in a comment on the original post, IIRC i find your attitude as expressed here to be problematic in many ways
  - alphan0n 2y ago
    CommonCrawl archives robots.txt For convenience, you can view the extracted data here: https://pastebin.com/VSHMTThJ https://pastebin.com/VSHMTThJ You are welcome to verify for yourself by searching for “wiki.diasporafoundation.org/robots.txt” in the CommonCrawl index here: https://index.commoncrawl.org/ https://index.commoncrawl.org/ The index contains a file name that you can append to the CommonCrawl url to download the archive and view. More detailed information on downloading archives here: https://commoncrawl.org/get-started https://commoncrawl.org/get-started From September to December, the robots.txt at wiki.diasporafoundation.org contained this, and only this: >User-agent: * >Disallow: /w/ Apologies for my attitude, I find defenders of the dishonest in the face of clear evidence even more problematic.
    - shkkmo 2y ago
      Your attitude is inappropriate and violates the sitewide guidelines for discussion.
      - alphan0n 2y ago
        There are currently two references to “Mangion-ing” OpenAI board members in this thread, several more from Reddit, based on the falsehoods being perpetrated by the author. Is this really someone you want to conspire with? Is calling this out more egregious than the witch hunt being organized here?
        shkkmo 2y ago
        "conspire" and "witch hunt", are not terms of productive discourse. If you are legitimately trying to correct misinformation, your attitude, tone and language are counter productive. You would be much better seved by taking that energy and crafting an actually persuasive argument. You come across as unreasonable and unwilling to listen, not someone with a good grasp of the technical specifics. I don't have a horse in the race. I'm fairly technical, but I did not find your arguments persuasive. This doesn't mean they are wrong, but it does mean that you didn't do a good job of explaining them.

by loeg 2y ago

I'd kind of like to see that claim substantiated a little more. Is it all crawlers that switch to a non-bot UA, or how are they determining it's the same bot? What non-bot UA do they claim?

denschub 2y ago

> Is it all crawlers that switch to a non-bot UA I've observed only one of them do this with high confidence. > how are they determining it's the same bot? it's fairly easy to determine that it's the same bot, because as soon as I blocked the "official" one, a bunch of AWS IPs started crawling the same URL patterns - in this case, mediawiki's diff view (`/wiki/index.php?title=[page]&diff=[new-id]&oldid=[old-id]`), that absolutely no bot ever crawled before. > What non-bot UA do they claim? Latest Chrome on Windows.

loeg 2y ago
Thanks.

untitaker_ 2y ago

Presumably they switch UA to Mozilla/something but tell on themselves by still using the same IP range or ASN. Unfortunately this has become common practice for feed readers as well.

alphan0n 2y ago

I would take anything the author said with a grain of salt. They straight up lied about the configuration of the robots.txt file. https://news.ycombinator.com/item?id=42551628 https://news.ycombinator.com/item?id=42551628

mplewis 2y ago
What is causing you to be so unnecessarily aggressive?
- alphan0n 2y ago
  Liars should be called out, necessarily. Intellectual dishonesty is cancer. I could be more aggressive if it were something that really mattered.
  - nkrisc 2y ago
    Lying requires intent to deceive. How have you determined their intent?
    - n144q 2y ago
      > Lying requires intent to deceive Since when do we ask people to guess other people's intent when they have better things to show, which is called evidence? Surely we should talk about things with substantiated matter?
      - nkrisc 2y ago
        Because there’s a meaningful difference between being wrong and lying. There’s evidence the statement was false, no evidence it was a lie.
    - alphan0n 2y ago
      When someone says: > Oh, and of course, they don't just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don't give a single flying fuck about robots.txt, because why should they. Their self righteous indignation and specificity of the pretend subject of that indignation precludes any doubt about intent. This guy made a whole public statement that is verifiably false. And then tried to toddler logic it away when he got called out.
      - nkrisc 2y ago
        That may all be true. That still doesn’t mean they intentionally lied.
ribadeo 2y ago
How do you know what the contextual configuration of their robots.txt is/was? Your accusation was directly addressed by the author in a comment on the original post, IIRC i find your attitude as expressed here to be problematic in many ways
- alphan0n 2y ago
  CommonCrawl archives robots.txt For convenience, you can view the extracted data here: https://pastebin.com/VSHMTThJ https://pastebin.com/VSHMTThJ You are welcome to verify for yourself by searching for “wiki.diasporafoundation.org/robots.txt” in the CommonCrawl index here: https://index.commoncrawl.org/ https://index.commoncrawl.org/ The index contains a file name that you can append to the CommonCrawl url to download the archive and view. More detailed information on downloading archives here: https://commoncrawl.org/get-started https://commoncrawl.org/get-started From September to December, the robots.txt at wiki.diasporafoundation.org contained this, and only this: >User-agent: * >Disallow: /w/ Apologies for my attitude, I find defenders of the dishonest in the face of clear evidence even more problematic.
  - shkkmo 2y ago
    Your attitude is inappropriate and violates the sitewide guidelines for discussion.
    - alphan0n 2y ago
      There are currently two references to “Mangion-ing” OpenAI board members in this thread, several more from Reddit, based on the falsehoods being perpetrated by the author. Is this really someone you want to conspire with? Is calling this out more egregious than the witch hunt being organized here?
      - shkkmo 2y ago
        "conspire" and "witch hunt", are not terms of productive discourse. If you are legitimately trying to correct misinformation, your attitude, tone and language are counter productive. You would be much better seved by taking that energy and crafting an actually persuasive argument. You come across as unreasonable and unwilling to listen, not someone with a good grasp of the technical specifics. I don't have a horse in the race. I'm fairly technical, but I did not find your arguments persuasive. This doesn't mean they are wrong, but it does mean that you didn't do a good job of explaining them.