4 ms·

I couldn't. Amusing that archive.ph's own crawler, evidently, went through just fine.

by ACCount37 2mo ago

Bender 2mo ago
I disabled blocking for a few hours then crawled several of my pages with their site.
- Barbing 2mo ago
  Oh, I assumed Archive had an automated* tool in their arsenal to overcome the blocks that would’ve affected them. Didja happen to confirm they were blocked initially? *I think they do something manually sometimes, like logging into a The Information Pro account and mirroring popular articles. (Unless they can use a visitor’s account somehow, like with some browser extension…) And suspected without evidence maybe The Information did some watermarking to ensure they could quickly ban accounts used like that. And thank you!
  - Bender 2mo ago
    Didja happen to confirm they were blocked initially They were the reason I started blocking data-centers. I wanted to see if I could block all of their addresses which turned into quite an exercise. Initially I was able to block them by blocking TCP SYN packets with an MSS of 1380 and 1300 but talking about it here was a mistake as their admin is on HN. They fixed that and made their MSS 1460 like a nominal home internet connection. The next thing I had to block was the Russian Federation. That seems to be where the controlling nodes are, I think. That made me even more curious who they were so I just kept iterating through the blocking process. I ended up blocking a number of ASN's and all of Russia. They are all in the linked archive file in the article but I don't specifically call out the ones that are archive.is. Most on HN think they operate on a shoe-string budget but I have my doubts. I suspect an asset of a technical arm of an intelligence agency but to what purpose and what end I have no clue. They are very clever whoever they are and certainly earned my respect. Either way I think their site is useful, I just wanted to see if I could block it.
    - inigyou 2mo ago
      Blocking anyone who uses a VPN. Ouch.
      - lukewarm707 2mo ago
        same applies. if its blocked just use a bot API XD. it can't see the bot traffic, it just looks like users
        inigyou 2mo ago
        Yep just pay some grey market business for a residential proxy
        deleted 2mo ago
        [deleted]
    - BLKNSLVR 2mo ago
      > Initially I was able to block them by blocking TCP SYN packets with an MSS of 1380 and 1300 but talking about it here was a mistake as their admin is on HN. They fixed that and made their MSS 1460 like a nominal home internet connection. For some reason I find this very amusing.
lukewarm707 2mo ago
you can use an ai crawler API to download it if you can't get through on your personal device

I couldn't. Amusing that archive.ph's own crawler, evidently, went through just fine.

Bender 2mo ago

I disabled blocking for a few hours then crawled several of my pages with their site.

Barbing 2mo ago
Oh, I assumed Archive had an automated* tool in their arsenal to overcome the blocks that would’ve affected them. Didja happen to confirm they were blocked initially? *I think they do something manually sometimes, like logging into a The Information Pro account and mirroring popular articles. (Unless they can use a visitor’s account somehow, like with some browser extension…) And suspected without evidence maybe The Information did some watermarking to ensure they could quickly ban accounts used like that. And thank you!
- Bender 2mo ago
  Didja happen to confirm they were blocked initially They were the reason I started blocking data-centers. I wanted to see if I could block all of their addresses which turned into quite an exercise. Initially I was able to block them by blocking TCP SYN packets with an MSS of 1380 and 1300 but talking about it here was a mistake as their admin is on HN. They fixed that and made their MSS 1460 like a nominal home internet connection. The next thing I had to block was the Russian Federation. That seems to be where the controlling nodes are, I think. That made me even more curious who they were so I just kept iterating through the blocking process. I ended up blocking a number of ASN's and all of Russia. They are all in the linked archive file in the article but I don't specifically call out the ones that are archive.is. Most on HN think they operate on a shoe-string budget but I have my doubts. I suspect an asset of a technical arm of an intelligence agency but to what purpose and what end I have no clue. They are very clever whoever they are and certainly earned my respect. Either way I think their site is useful, I just wanted to see if I could block it.
  - inigyou 2mo ago
    Blocking anyone who uses a VPN. Ouch.
    - lukewarm707 2mo ago
      same applies. if its blocked just use a bot API XD. it can't see the bot traffic, it just looks like users
      - inigyou 2mo ago
        Yep just pay some grey market business for a residential proxy
        deleted 2mo ago
        [deleted]
  - BLKNSLVR 2mo ago
    > Initially I was able to block them by blocking TCP SYN packets with an MSS of 1380 and 1300 but talking about it here was a mistake as their admin is on HN. They fixed that and made their MSS 1460 like a nominal home internet connection. For some reason I find this very amusing.

lukewarm707 2mo ago

you can use an ai crawler API to download it if you can't get through on your personal device