3 ms·

Accidentally deleting someone's entire website because they don't understand the difference between GET and POST requests is virtually a right of passage when w

Accidentally deleting someone's entire website because they don't understand the difference between GET and POST requests is virtually a right of passage when writing a web crawler.

chii 7y ago
if their 'delete website' link is reachable anonymously and performs using a GET, they deserve the result.
- sjwright 7y ago
  Does Google promise that its bot will never submit a POST even if it’s triggered by an <a onclick>?
  - epse 7y ago
    Googlebot has no javascript
    - adrianolek 7y ago
      Is that true, though? [1] [1] https://webmasters.googleblog.com/2019/05/the-new-evergreen-googlebot.html https://webmasters.googleblog.com/2019/05/the-new-evergreen-...
      - groestl 7y ago
        It's a masterpiece that Google convinced everyone to use their crawler engine as a browser.
- aidos 7y ago
  I’m trying to find a link to it but there was an incident based on this issue somewhere around 1999-2001 where Microsoft added a sort of prefetching thing to IE (or was it Netscape?!) and it would effectively click all the links on the page in order to get all the content in the cache. Lots of us really didn’t know what we were doing and we’d made all the action buttons in the listing screens regular links. As you can imagine, pandemonium ensued. Hey, at least we’d figured out that sql injection was a thing. It was a simpler time.
  - watersb 7y ago
    1997 we had this crazy notion of web "channels" (like RSS feeds), and offline viewing, where a client on a painfully slow dial-up connection could download and cache the resources required to display complex web pages. Microsoft did this via an explicit web manifest; the web page author needed to list all of the resources they wanted to use in offline or pre-cache mode. Netscape tried to do this by urging web authors to Be Very Careful with the links on a page, which usually required a specially-crafted offline-crawler-only version of the site. Predicably, hilarity ensued. The term of art at the time was "push technology" or "web push", the irony of which was not lost upon those tasked with making it work. 1997. Good times.
  - dragonwriter 7y ago
    While the timeframe and company are a little off, it sounds like you Google Web Accelerator (for IE and Firefox) in 2005. https://web.archive.org/web/20050505061702/http://webaccelerator.google.com/ https://web.archive.org/web/20050505061702/http://webacceler... The issue you described seems to match this: https://signalvnoise.com/archives2/google_web_accelerator_hey_not_so_fast_an_alert_for_web_app_designers.php https://signalvnoise.com/archives2/google_web_accelerator_he...
quickthrower2 7y ago
That's why it's good to crawl twice, so if the site got deleted by the first crawl, you can check and then discard the results. Saves a bit of disk space.
- 13of40 7y ago
  There's a similar problem with AV automatically unsubscribing all of your customers from their spam (and newsletters) the first time you start scanning emails for malicious links. It's a little less than completely solvable, in fact.
  - krageon 7y ago
    I'd say this is a happy solution, but that's as a target for these emails - not as a sender.

Accidentally deleting someone's entire website because they don't understand the difference between GET and POST requests is virtually a right of passage when writing a web crawler.

chii 7y ago

if their 'delete website' link is reachable anonymously and performs using a GET, they deserve the result.

sjwright 7y ago
Does Google promise that its bot will never submit a POST even if it’s triggered by an <a onclick>?
- epse 7y ago
  Googlebot has no javascript
  - adrianolek 7y ago
    Is that true, though? [1] [1] https://webmasters.googleblog.com/2019/05/the-new-evergreen-googlebot.html https://webmasters.googleblog.com/2019/05/the-new-evergreen-...
    - groestl 7y ago
      It's a masterpiece that Google convinced everyone to use their crawler engine as a browser.
aidos 7y ago
I’m trying to find a link to it but there was an incident based on this issue somewhere around 1999-2001 where Microsoft added a sort of prefetching thing to IE (or was it Netscape?!) and it would effectively click all the links on the page in order to get all the content in the cache. Lots of us really didn’t know what we were doing and we’d made all the action buttons in the listing screens regular links. As you can imagine, pandemonium ensued. Hey, at least we’d figured out that sql injection was a thing. It was a simpler time.
- watersb 7y ago
  1997 we had this crazy notion of web "channels" (like RSS feeds), and offline viewing, where a client on a painfully slow dial-up connection could download and cache the resources required to display complex web pages. Microsoft did this via an explicit web manifest; the web page author needed to list all of the resources they wanted to use in offline or pre-cache mode. Netscape tried to do this by urging web authors to Be Very Careful with the links on a page, which usually required a specially-crafted offline-crawler-only version of the site. Predicably, hilarity ensued. The term of art at the time was "push technology" or "web push", the irony of which was not lost upon those tasked with making it work. 1997. Good times.
- dragonwriter 7y ago
  While the timeframe and company are a little off, it sounds like you Google Web Accelerator (for IE and Firefox) in 2005. https://web.archive.org/web/20050505061702/http://webaccelerator.google.com/ https://web.archive.org/web/20050505061702/http://webacceler... The issue you described seems to match this: https://signalvnoise.com/archives2/google_web_accelerator_hey_not_so_fast_an_alert_for_web_app_designers.php https://signalvnoise.com/archives2/google_web_accelerator_he...

quickthrower2 7y ago

That's why it's good to crawl twice, so if the site got deleted by the first crawl, you can check and then discard the results. Saves a bit of disk space.

13of40 7y ago
There's a similar problem with AV automatically unsubscribing all of your customers from their spam (and newsletters) the first time you start scanning emails for malicious links. It's a little less than completely solvable, in fact.
- krageon 7y ago
  I'd say this is a happy solution, but that's as a target for these emails - not as a sender.