5 ms·

He takes this into account by generously considering any returned response code as “not dead”. > there’s a longtail of sites that had a variety of non-200 repo

He takes this into account by generously considering any returned response code as “not dead”. > there’s a longtail of sites that had a variety of non-200 reponse codes but just to be conservative we’ll assume that they are all valid

mort96 4y ago
That doesn't take this into account, no. `curl wixsite.com` returns a "Could not resolve host" error; it doesn't return a response code, so the author would consider it invalid, even though `curl www.wixsite.com` does return a response (a 301 redirect to www.wix.com).
- quickthrower2 4y ago
  Oh how does that work then? How does the browser get to the redirect when curl doesn't get any response at all? Is this a DNS thing?
  - chrisweekly 4y ago
    apex domain is different from www cname
    - quickthrower2 4y ago
      I just don't get how the browser gets a response like this (below), and then figures out what to do next. Sister comment said it might just try the common "www." prefix. $ dig wixsite.com ; <<>> DiG 9.16.1-Ubuntu <<>> wixsite.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65168 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;wixsite.com. IN A ;; Query time: 3 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) ;; WHEN: Sat Jul 16 09:12:42 AEST 2022 ;; MSG SIZE rcvd: 40 $ nslookup wixsite.com Server: 127.0.0.53 Address: 127.0.0.53#53 Non-authoritative answer: *** Can't find wixsite.com: No answer Does this mean that the WIX SEO team should really resurrect the record and do a 301, otherwise they are wasting their inbound links?
      - smugma 4y ago
        As someone hypothesizes above, it’s common for browsers to add www. to domains that don’t resolve.
        userbinator 4y ago
        I'm not sure if it's because I have my browsers set to generally "do what I say" or because I'm using a filtering proxy, but Firefox doesn't seem to try again if I just put "wixsite.com" in the address bar --- it gets a Host Not Found from the proxy and stops.
        account42 4y ago
        That is weird - it does the fixup from http://wixsite/ http://wixsite/ to http://www.wixsite.com/ http://www.wixsite.com/ but leaves http://wixsite.com/ http://wixsite.com/ alone.
      - remram 4y ago
        I think there isn't an inbound link, the crawler is choosing to hit http://wix.com/ http://wix.com/ if links to wix.com subdomains are common enough. It might be that there are millions of links to www.wix.com and docs.wix.com and user.api.wix.com and not a single (broken) link to wix.com, and they will crawl http://wix.com/ http://wix.com/ anyway and decide that "the site is dead". This is a problem with their methodology.
        quickthrower2 4y ago
        Yes this goes to “what is a site?” and “who/what is controlling what sub domains”. Especially with things like GitHub.io, and indeed wix. I think ignoring dead apex domains when a subdomain worked would have been a good extra pass for the methodology.
        account42 4y ago
        Perhaps that is the reason for the apex domain to be dead in the first place - to communicate that the subdomains are the real roots of separate sites. Similarly, TLDs themselves are not supposed to have any A records (although there are some that do).
  - Kwpolska 4y ago
    Browsers sometimes try adding things to URLs to try and make them work. Firefox tends to add https:// https:// if http:// http:// fails, perhaps some browsers are adding www.
    - quickthrower2 4y ago
      Funny, if true (doing some research...) I'll add this to the warchest for that classic interview question about what happens when you type a url into your browser.
      - account42 4y ago
        Definitely was true. Browsersers (at least FF) also used to add .com at the end. I think these days they all just send you to their ad-laden funding source instead if there is no TLD.(*) https://www.thewindowsclub.com/browser-automatically-adds-www-to-url https://www.thewindowsclub.com/browser-automatically-adds-ww... (*) Apparently at least firefox still does the domain fixup instead of search if you type the http:// http:// or https:// https:// prefix so e.g. http://example/ http://example/ will have you end up on http://www.example.com/ http://www.example.com/

mort96 4y ago

That doesn't take this into account, no. `curl wixsite.com` returns a "Could not resolve host" error; it doesn't return a response code, so the author would consider it invalid, even though `curl www.wixsite.com` does return a response (a 301 redirect to www.wix.com).

quickthrower2 4y ago
Oh how does that work then? How does the browser get to the redirect when curl doesn't get any response at all? Is this a DNS thing?
- chrisweekly 4y ago
  apex domain is different from www cname
  - quickthrower2 4y ago
    I just don't get how the browser gets a response like this (below), and then figures out what to do next. Sister comment said it might just try the common "www." prefix. $ dig wixsite.com ; <<>> DiG 9.16.1-Ubuntu <<>> wixsite.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65168 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;wixsite.com. IN A ;; Query time: 3 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) ;; WHEN: Sat Jul 16 09:12:42 AEST 2022 ;; MSG SIZE rcvd: 40 $ nslookup wixsite.com Server: 127.0.0.53 Address: 127.0.0.53#53 Non-authoritative answer: *** Can't find wixsite.com: No answer Does this mean that the WIX SEO team should really resurrect the record and do a 301, otherwise they are wasting their inbound links?
    - smugma 4y ago
      As someone hypothesizes above, it’s common for browsers to add www. to domains that don’t resolve.
      - userbinator 4y ago
        I'm not sure if it's because I have my browsers set to generally "do what I say" or because I'm using a filtering proxy, but Firefox doesn't seem to try again if I just put "wixsite.com" in the address bar --- it gets a Host Not Found from the proxy and stops.
        account42 4y ago
        That is weird - it does the fixup from http://wixsite/ http://wixsite/ to http://www.wixsite.com/ http://www.wixsite.com/ but leaves http://wixsite.com/ http://wixsite.com/ alone.
    - remram 4y ago
      I think there isn't an inbound link, the crawler is choosing to hit http://wix.com/ http://wix.com/ if links to wix.com subdomains are common enough. It might be that there are millions of links to www.wix.com and docs.wix.com and user.api.wix.com and not a single (broken) link to wix.com, and they will crawl http://wix.com/ http://wix.com/ anyway and decide that "the site is dead". This is a problem with their methodology.
      - quickthrower2 4y ago
        Yes this goes to “what is a site?” and “who/what is controlling what sub domains”. Especially with things like GitHub.io, and indeed wix. I think ignoring dead apex domains when a subdomain worked would have been a good extra pass for the methodology.
        account42 4y ago
        Perhaps that is the reason for the apex domain to be dead in the first place - to communicate that the subdomains are the real roots of separate sites. Similarly, TLDs themselves are not supposed to have any A records (although there are some that do).
- Kwpolska 4y ago
  Browsers sometimes try adding things to URLs to try and make them work. Firefox tends to add https:// https:// if http:// http:// fails, perhaps some browsers are adding www.
  - quickthrower2 4y ago
    Funny, if true (doing some research...) I'll add this to the warchest for that classic interview question about what happens when you type a url into your browser.
    - account42 4y ago
      Definitely was true. Browsersers (at least FF) also used to add .com at the end. I think these days they all just send you to their ad-laden funding source instead if there is no TLD.(*) https://www.thewindowsclub.com/browser-automatically-adds-www-to-url https://www.thewindowsclub.com/browser-automatically-adds-ww... (*) Apparently at least firefox still does the domain fixup instead of search if you type the http:// http:// or https:// https:// prefix so e.g. http://example/ http://example/ will have you end up on http://www.example.com/ http://www.example.com/