4 ms·

I would think that the number of people who (a) know how to create a valid robot.txt file, (b) have some idea of how to use the "crawl-delay" directive and (c)

I would think that the number of people who (a) know how to create a valid robot.txt file, (b) have some idea of how to use the "crawl-delay" directive and (c) write a "shoot-themselves-in-the-foot" worthy error is vanishingly small.

huxley 15y ago
As opposed to --for illustrative purposes-- the vanishingly small number of people who know how to block IP addresses and manage to get their site to disappear from Google's listings? ::cough:: A few years ago, I did pretty much the same thing myself. Thankfully the late summer was our slow season and the site recovered pretty quickly from my bone-headed move, but the split second after I realized what I've done was bone-chilling. I think just about everyone has thought at some point that they understood how something worked, only to have had things go pear-shaped on them. The lesson: people are not fully knowledgeable about everything, even the smart and talented ones.
- officemonkey 15y ago
  Perhaps I'm making an error assuming that a website as influential as Hacker News has a "real live Webmaster" to do things like write robot.txt files.
pilsetnieks 15y ago
Considering it's Google and we're talking about almost the whole population of the Earth, the vanishingly small percentage of the entire population of the planet would still be at least hundreds of thousands of people.
Matt_Cutts 15y ago
I alluded to some of the ways that I've seen people shoot themselves in the foot in a blog post a few years ago: http://www.mattcutts.com/blog/the-web-is-a-fuzz-test-patch-your-browser-and-your-web-server/ http://www.mattcutts.com/blog/the-web-is-a-fuzz-test-patch-y... "You would not believe the sort of weird, random, ill-formed stuff that some people put up on the web: everything from tables nested to infinity and beyond, to web documents with a filetype of exe, to executables returned as text documents. In a 1996 paper titled "An Investigation of Documents from the World Wide Web," Inktomi Eric Brewer and colleagues discovered that over 40% of web pages had at least one syntax error". We can often figure out the intent of the site owner, but mistakes do happen.
- adbge 15y ago
  The number of webpages with HTML that's just plain wrong (and renders fine!) is staggering. I often wonder what the web would be like if web browsers threw an error upon encountering a syntax error rather than making a best effort to render. If you're writing HTML, you should be validating it: http://validator.w3.org/ http://validator.w3.org/
  - MatthewPhillips 15y ago
    Google.com has 39 errors and 2 warnings. Among other things, they don't close their body or html tags. Is there any real downside to having syntax errors?
    - gcr 15y ago
      For the Google homepage, every byte counts. I'm not surprised.
      - alttag 15y ago
        The introduction of the menu bar and other javascripty-ness demonstrates this is less the case than it used to be.
    - ricardobeat 15y ago
      Not really, but there are many benefits to keeping your HTML clean. Google seems to just use whatever works, in able to support all kinds of ancient browsers.
    - dennisgorelik 15y ago
      It looks like Google Front page developers simply don't care about HTML compliance. There is no reason to allow most of these errors other than coding sloppiness. http://validator.w3.org/check?uri=http%3A%2F%2Fwww.google.com&charset=%28detect+automatically%29&doctype=Inline&group=0 http://validator.w3.org/check?uri=http%3A%2F%2Fwww.google.co...
    - thristian 15y ago

huxley 15y ago

As opposed to --for illustrative purposes-- the vanishingly small number of people who know how to block IP addresses and manage to get their site to disappear from Google's listings? ::cough:: A few years ago, I did pretty much the same thing myself. Thankfully the late summer was our slow season and the site recovered pretty quickly from my bone-headed move, but the split second after I realized what I've done was bone-chilling. I think just about everyone has thought at some point that they understood how something worked, only to have had things go pear-shaped on them. The lesson: people are not fully knowledgeable about everything, even the smart and talented ones.

officemonkey 15y ago
Perhaps I'm making an error assuming that a website as influential as Hacker News has a "real live Webmaster" to do things like write robot.txt files.

pilsetnieks 15y ago

Considering it's Google and we're talking about almost the whole population of the Earth, the vanishingly small percentage of the entire population of the planet would still be at least hundreds of thousands of people.

Matt_Cutts 15y ago

I alluded to some of the ways that I've seen people shoot themselves in the foot in a blog post a few years ago: http://www.mattcutts.com/blog/the-web-is-a-fuzz-test-patch-your-browser-and-your-web-server/ http://www.mattcutts.com/blog/the-web-is-a-fuzz-test-patch-y... "You would not believe the sort of weird, random, ill-formed stuff that some people put up on the web: everything from tables nested to infinity and beyond, to web documents with a filetype of exe, to executables returned as text documents. In a 1996 paper titled "An Investigation of Documents from the World Wide Web," Inktomi Eric Brewer and colleagues discovered that over 40% of web pages had at least one syntax error". We can often figure out the intent of the site owner, but mistakes do happen.

adbge 15y ago
The number of webpages with HTML that's just plain wrong (and renders fine!) is staggering. I often wonder what the web would be like if web browsers threw an error upon encountering a syntax error rather than making a best effort to render. If you're writing HTML, you should be validating it: http://validator.w3.org/ http://validator.w3.org/
- MatthewPhillips 15y ago
  Google.com has 39 errors and 2 warnings. Among other things, they don't close their body or html tags. Is there any real downside to having syntax errors?
  - gcr 15y ago
    For the Google homepage, every byte counts. I'm not surprised.
    - alttag 15y ago
      The introduction of the menu bar and other javascripty-ness demonstrates this is less the case than it used to be.
  - ricardobeat 15y ago
    Not really, but there are many benefits to keeping your HTML clean. Google seems to just use whatever works, in able to support all kinds of ancient browsers.
  - dennisgorelik 15y ago
    It looks like Google Front page developers simply don't care about HTML compliance. There is no reason to allow most of these errors other than coding sloppiness. http://validator.w3.org/check?uri=http%3A%2F%2Fwww.google.com&charset=%28detect+automatically%29&doctype=Inline&group=0 http://validator.w3.org/check?uri=http%3A%2F%2Fwww.google.co...
  - thristian 15y ago