4 ms·
Is all that HTML? Spiders generally only care about HTML -- they're not going to download the js and css and images and flash and whatever.
by joevandyk 17y ago
Is all that HTML? Spiders generally only care about HTML -- they're not going to download the js and css and images and flash and whatever.
- latch 17y agoduh, I didn't think about that :) HTML is way smaller
- bluesmoon 17y ago8 years ago, maybe. Today all search engine spiders pull down and execute javascript & css as well just so that they can index a real user's experience.
- code_duck 17y agoThe better ones may, but perhaps not for all pages. Generally, when viewing logs I see search engine spiders make a single request, for only the html content and not js or css.