32 ms·

Interesting that Google doesn't seem to use ipv6 for their crawlers. They seemed to be big supporters for ipv6, but they don't appear anywhere on the list. I'd

by dx034 10y ago

Interesting that Google doesn't seem to use ipv6 for their crawlers. They seemed to be big supporters for ipv6, but they don't appear anywhere on the list. I'd expect them to cause much more traffic than Facebook. Any clue why they only crawl via ipv4?

PudgePacket 10y ago
Does google use cloudflare? If not then their data probably wouldn't show up.
- driusan 10y ago
  They're in the chart. They're hovering near the bottom around 0%.
- brianwawok 10y ago
  Google is a partner of cloudflare. They have their own CDN though.
- dx034 10y ago
  Not directly, but their crawler will use Cloudflare. same as with Facebook. They don't use Cloudflare but crawl websites that use it.
johansch 10y ago
Because web sites won't be moving to ipv6 (only) any time soon, and the limiting factor here is residential connections, not Google? Are you proposing that Google should crawl ipv6-only web sites? And then serve those results to whom, exactly? (I don't think there is any reliable way of knowing whether an HTTP client that's connecting to you via ipv4 is also capable of ipv6?)
- dx034 10y ago
  Facebook seems to crawl ipv6 websites using an ipv6 connection. The Google crawler doesn't show up so it's probably working ipv4 only. I'm just wondering why Google doesn't seem to use ipv6 where available.
  - johansch 10y ago
    You mean they should prioritize ipv6 over ipv4 if a DNS lookup returns records of both? Why?
    - TorKlingberg 10y ago
      Many operating systems now prioritize IPv6 over IPv4. Otherwise there wouldn't be much IPv6 traffic at all.
      - johansch 10y ago
        If only v6 is available there is no point for them to include the site in their general index. If both v4 and v6 access is available, what is the extra value of doing the indexing using v6 rather than v4? (The indexing process itself would add very little to the global usage of v6, comparatively speaking.)
        dx034 10y ago
        In that case, what is the extra value of having ipv6 at all? And Facebook seems to prefer ipv6, I guess there's a reason for that.
        johansch 10y ago
        Facebook engineers also declared Mark Zuckerberg dead the other week. Maybe there's no deeper lesson here? http://www.theregister.co.uk/2016/11/11/facebook_zuckerberg_dead/ http://www.theregister.co.uk/2016/11/11/facebook_zuckerberg_...
    - zokier 10y ago
      That is what RFC6724 recommends as the default: > Another effect of the default policy table is to prefer communication using IPv6 addresses to communication using IPv4 addresses, if matching source addresses are available. https://tools.ietf.org/html/rfc6724#section-2.1 https://tools.ietf.org/html/rfc6724#section-2.1
- Nullabillity 10y ago
  > I don't think there is any reliable way of knowing whether an HTTP client that's connecting to you via ipv4 is also capable of ipv6? Send XHRs to a v4 subdomain and a v6 subdomain, check which requests succeed.
  - johansch 10y ago
    Sure, messy, but would generally work. I guess they could also move this checking to the user agent (because they control Chrome) to have the check performed before the user is actually waiting for results and milliseconds matter. Introduce some ugly but pragmatic HTTP header, like "Accept-IP: v6". :) So, this way they could serve v6-only results reliably to clients that can access either both v4 and v6. (People who can only access v6.. that must be a very special kind of people.)
zerognowl 10y ago
Because their crawler is so monolithic that it would be expensive and annoying overhauling it for IPV6. There is a great use-case for IPV6 for IOT where each device gets its own IPV6 address. IPV6 addresses are appearing more like MAC addresses at this rate as IPV6 is not exhausted yet.
- drcross 10y ago
  >is not exhausted yet. The world will be a completely unrecognizable place when this is even a slight concern.
  - SteveNuts 10y ago
    That's what they said about IPv4! But seriously, there's an astronomical # of addresses in IPv6. You're probably right that if we ever exhaust that space, we'll probably be communicating between planets by then.
    - baq 10y ago
      > That's what they said about IPv4! And they were right!
    - zokier 10y ago
      There is an astronomical number of individual IPv6 addresses, but in most cases that is not really the meaningful number to look at, at least right now. IPv6 is not really supposed to be subnetted beyond /64, so that already slashes the network space quite significantly. ISPs are supposed to hand out full /48s to customers (probably does not apply to consumers though), so there goes another 16 bits. The basic unit that RIRs give to ISPs is a /32 (afaik). Which leaves far less astronomical number of individual networks left. 2^32 - 2^48 is no doubt still a pretty big number, but not really as mindbogglingly humongous as 2^128.
- tombrossman 10y ago
  > There is a great use-case for IPV6 for IOT where each device gets its own IPV6 address. Do you really want your IOT devices to be directly addressable on the internet? It's my understanding that having devices behind a router is safer. I go a step further and disable UPnP on my routers and everything still 'just works' including network printing.
p1mrx 10y ago
The crawler supports IPv6-only servers, but IPv4 is preferred. For example: https://www.google.com/search?q=site:ipv6.whatismyv6.com https://www.google.com/search?q=site:ipv6.whatismyv6.com "You are connecting with an IPv6 Address of: 2001:4860:4801:6::1f."
- dx034 10y ago
  But why do they do that? That's what I don't understand. Google was one of the companies advertising ipv6 as the future, why don't they use it for their crawler?

by dx034 10y ago

PudgePacket 10y ago

Does google use cloudflare? If not then their data probably wouldn't show up.

driusan 10y ago
They're in the chart. They're hovering near the bottom around 0%.
brianwawok 10y ago
Google is a partner of cloudflare. They have their own CDN though.
dx034 10y ago
Not directly, but their crawler will use Cloudflare. same as with Facebook. They don't use Cloudflare but crawl websites that use it.

johansch 10y ago

Because web sites won't be moving to ipv6 (only) any time soon, and the limiting factor here is residential connections, not Google? Are you proposing that Google should crawl ipv6-only web sites? And then serve those results to whom, exactly? (I don't think there is any reliable way of knowing whether an HTTP client that's connecting to you via ipv4 is also capable of ipv6?)

dx034 10y ago
Facebook seems to crawl ipv6 websites using an ipv6 connection. The Google crawler doesn't show up so it's probably working ipv4 only. I'm just wondering why Google doesn't seem to use ipv6 where available.
- johansch 10y ago
  You mean they should prioritize ipv6 over ipv4 if a DNS lookup returns records of both? Why?
  - TorKlingberg 10y ago
    Many operating systems now prioritize IPv6 over IPv4. Otherwise there wouldn't be much IPv6 traffic at all.
    - johansch 10y ago
      If only v6 is available there is no point for them to include the site in their general index. If both v4 and v6 access is available, what is the extra value of doing the indexing using v6 rather than v4? (The indexing process itself would add very little to the global usage of v6, comparatively speaking.)
      - dx034 10y ago
        In that case, what is the extra value of having ipv6 at all? And Facebook seems to prefer ipv6, I guess there's a reason for that.
        johansch 10y ago
        Facebook engineers also declared Mark Zuckerberg dead the other week. Maybe there's no deeper lesson here? http://www.theregister.co.uk/2016/11/11/facebook_zuckerberg_dead/ http://www.theregister.co.uk/2016/11/11/facebook_zuckerberg_...
  - zokier 10y ago
    That is what RFC6724 recommends as the default: > Another effect of the default policy table is to prefer communication using IPv6 addresses to communication using IPv4 addresses, if matching source addresses are available. https://tools.ietf.org/html/rfc6724#section-2.1 https://tools.ietf.org/html/rfc6724#section-2.1
Nullabillity 10y ago
> I don't think there is any reliable way of knowing whether an HTTP client that's connecting to you via ipv4 is also capable of ipv6? Send XHRs to a v4 subdomain and a v6 subdomain, check which requests succeed.
- johansch 10y ago
  Sure, messy, but would generally work. I guess they could also move this checking to the user agent (because they control Chrome) to have the check performed before the user is actually waiting for results and milliseconds matter. Introduce some ugly but pragmatic HTTP header, like "Accept-IP: v6". :) So, this way they could serve v6-only results reliably to clients that can access either both v4 and v6. (People who can only access v6.. that must be a very special kind of people.)

zerognowl 10y ago

Because their crawler is so monolithic that it would be expensive and annoying overhauling it for IPV6. There is a great use-case for IPV6 for IOT where each device gets its own IPV6 address. IPV6 addresses are appearing more like MAC addresses at this rate as IPV6 is not exhausted yet.

drcross 10y ago
>is not exhausted yet. The world will be a completely unrecognizable place when this is even a slight concern.
- SteveNuts 10y ago
  That's what they said about IPv4! But seriously, there's an astronomical # of addresses in IPv6. You're probably right that if we ever exhaust that space, we'll probably be communicating between planets by then.
  - baq 10y ago
    > That's what they said about IPv4! And they were right!
  - zokier 10y ago
    There is an astronomical number of individual IPv6 addresses, but in most cases that is not really the meaningful number to look at, at least right now. IPv6 is not really supposed to be subnetted beyond /64, so that already slashes the network space quite significantly. ISPs are supposed to hand out full /48s to customers (probably does not apply to consumers though), so there goes another 16 bits. The basic unit that RIRs give to ISPs is a /32 (afaik). Which leaves far less astronomical number of individual networks left. 2^32 - 2^48 is no doubt still a pretty big number, but not really as mindbogglingly humongous as 2^128.
tombrossman 10y ago
> There is a great use-case for IPV6 for IOT where each device gets its own IPV6 address. Do you really want your IOT devices to be directly addressable on the internet? It's my understanding that having devices behind a router is safer. I go a step further and disable UPnP on my routers and everything still 'just works' including network printing.

p1mrx 10y ago

The crawler supports IPv6-only servers, but IPv4 is preferred. For example: https://www.google.com/search?q=site:ipv6.whatismyv6.com https://www.google.com/search?q=site:ipv6.whatismyv6.com "You are connecting with an IPv6 Address of: 2001:4860:4801:6::1f."

dx034 10y ago
But why do they do that? That's what I don't understand. Google was one of the companies advertising ipv6 as the future, why don't they use it for their crawler?