6 ms·

GoogleBot runs only in a few datacenters in certain locations in the world with amazing connectivity. It is a horrible predictor of load times.

ubernostrum 9y ago
Chrome's own dev tools can emulate slow connections for you. If their browser can do that, why not their crawler?
- ucaetano 9y ago
  Because the user isn't sitting at a datacenter. An user in Indonesia isn't at the closest datacenter in Singapore. It doesn't matter what you do, you can't, from Singapore, try to emulate what an user on a particular ISP in a particular city in Indonesia will face. In case it isn't clear: Consider 2 websites, both have hosting in Singapore, but one also uses a CDN in Jakarta. The Google crawler running on the Singapore datacenter connects to both websites, speeds are great because they are sitting just next to the datacenter. Both get great ratings. An user is in Jakarta, and tries to connect to both websites. One of them takes a century to load, because it has to load everything from Singapore, and international cables from Singapore to Jakarta are expensive, so connections are capped. Experience sucks. The second website uses a CDN in Jakarta, located at the local coloc facility to which the user's ISP is also connected to. The website loads instantly. CDNs today are located in every single small ISP, in all countries, in all medium-sized cities in the world.
  - SheinhardtWigCo 9y ago
    The poster was arguing that negative network conditions can be simulated. If the problem is a lack of information about global network conditions, then you can deploy canary machines around the world to gather that information.
    - ucaetano 9y ago
      They can be simulated, but not predicted. > If the problem is a lack of information about global network conditions, then you can deploy canary machines around the world to gather that information. You'd need to put machines in every single city and every single ISP in the world continuously testing every website in the world. Doesn't make any sense at all. That's not how networks work.
      - TeMPOraL 9y ago
        It's unfeasible, but also completely unnecessary. CDNs can give you some speedup, but they don't magically make a site smaller and less resource-intensive to use.
        deleted 9y ago
        [deleted]
      - SheinhardtWigCo 9y ago
        OP was talking about a "neutral" speed index. You're talking about solving for every possible combination of user, ISP, and website. There's probably a reasonable middle ground in there somewhere.
        ucaetano 9y ago
        But that's the point, it doesn't depend on a "neutral" speed index, it varies from website to website, ISP to ISP. If you're in Jakarta trying to access a certain website, if that website is available at the local CDN might make a bigger difference than which ISP or even which plan you're using. You could have a 1Gbps fiber connection from your ISP in Jakarta. If you're trying to access a website in Singapore the your ISP's international link is congested, that 1Gbps will be useless. OTOH, if you have a 10Mbps connection, and you're trying to access a website replicated at an Akamai CDN in your ISP's network, it will be way faster.
        joshmaker 9y ago
  - idoh 9y ago
    If a site is running Google Analytics, couldn't that information feed back into page loading speeds?
    - tempestn 9y ago
      You basically have the same problem then, of forcing sites to use google products in order to be ranked highly.
  - TeMPOraL 9y ago
    Beyond the fact that the network can be mapped (Google surely does this) and simulated, I feel the more important observation is that this doesn't matter. A good estimate of bloatiness should not require knowledge of network topology. CDNs are a hack that can make websites load a bit faster, but that's about it. It won't make them use less RAM, or consume less of your data cap. Ultimately, performance is about the size of the site, the resources it uses on the client, and the number of requests it makes. All of which is independent of the number and location of CDNs.
    - ucaetano 9y ago
      > A good estimate of bloatiness should not require knowledge of network topology. It does, because the definition of bloatness depends on the network speed. What counts as bloat for an user sitting in Singapore isn't the same as an user sitting in Jakarta. > Beyond the fact that the network can be mapped (Google surely does this) and simulated Network topology is irrelevant in this case. The ISP in Jakarta might have a satellite connection to Singapore, but have a CDN and it will load faster than in another ISP with a 10G international fiber link to Singapore. > All of which is independent of the number and location of CDNs. That's completely wrong. Today, the vast majority of internet traffic is served from CDNs, CDNs (and their placements) are critical for the internet as we know it.
      - TeMPOraL 9y ago
        > It does, because the definition of bloatness depends on the network speed. No, it doesn't. It depends on the proportion of actual content to the unnecessary cruft. The difference is that what a Singaporean considers bloat might prevent someone in Jakarta from accessing the site at all. > That's completely wrong. Today, the vast majority of internet traffic is served from CDNs, CDNs (and their placements) are critical for the internet as we know it. That is if you include streaming video, which is off-topic for this discussion. Imagine a perfect world with the ultimate CDN solution - something like IPFS, which ensures everyone is a CDN to everyone else (instead of wasting bandwidth with multiple downloads of the same content). The concept of bloat wouldn't disappear in such a world, because bloated websites will still require you to download unnecessary amount of data to your machine, and will reduce your computer's ability to multiprocess while also using up your battery life. I stand by what I said before - CDNs are a hack (basically a hand-rolled, rudimentary distributed system) to give you some extra download speed. They don't solve the problems of download size and resource usage.
  - ubernostrum 9y ago
    None of this is relevant, though. Simply simulating a globally slowed-down connection will tell you what's going to be a problem. And, really, common sense will tell you what's going to be a problem, too -- if a site "needs" to load 100MB of ads and trackers and other JavaScript to display a couple kilobytes of actual text content, then you have a problem. A user on a slower connection is going to see that site take forever to load even if there's a CDN across the street.
adtac 9y ago
This is probably the easiest problem to solve. Hosting a GoogleSpeedTestBot in every city with the sole purpose of speed testing a page would have laughably negligible impact on Google's resources.
- ucaetano 9y ago
  You're proposing hosting thousands and thousands of servers across every city in the world where each server would be accessing all the top 1M sites in the world every day and measuring the speed of each page in that site? Seriously?
  - adtac 9y ago
    Not every day; at the time of publishing, and maybe few days later. (This could be improved even further: you don't need to access every article. If you could identify a page that's representative of a real article, that could suffice. Maybe have periodic random checks to see if this is still representative.) Not every city; the top 500 cities would probably cover 90+% of the users. You're not understanding the scale of Google. They're incredibly good at exactly this stuff.
    - ucaetano 9y ago
      Oh, I am, but maybe you're not understanding the scale of the internet. Connections, cables, CDNs change on a daily and weekly basis. Is a cable is cut between Jakarta and Singapore, it can take months to fix as Indonesia only allows Indonesian-flagged cable layers to operate in its waters. Individual datacenters, CDNs, cables are going offline constantly, so you'd need to be continuously testing every site to rank it correctly. Not only that, each page in a site might have a very different structure, so you can't just rank a site, you need to rank individual pages and articles too, which means testing every new page and article from every city. I guess you're not getting the scale of that... AMP standardizes the CDN serving part, largely eliminating that issue. AMP articles can't prevent 3rd party CDNs from caching them, which means that they can be served from the closest CDN after the first time they are used.
- 9y ago
B-Con 9y ago
Counterpoint: You can already simulate the way bad Internet connections work. You don't want that introducing noise on top of the site's metrics. Measure the site itself, then apply a variety of scenarios for what practical fetches look like.
dx034 9y ago
If you have the number of roundtrips and size of documents you can easily calculate page load speed dependent on latency, packet loss rate and connection speed. That allows you to simulate nearly any connection in the world. Which is the reason why Google focuses so much on the number of roundtrips. For most, downloading 50kbyte doesn't cause a lag but 10 roundtrips before the site can be rendered do.

ubernostrum 9y ago

Chrome's own dev tools can emulate slow connections for you. If their browser can do that, why not their crawler?

ucaetano 9y ago
Because the user isn't sitting at a datacenter. An user in Indonesia isn't at the closest datacenter in Singapore. It doesn't matter what you do, you can't, from Singapore, try to emulate what an user on a particular ISP in a particular city in Indonesia will face. In case it isn't clear: Consider 2 websites, both have hosting in Singapore, but one also uses a CDN in Jakarta. The Google crawler running on the Singapore datacenter connects to both websites, speeds are great because they are sitting just next to the datacenter. Both get great ratings. An user is in Jakarta, and tries to connect to both websites. One of them takes a century to load, because it has to load everything from Singapore, and international cables from Singapore to Jakarta are expensive, so connections are capped. Experience sucks. The second website uses a CDN in Jakarta, located at the local coloc facility to which the user's ISP is also connected to. The website loads instantly. CDNs today are located in every single small ISP, in all countries, in all medium-sized cities in the world.
- SheinhardtWigCo 9y ago
  The poster was arguing that negative network conditions can be simulated. If the problem is a lack of information about global network conditions, then you can deploy canary machines around the world to gather that information.
  - ucaetano 9y ago
    They can be simulated, but not predicted. > If the problem is a lack of information about global network conditions, then you can deploy canary machines around the world to gather that information. You'd need to put machines in every single city and every single ISP in the world continuously testing every website in the world. Doesn't make any sense at all. That's not how networks work.
    - TeMPOraL 9y ago
      It's unfeasible, but also completely unnecessary. CDNs can give you some speedup, but they don't magically make a site smaller and less resource-intensive to use.
      - deleted 9y ago
        [deleted]
    - SheinhardtWigCo 9y ago
      OP was talking about a "neutral" speed index. You're talking about solving for every possible combination of user, ISP, and website. There's probably a reasonable middle ground in there somewhere.
      - ucaetano 9y ago
        But that's the point, it doesn't depend on a "neutral" speed index, it varies from website to website, ISP to ISP. If you're in Jakarta trying to access a certain website, if that website is available at the local CDN might make a bigger difference than which ISP or even which plan you're using. You could have a 1Gbps fiber connection from your ISP in Jakarta. If you're trying to access a website in Singapore the your ISP's international link is congested, that 1Gbps will be useless. OTOH, if you have a 10Mbps connection, and you're trying to access a website replicated at an Akamai CDN in your ISP's network, it will be way faster.
        joshmaker 9y ago
- idoh 9y ago
  If a site is running Google Analytics, couldn't that information feed back into page loading speeds?
  - tempestn 9y ago
    You basically have the same problem then, of forcing sites to use google products in order to be ranked highly.
- TeMPOraL 9y ago
  Beyond the fact that the network can be mapped (Google surely does this) and simulated, I feel the more important observation is that this doesn't matter. A good estimate of bloatiness should not require knowledge of network topology. CDNs are a hack that can make websites load a bit faster, but that's about it. It won't make them use less RAM, or consume less of your data cap. Ultimately, performance is about the size of the site, the resources it uses on the client, and the number of requests it makes. All of which is independent of the number and location of CDNs.
  - ucaetano 9y ago
    > A good estimate of bloatiness should not require knowledge of network topology. It does, because the definition of bloatness depends on the network speed. What counts as bloat for an user sitting in Singapore isn't the same as an user sitting in Jakarta. > Beyond the fact that the network can be mapped (Google surely does this) and simulated Network topology is irrelevant in this case. The ISP in Jakarta might have a satellite connection to Singapore, but have a CDN and it will load faster than in another ISP with a 10G international fiber link to Singapore. > All of which is independent of the number and location of CDNs. That's completely wrong. Today, the vast majority of internet traffic is served from CDNs, CDNs (and their placements) are critical for the internet as we know it.
    - TeMPOraL 9y ago
      > It does, because the definition of bloatness depends on the network speed. No, it doesn't. It depends on the proportion of actual content to the unnecessary cruft. The difference is that what a Singaporean considers bloat might prevent someone in Jakarta from accessing the site at all. > That's completely wrong. Today, the vast majority of internet traffic is served from CDNs, CDNs (and their placements) are critical for the internet as we know it. That is if you include streaming video, which is off-topic for this discussion. Imagine a perfect world with the ultimate CDN solution - something like IPFS, which ensures everyone is a CDN to everyone else (instead of wasting bandwidth with multiple downloads of the same content). The concept of bloat wouldn't disappear in such a world, because bloated websites will still require you to download unnecessary amount of data to your machine, and will reduce your computer's ability to multiprocess while also using up your battery life. I stand by what I said before - CDNs are a hack (basically a hand-rolled, rudimentary distributed system) to give you some extra download speed. They don't solve the problems of download size and resource usage.
- ubernostrum 9y ago
  None of this is relevant, though. Simply simulating a globally slowed-down connection will tell you what's going to be a problem. And, really, common sense will tell you what's going to be a problem, too -- if a site "needs" to load 100MB of ads and trackers and other JavaScript to display a couple kilobytes of actual text content, then you have a problem. A user on a slower connection is going to see that site take forever to load even if there's a CDN across the street.

adtac 9y ago

This is probably the easiest problem to solve. Hosting a GoogleSpeedTestBot in every city with the sole purpose of speed testing a page would have laughably negligible impact on Google's resources.

ucaetano 9y ago
You're proposing hosting thousands and thousands of servers across every city in the world where each server would be accessing all the top 1M sites in the world every day and measuring the speed of each page in that site? Seriously?
- adtac 9y ago
  Not every day; at the time of publishing, and maybe few days later. (This could be improved even further: you don't need to access every article. If you could identify a page that's representative of a real article, that could suffice. Maybe have periodic random checks to see if this is still representative.) Not every city; the top 500 cities would probably cover 90+% of the users. You're not understanding the scale of Google. They're incredibly good at exactly this stuff.
  - ucaetano 9y ago
    Oh, I am, but maybe you're not understanding the scale of the internet. Connections, cables, CDNs change on a daily and weekly basis. Is a cable is cut between Jakarta and Singapore, it can take months to fix as Indonesia only allows Indonesian-flagged cable layers to operate in its waters. Individual datacenters, CDNs, cables are going offline constantly, so you'd need to be continuously testing every site to rank it correctly. Not only that, each page in a site might have a very different structure, so you can't just rank a site, you need to rank individual pages and articles too, which means testing every new page and article from every city. I guess you're not getting the scale of that... AMP standardizes the CDN serving part, largely eliminating that issue. AMP articles can't prevent 3rd party CDNs from caching them, which means that they can be served from the closest CDN after the first time they are used.
9y ago

B-Con 9y ago

Counterpoint: You can already simulate the way bad Internet connections work. You don't want that introducing noise on top of the site's metrics. Measure the site itself, then apply a variety of scenarios for what practical fetches look like.

dx034 9y ago

If you have the number of roundtrips and size of documents you can easily calculate page load speed dependent on latency, packet loss rate and connection speed. That allows you to simulate nearly any connection in the world. Which is the reason why Google focuses so much on the number of roundtrips. For most, downloading 50kbyte doesn't cause a lag but 10 roundtrips before the site can be rendered do.