6 ms·

Chrome's own dev tools can emulate slow connections for you. If their browser can do that, why not their crawler?

ucaetano 9y ago
Because the user isn't sitting at a datacenter. An user in Indonesia isn't at the closest datacenter in Singapore. It doesn't matter what you do, you can't, from Singapore, try to emulate what an user on a particular ISP in a particular city in Indonesia will face. In case it isn't clear: Consider 2 websites, both have hosting in Singapore, but one also uses a CDN in Jakarta. The Google crawler running on the Singapore datacenter connects to both websites, speeds are great because they are sitting just next to the datacenter. Both get great ratings. An user is in Jakarta, and tries to connect to both websites. One of them takes a century to load, because it has to load everything from Singapore, and international cables from Singapore to Jakarta are expensive, so connections are capped. Experience sucks. The second website uses a CDN in Jakarta, located at the local coloc facility to which the user's ISP is also connected to. The website loads instantly. CDNs today are located in every single small ISP, in all countries, in all medium-sized cities in the world.
- SheinhardtWigCo 9y ago
  The poster was arguing that negative network conditions can be simulated. If the problem is a lack of information about global network conditions, then you can deploy canary machines around the world to gather that information.
  - ucaetano 9y ago
    They can be simulated, but not predicted. > If the problem is a lack of information about global network conditions, then you can deploy canary machines around the world to gather that information. You'd need to put machines in every single city and every single ISP in the world continuously testing every website in the world. Doesn't make any sense at all. That's not how networks work.
    - TeMPOraL 9y ago
      It's unfeasible, but also completely unnecessary. CDNs can give you some speedup, but they don't magically make a site smaller and less resource-intensive to use.
      - deleted 9y ago
        [deleted]
    - SheinhardtWigCo 9y ago
      OP was talking about a "neutral" speed index. You're talking about solving for every possible combination of user, ISP, and website. There's probably a reasonable middle ground in there somewhere.
      - ucaetano 9y ago
        But that's the point, it doesn't depend on a "neutral" speed index, it varies from website to website, ISP to ISP. If you're in Jakarta trying to access a certain website, if that website is available at the local CDN might make a bigger difference than which ISP or even which plan you're using. You could have a 1Gbps fiber connection from your ISP in Jakarta. If you're trying to access a website in Singapore the your ISP's international link is congested, that 1Gbps will be useless. OTOH, if you have a 10Mbps connection, and you're trying to access a website replicated at an Akamai CDN in your ISP's network, it will be way faster.
        joshmaker 9y ago
        If we accept your argument that Google can't measure the speed of a webpage then wouldn't it also follow that Google can't know if the AMP version of a page is actually faster than the non-AMP version?
        ucaetano 9y ago
        > If we accept your argument that Google can't measure the speed of a webpage Correction: Google can't predict the speed of a random webpage to a random user around the world. But you can reasonably predict that pages hosted at widely available CDNs around the world will load faster on average than pages not available.
- idoh 9y ago
  If a site is running Google Analytics, couldn't that information feed back into page loading speeds?
  - tempestn 9y ago
    You basically have the same problem then, of forcing sites to use google products in order to be ranked highly.
- TeMPOraL 9y ago
  Beyond the fact that the network can be mapped (Google surely does this) and simulated, I feel the more important observation is that this doesn't matter. A good estimate of bloatiness should not require knowledge of network topology. CDNs are a hack that can make websites load a bit faster, but that's about it. It won't make them use less RAM, or consume less of your data cap. Ultimately, performance is about the size of the site, the resources it uses on the client, and the number of requests it makes. All of which is independent of the number and location of CDNs.
  - ucaetano 9y ago
    > A good estimate of bloatiness should not require knowledge of network topology. It does, because the definition of bloatness depends on the network speed. What counts as bloat for an user sitting in Singapore isn't the same as an user sitting in Jakarta. > Beyond the fact that the network can be mapped (Google surely does this) and simulated Network topology is irrelevant in this case. The ISP in Jakarta might have a satellite connection to Singapore, but have a CDN and it will load faster than in another ISP with a 10G international fiber link to Singapore. > All of which is independent of the number and location of CDNs. That's completely wrong. Today, the vast majority of internet traffic is served from CDNs, CDNs (and their placements) are critical for the internet as we know it.
    - TeMPOraL 9y ago
      > It does, because the definition of bloatness depends on the network speed. No, it doesn't. It depends on the proportion of actual content to the unnecessary cruft. The difference is that what a Singaporean considers bloat might prevent someone in Jakarta from accessing the site at all. > That's completely wrong. Today, the vast majority of internet traffic is served from CDNs, CDNs (and their placements) are critical for the internet as we know it. That is if you include streaming video, which is off-topic for this discussion. Imagine a perfect world with the ultimate CDN solution - something like IPFS, which ensures everyone is a CDN to everyone else (instead of wasting bandwidth with multiple downloads of the same content). The concept of bloat wouldn't disappear in such a world, because bloated websites will still require you to download unnecessary amount of data to your machine, and will reduce your computer's ability to multiprocess while also using up your battery life. I stand by what I said before - CDNs are a hack (basically a hand-rolled, rudimentary distributed system) to give you some extra download speed. They don't solve the problems of download size and resource usage.
- ubernostrum 9y ago
  None of this is relevant, though. Simply simulating a globally slowed-down connection will tell you what's going to be a problem. And, really, common sense will tell you what's going to be a problem, too -- if a site "needs" to load 100MB of ads and trackers and other JavaScript to display a couple kilobytes of actual text content, then you have a problem. A user on a slower connection is going to see that site take forever to load even if there's a CDN across the street.

Because the user isn't sitting at a datacenter. An user in Indonesia isn't at the closest datacenter in Singapore. It doesn't matter what you do, you can't, from Singapore, try to emulate what an user on a particular ISP in a particular city in Indonesia will face. In case it isn't clear: Consider 2 websites, both have hosting in Singapore, but one also uses a CDN in Jakarta. The Google crawler running on the Singapore datacenter connects to both websites, speeds are great because they are sitting just next to the datacenter. Both get great ratings. An user is in Jakarta, and tries to connect to both websites. One of them takes a century to load, because it has to load everything from Singapore, and international cables from Singapore to Jakarta are expensive, so connections are capped. Experience sucks. The second website uses a CDN in Jakarta, located at the local coloc facility to which the user's ISP is also connected to. The website loads instantly. CDNs today are located in every single small ISP, in all countries, in all medium-sized cities in the world.

SheinhardtWigCo 9y ago
The poster was arguing that negative network conditions can be simulated. If the problem is a lack of information about global network conditions, then you can deploy canary machines around the world to gather that information.
- ucaetano 9y ago
  They can be simulated, but not predicted. > If the problem is a lack of information about global network conditions, then you can deploy canary machines around the world to gather that information. You'd need to put machines in every single city and every single ISP in the world continuously testing every website in the world. Doesn't make any sense at all. That's not how networks work.
  - TeMPOraL 9y ago
    It's unfeasible, but also completely unnecessary. CDNs can give you some speedup, but they don't magically make a site smaller and less resource-intensive to use.
    - deleted 9y ago
      [deleted]
  - SheinhardtWigCo 9y ago
    OP was talking about a "neutral" speed index. You're talking about solving for every possible combination of user, ISP, and website. There's probably a reasonable middle ground in there somewhere.
    - ucaetano 9y ago
      But that's the point, it doesn't depend on a "neutral" speed index, it varies from website to website, ISP to ISP. If you're in Jakarta trying to access a certain website, if that website is available at the local CDN might make a bigger difference than which ISP or even which plan you're using. You could have a 1Gbps fiber connection from your ISP in Jakarta. If you're trying to access a website in Singapore the your ISP's international link is congested, that 1Gbps will be useless. OTOH, if you have a 10Mbps connection, and you're trying to access a website replicated at an Akamai CDN in your ISP's network, it will be way faster.
      - joshmaker 9y ago
        If we accept your argument that Google can't measure the speed of a webpage then wouldn't it also follow that Google can't know if the AMP version of a page is actually faster than the non-AMP version?
        ucaetano 9y ago
        > If we accept your argument that Google can't measure the speed of a webpage Correction: Google can't predict the speed of a random webpage to a random user around the world. But you can reasonably predict that pages hosted at widely available CDNs around the world will load faster on average than pages not available.
idoh 9y ago
If a site is running Google Analytics, couldn't that information feed back into page loading speeds?
- tempestn 9y ago
  You basically have the same problem then, of forcing sites to use google products in order to be ranked highly.
TeMPOraL 9y ago
Beyond the fact that the network can be mapped (Google surely does this) and simulated, I feel the more important observation is that this doesn't matter. A good estimate of bloatiness should not require knowledge of network topology. CDNs are a hack that can make websites load a bit faster, but that's about it. It won't make them use less RAM, or consume less of your data cap. Ultimately, performance is about the size of the site, the resources it uses on the client, and the number of requests it makes. All of which is independent of the number and location of CDNs.
- ucaetano 9y ago
  > A good estimate of bloatiness should not require knowledge of network topology. It does, because the definition of bloatness depends on the network speed. What counts as bloat for an user sitting in Singapore isn't the same as an user sitting in Jakarta. > Beyond the fact that the network can be mapped (Google surely does this) and simulated Network topology is irrelevant in this case. The ISP in Jakarta might have a satellite connection to Singapore, but have a CDN and it will load faster than in another ISP with a 10G international fiber link to Singapore. > All of which is independent of the number and location of CDNs. That's completely wrong. Today, the vast majority of internet traffic is served from CDNs, CDNs (and their placements) are critical for the internet as we know it.
  - TeMPOraL 9y ago
    > It does, because the definition of bloatness depends on the network speed. No, it doesn't. It depends on the proportion of actual content to the unnecessary cruft. The difference is that what a Singaporean considers bloat might prevent someone in Jakarta from accessing the site at all. > That's completely wrong. Today, the vast majority of internet traffic is served from CDNs, CDNs (and their placements) are critical for the internet as we know it. That is if you include streaming video, which is off-topic for this discussion. Imagine a perfect world with the ultimate CDN solution - something like IPFS, which ensures everyone is a CDN to everyone else (instead of wasting bandwidth with multiple downloads of the same content). The concept of bloat wouldn't disappear in such a world, because bloated websites will still require you to download unnecessary amount of data to your machine, and will reduce your computer's ability to multiprocess while also using up your battery life. I stand by what I said before - CDNs are a hack (basically a hand-rolled, rudimentary distributed system) to give you some extra download speed. They don't solve the problems of download size and resource usage.
ubernostrum 9y ago
None of this is relevant, though. Simply simulating a globally slowed-down connection will tell you what's going to be a problem. And, really, common sense will tell you what's going to be a problem, too -- if a site "needs" to load 100MB of ads and trackers and other JavaScript to display a couple kilobytes of actual text content, then you have a problem. A user on a slower connection is going to see that site take forever to load even if there's a CDN across the street.