5 ms·

Response time data is useless without other data: - Where are the monitors located? - Where are the sites located? - How was the site monitored? Did they acc

Response time data is useless without other data: - Where are the monitors located? - Where are the sites located? - How was the site monitored? Did they account for DNS lookup and connection times in this time? 8ms for skysheet.com makes me think no? - Are error times rolled into these times? Errors typically will respond quickly. Having built www.yottaa.com, we worked hard to provide data that users can easily take actions against. In order to that understanding the questions above is important.

delano 16y ago
This is an early version of the site so there's still a lot of work to do to explain this stuff on the site itself. To answer your questions: - All monitoring is from EC2's us-east-1 region. - I don't track the location of the sites, other than specifying the vendor. - Data is generated by an open source tool called Stella. I'm not currently tracking DNS lookups but probably will eventually. - Tests with errors are not included in the response times. The specific response time is less interesting than looking at how a site performs over time (the number and duration of downtime, errors, etc). There are a lot of rabbit holes in testing so what I'm focusing on now is the simplest possible solution that works.
- emmett 16y ago
  So basically, anyone hosting their site on EC2 will appear to have a fantastic response time, compared to anyone else. Which makes the test mostly useless for comparison between startups (though it could be interesting/valid data for a single startup over time, which is what pingdom sells)
  - delano 16y ago
    I'm just one guy so I have to start somewhere :] Sometime soon though, it will be possible to select the location.
    - mayank 16y ago
      Unfortunately, I think the parent commenter is right -- your numbers are (currently) meaningless. Statistically, your sample size for estimating response time is 1 for each of those sites. You need at least a dozen or so monitor sites, preferably distributed in the same way that Internet users are, for anything meaningful information to be gained. Ideally, you'd also want to distribute your monitors taking peering agreements of the ISP, etc. into account for a truly representative picture of response times. As it stands, you're doing a disservice to startups that aren't on EC2, for no good reason. You solution "works" on a technical level, but the data is statistically meaningless.
      - delano 16y ago
        Combining monitoring data from multiple locations would give a more accurate response time overall. But as a site owner, that number is not very useful because it's not actionable. For example, if response times to my site become slow from say, Dallas or Ireland, there's is nothing I can do about it other than be aware that it's happening. Monitoring from a single location is interesting because it gives a metric for how well my application is performing.
        mayank 16y ago
        Certainly, but I would say that there are very few Internet startups that cater to a single (or even a few) geographical locations. Your numbers currently indicate quality of service for customers in the same hosting block as Amazon EC2. If you have enough monitors and spread them out well (which shouldn't be too hard, since you could just sign up for Perl/PHP enabled hosting accounts in different locations), you can still diagnose where the slow geographical regions are. In short, having more data can only give you a more accurate picture, depending on how you analyze it.
    - IgorPartola 16y ago
      I know how you feel. I built http://www.pingbrigade.com/ http://www.pingbrigade.com/ which is similar. Mine runs of VPS's though.
    - 16y ago
steadicat 16y ago
You forget HTTPS. The SSL handshake adds a significant delay to the connection time.

delano 16y ago

This is an early version of the site so there's still a lot of work to do to explain this stuff on the site itself. To answer your questions: - All monitoring is from EC2's us-east-1 region. - I don't track the location of the sites, other than specifying the vendor. - Data is generated by an open source tool called Stella. I'm not currently tracking DNS lookups but probably will eventually. - Tests with errors are not included in the response times. The specific response time is less interesting than looking at how a site performs over time (the number and duration of downtime, errors, etc). There are a lot of rabbit holes in testing so what I'm focusing on now is the simplest possible solution that works.

emmett 16y ago
So basically, anyone hosting their site on EC2 will appear to have a fantastic response time, compared to anyone else. Which makes the test mostly useless for comparison between startups (though it could be interesting/valid data for a single startup over time, which is what pingdom sells)
- delano 16y ago
  I'm just one guy so I have to start somewhere :] Sometime soon though, it will be possible to select the location.
  - mayank 16y ago
    Unfortunately, I think the parent commenter is right -- your numbers are (currently) meaningless. Statistically, your sample size for estimating response time is 1 for each of those sites. You need at least a dozen or so monitor sites, preferably distributed in the same way that Internet users are, for anything meaningful information to be gained. Ideally, you'd also want to distribute your monitors taking peering agreements of the ISP, etc. into account for a truly representative picture of response times. As it stands, you're doing a disservice to startups that aren't on EC2, for no good reason. You solution "works" on a technical level, but the data is statistically meaningless.
    - delano 16y ago
      Combining monitoring data from multiple locations would give a more accurate response time overall. But as a site owner, that number is not very useful because it's not actionable. For example, if response times to my site become slow from say, Dallas or Ireland, there's is nothing I can do about it other than be aware that it's happening. Monitoring from a single location is interesting because it gives a metric for how well my application is performing.
      - mayank 16y ago
        Certainly, but I would say that there are very few Internet startups that cater to a single (or even a few) geographical locations. Your numbers currently indicate quality of service for customers in the same hosting block as Amazon EC2. If you have enough monitors and spread them out well (which shouldn't be too hard, since you could just sign up for Perl/PHP enabled hosting accounts in different locations), you can still diagnose where the slow geographical regions are. In short, having more data can only give you a more accurate picture, depending on how you analyze it.
  - IgorPartola 16y ago
    I know how you feel. I built http://www.pingbrigade.com/ http://www.pingbrigade.com/ which is similar. Mine runs of VPS's though.
  - 16y ago

steadicat 16y ago

You forget HTTPS. The SSL handshake adds a significant delay to the connection time.