5 ms·

>Now we run automated tests to monitor server uptime and check server for problems every 30 seconds. Unfortunately automated test scripts were happily getting H

>Now we run automated tests to monitor server uptime and check server for problems every 30 seconds. Unfortunately automated test scripts were happily getting HTTP/200 replies while people using the Chrome browser were being told this is a scam business trying to steal their bank account information. I was surprised this wasn't part of the lessons learned. But it seems the monitoring basically failed but that wasn't a lesson. I feel like majority of uptime monitors are falling for this same trap. One of the reasons why for my monitoring service I choose to do full page load monitoring via Chrome instead of just a http request via Curl or whatever. Main reason, people care if the webpage loads or not. People care how long it takes for their webpage to load. Having a website respond in 200ms is great but if it takes 8000ms for all the JS to load and process your website is still slow. I get why sites are just doing curl requests because it's way cheaper but really you're monitoring one part of the stack while really caring about all of it. If your website starts producing javascript errors you want to know, etc. [1] https://www.ootliers.com https://www.ootliers.com (The landing page and everything are terrible and I'm working on improving that)

nchelluri 6y ago
It's funny you read it that way, you may understand it correctly but I came away with a different interpretation, that they allow-listed the developer's IP and returned good non-phishing-warning responses to the monitoring check, but not to end-users.
- that_guy_iain 6y ago
  They said their test scripts worked but people using Chrome got an error. So I take that as in their scripts weren't using Chrome at all. To be fair, I've not had this happen yet so I am going to try and find a site that chrome won't let me visit and see what happens when I visit it programmatically.
  - imwillofficial 6y ago
    When that warning page is thrown, is a 200 returned? It could “load” ok, but be blocked by a flag for chrome that isn’t http flag. Total guess. Anyone have any insight on that page showing up?
    - that_guy_iain 6y ago
      After a bunch of searching I found a test URL. https://testsafebrowsing.appspot.com/s/malware.html https://testsafebrowsing.appspot.com/s/malware.html via my Chrome script I get a 500. And when ignoring it in chrome manually it returns a 200 so the web url works. This is totally an edge case I didn't even think of until I read that blog. Super happy that my monitoring approach picks up on it. For others wanting to do the same I'm using chromedp. It does take up way more resources tho. I worked out I can do 90 per minute per 8-core 16gb server. [1] https://github.com/chromedp/chromedp https://github.com/chromedp/chromedp
klingon79 6y ago
How would Google know that a site is curling another site? Why would they flag that as a phishing site? I’m just having a difficult time determining how this situation is not the fault of the site/app; we don’t even know that any of this is true and it looks more scripted than an offended rant.
- stevarino 6y ago
  The curl'ing is not necessarily bad (and can continue just fine), but rather its blind to these types of problems. A false positive in your testing framework (especially something like this) is the worst case scenario.
- that_guy_iain 6y ago
  Google wouldn't know that someone is curling it which their script said everything was ok. While the website was basically down because Chrome and Firefox will both block a site based on Google's safe browsing list. They could use Google's safe browsing api to check if they're on that list as well as curl.
  - scrollaway 6y ago
    Frankly, Google's Safe Browsing list is one of those things which should be broken out of Google and ran by an independent entity, much like the Let's Encrypt model. I don't know how to help make this happen.
    - sterlind 6y ago
      I'd suggest: 1. Mozilla and the Chrome clones (Edge, Brave, etc.) partner to make an (open as possible) standard for blocking and reviewing, and start maintaining their own list upstream from Google. 2. When Google adds to their blocklist, independently check it according to the consortium's own standards. 3. Cut a deal with Bing or Yandex to scan for malware as part of their crawls, to get technology independence. 4. Put pressure on Google to get onboard. Step 4 is the hardest.
core-utility 6y ago
When talking about checks on the order of twice a minute, curl is probably the right approach. You can/should still do a full check, but that can be done at a lower frequency.
- that_guy_iain 6y ago
  To be fair, my page load checks aren't just is the site up and responding 200 but also page performance anomaly detection and stuff. So I want/need to see the performance minutely to be able to detectly in a quickish amount of time if the performance has degraded for the entire pageload. You could have a minute where it's slow and that is ok but if you have 5 minutes of it being slower than normal one after another then you have an issue. I feel if you're doing checks every few minutes your data won't be as good as doing it every 30 seconds. It is way more resources but honestly, I think it's the future of monitoring. Also, having multiple types of ways of calling a site via http to monitor it is way more complex. The main offer is for order monitoring but I am in the middle of creating a just page load monitoring offer for others since I think that service by itself is super useful.
- zepearl 6y ago
  Not sure why you're getting downvoted, personally I agree. For example: - a frequent/simple check dealing directly (on the internal network) with the webserver ("does it work well yes/no, what's the raw response time, etc..."). Here is where I would definitely use "curl". - another less frequent test involving as well the DNS and the external network. - another end-to-end test (e.g. once every 10 minutes?) involving as well one or more real browsers (this would test as well for example revoked SSL certs). => all these infos/metrics should be quite helpful to identify problems, or at least to shrink the potential area that is causing it.
- brongondwana 6y ago
  Yep, for sure. Fastmail still uses a once-every-10-minutes frequency for the full end-to-end "can log into the website, compose an email, receive the email, trigger an automated background fetch, receive that email too" tests, though the "can connect to service" tests run much more often.
noisy_boy 6y ago
Why not run a fully functional monitoring job i.e. upload file with a monitoring account and check the results to validate that it is working end to end. Doing this even once a minute shouldn't put any load and is a much more reliable test.
- exikyut 6y ago
  It would have needed to be done through Chromium in non-headless (likely full Xvfb) mode, with step screenshots, and screenshot comparison (always flaky!), for the Safe Browsing interstitial to have generated an alert.
  - noisy_boy 6y ago
    How about headless mode with validating response by parsing expected output? I don't have a lot of experience so not sure if what I'm saying is feasible.
    - exikyut 6y ago
      FWIW, the Safe Browsing interstitials are managed by a slightly disorientating bunch of browser/renderer interactions mostly initiated by C++ that basically delivers chrome://interstitials/safebrowsing?type=malware instead of the page in question, IIRC with the renderer setup so you can't right-click and view source etc. From a pure perspective, yes, you should be able to use headless mode and run some sort of validation on the HTML. That would make things really straightforward. In practice... the easiest explanation is that I just spent about 35 minutes trying to find a live test/example safe-browsing trigger so I could check whether right-click is disabled, and I couldn't find one. Even after firing up a separate Chrome profile and verifying Safe Browsing was enabled at chrome://safe-browsing/, modifying /etc/hosts to point http://malware.testing.google.test/testing/malware/ http://malware.testing.google.test/testing/malware/* anywhere did not work, none of the links https://testsafebrowsing.appspot.com/ https://testsafebrowsing.appspot.com/ generate any scary red warnings (and now my Downloads folder is full of EXEs), and I'm kinda sitting dazedly scratching my head a bit. Maybe all this requires some magic sauce I'm not aware of, or maybe it's just quietly bitrotted (doesn't seem to be the case?)... but the overwhelming lack of determinism is what makes trying to automate this next to impossible. You just have to shove the website into a browser and screenshot the output. Because the chain between "load URL" and "user sees red" is that rickety, that there's no clean way to test it. D:
phist_mcgee 6y ago
I wonder if there are page monitoring utilities that can download the full html payload, render the page via a headless chrome browser, and then perform a diff against the content? Most landing pages are quite static, and like you said, it's about full page load, so measuring how long the entire page took to load, along with each asset's network request and subsequent content paints, you could get a pretty good idea of your holistic page load health.
- that_guy_iain 6y ago
  I've seen automated testing tools for that. I was thinking about it for the future once I get other monitoring I want. I'm literally building this to be a monitoring tool I want to use. But you would have to make the montioring system aware of deployments or design changes. But those are the things that generally break things. But the key thing for me is the "goal" montioring. So for ecommerce it's orders but for other systems it's different. That's the thing I really want to monitor. If other things break they'll affect those so you can detect lots of failures. The only issue is finding out what the cause is. But first I'll improve the anomaly detection a bunch before looking into root cause detection.
  - phist_mcgee 6y ago
    It's certainly ok for things to break if it is a deliberate design change. Visual regression testing in your build/deployment pipeline already can flag a change to you as part of a github merge, I don't see why we can't have similar tooling in that pipeline that focuses on efficiency.

nchelluri 6y ago

It's funny you read it that way, you may understand it correctly but I came away with a different interpretation, that they allow-listed the developer's IP and returned good non-phishing-warning responses to the monitoring check, but not to end-users.

that_guy_iain 6y ago
They said their test scripts worked but people using Chrome got an error. So I take that as in their scripts weren't using Chrome at all. To be fair, I've not had this happen yet so I am going to try and find a site that chrome won't let me visit and see what happens when I visit it programmatically.
- imwillofficial 6y ago
  When that warning page is thrown, is a 200 returned? It could “load” ok, but be blocked by a flag for chrome that isn’t http flag. Total guess. Anyone have any insight on that page showing up?
  - that_guy_iain 6y ago
    After a bunch of searching I found a test URL. https://testsafebrowsing.appspot.com/s/malware.html https://testsafebrowsing.appspot.com/s/malware.html via my Chrome script I get a 500. And when ignoring it in chrome manually it returns a 200 so the web url works. This is totally an edge case I didn't even think of until I read that blog. Super happy that my monitoring approach picks up on it. For others wanting to do the same I'm using chromedp. It does take up way more resources tho. I worked out I can do 90 per minute per 8-core 16gb server. [1] https://github.com/chromedp/chromedp https://github.com/chromedp/chromedp

klingon79 6y ago

How would Google know that a site is curling another site? Why would they flag that as a phishing site? I’m just having a difficult time determining how this situation is not the fault of the site/app; we don’t even know that any of this is true and it looks more scripted than an offended rant.

stevarino 6y ago
The curl'ing is not necessarily bad (and can continue just fine), but rather its blind to these types of problems. A false positive in your testing framework (especially something like this) is the worst case scenario.
that_guy_iain 6y ago
Google wouldn't know that someone is curling it which their script said everything was ok. While the website was basically down because Chrome and Firefox will both block a site based on Google's safe browsing list. They could use Google's safe browsing api to check if they're on that list as well as curl.
- scrollaway 6y ago
  Frankly, Google's Safe Browsing list is one of those things which should be broken out of Google and ran by an independent entity, much like the Let's Encrypt model. I don't know how to help make this happen.
  - sterlind 6y ago
    I'd suggest: 1. Mozilla and the Chrome clones (Edge, Brave, etc.) partner to make an (open as possible) standard for blocking and reviewing, and start maintaining their own list upstream from Google. 2. When Google adds to their blocklist, independently check it according to the consortium's own standards. 3. Cut a deal with Bing or Yandex to scan for malware as part of their crawls, to get technology independence. 4. Put pressure on Google to get onboard. Step 4 is the hardest.

core-utility 6y ago

When talking about checks on the order of twice a minute, curl is probably the right approach. You can/should still do a full check, but that can be done at a lower frequency.

that_guy_iain 6y ago
To be fair, my page load checks aren't just is the site up and responding 200 but also page performance anomaly detection and stuff. So I want/need to see the performance minutely to be able to detectly in a quickish amount of time if the performance has degraded for the entire pageload. You could have a minute where it's slow and that is ok but if you have 5 minutes of it being slower than normal one after another then you have an issue. I feel if you're doing checks every few minutes your data won't be as good as doing it every 30 seconds. It is way more resources but honestly, I think it's the future of monitoring. Also, having multiple types of ways of calling a site via http to monitor it is way more complex. The main offer is for order monitoring but I am in the middle of creating a just page load monitoring offer for others since I think that service by itself is super useful.
zepearl 6y ago
Not sure why you're getting downvoted, personally I agree. For example: - a frequent/simple check dealing directly (on the internal network) with the webserver ("does it work well yes/no, what's the raw response time, etc..."). Here is where I would definitely use "curl". - another less frequent test involving as well the DNS and the external network. - another end-to-end test (e.g. once every 10 minutes?) involving as well one or more real browsers (this would test as well for example revoked SSL certs). => all these infos/metrics should be quite helpful to identify problems, or at least to shrink the potential area that is causing it.
brongondwana 6y ago
Yep, for sure. Fastmail still uses a once-every-10-minutes frequency for the full end-to-end "can log into the website, compose an email, receive the email, trigger an automated background fetch, receive that email too" tests, though the "can connect to service" tests run much more often.

noisy_boy 6y ago

Why not run a fully functional monitoring job i.e. upload file with a monitoring account and check the results to validate that it is working end to end. Doing this even once a minute shouldn't put any load and is a much more reliable test.

exikyut 6y ago
It would have needed to be done through Chromium in non-headless (likely full Xvfb) mode, with step screenshots, and screenshot comparison (always flaky!), for the Safe Browsing interstitial to have generated an alert.
- noisy_boy 6y ago
  How about headless mode with validating response by parsing expected output? I don't have a lot of experience so not sure if what I'm saying is feasible.
  - exikyut 6y ago
    FWIW, the Safe Browsing interstitials are managed by a slightly disorientating bunch of browser/renderer interactions mostly initiated by C++ that basically delivers chrome://interstitials/safebrowsing?type=malware instead of the page in question, IIRC with the renderer setup so you can't right-click and view source etc. From a pure perspective, yes, you should be able to use headless mode and run some sort of validation on the HTML. That would make things really straightforward. In practice... the easiest explanation is that I just spent about 35 minutes trying to find a live test/example safe-browsing trigger so I could check whether right-click is disabled, and I couldn't find one. Even after firing up a separate Chrome profile and verifying Safe Browsing was enabled at chrome://safe-browsing/, modifying /etc/hosts to point http://malware.testing.google.test/testing/malware/ http://malware.testing.google.test/testing/malware/* anywhere did not work, none of the links https://testsafebrowsing.appspot.com/ https://testsafebrowsing.appspot.com/ generate any scary red warnings (and now my Downloads folder is full of EXEs), and I'm kinda sitting dazedly scratching my head a bit. Maybe all this requires some magic sauce I'm not aware of, or maybe it's just quietly bitrotted (doesn't seem to be the case?)... but the overwhelming lack of determinism is what makes trying to automate this next to impossible. You just have to shove the website into a browser and screenshot the output. Because the chain between "load URL" and "user sees red" is that rickety, that there's no clean way to test it. D:

phist_mcgee 6y ago

I wonder if there are page monitoring utilities that can download the full html payload, render the page via a headless chrome browser, and then perform a diff against the content? Most landing pages are quite static, and like you said, it's about full page load, so measuring how long the entire page took to load, along with each asset's network request and subsequent content paints, you could get a pretty good idea of your holistic page load health.

that_guy_iain 6y ago
I've seen automated testing tools for that. I was thinking about it for the future once I get other monitoring I want. I'm literally building this to be a monitoring tool I want to use. But you would have to make the montioring system aware of deployments or design changes. But those are the things that generally break things. But the key thing for me is the "goal" montioring. So for ecommerce it's orders but for other systems it's different. That's the thing I really want to monitor. If other things break they'll affect those so you can detect lots of failures. The only issue is finding out what the cause is. But first I'll improve the anomaly detection a bunch before looking into root cause detection.
- phist_mcgee 6y ago
  It's certainly ok for things to break if it is a deliberate design change. Visual regression testing in your build/deployment pipeline already can flag a change to you as part of a github merge, I don't see why we can't have similar tooling in that pipeline that focuses on efficiency.