7 ms·

Monitor web page changes with Go

93 points by ssimunic 10y ago

nemo1618 10y ago
>we will need to define two global variables Or just pass them to the request functions. And why define the global LOOP_EVERY_SECONDS in terms of seconds, rather than a time.Duration? This code also doesn't handle errors at all. And it can exit any time from within checkChanges. And it compares data by converting the []byte to a string. Use bytes.Equal instead. Finally, wouldn't this be a bit more useful if it printed a diff of the old-vs-new?
- bootload 10y ago
  "wouldn't this be a bit more useful if it printed a diff of the old-vs-new?" I like the idea. If I had to do this quickly, I'd be tempted to use cron, diff and wget as a script. I can't see any handling of HTTP response codes [0] so lots of things could happen that need handling. Does Go have equivalent to the fabulous python ^Requests^ by Kenneth Reitz? [1] Page diffs is a nice feature I might add to a page size tool I hacked up. [2] Reference [0] https://tools.ietf.org/html/rfc2616#section-6 https://tools.ietf.org/html/rfc2616#section-6 [1] https://github.com/kennethreitz/requests https://github.com/kennethreitz/requests [2] https://github.com/peterrenshaw/pageglut https://github.com/peterrenshaw/pageglut
- pimlottc 10y ago
  Handling errors is 90% of the job for a page monitor like this. Have fun getting spammed when the site goes offline briefly, or when your internet connection goes down.
  - ssimunic 10y ago
    Good point. Added error checking.
zlagen 10y ago
This will break in a lot of pages as the html will be different on each request in most pages although the content will be the same. Perhaps it would be better to compare visible text rather than the raw html.
- arjie 10y ago
  Neat. Do you have a recommendation for a headless browser library that would simplify this? The difference between checking the raw HTML and visible HTML can be pretty big.
  - altavox 10y ago
    In client-side Javascript, document.documentElement.innerText yields only the rendered text, without markup, CSS or JavaScript source.
    - arjie 10y ago
      But the theory was that the source would be different, and only the rendered content would be the same at the sample place, right? How would you find the 'same place' without fully rendering the page?
      - tedmiston 10y ago
        One way is for the source to use cache control headers so you can just do a quick HEAD instead of a GET on the whole contents.
  - zlagen 10y ago
    lxml provides a module to perform html diffs mainly looking at the content rather than markup. It's not perfect but most of the times works. http://lxml.de/api/lxml.html.diff-module.html http://lxml.de/api/lxml.html.diff-module.html For Go I don't know any. When I was working on a project like this another issue that I found regularly is that there are regions of the page(counters, advertising, etc.) that change all the time too but they are part of the content. For this I would make several requests to the page and find the regions that would change in every request and disregard them when doing the diff.
Xeoncross 10y ago
When making HTTP requests with Go (or any language) you should at least consider the following: - Setting a User Agent (req.Header.Set(Key, Value)) - Setting Timeouts (https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/ https://blog.cloudflare.com/the-complete-guide-to-golang-net...) - Avoiding ioutil.ReadAll() and checking body length (or using https://golang.org/pkg/io/#LimitReader https://golang.org/pkg/io/#LimitReader) - Caching DNS lookups (https://github.com/viki-org/dnscache https://github.com/viki-org/dnscache)
- tedmiston 10y ago
  + respecting expires or cache-control headers
- thatusernametho 10y ago
  Quick question: Since Go's html parsing requires utf-8 encoding is there a way to convert the response to utf-8 without ioutil.ReadAll()? (Still pretty new to Go).
  - zalmoxes 10y ago
    html.Parse takes a io.Reader https://github.com/golang/net/blob/master/html/parse.go#L2016 https://github.com/golang/net/blob/master/html/parse.go#L201... an http Response.Body is a io.ReadCloser so you can pass it directly with `html.Parse(response.Body)` But don't forget to defer io.Close the body.
    - thatusernametho 10y ago
      I guess I wasn't specific enough. I'd like to stream the response so as to use less memory. I also need to ensure the body is utf-8 encoded because that makes Go happy. https://mionskowski.pl/html-golang-stream-processing https://mionskowski.pl/html-golang-stream-processing
- niftich 10y ago
  If only there were libraries that encapsulate best-practice plumbing and glue logic and offer a high-level API with sane defaults! A perusal of the awesome-go list [1] (allegedly, a "curated list", which may merit a debate) names a few: https://github.com/parnurzeal/gorequest https://github.com/parnurzeal/gorequest https://github.com/smallnest/goreq https://github.com/smallnest/goreq https://github.com/franela/goreq https://github.com/franela/goreq https://github.com/sethgrid/pester https://github.com/sethgrid/pester https://github.com/mozillazg/request https://github.com/mozillazg/request https://github.com/levigross/grequests https://github.com/levigross/grequests https://github.com/go-resty/resty https://github.com/go-resty/resty https://github.com/dghubble/sling https://github.com/dghubble/sling [1] https://github.com/avelino/awesome-go https://github.com/avelino/awesome-go
ssimunic 10y ago
Just a note and reply to the comments made so far: I agree that there are flaws with this as zlagen said (where the page is different in some other parts of HTML). I've made this according to my need, and that was to check a page for only one number change, where everything else wasn't changing. This is meant to demonstrate general idea behind making something like this. nemo1618, I agree that code should handle errors and you should definitely do that. For printing out old vs new version, as I said, for my needs, that wasn't needed since I just wanted to get emailed once one number on page changes.
AdamSC1 10y ago
Does anyone have insights on how to refine this to only monitor specific elements of the page rather than the entire page? I may not care if they change a template but would care if they change a price or a warrant canary etc?
- tedmiston 10y ago
  You could use an XPath or similar to extract the part of the HTML you care about. Similarly with a library like Beautiful Soup you can traverse the HTML a lot like a jQuery selector.
- wlkr 10y ago
  I suspect you'd want to search for a specific attribute or tag combination, I used the scrape package [0] recently and found it a pleasure to use. Beyond the example provided in the README the documentation [1] is also good. [0] https://github.com/yhat/scrape https://github.com/yhat/scrape [1] https://godoc.org/github.com/yhat/scrape https://godoc.org/github.com/yhat/scrape
jasode 10y ago
This is an interesting submission for educational purposes but I want to warn folks who might be inspired to "roll their own" website change monitoring tool. I started writing a similar tool in C# and since I was hoping to create a generalized tool for any website, I had to deal with lots of tricky edge cases. (E.g. raw HTML that changes vs visible text that doesn't, copyright dates on footers that get changed but the real meat of the page is still exactly the same triggering false alarms, etc.) I gave up and simply used Aignes Website-Watcher.[1] If you're still itching to program your own webpage change detector, I recommend reading through their 16 years of release history[2] to get an idea of the various bugs they had to solve. The scope of edge cases can be overwhelming if you want to write something very robust. [1] http://aignes.com/index.htm http://aignes.com/index.htm [2] http://aignes.com/wsw_history.htm http://aignes.com/wsw_history.htm
- Xeoncross 10y ago
  +1 Read the release logs of existing projects. I wish more projects actually took the time to talk about the problems they solved since those issues usually outlive their projects as new alternatives pop up.
dbg31415 10y ago
How would one use this with pages that have dynamic ads on them? Every production site should have content (at least for any page that gets more than 5% total traffic), uptime / load time, application performance, server health, and security monitoring. Can't count the number of times content monitoring bailed me out... a CMS change went live to early, or someone released the wrong branch to prod, or a DNS subscription I didn't control expired... I usually just string-check the Title or H1 on the page but would be great to have more advanced tools.
plg 10y ago
I have always wanted a program/script to check for changes in the NIH era commons site, specifically a page that indicates for a particular grant application, what its score is. Problem is, to get to this magic page (which is, I think, dynamically created), you have to: (1) log into the main era commons site with a username and pw (2) then a new page opens up, on which you have to click on a particular link to get a list of your grants (3) then click on the grant of interest (4) then click on the competition date (5) then the magic page opens up. It's this page I want to check for changes I've done the requisite google searching and found lots of generics suggestions about cookies, etc, but I don't have the skills, obviously. Is there a tool whereby I can click "record" and go through my various steps, and then the tool will keep track of the programmatic aspects of this surf-and-click dance, such that I can execute that script later (e.g. as a cron job)? Thanks!
- irishcoffee 10y ago
  An incredibly crude solution: http://sikulix.com/ http://sikulix.com/
- kej 10y ago
  I like the idea of something that records your actions and creates a script, but for your particular case this would be super easy to throw together with Python and the Requests library. You just create a Session object and tell it to perform the series of posts and gets that take you to the desired page, and then search the text of that page for whatever info you want.
- abhirag 10y ago
  Another option could be to use mechanize (https://pypi.python.org/pypi/mechanize https://pypi.python.org/pypi/mechanize). Originally written in Perl, then ported to Python as well as Ruby.
  - plg 10y ago
    Got it done with Python/Mechanize! Thanks for the suggestion!
- 10y ago
spraak 10y ago
Tangent: why is the Go mascot always so deformed[1]? It makes me sad because I do love Go but having such an awkward mascot makes me feel like the language is deformed, too (which some people would say is true, I admit) [1] Yes, I'm familiar with the history of its creation, I just wish it weren't so
- artpar 10y ago
  Do you judge a product from its logo/mascot ?
  - grzm 10y ago
    And books by their covers! Your parent clearly mentions it's a tangent, and likely not a big deal. I'll go a bit further and admit that the choice of logo or mascot does indicate something about the project, whether it be a sense of style or respect for tradition or history. It's by no means the most important criteria, but it's a datum nonetheless. Language creator hairstyles are also important signals, by the way.
    - spraak 10y ago
      Thanks for understanding. I think that "don't judge a book by its cover" isn't an absolute maxim to live by. In fact, yes, I do judge a book by its cover. Not ENTIRELY by its cover, but as you point out, it's not irrelevant.
  - spraak 10y ago
    I think this phrase is so infected in common speech and mind that we don't question it with books, but how many would disagree that with e.g. SaaS, having a nice looking landing/marketing page is important. Again, there are still successful SaaS apps that have awkward marketing or even UIs, as it's not the only determining factor, but it is indeed important. In this case Go's logo has much less to do with the language than a SaaS app's landing page, but as another comment below shares, it still shows some level of influence in the overall project.
- deleted 10y ago
jhoechtl 10y ago
A cryptographic hash is overkill, murmur or cityhash would be much more appopriate.Furtheron, it might make sense when using one of the mentioned hashes to accept a minimum of change, like a timestamp which has changed on the site , but that varies of course on the use case.

93 points by ssimunic 10y ago

>we will need to define two global variables Or just pass them to the request functions. And why define the global LOOP_EVERY_SECONDS in terms of seconds, rather than a time.Duration? This code also doesn't handle errors at all. And it can exit any time from within checkChanges. And it compares data by converting the []byte to a string. Use bytes.Equal instead. Finally, wouldn't this be a bit more useful if it printed a diff of the old-vs-new?

bootload 10y ago
"wouldn't this be a bit more useful if it printed a diff of the old-vs-new?" I like the idea. If I had to do this quickly, I'd be tempted to use cron, diff and wget as a script. I can't see any handling of HTTP response codes [0] so lots of things could happen that need handling. Does Go have equivalent to the fabulous python ^Requests^ by Kenneth Reitz? [1] Page diffs is a nice feature I might add to a page size tool I hacked up. [2] Reference [0] https://tools.ietf.org/html/rfc2616#section-6 https://tools.ietf.org/html/rfc2616#section-6 [1] https://github.com/kennethreitz/requests https://github.com/kennethreitz/requests [2] https://github.com/peterrenshaw/pageglut https://github.com/peterrenshaw/pageglut
pimlottc 10y ago
Handling errors is 90% of the job for a page monitor like this. Have fun getting spammed when the site goes offline briefly, or when your internet connection goes down.
- ssimunic 10y ago
  Good point. Added error checking.

zlagen 10y ago

This will break in a lot of pages as the html will be different on each request in most pages although the content will be the same. Perhaps it would be better to compare visible text rather than the raw html.

arjie 10y ago
Neat. Do you have a recommendation for a headless browser library that would simplify this? The difference between checking the raw HTML and visible HTML can be pretty big.
- altavox 10y ago
  In client-side Javascript, document.documentElement.innerText yields only the rendered text, without markup, CSS or JavaScript source.
  - arjie 10y ago
    But the theory was that the source would be different, and only the rendered content would be the same at the sample place, right? How would you find the 'same place' without fully rendering the page?
    - tedmiston 10y ago
      One way is for the source to use cache control headers so you can just do a quick HEAD instead of a GET on the whole contents.
- zlagen 10y ago
  lxml provides a module to perform html diffs mainly looking at the content rather than markup. It's not perfect but most of the times works. http://lxml.de/api/lxml.html.diff-module.html http://lxml.de/api/lxml.html.diff-module.html For Go I don't know any. When I was working on a project like this another issue that I found regularly is that there are regions of the page(counters, advertising, etc.) that change all the time too but they are part of the content. For this I would make several requests to the page and find the regions that would change in every request and disregard them when doing the diff.

Xeoncross 10y ago

When making HTTP requests with Go (or any language) you should at least consider the following: - Setting a User Agent (req.Header.Set(Key, Value)) - Setting Timeouts (https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/ https://blog.cloudflare.com/the-complete-guide-to-golang-net...) - Avoiding ioutil.ReadAll() and checking body length (or using https://golang.org/pkg/io/#LimitReader https://golang.org/pkg/io/#LimitReader) - Caching DNS lookups (https://github.com/viki-org/dnscache https://github.com/viki-org/dnscache)

tedmiston 10y ago
+ respecting expires or cache-control headers
thatusernametho 10y ago
Quick question: Since Go's html parsing requires utf-8 encoding is there a way to convert the response to utf-8 without ioutil.ReadAll()? (Still pretty new to Go).
- zalmoxes 10y ago
  html.Parse takes a io.Reader https://github.com/golang/net/blob/master/html/parse.go#L2016 https://github.com/golang/net/blob/master/html/parse.go#L201... an http Response.Body is a io.ReadCloser so you can pass it directly with `html.Parse(response.Body)` But don't forget to defer io.Close the body.
  - thatusernametho 10y ago
    I guess I wasn't specific enough. I'd like to stream the response so as to use less memory. I also need to ensure the body is utf-8 encoded because that makes Go happy. https://mionskowski.pl/html-golang-stream-processing https://mionskowski.pl/html-golang-stream-processing
niftich 10y ago
If only there were libraries that encapsulate best-practice plumbing and glue logic and offer a high-level API with sane defaults! A perusal of the awesome-go list [1] (allegedly, a "curated list", which may merit a debate) names a few: https://github.com/parnurzeal/gorequest https://github.com/parnurzeal/gorequest https://github.com/smallnest/goreq https://github.com/smallnest/goreq https://github.com/franela/goreq https://github.com/franela/goreq https://github.com/sethgrid/pester https://github.com/sethgrid/pester https://github.com/mozillazg/request https://github.com/mozillazg/request https://github.com/levigross/grequests https://github.com/levigross/grequests https://github.com/go-resty/resty https://github.com/go-resty/resty https://github.com/dghubble/sling https://github.com/dghubble/sling [1] https://github.com/avelino/awesome-go https://github.com/avelino/awesome-go

ssimunic 10y ago

Just a note and reply to the comments made so far: I agree that there are flaws with this as zlagen said (where the page is different in some other parts of HTML). I've made this according to my need, and that was to check a page for only one number change, where everything else wasn't changing. This is meant to demonstrate general idea behind making something like this. nemo1618, I agree that code should handle errors and you should definitely do that. For printing out old vs new version, as I said, for my needs, that wasn't needed since I just wanted to get emailed once one number on page changes.

AdamSC1 10y ago

Does anyone have insights on how to refine this to only monitor specific elements of the page rather than the entire page? I may not care if they change a template but would care if they change a price or a warrant canary etc?

tedmiston 10y ago
You could use an XPath or similar to extract the part of the HTML you care about. Similarly with a library like Beautiful Soup you can traverse the HTML a lot like a jQuery selector.
wlkr 10y ago
I suspect you'd want to search for a specific attribute or tag combination, I used the scrape package [0] recently and found it a pleasure to use. Beyond the example provided in the README the documentation [1] is also good. [0] https://github.com/yhat/scrape https://github.com/yhat/scrape [1] https://godoc.org/github.com/yhat/scrape https://godoc.org/github.com/yhat/scrape

jasode 10y ago

This is an interesting submission for educational purposes but I want to warn folks who might be inspired to "roll their own" website change monitoring tool. I started writing a similar tool in C# and since I was hoping to create a generalized tool for any website, I had to deal with lots of tricky edge cases. (E.g. raw HTML that changes vs visible text that doesn't, copyright dates on footers that get changed but the real meat of the page is still exactly the same triggering false alarms, etc.) I gave up and simply used Aignes Website-Watcher.[1] If you're still itching to program your own webpage change detector, I recommend reading through their 16 years of release history[2] to get an idea of the various bugs they had to solve. The scope of edge cases can be overwhelming if you want to write something very robust. [1] http://aignes.com/index.htm http://aignes.com/index.htm [2] http://aignes.com/wsw_history.htm http://aignes.com/wsw_history.htm

Xeoncross 10y ago
+1 Read the release logs of existing projects. I wish more projects actually took the time to talk about the problems they solved since those issues usually outlive their projects as new alternatives pop up.

dbg31415 10y ago

How would one use this with pages that have dynamic ads on them? Every production site should have content (at least for any page that gets more than 5% total traffic), uptime / load time, application performance, server health, and security monitoring. Can't count the number of times content monitoring bailed me out... a CMS change went live to early, or someone released the wrong branch to prod, or a DNS subscription I didn't control expired... I usually just string-check the Title or H1 on the page but would be great to have more advanced tools.

plg 10y ago

I have always wanted a program/script to check for changes in the NIH era commons site, specifically a page that indicates for a particular grant application, what its score is. Problem is, to get to this magic page (which is, I think, dynamically created), you have to: (1) log into the main era commons site with a username and pw (2) then a new page opens up, on which you have to click on a particular link to get a list of your grants (3) then click on the grant of interest (4) then click on the competition date (5) then the magic page opens up. It's this page I want to check for changes I've done the requisite google searching and found lots of generics suggestions about cookies, etc, but I don't have the skills, obviously. Is there a tool whereby I can click "record" and go through my various steps, and then the tool will keep track of the programmatic aspects of this surf-and-click dance, such that I can execute that script later (e.g. as a cron job)? Thanks!

irishcoffee 10y ago
An incredibly crude solution: http://sikulix.com/ http://sikulix.com/
kej 10y ago
I like the idea of something that records your actions and creates a script, but for your particular case this would be super easy to throw together with Python and the Requests library. You just create a Session object and tell it to perform the series of posts and gets that take you to the desired page, and then search the text of that page for whatever info you want.
abhirag 10y ago
Another option could be to use mechanize (https://pypi.python.org/pypi/mechanize https://pypi.python.org/pypi/mechanize). Originally written in Perl, then ported to Python as well as Ruby.
- plg 10y ago
  Got it done with Python/Mechanize! Thanks for the suggestion!
10y ago

spraak 10y ago

Tangent: why is the Go mascot always so deformed[1]? It makes me sad because I do love Go but having such an awkward mascot makes me feel like the language is deformed, too (which some people would say is true, I admit) [1] Yes, I'm familiar with the history of its creation, I just wish it weren't so

artpar 10y ago
Do you judge a product from its logo/mascot ?
- grzm 10y ago
  And books by their covers! Your parent clearly mentions it's a tangent, and likely not a big deal. I'll go a bit further and admit that the choice of logo or mascot does indicate something about the project, whether it be a sense of style or respect for tradition or history. It's by no means the most important criteria, but it's a datum nonetheless. Language creator hairstyles are also important signals, by the way.
  - spraak 10y ago
    Thanks for understanding. I think that "don't judge a book by its cover" isn't an absolute maxim to live by. In fact, yes, I do judge a book by its cover. Not ENTIRELY by its cover, but as you point out, it's not irrelevant.
- spraak 10y ago
  I think this phrase is so infected in common speech and mind that we don't question it with books, but how many would disagree that with e.g. SaaS, having a nice looking landing/marketing page is important. Again, there are still successful SaaS apps that have awkward marketing or even UIs, as it's not the only determining factor, but it is indeed important. In this case Go's logo has much less to do with the language than a SaaS app's landing page, but as another comment below shares, it still shows some level of influence in the overall project.
deleted 10y ago

jhoechtl 10y ago

A cryptographic hash is overkill, murmur or cityhash would be much more appopriate.Furtheron, it might make sense when using one of the mentioned hashes to accept a minimum of change, like a timestamp which has changed on the site , but that varies of course on the use case.