3 ms·
This will break in a lot of pages as the html will be different on each request in most pages although the content will be the same. Perhaps it would be better
by zlagen 10y ago
This will break in a lot of pages as the html will be different on each request in most pages although the content will be the same. Perhaps it would be better to compare visible text rather than the raw html.
- arjie 10y agoNeat. Do you have a recommendation for a headless browser library that would simplify this? The difference between checking the raw HTML and visible HTML can be pretty big.
- altavox 10y agoIn client-side Javascript, document.documentElement.innerText yields only the rendered text, without markup, CSS or JavaScript source.
- arjie 10y agoBut the theory was that the source would be different, and only the rendered content would be the same at the sample place, right? How would you find the 'same place' without fully rendering the page?
- tedmiston 10y agoOne way is for the source to use cache control headers so you can just do a quick HEAD instead of a GET on the whole contents.
- zlagen 10y agolxml provides a module to perform html diffs mainly looking at the content rather than markup. It's not perfect but most of the times works. http://lxml.de/api/lxml.html.diff-module.html http://lxml.de/api/lxml.html.diff-module.html For Go I don't know any. When I was working on a project like this another issue that I found regularly is that there are regions of the page(counters, advertising, etc.) that change all the time too but they are part of the content. For this I would make several requests to the page and find the regions that would change in every request and disregard them when doing the diff.