6 ms·

Show HN: Changedetection.io detect changes in websites and JSON feeds

147 points by biggunz 5y ago

sharmin123 5y ago
[dead]
eterps 5y ago
Nice! Does it have an API for adding URLs to monitor? Would be nice to have a browser extension to do that easily.
- biggunz 5y ago
  Yeah, coming in the next week or so actually, or feel free to open a PR! also adding an API to retrieve snapshots (text/JSON) too
leroman 5y ago
Looks nice! Good luck. Were in a similar space, coming out with a SaaS in a few weeks.
derefr 5y ago
There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their scraping algorithms in ways that make previously-working configs break; etc. Is there a change-detection service like this, being provided with stable SLAs by one of the cloud IaaS providers? I suppose I could always find a self-hosted software product for this, and deploy it myself. But though the software would be stable, that's not very "set and forget" either; I'd be on the hook for applying updates, etc. I want something that is to the "HTTP response change detection" abstraction, as S3-alike object-lifecycle event notifications are to the "object-storage change detection" abstraction. Some simple plumbing primitive that does exactly one thing, exactly one (obvious) way, that can be used to build the higher-level abstractions I want, with nine nines of uptime behind it, and the expectation it'll stick around for at least the next decade.
- blacksmith_tb 5y ago
  So you just need to set up two of these services, and have each monitor the other... Have to agree, it does feel like a case where rolling your own may be less elegant, but at least you'll know how the plumbing works (or if it doesn't).
- toomuchtodo 5y ago
  Feels like you need something like a non profit providing this service, similar to Let's Encrypt, with enough funds to keep a two person team (bus factor) compensated for maintenance and ongoing support.
- hackersnewses 5y ago
  > Is there a change-detection service like this, being provided with stable SLAs by one of the cloud IaaS providers? You're looking for Zyte (formerly ScrapingHub).
  - CharlesW 5y ago
    > You're looking for Zyte (formerly ScrapingHub). Are you sure? it seems focused on scraping/crawling/data extraction, and I'm not seeing any built-in capabilities to (for example) simply trigger a callback when a resource changes.
    - biggunz 5y ago
      changedetection.io supports filters and triggers which all can result in a notification to a JSON call or anyone of the existing hundreds of other services (email, discord, etc)
- iamstupidsimple 5y ago
  StackDriver supports health checks: https://cloud.google.com/monitoring/uptime-checks https://cloud.google.com/monitoring/uptime-checks
- layer8 5y ago
  The linked software is for self-hosting. In my experience self-hosting is the safest bet. When self-hosting it's of course best to use software packages that automatically get security updates from the distribution. On Debian, for example, urlwatch would be an option.
olex 5y ago
Nice and easy to self-host with Docker, thanks. Might save me more than a few bookmarks that I check every couple weeks manually. Small feature wish: optionally fetch and display favicons next to the URLs or <title>s in the website list.
- biggunz 5y ago
  Great idea!
deleted 5y ago
[deleted]
TheKnack 5y ago
I used this to replace Visualping.io. The only features it lacks compared to VisualPing is a GUI for selecting page elements to monitor and fully rendered diff results. It either monitors the whole page or you can add a CSS selector manually.
- biggunz 5y ago
  It accepts CSS filters - click on EDIT, Then "Filters & Triggers" and there you can fill in a CSS filter, more stuff coming soon :)
s-xyz 5y ago
How did you create the screenshot difference? https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/screenshot-diff.png https://raw.githubusercontent.com/dgtlmoon/changedetection.i...
- biggunz 5y ago
  Just wait for a change to be detected and then clicked "DIFF" button in the main list, easy.. the screenshot is just from my browser of my local installation of the software
visarga 5y ago
This would fit well as a browser extension - generate your own feeds with or without RSS. Coupled with a simple topic classifier with user defined topics it would bring some order to the chaos.
pabs3 5y ago
I'm using urlwatch for this: https://thp.io/2008/urlwatch/ https://thp.io/2008/urlwatch/
ianschmitz 5y ago
I tried this project recently, but it didn’t seem to handle pages that load content dynamically via XHR. At least the couple pages I tried at the time.
- biggunz 5y ago
  https://github.com/dgtlmoon/changedetection.io/wiki/Fetching-pages-with-WebDriver https://github.com/dgtlmoon/changedetection.io/wiki/Fetching...
  - ianschmitz 5y ago
    Indeed I had it setup using a separate selenium chrome container. Still had the same issue
jimmydorry 5y ago
Nice little project. I played around with it for a bit, but couldn't figure out how to extract snippets (e.g. a list of headlines, like from the hackernews home page). It seems to be good at grabbing a dumb dump of text within a given CSS selector, but no regex pattern matching within a line (the regex fields appear to toss out all lines that match the pattern).

147 points by biggunz 5y ago

sharmin123 5y ago

[dead]

eterps 5y ago

Nice! Does it have an API for adding URLs to monitor? Would be nice to have a browser extension to do that easily.

biggunz 5y ago
Yeah, coming in the next week or so actually, or feel free to open a PR! also adding an API to retrieve snapshots (text/JSON) too

leroman 5y ago

Looks nice! Good luck. Were in a similar space, coming out with a SaaS in a few weeks.

derefr 5y ago

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their scraping algorithms in ways that make previously-working configs break; etc. Is there a change-detection service like this, being provided with stable SLAs by one of the cloud IaaS providers? I suppose I could always find a self-hosted software product for this, and deploy it myself. But though the software would be stable, that's not very "set and forget" either; I'd be on the hook for applying updates, etc. I want something that is to the "HTTP response change detection" abstraction, as S3-alike object-lifecycle event notifications are to the "object-storage change detection" abstraction. Some simple plumbing primitive that does exactly one thing, exactly one (obvious) way, that can be used to build the higher-level abstractions I want, with nine nines of uptime behind it, and the expectation it'll stick around for at least the next decade.

blacksmith_tb 5y ago
So you just need to set up two of these services, and have each monitor the other... Have to agree, it does feel like a case where rolling your own may be less elegant, but at least you'll know how the plumbing works (or if it doesn't).
toomuchtodo 5y ago
Feels like you need something like a non profit providing this service, similar to Let's Encrypt, with enough funds to keep a two person team (bus factor) compensated for maintenance and ongoing support.
hackersnewses 5y ago
> Is there a change-detection service like this, being provided with stable SLAs by one of the cloud IaaS providers? You're looking for Zyte (formerly ScrapingHub).
- CharlesW 5y ago
  > You're looking for Zyte (formerly ScrapingHub). Are you sure? it seems focused on scraping/crawling/data extraction, and I'm not seeing any built-in capabilities to (for example) simply trigger a callback when a resource changes.
  - biggunz 5y ago
    changedetection.io supports filters and triggers which all can result in a notification to a JSON call or anyone of the existing hundreds of other services (email, discord, etc)
iamstupidsimple 5y ago
StackDriver supports health checks: https://cloud.google.com/monitoring/uptime-checks https://cloud.google.com/monitoring/uptime-checks
layer8 5y ago
The linked software is for self-hosting. In my experience self-hosting is the safest bet. When self-hosting it's of course best to use software packages that automatically get security updates from the distribution. On Debian, for example, urlwatch would be an option.

olex 5y ago

Nice and easy to self-host with Docker, thanks. Might save me more than a few bookmarks that I check every couple weeks manually. Small feature wish: optionally fetch and display favicons next to the URLs or <title>s in the website list.

biggunz 5y ago
Great idea!

deleted 5y ago

[deleted]

TheKnack 5y ago

I used this to replace Visualping.io. The only features it lacks compared to VisualPing is a GUI for selecting page elements to monitor and fully rendered diff results. It either monitors the whole page or you can add a CSS selector manually.

biggunz 5y ago
It accepts CSS filters - click on EDIT, Then "Filters & Triggers" and there you can fill in a CSS filter, more stuff coming soon :)

s-xyz 5y ago

How did you create the screenshot difference? https://raw.githubusercontent.com/dgtlmoon/changedetection.io/master/screenshot-diff.png https://raw.githubusercontent.com/dgtlmoon/changedetection.i...

biggunz 5y ago
Just wait for a change to be detected and then clicked "DIFF" button in the main list, easy.. the screenshot is just from my browser of my local installation of the software

visarga 5y ago

This would fit well as a browser extension - generate your own feeds with or without RSS. Coupled with a simple topic classifier with user defined topics it would bring some order to the chaos.

pabs3 5y ago

I'm using urlwatch for this: https://thp.io/2008/urlwatch/ https://thp.io/2008/urlwatch/

ianschmitz 5y ago

I tried this project recently, but it didn’t seem to handle pages that load content dynamically via XHR. At least the couple pages I tried at the time.

biggunz 5y ago
https://github.com/dgtlmoon/changedetection.io/wiki/Fetching-pages-with-WebDriver https://github.com/dgtlmoon/changedetection.io/wiki/Fetching...
- ianschmitz 5y ago
  Indeed I had it setup using a separate selenium chrome container. Still had the same issue

jimmydorry 5y ago

Nice little project. I played around with it for a bit, but couldn't figure out how to extract snippets (e.g. a list of headlines, like from the hackernews home page). It seems to be good at grabbing a dumb dump of text within a given CSS selector, but no regex pattern matching within a line (the regex fields appear to toss out all lines that match the pattern).