4 ms·

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them.

by derefr 5y ago

There seem to be a lot of these services ("scrape X in order to generate an RSS/webhook/etc. change feed for X"), but I feel that I can't rely on any of them. For anything that I would need this particular service for, my highest-concern requirement is that I'm able able to "set and forget" the service for literal years at a time. But almost always, services like this just drop dead eventually; or they tweak their scraping algorithms in ways that make previously-working configs break; etc. Is there a change-detection service like this, being provided with stable SLAs by one of the cloud IaaS providers? I suppose I could always find a self-hosted software product for this, and deploy it myself. But though the software would be stable, that's not very "set and forget" either; I'd be on the hook for applying updates, etc. I want something that is to the "HTTP response change detection" abstraction, as S3-alike object-lifecycle event notifications are to the "object-storage change detection" abstraction. Some simple plumbing primitive that does exactly one thing, exactly one (obvious) way, that can be used to build the higher-level abstractions I want, with nine nines of uptime behind it, and the expectation it'll stick around for at least the next decade.

blacksmith_tb 5y ago
So you just need to set up two of these services, and have each monitor the other... Have to agree, it does feel like a case where rolling your own may be less elegant, but at least you'll know how the plumbing works (or if it doesn't).
toomuchtodo 5y ago
Feels like you need something like a non profit providing this service, similar to Let's Encrypt, with enough funds to keep a two person team (bus factor) compensated for maintenance and ongoing support.
hackersnewses 5y ago
> Is there a change-detection service like this, being provided with stable SLAs by one of the cloud IaaS providers? You're looking for Zyte (formerly ScrapingHub).
- CharlesW 5y ago
  > You're looking for Zyte (formerly ScrapingHub). Are you sure? it seems focused on scraping/crawling/data extraction, and I'm not seeing any built-in capabilities to (for example) simply trigger a callback when a resource changes.
  - biggunz 5y ago
    changedetection.io supports filters and triggers which all can result in a notification to a JSON call or anyone of the existing hundreds of other services (email, discord, etc)
iamstupidsimple 5y ago
StackDriver supports health checks: https://cloud.google.com/monitoring/uptime-checks https://cloud.google.com/monitoring/uptime-checks
layer8 5y ago
The linked software is for self-hosting. In my experience self-hosting is the safest bet. When self-hosting it's of course best to use software packages that automatically get security updates from the distribution. On Debian, for example, urlwatch would be an option.
- biggunz 5y ago
  Self-hosting is the only safest bet, or share a real server somewhere with some friends, or however you like
andkon 5y ago
I think you're right that the services themselves break, but more often than not, the failure mode is that what page you're monitoring changes its structure, and then every refresh gives you data that is incommensurate with what came before. It's why we version APIs, but there's no such versioning we expose on websites. I think that also points to why what you propose would be very hard to do: in HTML pages, even if the page structure (a new div level) changes, it's hard to say if that's actually a fundamental change to how they've architected the information hierarchy you're trying to get notified out, or if that is indeed the information itself. It's ambiguous. Maybe that's okay? Like, if all you want is a notification about every time a thing changes, regardless of the meaning of the change, then what you're describing is possible. But I think the meaning of the change is important, and that's ultimately what shifts over time in ways computers cannot grok.
- biggunz 5y ago
  "But I think the meaning of the change is important" this is exactly right, is it a whitespace? is it the text? what is "change" ?
biggunz 5y ago
I've been running this software locally as a docker container on my laptop since about February 2021, obviously with a few updates/restarts, but generally it is set and forget
asdfgeoff 5y ago
It sounds like you value permanence and reliability over feature set. You may want to consider https://www.followthatpage.com/ https://www.followthatpage.com/ I have been using it for years, and—judging from it's visual design—it has been around for years longer.
deleted 5y ago
[deleted]
tedmiston 5y ago
Visualping.io works pretty well as a SaaS. They have an okay free tier (65 checks per month). https://visualping.io/pricing https://visualping.io/pricing I just wish their pricing model was less check-based. I would use it a lot more if I could pay $10–20/month for unlimited checks, but that's not something they offer. Their pricing model scales up really aggressively compared to the underlying resource usage.

by derefr 5y ago

blacksmith_tb 5y ago

So you just need to set up two of these services, and have each monitor the other... Have to agree, it does feel like a case where rolling your own may be less elegant, but at least you'll know how the plumbing works (or if it doesn't).

toomuchtodo 5y ago

Feels like you need something like a non profit providing this service, similar to Let's Encrypt, with enough funds to keep a two person team (bus factor) compensated for maintenance and ongoing support.

hackersnewses 5y ago

> Is there a change-detection service like this, being provided with stable SLAs by one of the cloud IaaS providers? You're looking for Zyte (formerly ScrapingHub).

CharlesW 5y ago
> You're looking for Zyte (formerly ScrapingHub). Are you sure? it seems focused on scraping/crawling/data extraction, and I'm not seeing any built-in capabilities to (for example) simply trigger a callback when a resource changes.
- biggunz 5y ago
  changedetection.io supports filters and triggers which all can result in a notification to a JSON call or anyone of the existing hundreds of other services (email, discord, etc)

iamstupidsimple 5y ago

StackDriver supports health checks: https://cloud.google.com/monitoring/uptime-checks https://cloud.google.com/monitoring/uptime-checks

layer8 5y ago

The linked software is for self-hosting. In my experience self-hosting is the safest bet. When self-hosting it's of course best to use software packages that automatically get security updates from the distribution. On Debian, for example, urlwatch would be an option.

biggunz 5y ago
Self-hosting is the only safest bet, or share a real server somewhere with some friends, or however you like

andkon 5y ago

I think you're right that the services themselves break, but more often than not, the failure mode is that what page you're monitoring changes its structure, and then every refresh gives you data that is incommensurate with what came before. It's why we version APIs, but there's no such versioning we expose on websites. I think that also points to why what you propose would be very hard to do: in HTML pages, even if the page structure (a new div level) changes, it's hard to say if that's actually a fundamental change to how they've architected the information hierarchy you're trying to get notified out, or if that is indeed the information itself. It's ambiguous. Maybe that's okay? Like, if all you want is a notification about every time a thing changes, regardless of the meaning of the change, then what you're describing is possible. But I think the meaning of the change is important, and that's ultimately what shifts over time in ways computers cannot grok.

biggunz 5y ago
"But I think the meaning of the change is important" this is exactly right, is it a whitespace? is it the text? what is "change" ?

biggunz 5y ago

I've been running this software locally as a docker container on my laptop since about February 2021, obviously with a few updates/restarts, but generally it is set and forget

asdfgeoff 5y ago

It sounds like you value permanence and reliability over feature set. You may want to consider https://www.followthatpage.com/ https://www.followthatpage.com/ I have been using it for years, and—judging from it's visual design—it has been around for years longer.

deleted 5y ago

[deleted]

tedmiston 5y ago

Visualping.io works pretty well as a SaaS. They have an okay free tier (65 checks per month). https://visualping.io/pricing https://visualping.io/pricing I just wish their pricing model was less check-based. I would use it a lot more if I could pay $10–20/month for unlimited checks, but that's not something they offer. Their pricing model scales up really aggressively compared to the underlying resource usage.