6 ms·

ChangeDetection, monitor any website change

253 points by serhack_ 3y ago

hdivider 3y ago
Good stuff. Niche idea for ya: consider government contractors. Gov sites tend to be atrocious. Manual change tracking can mean you find a $1M RFP weeks earlier, which increases chance of winning. (Consider mda.mil. There's no email signup. What they consider a 'newsletter' is in fact a list of PDFs hosted on their site.) And that's federal gov contracting. State/local is even more of a Wild West. So much so that many, maany companies just never have the bandwidth to even find RFPs and navigate those ancient sites.
- solardev 3y ago
  Funny how much better government might get if we just let their employees post on Craigslist
- MoOmer 3y ago
  While that may be true if contractors/competitors for contracts were generally unaware of the contract until posting, that’s usually not the reality. A lot of government contracting advice from experienced contractors is that if you’re discovering it via a public source (i.e. the website) you’re too late.
  - hdivider 3y ago
    This I am afraid is not true either, though. I know many dual-use companies who discovered an RFP (not just on SAM), late -- i.e. with a proposal deathmarch -- and got through.
    - MoOmer 3y ago
      My understanding is that those scenarios are the exception, and finding obscure contracts that have gone undiscovered isn’t really an untapped fount of free, missed, opportunity. That said, aggregators like bonfire etc. can get you pretty far if that’s your strategy!
      - hdivider 3y ago
        Yeah it's probably true that such things are the exception. But for a startup, a chance at e.g. a $256k SBIR Phase I and then $1.9M Phase II and the modern follow-on options in e.g. the Army (called the CATALYST program and other things) -- this one exception turns the company into something real. So because the amounts are large (compared to commercial, in terms of when a customer first buys from a small company), those cases really make a huge difference.
        MoOmer 3y ago
        Oh yeah, grants are great, but a completely different beast!
        hdivider 3y ago
        I have to counter yet again, apologies. :) SBIRs yes are considered grants, but I've seen the same phenomenon for e.g. CRADAs and OTAs. Functionally this is all similar: we find them often on bad gov websites, and an automated tool to track changes can greatly help many companies to win them.
  - chatmasta 3y ago
    How can you be "too late?" Isn't the point of RFP postings that they're the first step in a process of soliciting proposals (bids) for the requested project? They're certainly not supposed to be granted on any kind of first-come, first-served basis... combatting that sort of corrupt insider dealing is the whole reason there are legal requirements to post them in the first place. So while I don't doubt the premise of your cynical truism that "if you're discovering it via a public source you're too late," I hope that this is a separate issue from the RFP process itself, because any project where that's the case must be a project where the RFP was meaningless in the first place. And that behavior is also explicitly illegal, even if it's unfortunately prevalent and infrequently prosecuted.
serhack_ 3y ago
I'm OP. I'm not the developer of the project, but I think it deserved a post on HN because it's pretty an useful tool to track website changes. Setup can be done in a few minutes and the lead developer is always looking for solid feedbacks.
- TekMol 3y ago
  It cannot be a Show HN then: https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html Please change the title.
  - serhack_ 3y ago
    Uops, sorry
kierenj 3y ago
First time I've noticed a GH repo having so much SEO in the About section! Not sure how I feel about it..
- serhack_ 3y ago
  I actually found it via the about section so I guess it helped haha
- realPubkey 3y ago
  They should add an url into the description text. This will render as backlink any there are many pages out there which replicate github-topic pages which will then give your site many links for free.
  - metadat 3y ago
    IIRC, GitHub adds the nofollow attribute to all user-supplied links links.
- Lalabadie 3y ago
  Right? First time I run into a README file on Github and I think "Oh, this text is for search engines, not readers".
- mnstngr 3y ago
  It's actually refreshing to see end-user documentation and answering the basic questions of “what can this do” and “why should I care” in the readme. Most projects jump straight into how to build it locally, with not even a screenshot of what they’ve created.
champagnepapi 3y ago
How does this handle A/B tests that sites are running?
- masukomi 3y ago
  wouldn't that depend entirely on which group its request got put in, which would be something it would have no control over?
Aulig 3y ago
I've been self-hosting this for a year or so and it's pretty neat. It happens quite often that I need to wait for something which doesn't have a builtin alert. Currently I'm for example using it to get an alert when a flutter package has been updated on pub.dev
wkat4242 3y ago
Interesting! I've been making some scripts privately for specific tasks, like an Nvidia video card I wanted to buy. However many sites are really hostile to scraping these days. I'll give it a try,.
- weinzierl 3y ago
  camelcamelcamel is a really old service, but it's still around and still works well for product price changes.
  - wkat4242 3y ago
    Yes but they collaborate with Amazon. For example during the pandemic they turned it all off for a few months because Amazon wanted it. They're still around because Amazon lets them be around.
    - somsak2 3y ago
      source?
      - wkat4242 3y ago
        https://camelcamelcamel.com/blog/amazon-eu-covid-19/ https://camelcamelcamel.com/blog/amazon-eu-covid-19/
        weinzierl 3y ago
        This is interesting. I did not log into it for ages but sure still get the alerts for my existing trackers. That's why I assumed that they are still around.
Mo3 3y ago
Kudos, this is a very feature rich solution. I like it. 8.99 seems a little bit much, but I guess it's probably the sweet spot on the curve.
- serhack_ 3y ago
  You can install it for free on your machine. I think it's a decent price, considering that they offer you to check a website even with a ratio of seconds.
- johnnyworker 3y ago
  For Windows, there is the OG https://www.aignes.com/ https://www.aignes.com/
abbaselmas 3y ago
my favorite one is visualping.io
_flux 3y ago
I've used https://urlwatch.readthedocs.io https://urlwatch.readthedocs.io but this definitely seems much easier to use—though possibly not quite as powerful regarding page filtering. But at least ChangeDetection supports jq which is already quite a nice feature in that department.
shortrounddev2 3y ago
A few years ago when I was a Junior engineer, I worked for a company which would provide us with free tickets to any sports team in our city. I'm not a sports fan, but my stepdad loves hockey. When the tickets went up, they got reserved pretty fast. I noticed that the site we used to reserve the tickets had a predictable slug like: `{company}-hockey-tickets-2016-season`, and `{company}-hockey-tickets-2015-season`. The sites also didn't change between each refresh. So, when it was nearly time for the 2017 season to start, I wrote a script which hashed a GET request of the site at `{company}-hockey-tickets-2017-season`. If the hash changed, it would send me a text message using twilio, and I would know to immediately get on a machine and reserve the tickets we wanted. After a couple false positives, the page eventually went up before it was announced and I reserved the tickets I wanted. Unfortunately, the office manager who did these things told me I couldn't jump the gun and un-reserved my tickets. After the official announcement went up, the spots I had originally reserved quickly went to someone else
- HardlyCurious 3y ago
  Yeah, that office manager probably knew who would be getting the tickets in advance.
  - alach11 3y ago
    Working hard: writing a script to detect when the website goes live. Working smart: bring the office manager a pastry, ask them about their day, and drop hints about how excited you are for the tickets this year.
    - nkko 3y ago
      Engineer vs Non-Engineer approach.
      - deedree 3y ago
        You mean social engineering.
    - 3y ago
accidbuddy 3y ago
Anyone has been utilized this tool to monitoring Amazon Prices (with login)?
- stewx 3y ago
  Use CamelCamelCamel
  - accidbuddy 3y ago
    As far as I know, doesn't work to Amazon.br (Brazil). Thanks by the reply.
    - robk 3y ago
      Have you tried keepa?
cloudyporpoise 3y ago
Curious how this handles sites behind CloudFlare using bot detection.
- _lvbh 3y ago
  I see selenium and playwright in the dependencies. I would’ve just gone with tls-client
- _boffin_ 3y ago
  It’s not too difficult to get past cloudflare using puppeteer. Being normal is the key.
  - mcstempel 3y ago
    Yeah, +1. Even vanilla puppeteer is pretty successful against Cloudflare
sccxy 3y ago
I have used it for some time now. It has been very useful for me. Some use cases for me: * price changes * calendar changes for sport events * document changes for local gov * new firmware releases * terms of conditions changes
ephbit 3y ago
Here's an addon for Firefox that also checks websites in regular intervals to detect changes: https://github.com/WaldiPL/webpageScanner https://github.com/WaldiPL/webpageScanner Not affiliated with it and I don't know if it still works, haven't used it lately. When I used it, it worked to my satisfaction.
mchenier 3y ago
https://visualping.io https://visualping.io does a nice similar job and is free for moderate personal use.
- davidinosauro 3y ago
  On the theme of alternatives, I've been using https://www.followthatpage.com/ https://www.followthatpage.com/ on and off for years. In particular for my partner's job hunt, watching some local trade association billboards or single school's `/jobs.html` pages. Also has a reasonable free plan for personal use (20 pages daily + 20 pages weekly + 1 page hourly). Definitely much simpler in terms of diffing (only text), but it has this 2000s vibe.

253 points by serhack_ 3y ago

Good stuff. Niche idea for ya: consider government contractors. Gov sites tend to be atrocious. Manual change tracking can mean you find a $1M RFP weeks earlier, which increases chance of winning. (Consider mda.mil. There's no email signup. What they consider a 'newsletter' is in fact a list of PDFs hosted on their site.) And that's federal gov contracting. State/local is even more of a Wild West. So much so that many, maany companies just never have the bandwidth to even find RFPs and navigate those ancient sites.

solardev 3y ago
Funny how much better government might get if we just let their employees post on Craigslist
MoOmer 3y ago
While that may be true if contractors/competitors for contracts were generally unaware of the contract until posting, that’s usually not the reality. A lot of government contracting advice from experienced contractors is that if you’re discovering it via a public source (i.e. the website) you’re too late.
- hdivider 3y ago
  This I am afraid is not true either, though. I know many dual-use companies who discovered an RFP (not just on SAM), late -- i.e. with a proposal deathmarch -- and got through.
  - MoOmer 3y ago
    My understanding is that those scenarios are the exception, and finding obscure contracts that have gone undiscovered isn’t really an untapped fount of free, missed, opportunity. That said, aggregators like bonfire etc. can get you pretty far if that’s your strategy!
    - hdivider 3y ago
      Yeah it's probably true that such things are the exception. But for a startup, a chance at e.g. a $256k SBIR Phase I and then $1.9M Phase II and the modern follow-on options in e.g. the Army (called the CATALYST program and other things) -- this one exception turns the company into something real. So because the amounts are large (compared to commercial, in terms of when a customer first buys from a small company), those cases really make a huge difference.
      - MoOmer 3y ago
        Oh yeah, grants are great, but a completely different beast!
        hdivider 3y ago
        I have to counter yet again, apologies. :) SBIRs yes are considered grants, but I've seen the same phenomenon for e.g. CRADAs and OTAs. Functionally this is all similar: we find them often on bad gov websites, and an automated tool to track changes can greatly help many companies to win them.
- chatmasta 3y ago
  How can you be "too late?" Isn't the point of RFP postings that they're the first step in a process of soliciting proposals (bids) for the requested project? They're certainly not supposed to be granted on any kind of first-come, first-served basis... combatting that sort of corrupt insider dealing is the whole reason there are legal requirements to post them in the first place. So while I don't doubt the premise of your cynical truism that "if you're discovering it via a public source you're too late," I hope that this is a separate issue from the RFP process itself, because any project where that's the case must be a project where the RFP was meaningless in the first place. And that behavior is also explicitly illegal, even if it's unfortunately prevalent and infrequently prosecuted.

serhack_ 3y ago

I'm OP. I'm not the developer of the project, but I think it deserved a post on HN because it's pretty an useful tool to track website changes. Setup can be done in a few minutes and the lead developer is always looking for solid feedbacks.

TekMol 3y ago
It cannot be a Show HN then: https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html Please change the title.
- serhack_ 3y ago
  Uops, sorry

kierenj 3y ago

First time I've noticed a GH repo having so much SEO in the About section! Not sure how I feel about it..

serhack_ 3y ago
I actually found it via the about section so I guess it helped haha
realPubkey 3y ago
They should add an url into the description text. This will render as backlink any there are many pages out there which replicate github-topic pages which will then give your site many links for free.
- metadat 3y ago
  IIRC, GitHub adds the nofollow attribute to all user-supplied links links.
Lalabadie 3y ago
Right? First time I run into a README file on Github and I think "Oh, this text is for search engines, not readers".
mnstngr 3y ago
It's actually refreshing to see end-user documentation and answering the basic questions of “what can this do” and “why should I care” in the readme. Most projects jump straight into how to build it locally, with not even a screenshot of what they’ve created.

champagnepapi 3y ago

How does this handle A/B tests that sites are running?

masukomi 3y ago
wouldn't that depend entirely on which group its request got put in, which would be something it would have no control over?

Aulig 3y ago

I've been self-hosting this for a year or so and it's pretty neat. It happens quite often that I need to wait for something which doesn't have a builtin alert. Currently I'm for example using it to get an alert when a flutter package has been updated on pub.dev

wkat4242 3y ago

Interesting! I've been making some scripts privately for specific tasks, like an Nvidia video card I wanted to buy. However many sites are really hostile to scraping these days. I'll give it a try,.

weinzierl 3y ago
camelcamelcamel is a really old service, but it's still around and still works well for product price changes.
- wkat4242 3y ago
  Yes but they collaborate with Amazon. For example during the pandemic they turned it all off for a few months because Amazon wanted it. They're still around because Amazon lets them be around.
  - somsak2 3y ago
    source?
    - wkat4242 3y ago
      https://camelcamelcamel.com/blog/amazon-eu-covid-19/ https://camelcamelcamel.com/blog/amazon-eu-covid-19/
      - weinzierl 3y ago
        This is interesting. I did not log into it for ages but sure still get the alerts for my existing trackers. That's why I assumed that they are still around.

Mo3 3y ago

Kudos, this is a very feature rich solution. I like it. 8.99 seems a little bit much, but I guess it's probably the sweet spot on the curve.

serhack_ 3y ago
You can install it for free on your machine. I think it's a decent price, considering that they offer you to check a website even with a ratio of seconds.
johnnyworker 3y ago
For Windows, there is the OG https://www.aignes.com/ https://www.aignes.com/

abbaselmas 3y ago

my favorite one is visualping.io

_flux 3y ago

I've used https://urlwatch.readthedocs.io https://urlwatch.readthedocs.io but this definitely seems much easier to use—though possibly not quite as powerful regarding page filtering. But at least ChangeDetection supports jq which is already quite a nice feature in that department.

shortrounddev2 3y ago

A few years ago when I was a Junior engineer, I worked for a company which would provide us with free tickets to any sports team in our city. I'm not a sports fan, but my stepdad loves hockey. When the tickets went up, they got reserved pretty fast. I noticed that the site we used to reserve the tickets had a predictable slug like: `{company}-hockey-tickets-2016-season`, and `{company}-hockey-tickets-2015-season`. The sites also didn't change between each refresh. So, when it was nearly time for the 2017 season to start, I wrote a script which hashed a GET request of the site at `{company}-hockey-tickets-2017-season`. If the hash changed, it would send me a text message using twilio, and I would know to immediately get on a machine and reserve the tickets we wanted. After a couple false positives, the page eventually went up before it was announced and I reserved the tickets I wanted. Unfortunately, the office manager who did these things told me I couldn't jump the gun and un-reserved my tickets. After the official announcement went up, the spots I had originally reserved quickly went to someone else

HardlyCurious 3y ago
Yeah, that office manager probably knew who would be getting the tickets in advance.
- alach11 3y ago
  Working hard: writing a script to detect when the website goes live. Working smart: bring the office manager a pastry, ask them about their day, and drop hints about how excited you are for the tickets this year.
  - nkko 3y ago
    Engineer vs Non-Engineer approach.
    - deedree 3y ago
      You mean social engineering.
  - 3y ago

accidbuddy 3y ago

Anyone has been utilized this tool to monitoring Amazon Prices (with login)?

stewx 3y ago
Use CamelCamelCamel
- accidbuddy 3y ago
  As far as I know, doesn't work to Amazon.br (Brazil). Thanks by the reply.
  - robk 3y ago
    Have you tried keepa?

cloudyporpoise 3y ago

Curious how this handles sites behind CloudFlare using bot detection.

_lvbh 3y ago
I see selenium and playwright in the dependencies. I would’ve just gone with tls-client
_boffin_ 3y ago
It’s not too difficult to get past cloudflare using puppeteer. Being normal is the key.
- mcstempel 3y ago
  Yeah, +1. Even vanilla puppeteer is pretty successful against Cloudflare

sccxy 3y ago

I have used it for some time now. It has been very useful for me. Some use cases for me: * price changes * calendar changes for sport events * document changes for local gov * new firmware releases * terms of conditions changes

ephbit 3y ago

Here's an addon for Firefox that also checks websites in regular intervals to detect changes: https://github.com/WaldiPL/webpageScanner https://github.com/WaldiPL/webpageScanner Not affiliated with it and I don't know if it still works, haven't used it lately. When I used it, it worked to my satisfaction.

mchenier 3y ago

https://visualping.io https://visualping.io does a nice similar job and is free for moderate personal use.

davidinosauro 3y ago
On the theme of alternatives, I've been using https://www.followthatpage.com/ https://www.followthatpage.com/ on and off for years. In particular for my partner's job hunt, watching some local trade association billboards or single school's `/jobs.html` pages. Also has a reasonable free plan for personal use (20 pages daily + 20 pages weekly + 1 page hourly). Definitely much simpler in terms of diffing (only text), but it has this 2000s vibe.