chadwebscraper

5 ms·

1.
by chadwebscraper 8mo ago
Just did - great chatting with you. Best of luck & looking forward to (potentially) hearing back!
2.
by chadwebscraper 8mo ago
First off, really sorry to hear that it didn't work. Edit: it looks like you hit an edge case that I didn't see in testing. Happy to explain more, but it was skipping the extraction due to some pre-processing that craigslist was f
3.
by chadwebscraper 8mo ago
This is actually a really interesting though - like an embedder with live data?
4.
by chadwebscraper 8mo ago
Shoot me your site and I can blacklist it
5.
by chadwebscraper 8mo ago
Thank you for the feedback - agreed. It’s an extraction pattern for a certain site, so you can reuse it. Think a pattern to extract all forum posts - then using that on different pages with the same format. Like show new, show, new posts on
6.
by chadwebscraper 8mo ago
Let me clarify, it just reindexes if the structured data changes, so it ignores layout changes. So it diffs the extractions. Would be curious to try it out on your sites if you want to shoot me a few over - I can share my email. It does use
7.
by chadwebscraper 8mo ago
So with APIs, it adjusts. For HTML layouts, it looks at the previous diffs to catch potential errors and then re-indexes.
8.
by chadwebscraper 8mo ago
lol it probably does unless their filtering is great
9.
by chadwebscraper 8mo ago
This is a good callout - I’ve tried my best thus far to limit the use of proxies unless absolutely necessary and then focus on reputable providers (even though these are a bit more pricey). Definitely going to give this more thought though,
10.
by chadwebscraper 8mo ago
I appreciate the response! I also agree - happy to add some clarity to this stuff. Bot protection - this is handled in a few ways, the basic form bypasses most bot protections and that’s what you can use on the site today. For tougher sites
11.
Turn any website into a live, structured data feed(meter.sh)
36 points by chadwebscraper 8mo ago|32 comments
12.
by chadwebscraper 8mo ago
Here’s how it works: 1. Paste a URL in, describe what you want 2. Define an interval to monitor 3. Get real time webhooks of any changes in JSON Lots of customers are using this across different domains to get consistent, repeatable JSON ou
13.
by chadwebscraper 8mo ago
Thank you! Oops - it’s just https://docs.meter.sh - going to fix this.
14.
Turn any website into a live data feed(github.com)
2 points by chadwebscraper 8mo ago|3 comments
15.
by chadwebscraper 8mo ago
I recently needed to extract a sites content + track any new content. I didn’t like the current tools, so I built meter. It syncs a website and fires webhooks whenever content changes. To start, you just need to plug a site URL in. It will
16.
Show HN: Scrape (auto-discover APIs or HTML) & Monitor changes on any site(github.com)
1 points by chadwebscraper 8mo ago|0 comments