4 ms·

Add to that the removal of public APIs and things like RSS feeds. I would much rather use an API than scrape you, I will even pay a small fee, but if you don’t

Add to that the removal of public APIs and things like RSS feeds. I would much rather use an API than scrape you, I will even pay a small fee, but if you don’t provide anything, then you’re getting scraped.

1vuio0pswjnm7 3y ago
"Add to that the removal of public APIs and things like RSS feeds." You mean like this public API https://hacker-news.firebaseio.com/v0/item/39421253.json https://hacker-news.firebaseio.com/v0/item/39421253.json Or this RSS feed https://www.theverge.com/rss/index.xml https://www.theverge.com/rss/index.xml
- WirelessGigabit 3y ago
  Reddit recently started blocking anonymous GETs to RSS feeds. And if you use your cookies they temp-block your account after x (like x every 30 minutes) times. And I didn't even care that there was no content on the RSS feed, I just wanted the title / notification so I don't have to use ... (I can't put anything in here as there is no alternative that allows me to track everything).
- deleted 3y ago
  [deleted]
- 1vuio0pswjnm7 3y ago
  Another one. This one has a rate limit. curl -si40A "" https://old.reddit.com/new.rss > 1.json sed 10q 1.json HTTP/1.1 200 OK content-length: 40582 content-type: application/atom+xml; charset=UTF-8 x-ua-compatible: IE=edge x-frame-options: SAMEORIGIN x-ratelimit-remaining: 93 x-ratelimit-used: 3 x-ratelimit-reset: 96 x-reddit-pod-ip: 10.104.158.170:80 x-reddit-internal-ratelimit-rls-type: ip-standard In theory, APIs to retrieve public information suck because they are designed to be rate-limited or subject to quotas. Whereeas IME public-facing websites are much less likely to set rate limits and enforce quotas. I prefer to use the public-facing website instead of "Web APIs" and convert HTML to SQLite which I prefer over non-LD JSON. If the public-facing website changes their HTML, I change the code in the filter. This is rare. It is very simple for me to do.

1vuio0pswjnm7 3y ago

"Add to that the removal of public APIs and things like RSS feeds." You mean like this public API https://hacker-news.firebaseio.com/v0/item/39421253.json https://hacker-news.firebaseio.com/v0/item/39421253.json Or this RSS feed https://www.theverge.com/rss/index.xml https://www.theverge.com/rss/index.xml

WirelessGigabit 3y ago
Reddit recently started blocking anonymous GETs to RSS feeds. And if you use your cookies they temp-block your account after x (like x every 30 minutes) times. And I didn't even care that there was no content on the RSS feed, I just wanted the title / notification so I don't have to use ... (I can't put anything in here as there is no alternative that allows me to track everything).
deleted 3y ago
[deleted]
1vuio0pswjnm7 3y ago
Another one. This one has a rate limit. curl -si40A "" https://old.reddit.com/new.rss > 1.json sed 10q 1.json HTTP/1.1 200 OK content-length: 40582 content-type: application/atom+xml; charset=UTF-8 x-ua-compatible: IE=edge x-frame-options: SAMEORIGIN x-ratelimit-remaining: 93 x-ratelimit-used: 3 x-ratelimit-reset: 96 x-reddit-pod-ip: 10.104.158.170:80 x-reddit-internal-ratelimit-rls-type: ip-standard In theory, APIs to retrieve public information suck because they are designed to be rate-limited or subject to quotas. Whereeas IME public-facing websites are much less likely to set rate limits and enforce quotas. I prefer to use the public-facing website instead of "Web APIs" and convert HTML to SQLite which I prefer over non-LD JSON. If the public-facing website changes their HTML, I change the code in the filter. This is rare. It is very simple for me to do.