3 ms·
HTML is a perfectly good API.
by matheusmoreira 3y ago
HTML is a perfectly good API.
- VWWHFSfQ 3y agoReddit is a SPA though right
- matheusmoreira 3y agoDon't know to be honest. I assume new reddit is while old reddit isn't. Truth is the job is even easier in case of an SPA: you just need to use whatever internal APIs they built for themselves. Can't add idiotic restrictions to that without affecting the normal web service.
- delfinom 3y agoSPA being a problem was last decade. Headless chromium is pretty standard for scraping nowadays.
- VWWHFSfQ 3y agoHow does that work for clients though. Should Apollo ship a headless chromium in their mobile app
- rovr138 3y agoCan they launch a browser view hidden and scrape it? I have no idea if they can read from it.
- moneywoes 3y agoIsn’t that very expensive to run at scale especially mixing in residential IPs to avoid blocking
- ipaddr 3y agoEach person would spider for their own needs and most would use residential ips
- devjab 3y agoYes and no, the terrible attempts to build a new front-end are, but the old front-end that runs on python with Pylons (I believe) as it’s “front” isn’t. I like react, and I love typescript, but sites like Wikipedia, old.Reddit.com, stackoverflow, hacker news and so on are nice showcases of how you should never be afraid of the page reload, because your users won’t be unless you’re building something where you need to update screen states without user inputs. Like when your user receives a mail and can’t just reload the page because your users input would be lost if you do so. I think this last part is the primary reason (along with mobile) that Reddit has been attempting to move to React, because the “social” parts like chats and private messages don’t instantly show up for users in the old front-end. Unfortunately they haven’t been very successful so far. You probably can scrape their current or their new.Reddit front-ends since you can scrape SPAs, but it’s much, much, easier to scrape the old.Reddit front.
- mr_toad 3y agoMost HTML is undocumented and unstable, so I’d say it’s far from perfect.
- matheusmoreira 3y agoIt's perfect in that there's no way for them to differentiate from normal browser traffic. This is adversarial interoperability which is exactly how we're supposed to deal with corporations and their idiotic "terms". Nobody is forced to accept their "take it or leave it" BS. https://www.eff.org/deeplinks/2019/10/adversarial-interoperability https://www.eff.org/deeplinks/2019/10/adversarial-interopera... If it's important enough, someone somewhere will care enough to fix it when it inevitably breaks. Look at youtube-dl and its variations, there's even a youtube.js now. https://news.ycombinator.com/item?id=31021611 https://news.ycombinator.com/item?id=31021611
- great_psy 3y agoI would be very curious to know how many engineers inside google have as their sole responsibility to fight programs like YouTube-dl and so on. I am willing to bet it’s a whole division.
- matheusmoreira 3y agoThey can fight all they want. In the end it doesn't matter how many engineers they throw at the problem. The only way they can win is for the world to descend ever deeper into DRM tyranny. Google must literally own your computer in order to prevent you from copying YouTube videos. It has to come pwned straight off the factory so that Google can literally dictate that you the user are prohibited from running certain bad programs that are known to hurt their bottom line. I realize we're not too far off from that dystopia but it's still a battle that deserves to be fought for the principle of it.
- moneywoes 3y agoCan’t they just fingerprint the incoming requests based on a litany of variables such as headers ip etc to prevent this “scraping”? Sorry I don’t follow