3 ms·

I only dabble in reverse engineering web apis these days, but wouldn't it be exceedingly possible to write an unofficial API and run an app off of that? It's n

by Zetice 3y ago

I only dabble in reverse engineering web apis these days, but wouldn't it be exceedingly possible to write an unofficial API and run an app off of that? It's not pleasant work, but especially with LLM advances, I feel like keeping up with the shifting DOM would for sure be doable, and in fact one of the main reasons Reddit started out with an API: better to provide a path for programmatic access than make the only option outside of your control...

crabbone 3y ago
DOM is not very "shifting" in projects like these... The longer the project lives, the more tests it accumulates. A "good" practice is to rely on element ids for automation (i.e. clicking, finding form inputs etc), but often times it's not possible / hard to know in advance what the id will be, especially with all the HTML generation going on... so, tests would often use XPath to find where to click. And nobody really wants to change those tests... I mean, yeah, there will be changes ever now and then, but not a lot and not enough to render out-of-date scrappers completely useless.
djbusby 3y ago
Its a cat-mouse game doing it that way. Its a crappy path for any dev.
birracerveza 3y ago
Well, yeah, but the problem is the content and the users. Can't code those.
sigmoid10 3y ago
Possible? Yes. Reasonable? Almost certainly no. It's not just that you'd have to constantly keep up with changing content (which might even be hidden through endless obscure javascript) and all sorts of possible filtering mechanisms on reddit's server side, you'd also make yourself vulnerable to a boatload of issues if you want to publish your app in official channels, since you are violating reddit's TOS. The cost to benefit ratio is exceedingly unfavourable. Since reddit has a huge following, some people might try, but I wouldn't bet on anyone succeeding in a big way.
- skeaker 3y ago
  Seems like the exact sort of problem that an open source codebase is designed to handle. Even just 10 random devs sending PRs to a github repo will always outpace the changes reddit makes since they're bottlenecked by having to push to millions of users. Coincidentally, the types that would contribute most likely wouldn't care about publishing it either, since it would be FOSS anyways giving users little incentive to buy it.
tayo42 3y ago
I started experimenting with this, I don't think it's that bad to do. I don't think scale will be overwhelming either. Mostly my problem is motivation. Trying to efficiently store and retrieve comments is an interesting problem.
indrora 3y ago
It's worth noting that Reddit's own app uses their official API. The whole thing is OAuth2. They could have easily just said "Only Reddit Gold users can use external apps" and have avoided all this fracas. Now, it would take very little for someone to dig around and find the official Reddit API keys (OpenID/OAuth2 client key/secret) and just impersonate the official Reddit apps. I would never suggest that someone do some googling and discover threads like https://github.com/TwidereProject/Twidere-Android/issues/1475 https://github.com/TwidereProject/Twidere-Android/issues/147... where such things are discussed.

by Zetice 3y ago

crabbone 3y ago

DOM is not very "shifting" in projects like these... The longer the project lives, the more tests it accumulates. A "good" practice is to rely on element ids for automation (i.e. clicking, finding form inputs etc), but often times it's not possible / hard to know in advance what the id will be, especially with all the HTML generation going on... so, tests would often use XPath to find where to click. And nobody really wants to change those tests... I mean, yeah, there will be changes ever now and then, but not a lot and not enough to render out-of-date scrappers completely useless.

djbusby 3y ago

Its a cat-mouse game doing it that way. Its a crappy path for any dev.

birracerveza 3y ago

Well, yeah, but the problem is the content and the users. Can't code those.

sigmoid10 3y ago

Possible? Yes. Reasonable? Almost certainly no. It's not just that you'd have to constantly keep up with changing content (which might even be hidden through endless obscure javascript) and all sorts of possible filtering mechanisms on reddit's server side, you'd also make yourself vulnerable to a boatload of issues if you want to publish your app in official channels, since you are violating reddit's TOS. The cost to benefit ratio is exceedingly unfavourable. Since reddit has a huge following, some people might try, but I wouldn't bet on anyone succeeding in a big way.

skeaker 3y ago
Seems like the exact sort of problem that an open source codebase is designed to handle. Even just 10 random devs sending PRs to a github repo will always outpace the changes reddit makes since they're bottlenecked by having to push to millions of users. Coincidentally, the types that would contribute most likely wouldn't care about publishing it either, since it would be FOSS anyways giving users little incentive to buy it.

tayo42 3y ago

I started experimenting with this, I don't think it's that bad to do. I don't think scale will be overwhelming either. Mostly my problem is motivation. Trying to efficiently store and retrieve comments is an interesting problem.

indrora 3y ago

It's worth noting that Reddit's own app uses their official API. The whole thing is OAuth2. They could have easily just said "Only Reddit Gold users can use external apps" and have avoided all this fracas. Now, it would take very little for someone to dig around and find the official Reddit API keys (OpenID/OAuth2 client key/secret) and just impersonate the official Reddit apps. I would never suggest that someone do some googling and discover threads like https://github.com/TwidereProject/Twidere-Android/issues/1475 https://github.com/TwidereProject/Twidere-Android/issues/147... where such things are discussed.