4 ms·

Does anything stops devs from using the Web interface instead of the API? Sure it makes it a bit more challenging because parsing is involved but AFAIK it is co

by mrtksn 4y ago

Does anything stops devs from using the Web interface instead of the API? Sure it makes it a bit more challenging because parsing is involved but AFAIK it is completely legal to parse and display HTML output however you want. It's essentially like a browser add-on. I wonder if Twitter can be killed by the "embrace, extend, extinguish" strategy? So: 1. Make a client that combines Mastadon/Others and Twitter feeds, making it switching out of Twitter frictionless. 2. Provide extra functionality for non-Twitter posts, encouraging the use of the non-Twitter services. 3. Once have a good traction, penalize Twitter post to further push non-Twitter adoption. Eventually remove Twitter support completely. 4. Profit??

black_puppydog 4y ago
you get into an armsrace where twitter will obfuscate the markup to just wear out the scraper devs. which they can, because $$$
- mrtksn 4y ago
  Then just OCR the rendered output. It's cheap and easy these days.
  - mlatu 4y ago
    i heard it works best on iphones, best of luck
    - mrtksn 4y ago
      Apple’s vision framework is actually very good, so effective on-device extraction is really just a few lines of code on iPhone(all Apple device tbh).
  - tsimionescu 4y ago
    Good luck OCR-ing images and videos, though. Twitter isn't just text.
    - mrtksn 4y ago
      Why do I need to OCR images and videos? Why not display those as-is?
      - tsimionescu 4y ago
        Which one do you show for which tweet, if all you have is the OCR (that is, you can't read the markup)?
        mrtksn 4y ago
        Apple's vision framework is not limited to text, can detect shapes and other things. Twitter WebUI loads images and videos when you scroll. Which means you have a good signal what image and video is associated with what tweet.
- actualwitch 4y ago
  Doing this requires a fair amount of development time, especially when you can break something else with this and you previously removed all the guardrails.
  - Mindwipe 4y ago
    That didn't stop Facebook doing it (and breaking all kinds of stuff in the process).
    - Hamuko 4y ago
      Did Facebook lay off half of the company before they did it?
    - raydev 4y ago
      Facebook had far more engineers available to do this sort of work.
- nkozyra 4y ago
  They don't do this when they can just disable the app without being logged in.
samwillis 4y ago
While it's quite possible to build an app that way, using scraping. The danger is that users of that app will get suspended from Twitter (users have explicitly agreed not to scrape), once thats happened to a couple of users, thats the end of your app. Obviously you can try to obscure that you are scraping, but that is difficult to get right and there is no way back once you get it wrong.
- mrtksn 4y ago
  On Apple devices that wouldn’t be an issue because you you can load the Twitter Web interface into a real Safari and get the markup from there for processing. Twitter can try things like CAPTCHA but this will degrade the experience for all Twitter users and you can simply delegate this CAPTCHA to your user to solve anyway.
  - samwillis 4y ago
    While for reading that works, there is an easy why via posting to spot people use that app. They can change at any moment the input names so that when posting a tweet you are exposing that you are using the scrapping client.
    - mrtksn 4y ago
      Remember that you are loading full Twitter website in the background. You will use their original interface to post Tweets.
      - majewsky 4y ago
        Yes, but the app is not a human and will likely interact with the website in a more predictable (and thus, detectable) way than the human itself would.
        mrtksn 4y ago
        But you also have a human using the app, you can transfer the interactions with your app to the Twitter's Web UI.
amelius 4y ago
Where can I vote for this project?
Nextgrid 4y ago
You will have 2 problems with this: 1) the law, at least in the US. Laws such as copyright and maybe even the CFAA have been abused to outlaw adversarial interoperability. You'd need to either practice good opsec and operate anonymously or be based out of reach of US-allied law enforcement. 2) Apple (and maybe Google too?) - they will remove any client for a third-party service unless said client has explicit approval from the third-party, all the way up to ridiculous levels such as rejecting an app using the LAN API of smart home devices explicitly designed for this purpose: https://community.lifx.com/t/app-store-rejection-permission-from-lifx/5830 https://community.lifx.com/t/app-store-rejection-permission-...
- mrtksn 4y ago
  Can you elaborate on the first point? Apple might not be such a big issue because they allow reader apps. I guess they can ban you to improve their relationship with Twitter but you can also throw a subscription service as an IAP to improve your odds :)
- counttheforks 4y ago
  Another point to show what Apple's true goal is: To kill general computing. If you buy an iPhone you should be able to run whatever app you want on it. But no, Apple knows best and will not permit you to do anything it doesn't approve of. As for the APIs being copyrightable see Google vs Oracle.
  - deleted 4y ago
    [deleted]
  - paulcole 4y ago
    > If you buy an iPhone you should be able to run whatever app you want on it. I hear this sentiment quite often on HN. Maybe I’m missing something but was this tacked onto the end of those stone tablets that Moses brought down from the mountain? If not, then it seems like people want this to be a statement of fact when the reality is much closer to: > If I buy an iPhone I wish I could run whatever app I want on it.
prirun 4y ago
I think you'll have the same problem as Twitter with #4!
- mrtksn 4y ago
  Fair point. Maybe have a paid subscription with features that benefit social media professionals? There are some popular and profitable services economy built around social media.
- addandsubtract 4y ago
  I think the tools running on top of twitter are more profitable than twitter itself.
dom96 4y ago
I’ve done this recently. Essentially created a browser extension that combines Mastodon and Twitter on twitter.com. It works surprisingly well, but I’m guessing catching up with Twitter.com changes is bound to be painful. You can find it here: https://chrome.google.com/webstore/detail/mastodon-chirper/lgiffpmohlpjlkclpmelkkfmjnlpafne?hl=en-GB&authuser=0 https://chrome.google.com/webstore/detail/mastodon-chirper/l...
- amelius 4y ago
  > catching up with Twitter.com changes is bound to be painful. Perhaps it should be handled in the same way the YouTubeDL (YT DLP) team handles changes in the html.
  - heavyset_go 4y ago
    I'm not familiar with how they do that, can you expound on it?

by mrtksn 4y ago

black_puppydog 4y ago

you get into an armsrace where twitter will obfuscate the markup to just wear out the scraper devs. which they can, because $$$

mrtksn 4y ago
Then just OCR the rendered output. It's cheap and easy these days.
- mlatu 4y ago
  i heard it works best on iphones, best of luck
  - mrtksn 4y ago
    Apple’s vision framework is actually very good, so effective on-device extraction is really just a few lines of code on iPhone(all Apple device tbh).
- tsimionescu 4y ago
  Good luck OCR-ing images and videos, though. Twitter isn't just text.
  - mrtksn 4y ago
    Why do I need to OCR images and videos? Why not display those as-is?
    - tsimionescu 4y ago
      Which one do you show for which tweet, if all you have is the OCR (that is, you can't read the markup)?
      - mrtksn 4y ago
        Apple's vision framework is not limited to text, can detect shapes and other things. Twitter WebUI loads images and videos when you scroll. Which means you have a good signal what image and video is associated with what tweet.
actualwitch 4y ago
Doing this requires a fair amount of development time, especially when you can break something else with this and you previously removed all the guardrails.
- Mindwipe 4y ago
  That didn't stop Facebook doing it (and breaking all kinds of stuff in the process).
  - Hamuko 4y ago
    Did Facebook lay off half of the company before they did it?
  - raydev 4y ago
    Facebook had far more engineers available to do this sort of work.
nkozyra 4y ago
They don't do this when they can just disable the app without being logged in.

samwillis 4y ago

While it's quite possible to build an app that way, using scraping. The danger is that users of that app will get suspended from Twitter (users have explicitly agreed not to scrape), once thats happened to a couple of users, thats the end of your app. Obviously you can try to obscure that you are scraping, but that is difficult to get right and there is no way back once you get it wrong.

mrtksn 4y ago
On Apple devices that wouldn’t be an issue because you you can load the Twitter Web interface into a real Safari and get the markup from there for processing. Twitter can try things like CAPTCHA but this will degrade the experience for all Twitter users and you can simply delegate this CAPTCHA to your user to solve anyway.
- samwillis 4y ago
  While for reading that works, there is an easy why via posting to spot people use that app. They can change at any moment the input names so that when posting a tweet you are exposing that you are using the scrapping client.
  - mrtksn 4y ago
    Remember that you are loading full Twitter website in the background. You will use their original interface to post Tweets.
    - majewsky 4y ago
      Yes, but the app is not a human and will likely interact with the website in a more predictable (and thus, detectable) way than the human itself would.
      - mrtksn 4y ago
        But you also have a human using the app, you can transfer the interactions with your app to the Twitter's Web UI.

amelius 4y ago

Where can I vote for this project?

Nextgrid 4y ago

You will have 2 problems with this: 1) the law, at least in the US. Laws such as copyright and maybe even the CFAA have been abused to outlaw adversarial interoperability. You'd need to either practice good opsec and operate anonymously or be based out of reach of US-allied law enforcement. 2) Apple (and maybe Google too?) - they will remove any client for a third-party service unless said client has explicit approval from the third-party, all the way up to ridiculous levels such as rejecting an app using the LAN API of smart home devices explicitly designed for this purpose: https://community.lifx.com/t/app-store-rejection-permission-from-lifx/5830 https://community.lifx.com/t/app-store-rejection-permission-...

mrtksn 4y ago
Can you elaborate on the first point? Apple might not be such a big issue because they allow reader apps. I guess they can ban you to improve their relationship with Twitter but you can also throw a subscription service as an IAP to improve your odds :)
counttheforks 4y ago
Another point to show what Apple's true goal is: To kill general computing. If you buy an iPhone you should be able to run whatever app you want on it. But no, Apple knows best and will not permit you to do anything it doesn't approve of. As for the APIs being copyrightable see Google vs Oracle.
- deleted 4y ago
  [deleted]
- paulcole 4y ago
  > If you buy an iPhone you should be able to run whatever app you want on it. I hear this sentiment quite often on HN. Maybe I’m missing something but was this tacked onto the end of those stone tablets that Moses brought down from the mountain? If not, then it seems like people want this to be a statement of fact when the reality is much closer to: > If I buy an iPhone I wish I could run whatever app I want on it.

prirun 4y ago

I think you'll have the same problem as Twitter with #4!

mrtksn 4y ago
Fair point. Maybe have a paid subscription with features that benefit social media professionals? There are some popular and profitable services economy built around social media.
addandsubtract 4y ago
I think the tools running on top of twitter are more profitable than twitter itself.

dom96 4y ago

I’ve done this recently. Essentially created a browser extension that combines Mastodon and Twitter on twitter.com. It works surprisingly well, but I’m guessing catching up with Twitter.com changes is bound to be painful. You can find it here: https://chrome.google.com/webstore/detail/mastodon-chirper/lgiffpmohlpjlkclpmelkkfmjnlpafne?hl=en-GB&authuser=0 https://chrome.google.com/webstore/detail/mastodon-chirper/l...

amelius 4y ago
> catching up with Twitter.com changes is bound to be painful. Perhaps it should be handled in the same way the YouTubeDL (YT DLP) team handles changes in the html.
- heavyset_go 4y ago
  I'm not familiar with how they do that, can you expound on it?