7 ms·

It's simple: Use the website. It will never stop working because people use it and you can automate and scrape it just fine. Kind of sucks that you have to go t

It's simple: Use the website. It will never stop working because people use it and you can automate and scrape it just fine. Kind of sucks that you have to go through the effort - but then there is no social media corporation that isn't toxic in some way to it's userbase.

1vuio0pswjnm7 4y ago
Does not take much effort. Below is an example using curl. For reading Twitter feeds I just get the JSON and read the "full_text" objects. I have simple custom program I wrote that turns JSON of unlimited size into something like line-delimited JSON so I can use sed, grep and awk on it but HN readers probably prefer jq. For checking out t.co URLs I use HTTP/1.1 pipelining. Usage for reading is something like (but not identical to) 1.sh screen_name > 1.json yy059 < 1.json|grep full_text > 1.txt less 1.txt Usage for checking out URLs is something like (but not identical to) unset connection export Connection=keep-alive yy059 < 1.json|grep full_text \ |yy030 \ |grep "https://t.co/.{10}$" \ |uniq \ |yy025 \ |nc -vv h1b 80 \ |sed -n '/location: /s///p' \ |ahref > 1.htm links -no-connect 1.htm "ahref" is just a script turns URLs on stdin into simple HTML on stdout Alternatively if I do not trust the URLs I might use a script called "www" instead of ahref. It takes URLs on stdin and fetches archive.org URLs wrapped in simple HTML to stdout, using the IA's cdx API. #!/bin/sh SCREEN_NAME=$1 COUNT=500 PUBLIC_TOKEN="Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA" GT=$(exec curl -A "" -s https://twitter.com/$SCREEN_NAME|sed -n '/gt=/{s/.*gt=//;s/;.*//p;}'); echo "x-guest-token: $GT" >&2; REST_ID=$(exec curl -A "" -H "authorization: $PUBLIC_TOKEN" -H "content-type: application/json" -H "x-guest-token: $GT" -s "https://twitter.com/i/api/graphql/mCbpQvZAw6zu_4PvuAUVVQ/UserByScreenName?variables=%7B%22screen_name%22%3A%22$SCREEN_NAME%22%2C%22withSafetyModeUserFields%22%3Atrue%2C%22withSuperFollowsUserFields%22%3Atrue%7D"|sed 's/$rest_id\":\"[0-9]*$$.*$/\1/;s/.*\"//'); echo "rest_id: $REST_ID" >&2; curl -A "" -H "authorization: $PUBLIC_TOKEN" -H "content-type: application/json" -H "x-guest-token: $GT" -s "https://twitter.com/i/api/graphql/3ywp9kIIW-VQOssauKmLiQ/UserTweets?variables=%7B%22userId%22%3A%22${REST_ID}%22%2C%22count%22%3A$COUNT%2C%22includePromotedContent%22%3Atrue%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Atrue%7D&features=%7B%22dont_mention_me_view_api_enabled%22%3Atrue%2C%22interactive_text_enabled%22%3Atrue%2C%22responsive_web_uc_gql_enabled%22%3Afalse%2C%22vibe_tweet_context_enabled%22%3Afalse%2C%22responsive_web_edit_tweet_api_enabled%22%3Afalse%2C%22standardized_nudges_misinfo%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%2C%22include_rts%22%3Atrue%7D" There's no way I would use the Twitter website as it requires enabling Javascript and not for the user's benefit. This solution isn't pretty but I can easily keep tabs on Twitter feeds without any need for a Twitter account, a Twitter "API key" or a so-called "modern" browser.
- altdataseller 4y ago
  Can u scrape search feeds too? Ie tweets that match a certain string?
  - Hakkin 4y ago
    You can scrape anything you see in the UI (and sometimes stuff you cannot see). Twitter makes almost no effort to stop people from using their internal APIs, which is why them saying discontinuing the free public API is to stop malicious bots is pretty laughable. Unless they seriously increase their detection abilities for non-approved clients using their internal API, it would take any malicious actor all of a few hours to transition to using the internal API for whatever they want. Honestly, I assumed most bad actors would already be doing this, since things like spamming were already against the ToS of the public API.
- zimpenfish 4y ago
  What happens if/when they block that Bearer token?
  - deleted 4y ago
    [deleted]
  - 1vuio0pswjnm7 4y ago
    The token has been the same since at least 2020 when Twitter started using GraphQL instead of REST. Every person visiting twitter.com is using this same token. The token is neither personal nor private. What would be the point of changing or blocking it.
    - zimpenfish 4y ago
      > Every person visiting twitter.com is using this same token. Be interesting to see if that stays the same when they're charging for the API but leaving a huge loophole with this token.
      - 4y ago
pessimizer 4y ago
The website is awful. It's horrific. It was the worst site I visited regularly until I started using Nitter instances. I'd rather not know what is going on than go to it. I wish they'd wind the UI back 10 years, to back when it was pleasant to browse.
- kibwen 4y ago
  I look forward to someone teaching Elon Musk what version control is, resulting in him wholesale rolling back Twitter's entire stack to 2008. At last, Twitter is written in Ruby again!
  - flir 4y ago
    I too miss the fail whale.
  - Consultant32452 4y ago
    Do you remember, I think it was 2020, when there was a big "hack" and tons of blue check accounts started posting fraud links? Two important take-aways from that: 1) The attackers just pulled the old "Hey, I'm from IT service desk and need your password." 2) Apparently every engineer had god mode in production I can't imagine what scary monsters are in that codebase.
- hgsgm 4y ago
  The point is that the website is an API, irrelevant if it is a good or bad UI.
  - omoikane 4y ago
    Depending on twitter's pricing, this might not be worthwhile. Scraping web pages is a last resort for when proper APIs are not available.
anigbrowl 4y ago
I actually don't mind scraping it that much, and even enjoy the adversarial aspects of writing scrapers. but it makes a lot of high level functionality like filtered streams or historical search much less accessible; I'd probably never have learned a lot of network analysis stuff over the last decade or so if I'd had to to pay to access streaming data first. Also, I think it's going to be harder for academic researchers to get institutional approval to scrape adversarially, so it could put a dent in a lot of social science research by forcing people to chase grants instead of focusing on their code.
- rkagerer 4y ago
  It's ironic, for a couple (non-Twitter) projects I wound up scraping because either a) they didn't have an API yet (e.g. early crypto pricing sites) or b) I wasn't confident the API would remain intact over the long term. Kind of depressing.
- geraldwhen 4y ago
  Also quality residential proxies are pricey. You need to rate limit and rotate both IP and puppeteer fingerprint when adversarially scraping.
- mindslight 4y ago
  > I think it's going to be harder for academic researchers to get institutional approval to scrape adversarially This is a good point about what might happen, but it seems worthwhile to address and fix directly. Personally I don't see why adversarial scraping of a publicly published website should require any more ethical consideration/review than using the suggested API would. Ethical concerns should revolve around humans, not the business desires of non-human entities.
olliej 4y ago
IIRC the twitter website goes to great lengths to mangle itself to prevent ad blockers presumably. Of course this used to have the side effect of breaking significant swathes of basic browser UX, especially in areas of accessibility. I assume they’re better now than they once were, but given musk’s historical behavior I assume he won’t consider breaking something like accessibility to be bad or problematic.
- dzhiurgis 4y ago
  > musk’s historical behavior I assume he won’t consider breaking something like accessibility to be bad or problematic. I don't see why would he spend more to make website less accessible. Not fixing bugs - yes, just about everyone does so. But breaking it intentionally costs money. Tesla's aren't THE most accessible (by cost) EV out there, but SpaceX def is.
  - optionalsquid 4y ago
    Supposedly the entire accessibility team got laid off [1], so there might not be anyone left to ensure that changes to existing features do not reduce accessibility nor ensure that new features are accessible. Not that I'd expect Musk to care a whole lot about concerns raised in that regard, considering that he supposedly went ahead with Twitter Blue despite significant (and well warranted) concerns from the Trust and Safety team [2]. So Twitter will probably become less and less accessible over time. [1] https://www.wired.com/story/twitter-layoffs-accessibility/ https://www.wired.com/story/twitter-layoffs-accessibility/ [2] https://www.businessinsider.com/twitter-sent-musk-risks-paid-verification-before-launch-2022-11 https://www.businessinsider.com/twitter-sent-musk-risks-paid...
  - dragonwriter 4y ago
    > Tesla's aren't THE most accessible (by cost) EV out there, but SpaceX def is. SpaceX is... an accessible EV? What?
    - kingofpandora 4y ago
      I'm pretty sure you know exactly what the poster meant ...
dmonitor 4y ago
I think it’s a terrible move to kill the stuff like postybirb that a lot of people use to make posts on multiple websites at once. they’ll have to either change their workflow to make a tweet (annoying) or just abandon twitter altogether. just makes the website worse for practically no benefit.

1vuio0pswjnm7 4y ago

Does not take much effort. Below is an example using curl. For reading Twitter feeds I just get the JSON and read the "full_text" objects. I have simple custom program I wrote that turns JSON of unlimited size into something like line-delimited JSON so I can use sed, grep and awk on it but HN readers probably prefer jq. For checking out t.co URLs I use HTTP/1.1 pipelining. Usage for reading is something like (but not identical to) 1.sh screen_name > 1.json yy059 < 1.json|grep full_text > 1.txt less 1.txt Usage for checking out URLs is something like (but not identical to) unset connection export Connection=keep-alive yy059 < 1.json|grep full_text \ |yy030 \ |grep "https://t.co/.{10}$" \ |uniq \ |yy025 \ |nc -vv h1b 80 \ |sed -n '/location: /s///p' \ |ahref > 1.htm links -no-connect 1.htm "ahref" is just a script turns URLs on stdin into simple HTML on stdout Alternatively if I do not trust the URLs I might use a script called "www" instead of ahref. It takes URLs on stdin and fetches archive.org URLs wrapped in simple HTML to stdout, using the IA's cdx API. #!/bin/sh SCREEN_NAME=$1 COUNT=500 PUBLIC_TOKEN="Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA" GT=$(exec curl -A "" -s https://twitter.com/$SCREEN_NAME|sed -n '/gt=/{s/.*gt=//;s/;.*//p;}'); echo "x-guest-token: $GT" >&2; REST_ID=$(exec curl -A "" -H "authorization: $PUBLIC_TOKEN" -H "content-type: application/json" -H "x-guest-token: $GT" -s "https://twitter.com/i/api/graphql/mCbpQvZAw6zu_4PvuAUVVQ/UserByScreenName?variables=%7B%22screen_name%22%3A%22$SCREEN_NAME%22%2C%22withSafetyModeUserFields%22%3Atrue%2C%22withSuperFollowsUserFields%22%3Atrue%7D"|sed 's/$rest_id\":\"[0-9]*$$.*$/\1/;s/.*\"//'); echo "rest_id: $REST_ID" >&2; curl -A "" -H "authorization: $PUBLIC_TOKEN" -H "content-type: application/json" -H "x-guest-token: $GT" -s "https://twitter.com/i/api/graphql/3ywp9kIIW-VQOssauKmLiQ/UserTweets?variables=%7B%22userId%22%3A%22${REST_ID}%22%2C%22count%22%3A$COUNT%2C%22includePromotedContent%22%3Atrue%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Atrue%7D&features=%7B%22dont_mention_me_view_api_enabled%22%3Atrue%2C%22interactive_text_enabled%22%3Atrue%2C%22responsive_web_uc_gql_enabled%22%3Afalse%2C%22vibe_tweet_context_enabled%22%3Afalse%2C%22responsive_web_edit_tweet_api_enabled%22%3Afalse%2C%22standardized_nudges_misinfo%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%2C%22include_rts%22%3Atrue%7D" There's no way I would use the Twitter website as it requires enabling Javascript and not for the user's benefit. This solution isn't pretty but I can easily keep tabs on Twitter feeds without any need for a Twitter account, a Twitter "API key" or a so-called "modern" browser.

altdataseller 4y ago
Can u scrape search feeds too? Ie tweets that match a certain string?
- Hakkin 4y ago
  You can scrape anything you see in the UI (and sometimes stuff you cannot see). Twitter makes almost no effort to stop people from using their internal APIs, which is why them saying discontinuing the free public API is to stop malicious bots is pretty laughable. Unless they seriously increase their detection abilities for non-approved clients using their internal API, it would take any malicious actor all of a few hours to transition to using the internal API for whatever they want. Honestly, I assumed most bad actors would already be doing this, since things like spamming were already against the ToS of the public API.
zimpenfish 4y ago
What happens if/when they block that Bearer token?
- deleted 4y ago
  [deleted]
- 1vuio0pswjnm7 4y ago
  The token has been the same since at least 2020 when Twitter started using GraphQL instead of REST. Every person visiting twitter.com is using this same token. The token is neither personal nor private. What would be the point of changing or blocking it.
  - zimpenfish 4y ago
    > Every person visiting twitter.com is using this same token. Be interesting to see if that stays the same when they're charging for the API but leaving a huge loophole with this token.
    - 4y ago

pessimizer 4y ago

The website is awful. It's horrific. It was the worst site I visited regularly until I started using Nitter instances. I'd rather not know what is going on than go to it. I wish they'd wind the UI back 10 years, to back when it was pleasant to browse.

kibwen 4y ago
I look forward to someone teaching Elon Musk what version control is, resulting in him wholesale rolling back Twitter's entire stack to 2008. At last, Twitter is written in Ruby again!
- flir 4y ago
  I too miss the fail whale.
- Consultant32452 4y ago
  Do you remember, I think it was 2020, when there was a big "hack" and tons of blue check accounts started posting fraud links? Two important take-aways from that: 1) The attackers just pulled the old "Hey, I'm from IT service desk and need your password." 2) Apparently every engineer had god mode in production I can't imagine what scary monsters are in that codebase.
hgsgm 4y ago
The point is that the website is an API, irrelevant if it is a good or bad UI.
- omoikane 4y ago
  Depending on twitter's pricing, this might not be worthwhile. Scraping web pages is a last resort for when proper APIs are not available.

anigbrowl 4y ago

I actually don't mind scraping it that much, and even enjoy the adversarial aspects of writing scrapers. but it makes a lot of high level functionality like filtered streams or historical search much less accessible; I'd probably never have learned a lot of network analysis stuff over the last decade or so if I'd had to to pay to access streaming data first. Also, I think it's going to be harder for academic researchers to get institutional approval to scrape adversarially, so it could put a dent in a lot of social science research by forcing people to chase grants instead of focusing on their code.

rkagerer 4y ago
It's ironic, for a couple (non-Twitter) projects I wound up scraping because either a) they didn't have an API yet (e.g. early crypto pricing sites) or b) I wasn't confident the API would remain intact over the long term. Kind of depressing.
geraldwhen 4y ago
Also quality residential proxies are pricey. You need to rate limit and rotate both IP and puppeteer fingerprint when adversarially scraping.
mindslight 4y ago
> I think it's going to be harder for academic researchers to get institutional approval to scrape adversarially This is a good point about what might happen, but it seems worthwhile to address and fix directly. Personally I don't see why adversarial scraping of a publicly published website should require any more ethical consideration/review than using the suggested API would. Ethical concerns should revolve around humans, not the business desires of non-human entities.

olliej 4y ago

IIRC the twitter website goes to great lengths to mangle itself to prevent ad blockers presumably. Of course this used to have the side effect of breaking significant swathes of basic browser UX, especially in areas of accessibility. I assume they’re better now than they once were, but given musk’s historical behavior I assume he won’t consider breaking something like accessibility to be bad or problematic.

dzhiurgis 4y ago
> musk’s historical behavior I assume he won’t consider breaking something like accessibility to be bad or problematic. I don't see why would he spend more to make website less accessible. Not fixing bugs - yes, just about everyone does so. But breaking it intentionally costs money. Tesla's aren't THE most accessible (by cost) EV out there, but SpaceX def is.
- optionalsquid 4y ago
  Supposedly the entire accessibility team got laid off [1], so there might not be anyone left to ensure that changes to existing features do not reduce accessibility nor ensure that new features are accessible. Not that I'd expect Musk to care a whole lot about concerns raised in that regard, considering that he supposedly went ahead with Twitter Blue despite significant (and well warranted) concerns from the Trust and Safety team [2]. So Twitter will probably become less and less accessible over time. [1] https://www.wired.com/story/twitter-layoffs-accessibility/ https://www.wired.com/story/twitter-layoffs-accessibility/ [2] https://www.businessinsider.com/twitter-sent-musk-risks-paid-verification-before-launch-2022-11 https://www.businessinsider.com/twitter-sent-musk-risks-paid...
- dragonwriter 4y ago
  > Tesla's aren't THE most accessible (by cost) EV out there, but SpaceX def is. SpaceX is... an accessible EV? What?
  - kingofpandora 4y ago
    I'm pretty sure you know exactly what the poster meant ...

dmonitor 4y ago

I think it’s a terrible move to kill the stuff like postybirb that a lot of people use to make posts on multiple websites at once. they’ll have to either change their workflow to make a tweet (annoying) or just abandon twitter altogether. just makes the website worse for practically no benefit.