5 ms·

Twitter archiver: Make your own simple, public, searchable Twitter archive

167 points by aaossa 4y ago

donohoe 4y ago
Sort of a shameless plug, but you can use this PHP script to generate a lazy-loading basic webpage with all your tweets that you can host: https://github.com/donohoe/twitter-archive https://github.com/donohoe/twitter-archive Example: https://donohoe.dev/twitter/ https://donohoe.dev/twitter/
pkulak 4y ago
Man, wish I did this before my account got suspended.
terhechte 4y ago
Another shameless plug: I made a cross platform "native (Rust with Webkit)" desktop app that does the same thing (Win/Lin/Mac). It also allows you to see your "profile" afterwards in the app, search within your Tweets, and allows backing up things like mentions, responses, likes, etc: https://github.com/terhechte/twitvault https://github.com/terhechte/twitvault
- terhechte 4y ago
  Also, the reason I made a Desktop app is that all other tools usually require somebody to know the Terminal, or how to set up a Webserver. Twitvault can also be used by non-tech people to archive their data.
  - notjoemama 4y ago
    Or journalists and activists will use to keep track of everything said on twitter to use in online hate campaigns when the time is right. Forgive me for jumping to malice, it's that I've been conditioned to assume the worst intention of activists and journalists, by activists and journalists.
nixcraft 4y ago
Also, you can archive your Tweets with the Wayback Machine https://help.archive.org/help/how-to-archive-your-tweets-with-the-wayback-machine/ https://help.archive.org/help/how-to-archive-your-tweets-wit...
- ElijahLynn 4y ago
  YES! Thank you, I have been looking for a way to archive my Twitter account. I've been considering deleting mine but this was THE blocker! Thanks.
- joe-collins 4y ago
  Does anyone know of a means of accomplishing this without access to the account? A friend passed away last year, and I would love to archive those memories.
  - thelamest 4y ago
    I hope you’ll get a better answer, but a minimal solution is to use snscrape (https://github.com/JustAnotherArchivist/snscrape https://github.com/JustAnotherArchivist/snscrape) to download a raw JSON dump of a user timeline. Some caveats: 1. This doesn’t work for all users (e.g. some accounts seem to remain deindexed after unsuspensions). 2. While you can get rid of 90% of the file size by removing fluff columns (e.g. related to processing media and emojis), keep a backup. When importing such JSON, I unwittingly did a lossy data type conversion, and it can be irreparable once the tweets are deleted. $ snscrape --progress --jsonl twitter-user jack > jack.json [PS With courtesy rate limiting, depending on the account size, you may then script a way to send each tweet URL for archival in the Wayback Machine.]
- nb_key 4y ago
  waybachmachine is a bless these days.
deleted 4y ago
[deleted]
cobertos 4y ago
Anyone know how to download a conversation? I want to mirror my tweets to my website with the comments. It doesn't come in the Twitter data export. It's also impossible to get a tweets comments from the API as a normal user? (> 7 days of conversation threading only available for academic institutions and enterprise)
- edent 4y ago
  You need to use one of the leaked API keys for Twitter's app. That will get you the ability to call the privileged APIs.
  - cobertos 4y ago
    Hmm, just tried that. Had to switch to the v1.1 API (v2 does not work with the extracted API keys, you get a `Sorry, official Twitter clients cannot use this API. Register your own.` error), but it still doesn't have access to the premium search API. You get a `Forbidden: Authentication succeeded but account is not authorized to access this resource.` error. Tried some other third-party leaked API keys as well and no luck. Standard search, even for the app keys, I'm pretty sure only gives you 7 days back (recent tweets)
    - cobertos 4y ago
      Figured it out. I found if you make a new developer account and application, you can fill out the Elevated credentials form and you will be automatically accepted (no waiting or manual process). This will get you access to the premium v1.1 API sandbox. You get 50 queries in there against the full history endpoint, which I used to grab all tweets replies to my account since the beginning. No recursive replies but meh, better than nothing
RobertoG 4y ago
Is there an easy way of downloading the favorites?
- jchook 4y ago
  v1 of the API only gives access to the last 3200 favs. v2 of the API improves on this a good bit, but I haven't tested it thoroughly yet.
- nikvdp 4y ago
  shameless plug, but try http://birdbear.app http://birdbear.app
- d4rkp4ttern 4y ago
  Also wondered the same. Even more useful might be a way to download all tweets from certain lists.
amatecha 4y ago
Nice haha remember when you could search for something on Twitter and it actually returned what you typed in? I've tried so many times to find an old tweet of mine and had no luck. Needless to say I recently grabbed an archive of my data so I can just refer back to that, and never need to go to the sub-par Twitter website anymore.
tmpburning 4y ago
The NSA should be required to give you all public information that they have on you... it would be a convenient place to get everything.

167 points by aaossa 4y ago

Sort of a shameless plug, but you can use this PHP script to generate a lazy-loading basic webpage with all your tweets that you can host: https://github.com/donohoe/twitter-archive https://github.com/donohoe/twitter-archive Example: https://donohoe.dev/twitter/ https://donohoe.dev/twitter/

pkulak 4y ago

Man, wish I did this before my account got suspended.

terhechte 4y ago

Another shameless plug: I made a cross platform "native (Rust with Webkit)" desktop app that does the same thing (Win/Lin/Mac). It also allows you to see your "profile" afterwards in the app, search within your Tweets, and allows backing up things like mentions, responses, likes, etc: https://github.com/terhechte/twitvault https://github.com/terhechte/twitvault

terhechte 4y ago
Also, the reason I made a Desktop app is that all other tools usually require somebody to know the Terminal, or how to set up a Webserver. Twitvault can also be used by non-tech people to archive their data.
- notjoemama 4y ago
  Or journalists and activists will use to keep track of everything said on twitter to use in online hate campaigns when the time is right. Forgive me for jumping to malice, it's that I've been conditioned to assume the worst intention of activists and journalists, by activists and journalists.

nixcraft 4y ago

Also, you can archive your Tweets with the Wayback Machine https://help.archive.org/help/how-to-archive-your-tweets-with-the-wayback-machine/ https://help.archive.org/help/how-to-archive-your-tweets-wit...

ElijahLynn 4y ago
YES! Thank you, I have been looking for a way to archive my Twitter account. I've been considering deleting mine but this was THE blocker! Thanks.
joe-collins 4y ago
Does anyone know of a means of accomplishing this without access to the account? A friend passed away last year, and I would love to archive those memories.
- thelamest 4y ago
  I hope you’ll get a better answer, but a minimal solution is to use snscrape (https://github.com/JustAnotherArchivist/snscrape https://github.com/JustAnotherArchivist/snscrape) to download a raw JSON dump of a user timeline. Some caveats: 1. This doesn’t work for all users (e.g. some accounts seem to remain deindexed after unsuspensions). 2. While you can get rid of 90% of the file size by removing fluff columns (e.g. related to processing media and emojis), keep a backup. When importing such JSON, I unwittingly did a lossy data type conversion, and it can be irreparable once the tweets are deleted. $ snscrape --progress --jsonl twitter-user jack > jack.json [PS With courtesy rate limiting, depending on the account size, you may then script a way to send each tweet URL for archival in the Wayback Machine.]
nb_key 4y ago
waybachmachine is a bless these days.

deleted 4y ago

[deleted]

cobertos 4y ago

Anyone know how to download a conversation? I want to mirror my tweets to my website with the comments. It doesn't come in the Twitter data export. It's also impossible to get a tweets comments from the API as a normal user? (> 7 days of conversation threading only available for academic institutions and enterprise)

edent 4y ago
You need to use one of the leaked API keys for Twitter's app. That will get you the ability to call the privileged APIs.
- cobertos 4y ago
  Hmm, just tried that. Had to switch to the v1.1 API (v2 does not work with the extracted API keys, you get a `Sorry, official Twitter clients cannot use this API. Register your own.` error), but it still doesn't have access to the premium search API. You get a `Forbidden: Authentication succeeded but account is not authorized to access this resource.` error. Tried some other third-party leaked API keys as well and no luck. Standard search, even for the app keys, I'm pretty sure only gives you 7 days back (recent tweets)
  - cobertos 4y ago
    Figured it out. I found if you make a new developer account and application, you can fill out the Elevated credentials form and you will be automatically accepted (no waiting or manual process). This will get you access to the premium v1.1 API sandbox. You get 50 queries in there against the full history endpoint, which I used to grab all tweets replies to my account since the beginning. No recursive replies but meh, better than nothing

RobertoG 4y ago

Is there an easy way of downloading the favorites?

jchook 4y ago
v1 of the API only gives access to the last 3200 favs. v2 of the API improves on this a good bit, but I haven't tested it thoroughly yet.
nikvdp 4y ago
shameless plug, but try http://birdbear.app http://birdbear.app
d4rkp4ttern 4y ago
Also wondered the same. Even more useful might be a way to download all tweets from certain lists.

amatecha 4y ago

Nice haha remember when you could search for something on Twitter and it actually returned what you typed in? I've tried so many times to find an old tweet of mine and had no luck. Needless to say I recently grabbed an archive of my data so I can just refer back to that, and never need to go to the sub-par Twitter website anymore.

tmpburning 4y ago

The NSA should be required to give you all public information that they have on you... it would be a convenient place to get everything.