6 ms·
Show HN: Download Twitter data without API keys
In April last year I started thinking about using Twitter in a smarter way. I wanted to do analytics on my tweets and find out more about people following me on Twitter. What kinds of things do people who follow me like and retweet? I decided to dig into the data and find out.
When I went to try and download Twitter data in raw form I found I quickly got bogged down writing API wrangling code and fiddling with API keys. I just wanted to crunch some data but here I was wrangling Twitter's API. This was such a frustrating experience it suddenly looked like an opportunity to me. Was there room for a product here? A product which does one simple thing: help people extract Twitter data painlessly without writing any code.
I did some research and discovered some tools that purported to do this, but none of them worked well for my use-case and all were badly designed and/or expensive. I decided to take a shot at it.
I worked on this as a side project in a "calm company" fashion. Each week for 26 weeks I would put aside one day to chip away at the features. I tried not to think about how the issue tracker was filling up more and more. Several times I pared back the feature set to try to really focus on the core use-case.
When my first user reached out and engaged I knew I might be onto something. I kept posting progress on Twitter and a few more people started to use it each week. Some of them came back, hinting at possible user retention. I hired a writer to write some articles to help with SEO and I kept working towards and MVP that I could use to test the market.
Finally the day arrived where all of the critical issues in my issue tracker were closed. That meant it was launch day. That was yesterday. So here I am releasing this on Hacker News to you, dear reader.
God speed little micro-SaaS, may the winds of fortune be at your back.
- deleted 5y ago[deleted]
- michaelmior 5y agoI'm not too familiar with the Twitter terms of service, but I would be surprised if selling access to their data wasn't a clear violation.
- PeterBarrett 5y agoWell they're not selling access to data, the data is freely available. They're selling a service that packages that data for you.
- vasco 5y agoData which probably was obtained through the API which from my interpretation of https://developer.twitter.com/en/developer-terms/agreement-and-policy https://developer.twitter.com/en/developer-terms/agreement-a... is not permitted.
- throwaway2016a 5y agoIf obtained through scraping vs the API it also isn't allowed via the regular ToS https://twitter.com/en/tos https://twitter.com/en/tos and their robots.txt So either way sounds like a ToS violation. Which Twitter is usually pretty aggressive about shutting down.
- altdataseller 5y agoI can’t see how this is true as there are tons of companies selling access to data obtained from the Twitter API such as social media monitoring companies (Sprinklr, Meltwater)
- simonw 5y agoThose companies have business development deals in place with Twitter that authorize them to do this.
- altdataseller 5y agoThere are plenty of other non-enterprisey companies that sell Twitter data as well* *I was a founder of one of them.
- chr15m 5y ago
- iKlsR 5y agoWhat assurances can you give that you will be around in a year because I think this could get crippled easily regardless of how you are doing the fetching on the backend (scraping, api, nitter). I'd love something like this to not have to wrangle with the api myself but it's giving uneasy vibes for longevity. Edit: I tried it, got the following for "tweets from a search result" Error: Request failed with code 429 Request exceeds account’s current package request limits. Please upgrade your package and retry or contact Twitter about enterprise access.
- nannal 5y agoSo they really were trying to sell access to the content you could fetch with a twitter api key?
- chr15m 5y agoYes that's correct. The value proposition of the site is a nice UI and access to Twitter data that does not require writing code or using a Twitter API key.
- mritchie712 5y agoThis will be gone in days if anyone uses it, it's clearly against the terms.
- chr15m 5y agoYou are right, API access has been suspended for my app. Do you know which rule or term the app violates? I am not able to find it in any of the documentation I read.
- chr15m 5y agoYour uneasy vibes for longevity turned out to be prescient. Less than 24 hours after launch Twitter suspended the account and locked my app's API access. I have appealed the suspension as I genuinely do not understand which rule the site violates, but I don't like the chances of getting it re-enabled. Thank you for checking out the project and taking the time to write a comment, much appreciated.
- meigwilym 5y agoGreat idea (despite the dubious legality). Perhaps a section on what can be done with the data? Your potential clients are probably the type of users who don't know why they'd need such a tool. And an explanation of the sentiment analysis?
- chr15m 5y agoThat's a great idea. If my app survives suspension I will do this. Thank you for the suggestion.
- spupe 5y agoWhat is the advantage of your implementation versus Twint, which is free?
- throwawayrev3 5y agoTwint's last commit was on March 2021.
- chr15m 5y agoFrom the Twint "install" documentation: > pip3 install --user ... > git clone --depth=1 ... > Noticed a lot of people are having issues installing (including me). Please use the Dockerfile temporarily while I look into them. Most people in the world don't know what git is, what a Dockerfile is or how to use the command line or Python. The advantage of my implementation is you don't have to know those things to get data out of the Twitter API. You can simply sign in with Twitter and you get a nice user interface for downloading data from the API without having to write a line of code.
- rank0 5y agoI like the spirit of this, but the monetization strategy rubs me the wrong way. You're basically reselling a UI wrapper for the twitter API. Nevertheless, good luck and I hope they don't shut you down too fast.
- ugjka 5y ago3... 2... 1... cease & desist
- deleted 5y ago[deleted]
- chr15m 5y agoYes, the account has been suspended. I have lodged an appeal as I do not understand which of their terms has been violated.
- chr15m 5y agoYes, I am selling a UI wrapper for the Twitter API. Why does that rub you up the wrong way? Many people do not have the privilege of being able to write code to interface with the Twitter API. I thought these people might appreciate a way to get Twitter data from the API anyway. It turns out some of them did but unfortunately as you predicted, access was shut down quickly.
- antisthenes 5y ago> I just wanted to crunch some data but here I was wrangling Twitter's API. > I worked on this as a side project in a "calm company" fashion. Each week for 26 weeks I would put aside one day to chip away at the features. So you wanted to crunch some data...and instead of spending 3-4 hours setting up the API properly...you instead spent 26 days writing an unofficial hack? It seems like what you really wanted was to spend 26 days on a coding hobby, not actually crunching some data.
- chr15m 5y agoCorrect. I wanted to spend 26 days on my bootstrapping hobby, not actually crunching data. I did that and this app is the result.
- new_stranger 5y ago> Up to 10k rows of data. uh, how is 10k tweets (1 second of data?) enough?
- chr15m 5y agoFor many use-cases 10k rows is more than enough. For example, most people have fewer than 10k followers and downloading a CSV of your followers is something many people want to do.
- donkarma 5y agotwint does this for free
- throwawayrev3 5y agoIt's last commit was on March 2021. Is the project abandoned?
- torbTurret 5y agoThat’s not even that long ago? What?
- throwawayrev3 5y agoIdk. I just genuinely wanted to know. Because there are many open issues. There is even an issue asking whether it's abandoned. I was thinking because it's twitter and it doesn't use an API, twitter might introduce breaking changes and Twint has to catch up often. Just like NewPipe, yt-dlp etc does for YouTube.
- chr15m 5y agoTwint is an amazing piece of software. It seems to be primarily for technical users who are able to use git, pip, Python, command line, and Docker. TweetFeast is for a different type of user who just wants a user interface to get the Twitter data they want.
- ryankrage77 5y agoCharging for web scraping feels scummy somehow. Can't quite put my finger on why, since I wouldn't feel guilty about scraping twitter myself.
- chr15m 5y agoThe app does not "scrape". It uses the Twitter API to access the data and packages it up in a nice format (CSV or JSON). There are literally hundreds of tools, paid and free, which do this for various platforms on the internet.
- accountofme 5y agoColour me curious, but why do I need to sign in with twitter to use this? And if I do and you get c&d'd do I get blocked by twitter? Edit: spelling
- chr15m 5y agoYou need to be signed in as the app makes the data requests on your behalf. My personal account was logged in and I did not get blocked by Twitter when they suspended the app account, so it seems unlikely anybody else would be.
- Zeik0s 5y agoYou were banned for either one or both of the following things: 1. Violating II. Restrictions on Use of Licensed Materials of the Developers Policy => A => ) provide use of the Twitter API on a service bureau, rental or managed services basis or permit other individuals or entities to create links to the Twitter API or "frame" or "mirror" the Twitter API on any other server, or wireless or Internet-based device, or otherwise make available to a third party, any token, key, password or other login credentials to the Twitter API; 2. violating the security model of the Twitter API, because with your Service Twitter is no longer in control of who can access their data over the API.
- chr15m 5y agoThank you very much for finding this. It seems the violation is this past specifically: > provide use of the Twitter API on a service bureau, rental or managed services basis