3 ms·
I learned a lot about machine learning in AI courses the past year and a half in college, and really wanted to apply it somewhere outside of the classroom. This
by dangrossman 17y ago
I learned a lot about machine learning in AI courses the past year and a half in college, and really wanted to apply it somewhere outside of the classroom. This was a one-day project app, so it's really functionally simple. It looks at your Twitter follower and following list, computes the difference, decides if each of those people is "spam" or "ham", then recommends which you are safe to follow and which you should block. Since the API calls can take a while, it does it all with a background worker and e-mails you when the results are ready, rather than doing the work synchronously while you wait at a browser.
The hard part was getting ready for it. I decided to do a very basic spam filter for Twitter, one which looked only at the profile information, and not the text of anyone's tweets, since that'd require too many API calls without special access beyond even a whitelisted API account. I wrote some scripts that would use Twitter's search API to find tweets directed at @spam for a week and a half. It took that long because looking at old tweets would yield accounts Twitter had already suspended, meaning I couldn't get any info about them through the API. I had to get the reports while they were fresh and download the profile info of known spammers to build a training corpus. For the "ham" examples I used the following lists of trusted friends with various interests.
I used that corpus to build and prune a decision tree, which tested to around 99% precision/recall accuracy with 2000 ham and 2000 spam examples. Then I manually rewrote the tree as simple conditionals in PHP to create the classifier for the website.
Note: I'm about to hit my rate limit (on a whitelisted account, so 20k calls) in only half an hour...
- timdorr 17y agoIf you authorize the requests, you'll get a boost of the individual user's available requests after you run out of whitelist API requests for your IP (20000 + (users * 150)). Also, make sure you're using the api.twitter.com/1/ URL format, as they're going to be upping the rate limit for users to 1000 "soon" (it was announced at Le Web 1.75 months ago...). Also, that only applies to OAuth, so you should get on that boat soon. They're deprecating basic auth this summer.
- dangrossman 17y agoFor many accounts with more followers than they follow, there are more than 150 profiles I need to download. Someone just passed me an account with 6000 followers he's not following back. That's 6k API calls. I thought at first of using the 'followers' timeline' request to get 100 at a time, but the profile information it gives for each follower is abbreviated. I need other information, so I have to make a separate API call for each and every profile I want. Using up someone's entire API usage in a flash while they need it for the rest of the hour to run their Twitter client isn't gonna fly. The best I can do right now is make 20k authorized requests from my whitelisted account, and 20k unauthorized requests from my whitelisted IP. I just built in some status checking on the homepage so it won't let more people add themselves to the queue when it's already overloaded for the hour.
- timdorr 17y agoThat's actually longstanding API request: http://code.google.com/p/twitter-api/issues/detail?id=1078&sort=-stars&colspec=ID%20Stars%20Type%20Status%20Priority%20Owner%20Summary%20Opened%20Modified%20Component http://code.google.com/p/twitter-api/issues/detail?id=1078&#... It's one I would benefit from for my app as well. One thing to do is to cache the user information locally so that when they're requested again, you don't have to make an API call. For your app, I would think spammy users wouldn't change profile details all that often.
- arikfr 17y agoAFAIK, Twitter's API rate limit is per IP + per user (i.e. you can do X api requests per user on each machine). Also, on a whitelisted IP you get 20K requests per user. There was a long discussion about that a few months ago on the dev mailing list, whether this by purpose or mistake. Back then, the official response was that it's by purpose, but verify if that's true. If it is, you have plenty of API calls you can do per user.
- dangrossman 17y agoIt does seem to be. In this case, I have added reason to implement OAuth!