5 ms·
You know, back before APIs were all the rage, people wrote clients for web services by scraping. Twitter really wouldn't be able to do anything about a Twitter
by quotemstr 14y ago
You know, back before APIs were all the rage, people wrote clients for web services by scraping. Twitter really wouldn't be able to do anything about a Twitter client that pretended to be IE9.
- artursapek 14y agoI've had a bot blocked by Myspace when I resorted to that. They started redirecting all my requests to " http://google.com http://google.com ". And I was just working on an old Thinkpad running Ubuntu server in my dorm room, so I have to think blocking me was an automated response. I'm sure Twitter knows how to block crawlers, especially ones that they're actually aware of.
- quotemstr 14y agoI'm sure Twitter would block bots doing unuserlike things. But isn't the point of a Twitter client to do userlike things?
- artursapek 14y agoI suppose, and I suppose if the client used a user's IP they couldn't block them the same way. But wouldn't a client that scrapes the full HTML to get the data you want be horribly inefficient, and hard to maintain? API's are all the rage because they give you just what you want. Think about bandwidth.
- wccrawford 14y agoHaving written scrapers in the past, I can tell you that changes are usually pretty easy to deal with. And if you write your scraper nicely, many small changes won't even matter. Since the client runs on the user's machine, performance isn't really an issue, either. (Trying to do it for all users on a server is a different story.) APIs are indeed "all the rage" because they're easier and don't change as much. But when given no other option... You do what you have to.
- halter73 14y agoI think the obvious solution is to have the client first check for updated scraping rules from a server operated by the client's creator. The client could check for these scraping rule updates every time the client starts up or on some interval (or maybe after certain errors even) and only pull down the whole thing when the cache is invalidated. Then, as you suggest, the client could do the actual scraping which would make it harder for twitter to block. Not as straightforward as an API to be sure, but not impossible by a long shot.
- eli 14y agoTo what end? You're going to base a business on scraping twitter.com? That sounds like a much worse idea than trying to work within the API.
- quotemstr 14y ago> You're going to base a business on scraping twitter.com? A business? Probably too high risk. A nice FOSS client, though? Why not?
- spoonmonkeyuk 14y agoWhy not? Because of liability to law suits for breach of the terms of service or copyright violation. Third parties have been sued for scraping in the past.
- aGHz 14y agoYou seem to have gotten downvoted without comment, so let me chime in. He's proposing a FOSS client. There's no third party here, the client is using this software to breach their own ToS (supposing there's a clause in there by which you're only allowed to access twitter.com from a browser).
- xentronium 14y agoConsidering how much javascript is involved in current twitter and how often can subtle things in markup change, I'd say it's just too brittle to rely on.
- jowiar 14y agoJavascript-filled sites are often cleaner to scrape. Usually, at some point in the process, a nice JSON structure containing all the useful data exists.
- dannyr 14y agoThis would only work for the public timeline. Most people would want to view their own timeline and that would be hard to since you'll have to create a browser instance for every user you have.
- pavel_lishin 14y agoBut a client would just run on their machine. Save cookies, and you're fine.
- timhaines 14y agoTwitter would blacklist the IP(s).
- LiveFanChat 14y agobut when you run the app on the end user machines they cant, its the same reason why they couldnt block MyTwitterButler