5 ms·

Why Twitter’s decision on Scraperwiki is bad for data democracy

31 points by frabcus 12y ago

wslh 12y ago
This issue is just an instance of a big problem: there is not such thing as a data democracy, only a theory. It's not about Scraperwiki only, the SLA for the free Twitter API is very restrictive (i.e: number of calls per hour). Twitter (and others) works like a cartel, imagine that you want to pay to increase the number of API calls per hour, you don't have a straightforward way to do it. Twitter has a lot to learn from what Google did almost a decade ago: make it easy to pay and work with your service instantly (AdWords).
- theworst 12y ago
  A little off topic, but I'm curious where the HN crowd lies on the ethics of scraping. I'm a bit on the copyleft side -- I say, if something is made public, it's fair game. If you want to protect your data from being scraped, you very easily indicate that by putting it behind a login. But in my opinion, anything google is allowed to access and put in the index, I can access.
  - wslh 12y ago
    I think at the startup level you can "hack the system". The issue comes when you grow.
    - theworst 12y ago
      As a practical, or ethical issue? IE, is it just as "wrong" in your opinion to scrape sites, regardless of size or profitability? Or are you espousing the practical argument that it's fine because you can get away with it when flying under the radar?
zAy0LfpBZLC8mAC 12y ago
What, a centralized service is not good for democracy? Who would have guessed!
markbnj 12y ago
Any links to information about the decision? Googling the two company names didn't turn up much. Has this been publicized?
- chippy 12y ago
  Here is a link from ScraperWiki from the article: https://blog.scraperwiki.com/2014/08/the-story-of-getting-twitter-data-and-its-missing-middle/ https://blog.scraperwiki.com/2014/08/the-story-of-getting-tw...
  - markbnj 12y ago
    Thanks!
angersock 12y ago
Or, you know, you could learn to code. That would open up a myriad of opportunities for you, and help prevent this sort of loss of independence in the future.
- mcherm 12y ago
  I really don't think that will help. It is not that Twitter is insisting that people stop using (unlimited, unmetered) scraping and move to (unlimited, unmetered) APIs. Rather, it is that Twitter has stopped offering unlimited, unmetered APIs and is now charging for use.
  - angersock 12y ago
    I agree that that is the real problem--that said, the article took the stance of "Why is Twitter being so much nicer to developers than non-coders?"
Patrick_Devine 12y ago
Couldn't Scraperwiki release a tool which allowed anyone to scrape Twitter and have it upload the data back to Scraperwiki? Would that get around the SLA?
- frabcus 12y ago
  Probably not. See Twitter's response to this blog post of somebody trying to do something similar: http://mashe.hawksey.info/2011/03/export-twitter-followers/ http://mashe.hawksey.info/2011/03/export-twitter-followers/ Even if we could, the users would have to register a Twitter developer application, making the user experience much worse than just OAuth, for no good reason.
metaphorm 12y ago
I don't get it. Is the author under the impression that Twitter data is some sort of essential common resource? That's ridiculous. Its private and and its for sale by a for-profit company. This isn't about "data democracy". Thats a weasel word the author used to try and sound persuasive despite his weak argument. This is about the author wanting to have cheaper access to Twitter data and whining about how much it will cost him to get that data.
- frabcus 12y ago
  Hiya! Francis from ScraperWiki here. There are a couple of things going on: Firstly, it is about inconsistency between how developers are treated and how people-who-can-use-Excel are treated. Secondly, it is that vetting of "data use" is so hard, it means low end users just can't buy it at all (financial and red tape reasons). Even if they are happy with the per-Tweet cost. Speaking for ScraperWiki, we want Twitter to make more money from their data. More details here: https://blog.scraperwiki.com/2014/08/the-story-of-getting-twitter-data-and-its-missing-middle/ https://blog.scraperwiki.com/2014/08/the-story-of-getting-tw...
  - metaphorm 12y ago
    Twitter seems to have a strategy for monetizing their data that is at odds with your company's strategy. This is not about "data democracy", its just a business dispute. You're making it sound like its ideological but its not. You want something from them and they said no. That's all.

31 points by frabcus 12y ago

wslh 12y ago

This issue is just an instance of a big problem: there is not such thing as a data democracy, only a theory. It's not about Scraperwiki only, the SLA for the free Twitter API is very restrictive (i.e: number of calls per hour). Twitter (and others) works like a cartel, imagine that you want to pay to increase the number of API calls per hour, you don't have a straightforward way to do it. Twitter has a lot to learn from what Google did almost a decade ago: make it easy to pay and work with your service instantly (AdWords).

theworst 12y ago
A little off topic, but I'm curious where the HN crowd lies on the ethics of scraping. I'm a bit on the copyleft side -- I say, if something is made public, it's fair game. If you want to protect your data from being scraped, you very easily indicate that by putting it behind a login. But in my opinion, anything google is allowed to access and put in the index, I can access.
- wslh 12y ago
  I think at the startup level you can "hack the system". The issue comes when you grow.
  - theworst 12y ago
    As a practical, or ethical issue? IE, is it just as "wrong" in your opinion to scrape sites, regardless of size or profitability? Or are you espousing the practical argument that it's fine because you can get away with it when flying under the radar?

zAy0LfpBZLC8mAC 12y ago

What, a centralized service is not good for democracy? Who would have guessed!

markbnj 12y ago

Any links to information about the decision? Googling the two company names didn't turn up much. Has this been publicized?

chippy 12y ago
Here is a link from ScraperWiki from the article: https://blog.scraperwiki.com/2014/08/the-story-of-getting-twitter-data-and-its-missing-middle/ https://blog.scraperwiki.com/2014/08/the-story-of-getting-tw...
- markbnj 12y ago
  Thanks!

angersock 12y ago

Or, you know, you could learn to code. That would open up a myriad of opportunities for you, and help prevent this sort of loss of independence in the future.

mcherm 12y ago
I really don't think that will help. It is not that Twitter is insisting that people stop using (unlimited, unmetered) scraping and move to (unlimited, unmetered) APIs. Rather, it is that Twitter has stopped offering unlimited, unmetered APIs and is now charging for use.
- angersock 12y ago
  I agree that that is the real problem--that said, the article took the stance of "Why is Twitter being so much nicer to developers than non-coders?"

Patrick_Devine 12y ago

Couldn't Scraperwiki release a tool which allowed anyone to scrape Twitter and have it upload the data back to Scraperwiki? Would that get around the SLA?

frabcus 12y ago
Probably not. See Twitter's response to this blog post of somebody trying to do something similar: http://mashe.hawksey.info/2011/03/export-twitter-followers/ http://mashe.hawksey.info/2011/03/export-twitter-followers/ Even if we could, the users would have to register a Twitter developer application, making the user experience much worse than just OAuth, for no good reason.

metaphorm 12y ago

I don't get it. Is the author under the impression that Twitter data is some sort of essential common resource? That's ridiculous. Its private and and its for sale by a for-profit company. This isn't about "data democracy". Thats a weasel word the author used to try and sound persuasive despite his weak argument. This is about the author wanting to have cheaper access to Twitter data and whining about how much it will cost him to get that data.

frabcus 12y ago
Hiya! Francis from ScraperWiki here. There are a couple of things going on: Firstly, it is about inconsistency between how developers are treated and how people-who-can-use-Excel are treated. Secondly, it is that vetting of "data use" is so hard, it means low end users just can't buy it at all (financial and red tape reasons). Even if they are happy with the per-Tweet cost. Speaking for ScraperWiki, we want Twitter to make more money from their data. More details here: https://blog.scraperwiki.com/2014/08/the-story-of-getting-twitter-data-and-its-missing-middle/ https://blog.scraperwiki.com/2014/08/the-story-of-getting-tw...
- metaphorm 12y ago
  Twitter seems to have a strategy for monetizing their data that is at odds with your company's strategy. This is not about "data democracy", its just a business dispute. You're making it sound like its ideological but its not. You want something from them and they said no. That's all.