7 ms·

Have some fairly strong feelings about how much of a bad idea this is for twitter in the long run - posted here: http://www.3scale.net/2015/04/loosing-innovatio

by njyx 12y ago

Have some fairly strong feelings about how much of a bad idea this is for twitter in the long run - posted here: http://www.3scale.net/2015/04/loosing-innovation-twitters-firehose-mistake/ http://www.3scale.net/2015/04/loosing-innovation-twitters-fi... I have no idea if DataSift and NTT were good partners, but not having anyone else access the Firehose suggests they think they can cover all significant use-cases. This almost always turns out to be wrong. It also sends another chilling signal to say "we're a platform, but when we feel we want to roll over part of the ecosystem we'll do it".

danmaz74 12y ago
Twitter is still selling data to analysis companies, they are "only" forcing them to buy the data directly from Twitter (aka Gnip).
MichaelGG 12y ago
It's not like Twitter can't offer some slower Firehose or more advanced filtering or whatever. Or offer some EC2 images to spin up to handle it. It's only 500M items a day. At that level you've got plenty of options as far as hardware and software. (You could even record it into a normalass SQL DB to log each one if you wanted.) You may need some useful buffering for high spikes, perhaps some message queues with lots of RAM.
- ultimape 12y ago
  This made me giggle
  - MichaelGG 12y ago
    Why? A couple years ago I was running a telecom, and we had around that volume of calls a day. Each call required not only routing instructions that hit several DBs, but also a transactional INSERT/UPDATE to account balances. Single node SQL Server handled it fine. (Though it was less than optimal at doing non-OLTP stuff.) Twitter can handle this for people (like DataSift, providing historical lookback), or they can provide template to help them spin up. It's not gonna cost thousands of dollars a month in hardware.
clay_to_n 12y ago
Gnip was DataSift's biggest competitor, and Twitter acquired Gnip. So firehose access is still totally available, but only through Twitter itself (b/c of the Gnip acquisition). When the big company you relies on buys your competitor, you should be very, very scared.
- bvm 12y ago
  i heard a (possibly apocryphal) explanation of that acqusition that had something do with a forgotten invoice on Twitter's part.

by njyx 12y ago

danmaz74 12y ago

Twitter is still selling data to analysis companies, they are "only" forcing them to buy the data directly from Twitter (aka Gnip).

MichaelGG 12y ago

It's not like Twitter can't offer some slower Firehose or more advanced filtering or whatever. Or offer some EC2 images to spin up to handle it. It's only 500M items a day. At that level you've got plenty of options as far as hardware and software. (You could even record it into a normalass SQL DB to log each one if you wanted.) You may need some useful buffering for high spikes, perhaps some message queues with lots of RAM.

ultimape 12y ago
This made me giggle
- MichaelGG 12y ago
  Why? A couple years ago I was running a telecom, and we had around that volume of calls a day. Each call required not only routing instructions that hit several DBs, but also a transactional INSERT/UPDATE to account balances. Single node SQL Server handled it fine. (Though it was less than optimal at doing non-OLTP stuff.) Twitter can handle this for people (like DataSift, providing historical lookback), or they can provide template to help them spin up. It's not gonna cost thousands of dollars a month in hardware.

clay_to_n 12y ago

Gnip was DataSift's biggest competitor, and Twitter acquired Gnip. So firehose access is still totally available, but only through Twitter itself (b/c of the Gnip acquisition). When the big company you relies on buys your competitor, you should be very, very scared.

bvm 12y ago
i heard a (possibly apocryphal) explanation of that acqusition that had something do with a forgotten invoice on Twitter's part.