5 ms·
I had the chance to test it out during the Oscars [1], receiving about 1.2M tweets via Twitter Streaming API in 6 hours, publishing them in Pub/Sub, pulling wit
by ecesena 12y ago
I had the chance to test it out during the Oscars [1], receiving about 1.2M tweets via Twitter Streaming API in 6 hours, publishing them in Pub/Sub, pulling with another process and storing in BigQuery. I still need to find time to blog about the experience, happy to collect a few extra questions here.
[1] http://ecesena.github.io/oscars2015/ http://ecesena.github.io/oscars2015/
- neumino 12y agoHow did you collect the data? Did you just use Twitter stream API with track?
- gtaylor 12y agoDid you get any feel for the latency between publishing and consumption? I'm definitely interested, but I'd be looking for no more than a few second delay between publish and potential consumption.
- ecesena 12y agoWe didn't collect precise timing data, but I hope to release the code soon, and it's pretty easy to setup a test similar to ours, e.g. with high volume hashtags such as #nowplaying. What I can tell you is that we published messages in batches, closing a batch every 5 seconds or 100 tweets (at the highest traffic rate, we were receiving and publishing about 100 tweets every 1-2 seconds). This is way below the quota limits. On the subscriber side, we received batches of at most 10 messages. I didn't investigate if this is a limit or a config param. The sensation was to receive the messages pretty much instantly (I was looking in parallel at 2 shells, one publishing and one receiving), but again, I don't have precise measurements.
- erikrothoff 12y agoHow much did it cost you?
- ecesena 12y agoPub/Sub is currently free. The whole experiment including 2 days of tests, between AppEngine, compute nodes and BigQuery was about 10$.