5 ms·
From the Twitter Hadoop to GCP talk that Boulos linked: - >500k cores - >300PB storage - >12,500 cluster size - >1T messages per day And there's also this
by fhoffa 8y ago
From the Twitter Hadoop to GCP talk that
Boulos linked:
- >500k cores
- >300PB storage
- >12,500 cluster size
- >1T messages per day
And there's also this other talk, "How Twitter Migrated its On-Prem Analytics to Google Cloud" - focused on their migration to BigQuery:
- https://www.youtube.com/watch?v=sitnQxyejUg https://www.youtube.com/watch?v=sitnQxyejUg
- 20 TB/ day of raw log data, >100k events/sec
- Loading ~1TB/hour into BigQuery.
- Serving 5,000+ complex queries / second. p99 ~300ms
Disclosure: I'm Felipe Hoffa and I work for Google Cloud https://twitter.com/felipehoffa https://twitter.com/felipehoffa.
- opportune 8y agoAt this scale I'm a little surprised Twitter saved money on the migration. My assumptions were that at this scale the upfront cost of migration trumped any kind of difference in annual maintenance, esp. considering you still do need an SRE team anyway who knows big data well enough to solve any and all software related problems
- Aeolun 8y agoAt that scale the team itself likely costs peanuts.
- opportune 8y agoyeah but you still need the IT team too. Not sure, from my conversations at cough cough other media giants the economics of moving to cloud don't always work out at massive scale
- Aeolun 8y agoOh, certainly. I was talking more about the fact that the monthly cost of 125.000 instances likely beats the cost of your IT team whatever it’s size is. I’d be interested in their cost/benefit analysis for this one.
- opportune 8y agoYeah, what I mean is it's not just cloud costs vs. local hosting costs. You have to come up with different processes for SRE and IT (removing mundane hardware work, accounting for cloud idiosyncrasies), benchmark, make sure your hosting cloud can handle scaling up to some upper bound estimate of your future user base, etc. At a company the size of twitter these become legitimate concerns. I've been throttled on certain services of certain cloud providers at 600 data nodes lol... of course twitter can negotiate higher allowances but massive scaling doesn't happen with the flick of a switch, sometimes you need big, long-term, expensive contracts
- Scarbutt 8y ago12,500, not 125,000. Still a lot ;)
- geggam 8y agoAutomation, the twitter team is full of SV veterans who have run into many problems at scale years ago. At least it used to be.
- gniv 8y ago> 1T messages per day How? I mean there are 7B people on earth. Are people that active on Twitter? Or does "messages" mean something else here?
- pgeorgi 8y agoThere are tons of bots on Twitter
- 24gttghh 8y agoBots, news organizations, and teenagers. ~143 messages/day per human doesn't seem unreasonable.
- regnerba 8y agoA message in this case probably doesn't mean a Tweet. A message if probably just a message on the bus/in the queue and any action on Twitter probably produces one. Like, retweet, tweet, follow, sign up, everything.
- gniv 8y agoOh if the likes are included then it makes sense.
- zjaffee 8y agoA message probably counts every tweet a person views at a minimum.
- JohnJamesRambo 8y agoIt's stunning how many resources we as a species are wasting on something of so little net positive value to humanity as Twitter.
- tyrust 8y agoI'm not a fan of or user of Twitter, but I can't deny that anything that allows people to communicate instantly, easily, and broadly is a benefit to humanity.
- JohnJamesRambo 8y agoSo in your opinion Facebook is a benefit to humanity also? Which conflicts with nearly every study of it ever done?