15 ms·

Disclosure: I work at Google Cloud (and directly with Derek and the Twitter team). I’ll try to edit this page tomorrow when I’m at a computer, but there’s much

by boulos 8y ago

Disclosure: I work at Google Cloud (and directly with Derek and the Twitter team). I’ll try to edit this page tomorrow when I’m at a computer, but there’s much more information in Derek’s talk at NEXT [1]. They (rightfully) didn’t want to get into a detailed “this is what we saw on <X>”, but Derek alludes to their careful benchmarking across providers. While you should always assume smart people make economically reasonable decisions, Derek’s point about savings is about list price differences that result from total system performance (and not any sort of special discounting). I’m hoping a follow-up talk will let us say more about how the migration is going, while this talk was focused on the decision to move part of (!) their Hadoop environment to GCP. [1] https://m.youtube.com/watch?v=4FLFcWgZdo4 https://m.youtube.com/watch?v=4FLFcWgZdo4

fhoffa 8y ago
From the Twitter Hadoop to GCP talk that Boulos linked: - >500k cores - >300PB storage - >12,500 cluster size - >1T messages per day And there's also this other talk, "How Twitter Migrated its On-Prem Analytics to Google Cloud" - focused on their migration to BigQuery: - https://www.youtube.com/watch?v=sitnQxyejUg https://www.youtube.com/watch?v=sitnQxyejUg - 20 TB/ day of raw log data, >100k events/sec - Loading ~1TB/hour into BigQuery. - Serving 5,000+ complex queries / second. p99 ~300ms Disclosure: I'm Felipe Hoffa and I work for Google Cloud https://twitter.com/felipehoffa https://twitter.com/felipehoffa.
- opportune 8y ago
  At this scale I'm a little surprised Twitter saved money on the migration. My assumptions were that at this scale the upfront cost of migration trumped any kind of difference in annual maintenance, esp. considering you still do need an SRE team anyway who knows big data well enough to solve any and all software related problems
  - Aeolun 8y ago
    At that scale the team itself likely costs peanuts.
    - opportune 8y ago
      yeah but you still need the IT team too. Not sure, from my conversations at cough cough other media giants the economics of moving to cloud don't always work out at massive scale
      - Aeolun 8y ago
        Oh, certainly. I was talking more about the fact that the monthly cost of 125.000 instances likely beats the cost of your IT team whatever it’s size is. I’d be interested in their cost/benefit analysis for this one.
        opportune 8y ago
        Yeah, what I mean is it's not just cloud costs vs. local hosting costs. You have to come up with different processes for SRE and IT (removing mundane hardware work, accounting for cloud idiosyncrasies), benchmark, make sure your hosting cloud can handle scaling up to some upper bound estimate of your future user base, etc. At a company the size of twitter these become legitimate concerns. I've been throttled on certain services of certain cloud providers at 600 data nodes lol... of course twitter can negotiate higher allowances but massive scaling doesn't happen with the flick of a switch, sometimes you need big, long-term, expensive contracts
        Scarbutt 8y ago
        12,500, not 125,000. Still a lot ;)
  - geggam 8y ago
    Automation, the twitter team is full of SV veterans who have run into many problems at scale years ago. At least it used to be.
- gniv 8y ago
  > 1T messages per day How? I mean there are 7B people on earth. Are people that active on Twitter? Or does "messages" mean something else here?
  - pgeorgi 8y ago
    There are tons of bots on Twitter
  - 24gttghh 8y ago
    Bots, news organizations, and teenagers. ~143 messages/day per human doesn't seem unreasonable.
  - regnerba 8y ago
    A message in this case probably doesn't mean a Tweet. A message if probably just a message on the bus/in the queue and any action on Twitter probably produces one. Like, retweet, tweet, follow, sign up, everything.
    - gniv 8y ago
      Oh if the likes are included then it makes sense.
  - zjaffee 8y ago
    A message probably counts every tweet a person views at a minimum.
- JohnJamesRambo 8y ago
  It's stunning how many resources we as a species are wasting on something of so little net positive value to humanity as Twitter.
  - tyrust 8y ago
    I'm not a fan of or user of Twitter, but I can't deny that anything that allows people to communicate instantly, easily, and broadly is a benefit to humanity.
    - JohnJamesRambo 8y ago
      So in your opinion Facebook is a benefit to humanity also? Which conflicts with nearly every study of it ever done?
omeid2 8y ago
It is interesting point to bring up but nonetheless discount pricing is not out of question. It is possible that while GCP gives a good value based on the writing on the tin, someone else might beat that with a good margin under special terms. That being said, I have reasons to think very few could beat Google on compute or storage game and definitely much harder to beat the big G on Networking side of things.
- jkaplowitz 8y ago
  Companies like Twitter would get a significant discount from any provider, though I imagine AWS as market leader would have less incentive than GCP to go quite as deep. Disclosure: I worked for GCP until early 2015, but I haven't worked for Google since then, have no inside info on this deal and am not relying on anything from my time at Google for this comment.
dijit 8y ago
FWIW I’m not a big cloud fan, but I was tasked with the obligation of finding the “least worst” cloud provider in terms of predictable throughout and total cost (inc dev/ops time) and GCP came out as a clear winner despite heavy financial incentives from both Azure and AWS. (We’re a very large corp). Due to our new allegiance to google cloud I’ve been given a little more privileged access to engineers (after the fact) and I can tell that I definitely made the right choice, the people I spoke to favoured having a clean/clear backend with actual quality of life features; but since they don’t go nicely into feature comparison charts people often think that GCP is less mature or featureful. I’m a real convert, and our company uses all three of the big cloud providers in some fashion- but my team only deals with GCP and we’ve had the least headaches. What I’m trying to say is: you guys are doing great. I’m really happy with the product for my use-cases.
- indigochill 8y ago
  In what way(s) do you find GCP better than AWS? My company was using both for a while but has migrated more towards AWS lately. But I'm just learning my way around cloud development now so I don't have insight into the major differences between providers.
  - dklsafhjskljfl 8y ago
    Aside from pricing (different workloads yield different results), I'd say that: GCP has much more modern APIs (they got to do "right" the first time around, while AWS had to learn from their "mistakes") GCP is less mature with documentation, but still excellent (AWS is first class). I've found bugs, and others I know have run into straight up wrong docs - a quick message to support yielded the correct information and docs update within the week. GCP is far more "opinionated" about how you should run a service. AWS is opinionated as well, but less so. What this means is that while GCP will sell you resources in traditional way, to go GCP native or from scratch really requires buying into the GCP "way" of doing things more than AWS does. Basically, K8S or go home. We use GCP and I enjoy it quite a bit. I have extensive AWS experience as well, but no strong preference.
    - 8y ago
deanCommie 8y ago
> Derek’s point about savings is about list price differences that result from total system performance (and not any sort of special discounting). Obviously that's the thing you'd choose to highlight in a public Google-hosted presentation after a strategic partnership that is a bonanza in marketing for Google Cloud. It wouldn't work so well to tell a room full of engineers "We moved to Google cuz they gave us a fat discount, one you can't get because you're not Twitter!" The fact that you allocated a top level URL for it is embarrassing (https://cloud.google.com/twitter/ https://cloud.google.com/twitter/). Can you imagine if Amazon advertised https://aws.amazon.com/netflix/ https://aws.amazon.com/netflix/?
- dustinmoris 8y ago
  > The fact that you allocated a top level URL for it is embarrassing (https://cloud.google.com/twitter/ https://cloud.google.com/twitter/). Can you imagine if Amazon advertised https://aws.amazon.com/netflix/ https://aws.amazon.com/netflix/? It's a case study. Google didn't create a one-off URL for Twitter, but have the same URL pattern for all their public case studies: - https://cloud.google.com/chilean-healthcare/ https://cloud.google.com/chilean-healthcare/ - https://cloud.google.com/now-ims/ https://cloud.google.com/now-ims/ - etc. Amazon has absolutely the same, except that their URL is slightly different, but I didn't think that there is a URL police which determines which URLs are considered embarrassing for case studies and which not: https://aws.amazon.com/solutions/case-studies/netflix/ https://aws.amazon.com/solutions/case-studies/netflix/
eruci 8y ago
I tried to use both GCP and AWS and got killed with egress charges from GCP. I suppose I got to pick one and stick with it, so I picked AWS.
- jkaplowitz 8y ago
  Both clouds charge significantly for egress. Were you sending a lot of network traffic to SaaS vendors that were hosted in AWS but not GCP? That would make your choice pretty natural, assuming you wanted to stick with those vendors.
  - eruci 8y ago
    Yes, the majority of my clients is on AWS so I have to stick with it.
rblion 8y ago
Only on HN would I see a comment start with > Disclosure: I work at Google Cloud (and directly with Derek and the Twitter team). Thanks for making this community awesome. I want to work at Google or with Google one day. One of the few companies I have always admired and always will.
UncleJo7 8y ago
well most probably twitter might not be able to cope up with future attacks or the cost to mitigate them.
burtonator 8y ago
I wrote a blog post on our implementation of cloud sync and migration to Google Cloud / Firebase. https://getpolarized.io/2019/01/03/building-cloud-sync-on-google-firebase.html https://getpolarized.io/2019/01/03/building-cloud-sync-on-go... I like the google storage API the most honestly. Firebase is really nice but the storage API is very polished and basically does exactly what you need it to do without the complexity of S3.

by boulos 8y ago

fhoffa 8y ago

From the Twitter Hadoop to GCP talk that Boulos linked: - >500k cores - >300PB storage - >12,500 cluster size - >1T messages per day And there's also this other talk, "How Twitter Migrated its On-Prem Analytics to Google Cloud" - focused on their migration to BigQuery: - https://www.youtube.com/watch?v=sitnQxyejUg https://www.youtube.com/watch?v=sitnQxyejUg - 20 TB/ day of raw log data, >100k events/sec - Loading ~1TB/hour into BigQuery. - Serving 5,000+ complex queries / second. p99 ~300ms Disclosure: I'm Felipe Hoffa and I work for Google Cloud https://twitter.com/felipehoffa https://twitter.com/felipehoffa.

opportune 8y ago
At this scale I'm a little surprised Twitter saved money on the migration. My assumptions were that at this scale the upfront cost of migration trumped any kind of difference in annual maintenance, esp. considering you still do need an SRE team anyway who knows big data well enough to solve any and all software related problems
- Aeolun 8y ago
  At that scale the team itself likely costs peanuts.
  - opportune 8y ago
    yeah but you still need the IT team too. Not sure, from my conversations at cough cough other media giants the economics of moving to cloud don't always work out at massive scale
    - Aeolun 8y ago
      Oh, certainly. I was talking more about the fact that the monthly cost of 125.000 instances likely beats the cost of your IT team whatever it’s size is. I’d be interested in their cost/benefit analysis for this one.
      - opportune 8y ago
        Yeah, what I mean is it's not just cloud costs vs. local hosting costs. You have to come up with different processes for SRE and IT (removing mundane hardware work, accounting for cloud idiosyncrasies), benchmark, make sure your hosting cloud can handle scaling up to some upper bound estimate of your future user base, etc. At a company the size of twitter these become legitimate concerns. I've been throttled on certain services of certain cloud providers at 600 data nodes lol... of course twitter can negotiate higher allowances but massive scaling doesn't happen with the flick of a switch, sometimes you need big, long-term, expensive contracts
      - Scarbutt 8y ago
        12,500, not 125,000. Still a lot ;)
- geggam 8y ago
  Automation, the twitter team is full of SV veterans who have run into many problems at scale years ago. At least it used to be.
gniv 8y ago
> 1T messages per day How? I mean there are 7B people on earth. Are people that active on Twitter? Or does "messages" mean something else here?
- pgeorgi 8y ago
  There are tons of bots on Twitter
- 24gttghh 8y ago
  Bots, news organizations, and teenagers. ~143 messages/day per human doesn't seem unreasonable.
- regnerba 8y ago
  A message in this case probably doesn't mean a Tweet. A message if probably just a message on the bus/in the queue and any action on Twitter probably produces one. Like, retweet, tweet, follow, sign up, everything.
  - gniv 8y ago
    Oh if the likes are included then it makes sense.
- zjaffee 8y ago
  A message probably counts every tweet a person views at a minimum.
JohnJamesRambo 8y ago
It's stunning how many resources we as a species are wasting on something of so little net positive value to humanity as Twitter.
- tyrust 8y ago
  I'm not a fan of or user of Twitter, but I can't deny that anything that allows people to communicate instantly, easily, and broadly is a benefit to humanity.
  - JohnJamesRambo 8y ago
    So in your opinion Facebook is a benefit to humanity also? Which conflicts with nearly every study of it ever done?

omeid2 8y ago

It is interesting point to bring up but nonetheless discount pricing is not out of question. It is possible that while GCP gives a good value based on the writing on the tin, someone else might beat that with a good margin under special terms. That being said, I have reasons to think very few could beat Google on compute or storage game and definitely much harder to beat the big G on Networking side of things.

jkaplowitz 8y ago
Companies like Twitter would get a significant discount from any provider, though I imagine AWS as market leader would have less incentive than GCP to go quite as deep. Disclosure: I worked for GCP until early 2015, but I haven't worked for Google since then, have no inside info on this deal and am not relying on anything from my time at Google for this comment.

dijit 8y ago

FWIW I’m not a big cloud fan, but I was tasked with the obligation of finding the “least worst” cloud provider in terms of predictable throughout and total cost (inc dev/ops time) and GCP came out as a clear winner despite heavy financial incentives from both Azure and AWS. (We’re a very large corp). Due to our new allegiance to google cloud I’ve been given a little more privileged access to engineers (after the fact) and I can tell that I definitely made the right choice, the people I spoke to favoured having a clean/clear backend with actual quality of life features; but since they don’t go nicely into feature comparison charts people often think that GCP is less mature or featureful. I’m a real convert, and our company uses all three of the big cloud providers in some fashion- but my team only deals with GCP and we’ve had the least headaches. What I’m trying to say is: you guys are doing great. I’m really happy with the product for my use-cases.

indigochill 8y ago
In what way(s) do you find GCP better than AWS? My company was using both for a while but has migrated more towards AWS lately. But I'm just learning my way around cloud development now so I don't have insight into the major differences between providers.
- dklsafhjskljfl 8y ago
  Aside from pricing (different workloads yield different results), I'd say that: GCP has much more modern APIs (they got to do "right" the first time around, while AWS had to learn from their "mistakes") GCP is less mature with documentation, but still excellent (AWS is first class). I've found bugs, and others I know have run into straight up wrong docs - a quick message to support yielded the correct information and docs update within the week. GCP is far more "opinionated" about how you should run a service. AWS is opinionated as well, but less so. What this means is that while GCP will sell you resources in traditional way, to go GCP native or from scratch really requires buying into the GCP "way" of doing things more than AWS does. Basically, K8S or go home. We use GCP and I enjoy it quite a bit. I have extensive AWS experience as well, but no strong preference.
  - 8y ago

deanCommie 8y ago

> Derek’s point about savings is about list price differences that result from total system performance (and not any sort of special discounting). Obviously that's the thing you'd choose to highlight in a public Google-hosted presentation after a strategic partnership that is a bonanza in marketing for Google Cloud. It wouldn't work so well to tell a room full of engineers "We moved to Google cuz they gave us a fat discount, one you can't get because you're not Twitter!" The fact that you allocated a top level URL for it is embarrassing (https://cloud.google.com/twitter/ https://cloud.google.com/twitter/). Can you imagine if Amazon advertised https://aws.amazon.com/netflix/ https://aws.amazon.com/netflix/?

dustinmoris 8y ago
> The fact that you allocated a top level URL for it is embarrassing (https://cloud.google.com/twitter/ https://cloud.google.com/twitter/). Can you imagine if Amazon advertised https://aws.amazon.com/netflix/ https://aws.amazon.com/netflix/? It's a case study. Google didn't create a one-off URL for Twitter, but have the same URL pattern for all their public case studies: - https://cloud.google.com/chilean-healthcare/ https://cloud.google.com/chilean-healthcare/ - https://cloud.google.com/now-ims/ https://cloud.google.com/now-ims/ - etc. Amazon has absolutely the same, except that their URL is slightly different, but I didn't think that there is a URL police which determines which URLs are considered embarrassing for case studies and which not: https://aws.amazon.com/solutions/case-studies/netflix/ https://aws.amazon.com/solutions/case-studies/netflix/

eruci 8y ago

I tried to use both GCP and AWS and got killed with egress charges from GCP. I suppose I got to pick one and stick with it, so I picked AWS.

jkaplowitz 8y ago
Both clouds charge significantly for egress. Were you sending a lot of network traffic to SaaS vendors that were hosted in AWS but not GCP? That would make your choice pretty natural, assuming you wanted to stick with those vendors.
- eruci 8y ago
  Yes, the majority of my clients is on AWS so I have to stick with it.

rblion 8y ago

Only on HN would I see a comment start with > Disclosure: I work at Google Cloud (and directly with Derek and the Twitter team). Thanks for making this community awesome. I want to work at Google or with Google one day. One of the few companies I have always admired and always will.

UncleJo7 8y ago

well most probably twitter might not be able to cope up with future attacks or the cost to mitigate them.

burtonator 8y ago

I wrote a blog post on our implementation of cloud sync and migration to Google Cloud / Firebase. https://getpolarized.io/2019/01/03/building-cloud-sync-on-google-firebase.html https://getpolarized.io/2019/01/03/building-cloud-sync-on-go... I like the google storage API the most honestly. Firebase is really nice but the storage API is very polished and basically does exactly what you need it to do without the complexity of S3.