4 ms·

I'm pretty surprised that Twitter is moving their homegrown stack to GCP. Twitter has a storied big data platform (Storm, Summingbird, Lambda Platform, Manhatta

I'm pretty surprised that Twitter is moving their homegrown stack to GCP. Twitter has a storied big data platform (Storm, Summingbird, Lambda Platform, Manhattan), and they have completely moved to the cloud. I wonder how the finances worked out; I'm sure Twitter paid a lot for their bespoke platform and Google must've been competitive. It was probably also easier as I imagine a lot of the engineers who considered those projects their children may have already moved on.

secondcoming 5y ago
Well, Google paid for my employer to migrate to GCP from AWS.
geodel 5y ago
Well all these projects seems Java/Scala codebase. I wouldn't be surprised as amount of data grew these systems would have started behaving unpredictably and unceremoniously dumped later on. Another thing as Twitter was (still is?) big on Scala so solid working software becomes secondary to circlejerking around Scala features and products eventually would have suffered. As a user the only Scala software I have used is Kafka and needless to say it comes across as half-assed cobbled system and not a thoughtfully designed, solid, reliable and performant system. So Google being competitive is not a must as Twitter itself does not anything reasonable for their growing needs.
funstuff007 5y ago
maybe they got a sweetheart deal being a marquee customer. Although, not as sweet as a deal that the CME got. > Google also makes $1B equity investment in CME Group https://www.cmegroup.com/media-room/press-releases/2021/11/04/cme_group_signs_10-yearpartnershipwithgooglecloudtotransformglob.html https://www.cmegroup.com/media-room/press-releases/2021/11/0...
chillfox 5y ago
Or it's the result of some good wine, dinners and gifts. The spreadsheets justifying it can always be engineered afterwards.
- markus_zhang 5y ago
  I always wonder how those lucrative corporate sales happen.
- marginalia_nu 5y ago
  The rationale is likely reminiscent of that motivating the outsourcing craze from 15 years ago. When all you care about is the next quarterly report, it's real easy to sell the idea of getting rid of all that expensive technical competence from the company payroll, letting someone else know that stuff instead, then hire some a team of cheap oompa loompas straight outta college to spend all day writing -X-M-L- YAML instead.
ovis 5y ago
For some reason I'm kind of sad seeing Twitter's (very cool) homegrown technologies in the "Old" diagram with the "New" architecture basically Google Cloud. I'm sure it makes sense internally, but it feels like the loss of an innovation center in the streaming space.
random314 5y ago
Sad to see Heron go

secondcoming 5y ago

Well, Google paid for my employer to migrate to GCP from AWS.

geodel 5y ago

Well all these projects seems Java/Scala codebase. I wouldn't be surprised as amount of data grew these systems would have started behaving unpredictably and unceremoniously dumped later on. Another thing as Twitter was (still is?) big on Scala so solid working software becomes secondary to circlejerking around Scala features and products eventually would have suffered. As a user the only Scala software I have used is Kafka and needless to say it comes across as half-assed cobbled system and not a thoughtfully designed, solid, reliable and performant system. So Google being competitive is not a must as Twitter itself does not anything reasonable for their growing needs.

funstuff007 5y ago

maybe they got a sweetheart deal being a marquee customer. Although, not as sweet as a deal that the CME got. > Google also makes $1B equity investment in CME Group https://www.cmegroup.com/media-room/press-releases/2021/11/04/cme_group_signs_10-yearpartnershipwithgooglecloudtotransformglob.html https://www.cmegroup.com/media-room/press-releases/2021/11/0...

chillfox 5y ago

Or it's the result of some good wine, dinners and gifts. The spreadsheets justifying it can always be engineered afterwards.

markus_zhang 5y ago
I always wonder how those lucrative corporate sales happen.
marginalia_nu 5y ago
The rationale is likely reminiscent of that motivating the outsourcing craze from 15 years ago. When all you care about is the next quarterly report, it's real easy to sell the idea of getting rid of all that expensive technical competence from the company payroll, letting someone else know that stuff instead, then hire some a team of cheap oompa loompas straight outta college to spend all day writing -X-M-L- YAML instead.

ovis 5y ago

For some reason I'm kind of sad seeing Twitter's (very cool) homegrown technologies in the "Old" diagram with the "New" architecture basically Google Cloud. I'm sure it makes sense internally, but it feels like the loss of an innovation center in the streaming space.

random314 5y ago

Sad to see Heron go