34 ms·

Twitter migrates data to Google Cloud

680 points by theBashShell 8y ago

ripvanwinkle 8y ago
Any inside stories on what twitter discovered when they benchmarked Google with other providers. I imagine they must have done a very exhaustive evaluation
- gkoberger 8y ago
  Or, Google thought the publicity was worth a gigantic discount. (EDIT: The provided quote from a Twitter employee hints at this directly: "the savings")
  - everdev 8y ago
    A comment from a Google Cloud employee seems to suggest otherwise. > Derek’s point about savings is about list price differences that result from total system performance (and not any sort of special discounting).
    - gkoberger 8y ago
      There's a lot of ways that both can be true. For example, metering variables that are friendly to how Google does things.
  - WaxProlix 8y ago
    I did a gig for <huge content producer/delivery company> and at the time mostly did AWS implementations (I work at AWS now full time, for what it's worth). The client was really interested in going with Google Cloud because they were promised essentially free storage, with no limit. This is a company with more petabytes of video footage in their data lake than I've seen before and GCP wanted to give them free storage for the prestige (and possible upside, longterm) of having Big Company on their list.
  - cheriot 8y ago
    Google may be playing the long game. I've seen an effect on my team after moving from physical hardware and centralized capacity provisioning to GCP with immediate gratification hardware. It's easier to spend money so we ask for more stuff more often.
    - pdelgallego 8y ago
      it is The Jevons paradox [1]. I have seen it as well happening when I migrated a client from a standard cloud architecture to a serverless architecture. >>> In economics, the Jevons paradox (sometimes Jevons effect) occurs when technological progress or government policy increases the efficiency with which a resource is used (reducing the amount necessary for any one use), but the rate of consumption of that resource rises due to increasing demand. [1] https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox
  - onetimemanytime 8y ago
    This is an interesting case, sorta like Amazon going to NYC. If you give discounts and lower your profit you get something, plus PR. If you don't you loose it all. But if all ask for the Twitter discount, you're screwed. However, by by next year no one can compare the offerings side by side. Better machines, cheaper offerings etc.
  - ernsheong 8y ago
    "savings" does not imply discounts, mind you. GCP is actually cheaper on most fronts compared to AWS, Azure. Quote from reply above: "Derek’s point about savings is about list price differences that result from total system performance (and not any sort of special discounting)"
- andrewguenther 8y ago
  For a company of Twitter's size, benchmarks likely played a small role compared to price negotiations. Being able to say Twitter runs in your cloud is worth a lot to any cloud provider. They're likely operating at steep discounts and probably played off all the major providers to get the best offer.
  - deleted 8y ago
    [deleted]
  - noahl 8y ago
    Agreed to all of what you said, but also, performance in some circumstances is a discount. One of the Google Cloud employees said "Derek’s point about savings is about list price differences that result from total system performance (and not any sort of special discounting)." I don't have any internal numbers, but imagine if Twitter could meet its needs with an N-node Redshift cluster or an N/2-node Dataproc cluster, because Dataproc is so much faster at flipping the bits or whatever Twitter wants to do (probably network performance really). Google could charge more per node and still come out cheaper because of performance.
- StudentStuff 8y ago
  Perhaps they did a solid analysis, but Twitter doesn't appear to be a rational actor. Look at Jack Dorsey's recent interview as an example, at one point he names 3 separate things as priority #1: https://www.huffingtonpost.com/entry/jack-dorsey-twitter-interview_us_5c3e2601e4b01c93e00e2a00 https://www.huffingtonpost.com/entry/jack-dorsey-twitter-int... Apparently this is a trend going back years, it doesn't seem to be doing anything good for Twitter or Square: https://sg.finance.yahoo.com/news/big-twitter-investor-defends-company-013248222.html https://sg.finance.yahoo.com/news/big-twitter-investor-defen...
  - afarrell 8y ago
    You mean Square. Stripe is led by the Collison brothers but is in the same industry.
    - StudentStuff 8y ago
      Sorry, all the payments companies just kinda blend together for me (having had to deal with too many of 'em). Corrected the post above. On a sidenote, Square, Stripe, Payment Ninja, <insert payments startup> so far seems to be lipstick on the pig that is the underlying debit/credit networks, often with a hefty price for said lipstick. The US badly needs a better version of Zelle that supports business transactions.
      - afarrell 8y ago
        For recurring/invoice transactions, you could use GoCardless: https://gocardless.com/en-ca/pricing/ https://gocardless.com/en-ca/pricing/ We don't use debit/credit card networks but bank account->bank account networks like ACH, SEPA, etc. Not yet available in the US. Coming soon though.
  - TheRealDunkirk 8y ago
    People have to come to realize that nothing that gets said at this level should be taken literally. At the C-level of Fortune 500 companies, everything is crafted to create a narrative. People claiming that Twitter or Facebook has a political bias are missing the real point. For example, I think that Zuck really, truly, doesn't give a fink about political bias. How could he? He'd never get any sleep! In his position, all he cares about is the money. If that takes making this or that comment or public statement or testimony to Congress, it doesn't matter. It's all geared towards and carefully crafted towards keeping the money flowing. In that sense, he and the top brass at a place like Facebook are all political whores, so to speak. Selling their convictions for whatever keeps the spigot open.
- tanilama 8y ago
  It will be simplistic to assume such decision is tech-driven. Per my experience, it is mostly business driven. Who offers the best deal with the lowest price, gets the business. You'd be surprised, if you are a big enough customer, even better if you are a reputable one, how much catering the cloud provider would be willing to offer.
- opportune 8y ago
  Hadoop is an offline storage mechanism used for analytics, random access benchmarks barely matter compared to the overall cost of storage+compute so long as access time aren't atrocious
- derekalyon 8y ago
  This is Derek. We shared a bunch of stuff in this video: https://youtu.be/4FLFcWgZdo4 https://youtu.be/4FLFcWgZdo4 One of the interesting benchmarks was GridMix performance at scale. And overall performance for throughput-oriented workloads like Hadoop was generally strong. Most cloud benchmarking I have seen sticks to micro-benchmarks and single-node benchmarks, so they miss this.
deleted 8y ago
[deleted]
slackoverflower 8y ago
Just makes the eventual Google acquisition one bit easier.
- mobilekid7 8y ago
  That crossed my mind the minute I read the headline too. I can imagine folks at Google getting excited about this first step...
- StudentStuff 8y ago
  With Twitter's current state and Google's moves to kill their social side, this is unlikely. Why buy a moderate sized social network when you just killed your similarly sized social network?
  - chipperyman573 8y ago
    If you're referring to G+, I don't think G+ was a "moderate sized social network" any more than twitter (although in opposite directions). Also, imo, calling them "similarly sized" would be incorrect, except perhaps in signed up users (from which google got a ton from the "login with google and your account is already setup", not daily active users)
    - StudentStuff 8y ago
      300 million monthly active users on G+ to Twitter's 335 million monthly active users, they were nearly identical in size. https://en.wikipedia.org/wiki/Google%2B#Growth https://en.wikipedia.org/wiki/Google%2B#Growth https://en.wikipedia.org/wiki/Twitter https://en.wikipedia.org/wiki/Twitter (see sidebar)
      - chipperyman573 8y ago
        Wow, really?? In that case, what caused G+ to fail? I was always under the impression that it was due to (low) DAUs.
echelon 8y ago
Correct me if I'm wrong, but this doesn't say what Twitter moved to the cloud. It could be literally anything, and may not in fact be core tweet or user data. This strikes me as a team (or teams) outgrowing the storage options that were offered internally and choosing to outsource to fit their needs. Isolated use for one business case, and not indicative of a broad movement to the cloud on the part of the whole org. I could be wrong, of course. I'd like to know more. And I'd certainly like to know what the engineers at Twitter think.
- sidcool 8y ago
  You are not wrong. The message here is that whatever data was transferred, Google was able to handle hundreds of petabytes of data. It could just be billions of photos of cats
  - londons_explore 8y ago
    'cold storage' It's going to be logs data, backups, and users original uploaded data (rather than reencoded versions) that they hope will be valuable one day.
    - fhoffa 8y ago
      you missed this part "and our flexible compute Hadoop clusters"
  - anvuong 8y ago
    billions photos of cats is invaluable data, mind you
    - inferiorhuman 8y ago
      Well Twitter is now part of the official White House record, isn't it?
- andrewguenther 8y ago
  The post on Twitter's side goes into more detail: https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/a-new-collaboration-with-google-cloud.html https://blog.twitter.com/engineering/en_us/topics/infrastruc... Relevant technical bits: > Today, we are excited to announce that we are working with Google Cloud to move cold data storage and our flexible compute Hadoop clusters to Google Cloud Platform. This will enable us to enhance the experience and productivity of our engineering teams working with our data platform. > Twitter runs multiple large Hadoop clusters that are among the biggest in the world. In fact, our Hadoop file systems host more than 300PB of data across tens of thousands of servers.
andrewguenther 8y ago
Is there anything new here? Twitter announced this move back in May of last year: https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/a-new-collaboration-with-google-cloud.html https://blog.twitter.com/engineering/en_us/topics/infrastruc... The video linked on the page is also from last August: https://www.youtube.com/watch?v=T1zjmNAuMjs https://www.youtube.com/watch?v=T1zjmNAuMjs Throw a 2018 tag on this?
boulos 8y ago
Disclosure: I work at Google Cloud (and directly with Derek and the Twitter team). I’ll try to edit this page tomorrow when I’m at a computer, but there’s much more information in Derek’s talk at NEXT [1]. They (rightfully) didn’t want to get into a detailed “this is what we saw on <X>”, but Derek alludes to their careful benchmarking across providers. While you should always assume smart people make economically reasonable decisions, Derek’s point about savings is about list price differences that result from total system performance (and not any sort of special discounting). I’m hoping a follow-up talk will let us say more about how the migration is going, while this talk was focused on the decision to move part of (!) their Hadoop environment to GCP. [1] https://m.youtube.com/watch?v=4FLFcWgZdo4 https://m.youtube.com/watch?v=4FLFcWgZdo4
- fhoffa 8y ago
  From the Twitter Hadoop to GCP talk that Boulos linked: - >500k cores - >300PB storage - >12,500 cluster size - >1T messages per day And there's also this other talk, "How Twitter Migrated its On-Prem Analytics to Google Cloud" - focused on their migration to BigQuery: - https://www.youtube.com/watch?v=sitnQxyejUg https://www.youtube.com/watch?v=sitnQxyejUg - 20 TB/ day of raw log data, >100k events/sec - Loading ~1TB/hour into BigQuery. - Serving 5,000+ complex queries / second. p99 ~300ms Disclosure: I'm Felipe Hoffa and I work for Google Cloud https://twitter.com/felipehoffa https://twitter.com/felipehoffa.
  - opportune 8y ago
    At this scale I'm a little surprised Twitter saved money on the migration. My assumptions were that at this scale the upfront cost of migration trumped any kind of difference in annual maintenance, esp. considering you still do need an SRE team anyway who knows big data well enough to solve any and all software related problems
    - Aeolun 8y ago
      At that scale the team itself likely costs peanuts.
ranman 8y ago
I have had a hard time dealing with twitter engineering since the account activity API debacle.
sudhirj 8y ago
One thing to remember with the benchmarks is that when evaluating a cloud provider, the benchmarks are useless if talked about with absolute numbers or with regards to hardware specs. Because they're so horizontally scalable, it's more a question of average cost per compute operation or cost per (peta) byte. Both AWS and Google clouds are perfectly capable of running Twitter, and running any software that Twitter uses, even if the number of machines is different. The only "benchmark" applicable is actually the total negotiated cost of the machines required to get the job done. So it's not about being able to do things on Google cloud with fewer processors or less RAM or faster hard drives - Google was willing to give them a lower total cost of ownership for reasons known only to Google. Do not assume that Google (or anybody else) will give you a similar preferential deal. Ignore the "benchmarks".
- sudhirj 8y ago
  If you want to check which cloud is best for you, run your application on all of them, measure your average+median+p95 cost per whatever (tweet/post/reader/dollar of revenue/ticket/?). Then factor in which platform is easier to work with, has better tooling and community support, because until you hit a certain scale your time will always be more expensive. After you do all that, just start with Heroku and Postgres with some AWS Lambda, then move to ALB + AWS Fargate + RDS|Aurora / DynamoDB as you get bigger, then to NLB + ECS with a cluster of 20-80 On Demand and Spot Instances, then to a cluster of ARM instances on Spot. If you need to build your own datacenter at that point, you'll know. And you would have built a 12Factor app to make all of the above work, so migrating will be easy.
- sciurus 8y ago
  I don't think it's purely a measure of cost per compute operation. That assumes you would run the same software on any cloud using plain VMs or a service that slightly abstracts them. That may be true in this case since Twitter will continue to use Hadoop/Spark, but sometimes you can get a real advantage from switching to a service that only one cloud provider offers. For instance, someone in these comments pointed out that Twitter already migrated their anayltics workload to BigQuery. Evaluating BigQuery vs Redshift vs something else is not as clear cut as cost per compute operation.
gigatexal 8y ago
Huge win for Google. I hope this helps Twitter focus on features and not firefighting infrastructure issues.
- SmellyGeekBoy 8y ago
  Does Twitter need more features? Perhaps they could focus more on customer support and moderation.
  - gigatexal 8y ago
    A downvote button would be nice. The ability to edit a tweet would also be nice. It’s 2019 after all, and we can edit things on other platforms. But yes, moderation would be great. Maybe they can leverage Google’s machine learning hardware and smarts.
rapsey 8y ago
Well hopefully google does not decide to abandon google cloud :)
- advisedwang 8y ago
  Hopefully Google doesn't abandon search!
  - rapsey 8y ago
    They are king of search and distant third in the cloud game.
  - samfisher83 8y ago
    Search+Ads:Google Windows+Office:microsoft Those are the sacred cows.
    - ocdtrekkie 8y ago
      Microsoft has been hacking off bits of dependency on Windows these days in order to push and support their cloud business, to the point I would argue Microsoft is aggressively trying to push people off of Windows+Office and onto Azure+365. Microsoft would far rather you use Linux on their cloud platform over having you use Windows literally anywhere else. I don't think Windows counts as a sacred cow anymore at Microsoft.
  - halfjoking 8y ago
    Hopefully Google doesn't abandon human civilization, and choose intelligent machines over biological ones!
synaesthesisx 8y ago
[Removed]
- jcmi 8y ago
  LOL
temp231239 8y ago
Hun. I was under impression that at higher scale of operations building your own cloud is more economical. I guess money is not an issue for Twitter.
- rjf72 8y ago
  This thread, and site, tells the tale. Google is using this for marketing purposes. They already have some trite 2 minute marketing ad running as well. The deal that Twitter received is going to come with extensive incentives unavailable to typical operations. Twitter also was likely able to play the typical providers against each other. 'Oh, well AWS is offering me [x,y,z]. Can you do better?'
  - rapsey 8y ago
    Well yeah but that is the case for anyone large enough.
- pas 8y ago
  They are moving batch jobs to GCP. Because there probably the occasional burst loads and the often underutilized capacity makes using a flexible compute fabric (aka cloud provider) a better choice. If they are moving everything, well, Netflix did that too. Maybe opportunity cost, maybe having your engineers work on your core product instead of running VMs is cheaper altogether.
  - tybit 8y ago
    Much of Netflix’s spend would presumably be on CDNs, which they run themselves not on AWS.
    - opportune 8y ago
      uh correct me if I'm wrong but all CDN's that Netflix purchases also need to have a large storage cache backing them right? Meaning each CDN Netflix uses for local caching also requires a colocated datastore to circumvent their centralized-bandwidth issue.
tanilama 8y ago
I think by this announcement, it is probably safe to say, even at the size of Twitter, it is probably still more economical to handle your operations to bigger players. Hosting your own infrastructure makes less and less sense going forward, the cloud is the new king now.
- kortilla 8y ago
  Probably not safe to conclude anything. Twitter has very different requirements than something like a bank, an insurance company, an auto manufacturer, etc.
- inapis 8y ago
  I dont believe that this becomes a nail in the coffin for hosting your own infrastructure. This should still be evaluated on a case by case basis.
pizza 8y ago
The scale of these megacorps is crazy! I wonder how you even plan to move that much data. I know that Amazon has a service called Snowmobile for stuff like this. They probably did something similar here. "You can transfer up to 100PB per Snowmobile, a 45-foot long rugged sized shipping container, pulled by a semi-trailer truck." https://www.youtube.com/watch?v=8vQmTZTq7nw https://www.youtube.com/watch?v=8vQmTZTq7nw
- fro0116 8y ago
  > Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. https://what-if.xkcd.com/31/ https://what-if.xkcd.com/31/
  - BerislavLopac 8y ago
    The mention of "SneakerNet" reminded me of old times when we had a bunch of old Macintoshes (pre-OSX) connected with a proprietary local network called LocalTalk. It was so unreliable that we had a couple of 3.5" floppy disks that we used to transfer files when it didn't work; the disks were labelled "LocalWalk". :D
- pojzon 8y ago
  Migrations like these take time and happen step by step. First you can try to replicate some data and move read instances. After that step by step the rest. Its really not possible to do that overnight or months even. Fyi: im currently migrating twice the Twitter size one in the world infrastructure from aws to azure now.
  - tpetry 8y ago
    Isn‘t exporting 600TB of data from aws ridicously expensive?
    - deleted 8y ago
      [deleted]
    - 8y ago
dustinmoris 8y ago
Reading most comments here I feel like nobody wants to give a tiny bit of credit to either Twitter or Google, which I find disenchanting. It is true that any cloud provider would get good publicity by being able to say that Twitter runs on their cloud, but that is precisely the same reason why we shouldn't just shrug this off as a publicity/business decision, because just like Google might have tried to persuade Twitter other cloud providers will most certainly have tried their best too. If Twitter still went with GCP then I think it is only fair to assume that there must have been some real advantage in going with GCP. I don't think anyone who says this must have been a pure business decision is entirely honest with themselves, because a cloud which cannot handle Twitter's volume is no good even if it was entirely for free. I have nothing to do with Twitter (actually don't even like Twitter because it is such a toxic social media platform) and I don't like many things which Google does, but as an engineer myself I can only say that after using Azure, AWS and GCP for many years in a commercial setting that GCP is indeed years ahead. The quality, speed and reliability of GCP is second to none from my experience and I honestly couldn't say the same about AWS and especially not at all about Azure.
- notyourwork 8y ago
  > The quality, speed and reliability of GCP is second to none from my experience Can you be more specific on what reliability GCP is years ahead of the others? I'm guessing numbers will be pretty comparable on all platforms so if we want to declare outright winners it would be useful to cite a number to indicate where you experienced this.
- privateSFacct 8y ago
  the quality, speed and reliability of GCP is second to none? I hope this is a joke. AS someone who started on GCP, and got tired of the TERRIBLE quality and reliability of their docs among other issues saying that they are ahead is an absolute joke. I've found some corner cases with AWS, but got quick resolution even on things they have in "beta" - and consistency from documentation through to system is high. Secondarily, AWS seems to support even old tech FOREVER. I have a simpledb based app. That tech is 9 years old now. Still ticking. When you are building stuff up over time and can't afford to rebuild on the new hotness every other year, this is nice. Anyone know the revenue AWS and GCP generate? Seriously hard to believe GCP is so many years ahead in this space from my own experience.
cyberferret 8y ago
I've scoured the comments and article, but I might have missed it... What did Twitter migrate from?? Did they have their own data centre before moving to Google Cloud?
- joatmon-snoo 8y ago
  Yep. https://www.datacenterknowledge.com/archives/2015/07/13/custom-servers-process-tweets-in-twitter-data-centers https://www.datacenterknowledge.com/archives/2015/07/13/cust...
  - userbinator 8y ago
    Another step backward for the decentralisation of the Internet... not that Twitter was decentralised in the first place, but it's unsettling to see Google taking over more and more of the "big pieces" of Internet.
    - zymhan 8y ago
      Funnily enough, Twitter and Google were/are neighbors in a QTS datacenter in Atlanta.
      - forgot-my-pw 8y ago
        Just moving across the street then? It's probably the fastest data transfer ever by physically moving all the hard drives!
Sample17 8y ago
<a href="https://luckypatchers.xyz/">lucky https://luckypatchers.xyz/">lucky patcher app</a>
thisgoodlife 8y ago
How are they gonna move all that 300 pb of data to Google's data center? A fleet of semitrailer trucks?
- aboutruby 8y ago
  For instance AWS has 50 Terabytes devices that they ship from your own data center to their data center: https://aws.amazon.com/snowball/ https://aws.amazon.com/snowball/
  - kondro 8y ago
    They also have the 100PB Snowmobile. https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/
- boulos 8y ago
  Disclosure: I work on Google Cloud (with Twitter). Pipes! That data isn't static either. As Derek said in a sibling comment, they setup hundreds of Gbps of peering. Each 100 Gbps gets you a Petabyte per day (roughly). Planning the move and doing the tooling to keep stuff synchronized takes longer than the actual transfer once you've got say 800 Gbps (just 30 days ish). And again, you're going to have lots of data churn per month, so you want pipes anyway.
prepend 8y ago
This really makes me wonder if Twitter finally made up their mind who they will merge with.
- partiallypro 8y ago
  I too was thinking this, does this mean a future acquisition is not out of the question? Frankly, I find a Google owned Twitter to be a terrifying prospect. Not that Jack is doing a great job there...but I'd much rather they be owned by multiple companies in a partnership than by one company. Twitter, imo, is far far more powerful than Facebook in driving world events given the leadership and journalists on the platform.
egorfine 8y ago
The next global Twitter downtime? Their credit card failed at Google Payments, acc gets marked for fraud and no one in support can help them recover. Twitter is gone for good.
- jasonvorhe 8y ago
  This has long been fixed.
  - marcinzm 8y ago
    Every few weeks there's a story about something in Google's automated machinery failing a customer with no recourse except to post an angry blog/twitter/hackernews note and hope it get's noticed. The core issue (bad and powerless customer service) isn't fixed and all that changes is the particular way it may bite you.
    - munchbunny 8y ago
      When you're Twitter sized, that was never a problem to begin with. For mere mortals, that continues to be a problem.
    - el_isma 8y ago
      Where's twitter gonna twit if they go down?
  - wil421 8y ago
    6-7 months isn’t too long[1]. If this is the fix you were taking about it happened last year. According to Wikipedia GCP started in 2008 and generally available in 2011 as App engine. It took them 7-8 years to solve billing? [1]https://medium.com/@serverpunch/why-you-should-not-use-google-cloud-75ea2aec00de https://medium.com/@serverpunch/why-you-should-not-use-googl...
  - egorfine 8y ago
    Not even close.
  - 8y ago
opportune 8y ago
side note, if you want to know why distributed infrastructure engineering has been the place to be recently, think about the fact that engineers making $200-500k/year are making decisions that can save big corporations $1-5mm per year in compute/storage costs
tome 8y ago
I'm curious what happens to everyone at Twitter who was employed in managing their old data centre.
- thecanman 8y ago
  Well I'm sure they still have amazing career prospects just based on their resumes.
  - tome 8y ago
    Sure, but what's the actual process for dealing with them? Will they be let go in small batches as the infrastructure moves over to GCP, or what?
- Rafuino 8y ago
  This isn't all of Twitter's workloads moving to GCP. Seems like just cold storage and some aspects of their Hadoop workloads.
maloneyg 8y ago
More beneficial for Google I believe.
patrickg_zill 8y ago
What Twitter is paying for and the level of service they receive from Google, is very different from the typical GCP pricing and service level that would be available for say, your company... Realistically, it can't be considered as the same service, can it?
yani 8y ago
This feels like marketing material. Twitter jumped over to GCC because of the offer they received.
- SketchySeaBeast 8y ago
  The first clue was that the news was being hosted on https://cloud.google.com/ https://cloud.google.com/
kerng 8y ago
The issue with GCP and with Google in general is support. I'm certain that's not an issue for Twitter as there are probably multiple people on call just for the Twitter deal. But it might mislead the average company down a bad path believing there is decent support with GCP. Any custom issues (like billing) and it's a headache dealing with Google.
keithnoizu 8y ago
GCE blows, good luck with that.
iheartpotatoes 8y ago
wow. talk about "loss of service challenges".
dugluak 8y ago
Next step - acquire twitter?
- smt88 8y ago
  I definitely wonder if Twitter's management considered this. Any price for Twitter goes down by at least a few million if no major cloud migration is necessary. That said, a few million is a rounding error on a price Google would pay for Twitter.
cr0sh 8y ago
I know my comment won't move this discussion forward in any fashion, but I have to ask anyway. What was this data, and why did they feel they needed to keep it? I mean - 300PB is nothing to sneeze at. I can understand wanting to keep important data around (IP, for instance, or recent server logs for the last 30 days or even a year) - but this amount seems to go well beyond that. In fact, it almost seems like they have stored every single tweet ever written. In addition to probably all of their logs, and who knows what else. Why are they keeping all of that information? If it is all the tweets ever written - they don't have copyright to it, so such information wouldn't belong to them... I'm not a big twitter user - can I search for a tweet on their system that I (or someone else) made say...5 years ago or longer? If this is a bunch of tweets - is that why they keep them around? Are people in general aware of this? That is - if they are tweets - that Twitter has stored all of them, forever and ever, and that they are all still accessible (if not by the general public, then by law enforcement at the very least)? If they are tweets - do users have any recourse at being able to remove those tweets (aside from those that they likely are legally bound to keep - I am thinking of the current POTUS's tweets, for instance)? What is the ultimate purpose behind keeping all of that data? Even if it wasn't a bunch of tweets, whatever it is can't really be that useful otherwise, except for maybe the most recent stuff created in the past year, at best. Things like that - logs, etc - might be destined for a rotated "cold storage" and would ultimately "age out". Accounting records (financial data, etc) probably isn't that great a percentage of the data (if at all?) - but whatever was there, too, would likely need to be saved for a greater range - but even there maybe only 10-15 years worth (and it certainly wouldn't even comprise a fraction of the bulk). So what is it? Why is it being kept? To what purpose? While 300PB isn't that great of an amount in physical terms (that is, the storage medium probably doesn't take up a great amount of space in a datacenter), it still isn't anything tiny, either (especially depending on what medium was used - I mean, I doubt they are storing it all on a ton of 256GB microSD cards - though that does bring up an interesting idea/thought to mind - but I digress). I'm of the thought that companies shouldn't just "throw away" data - but it should be tempered by "importance to humanity over time". Not everything should be kept - but, for instance, it would have been nice if we (humanity) had kept around the original transmissions from the surface of the moon, or the blueprints to the rocket that took humans there. But we probably don't need all the various interoffice memoranda and fluff at the various NASA contractors (we probably don't need the accounting records - but BOMs might be useful). That's just a couple of examples I can think of off the top of my head, but history is rife with instances of companies imploding or being bought out or restructured in such a manner that data is just "destroyed" instead of kept. Or lost, or otherwise made unretrievable to humanity. Here with Twitter - an active company to be sure - we have the seemingly exact opposite - almost like r/datahoarders were in charge. ...and that should concern people a bit, regardless of which company it is - but especially a company like Twitter.
yeukhon 8y ago
So what will happen to the small Cloud team in NY? I thought they were developing their own dc.

680 points by theBashShell 8y ago

Any inside stories on what twitter discovered when they benchmarked Google with other providers. I imagine they must have done a very exhaustive evaluation

gkoberger 8y ago
Or, Google thought the publicity was worth a gigantic discount. (EDIT: The provided quote from a Twitter employee hints at this directly: "the savings")
- everdev 8y ago
  A comment from a Google Cloud employee seems to suggest otherwise. > Derek’s point about savings is about list price differences that result from total system performance (and not any sort of special discounting).
  - gkoberger 8y ago
    There's a lot of ways that both can be true. For example, metering variables that are friendly to how Google does things.
- WaxProlix 8y ago
  I did a gig for <huge content producer/delivery company> and at the time mostly did AWS implementations (I work at AWS now full time, for what it's worth). The client was really interested in going with Google Cloud because they were promised essentially free storage, with no limit. This is a company with more petabytes of video footage in their data lake than I've seen before and GCP wanted to give them free storage for the prestige (and possible upside, longterm) of having Big Company on their list.
- cheriot 8y ago
  Google may be playing the long game. I've seen an effect on my team after moving from physical hardware and centralized capacity provisioning to GCP with immediate gratification hardware. It's easier to spend money so we ask for more stuff more often.
  - pdelgallego 8y ago
    it is The Jevons paradox [1]. I have seen it as well happening when I migrated a client from a standard cloud architecture to a serverless architecture. >>> In economics, the Jevons paradox (sometimes Jevons effect) occurs when technological progress or government policy increases the efficiency with which a resource is used (reducing the amount necessary for any one use), but the rate of consumption of that resource rises due to increasing demand. [1] https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox
- onetimemanytime 8y ago
  This is an interesting case, sorta like Amazon going to NYC. If you give discounts and lower your profit you get something, plus PR. If you don't you loose it all. But if all ask for the Twitter discount, you're screwed. However, by by next year no one can compare the offerings side by side. Better machines, cheaper offerings etc.
- ernsheong 8y ago
  "savings" does not imply discounts, mind you. GCP is actually cheaper on most fronts compared to AWS, Azure. Quote from reply above: "Derek’s point about savings is about list price differences that result from total system performance (and not any sort of special discounting)"
andrewguenther 8y ago
For a company of Twitter's size, benchmarks likely played a small role compared to price negotiations. Being able to say Twitter runs in your cloud is worth a lot to any cloud provider. They're likely operating at steep discounts and probably played off all the major providers to get the best offer.
- deleted 8y ago
  [deleted]
- noahl 8y ago
  Agreed to all of what you said, but also, performance in some circumstances is a discount. One of the Google Cloud employees said "Derek’s point about savings is about list price differences that result from total system performance (and not any sort of special discounting)." I don't have any internal numbers, but imagine if Twitter could meet its needs with an N-node Redshift cluster or an N/2-node Dataproc cluster, because Dataproc is so much faster at flipping the bits or whatever Twitter wants to do (probably network performance really). Google could charge more per node and still come out cheaper because of performance.
StudentStuff 8y ago
Perhaps they did a solid analysis, but Twitter doesn't appear to be a rational actor. Look at Jack Dorsey's recent interview as an example, at one point he names 3 separate things as priority #1: https://www.huffingtonpost.com/entry/jack-dorsey-twitter-interview_us_5c3e2601e4b01c93e00e2a00 https://www.huffingtonpost.com/entry/jack-dorsey-twitter-int... Apparently this is a trend going back years, it doesn't seem to be doing anything good for Twitter or Square: https://sg.finance.yahoo.com/news/big-twitter-investor-defends-company-013248222.html https://sg.finance.yahoo.com/news/big-twitter-investor-defen...
- afarrell 8y ago
  You mean Square. Stripe is led by the Collison brothers but is in the same industry.
  - StudentStuff 8y ago
    Sorry, all the payments companies just kinda blend together for me (having had to deal with too many of 'em). Corrected the post above. On a sidenote, Square, Stripe, Payment Ninja, <insert payments startup> so far seems to be lipstick on the pig that is the underlying debit/credit networks, often with a hefty price for said lipstick. The US badly needs a better version of Zelle that supports business transactions.
    - afarrell 8y ago
      For recurring/invoice transactions, you could use GoCardless: https://gocardless.com/en-ca/pricing/ https://gocardless.com/en-ca/pricing/ We don't use debit/credit card networks but bank account->bank account networks like ACH, SEPA, etc. Not yet available in the US. Coming soon though.
- TheRealDunkirk 8y ago
  People have to come to realize that nothing that gets said at this level should be taken literally. At the C-level of Fortune 500 companies, everything is crafted to create a narrative. People claiming that Twitter or Facebook has a political bias are missing the real point. For example, I think that Zuck really, truly, doesn't give a fink about political bias. How could he? He'd never get any sleep! In his position, all he cares about is the money. If that takes making this or that comment or public statement or testimony to Congress, it doesn't matter. It's all geared towards and carefully crafted towards keeping the money flowing. In that sense, he and the top brass at a place like Facebook are all political whores, so to speak. Selling their convictions for whatever keeps the spigot open.
tanilama 8y ago
It will be simplistic to assume such decision is tech-driven. Per my experience, it is mostly business driven. Who offers the best deal with the lowest price, gets the business. You'd be surprised, if you are a big enough customer, even better if you are a reputable one, how much catering the cloud provider would be willing to offer.
opportune 8y ago
Hadoop is an offline storage mechanism used for analytics, random access benchmarks barely matter compared to the overall cost of storage+compute so long as access time aren't atrocious
derekalyon 8y ago
This is Derek. We shared a bunch of stuff in this video: https://youtu.be/4FLFcWgZdo4 https://youtu.be/4FLFcWgZdo4 One of the interesting benchmarks was GridMix performance at scale. And overall performance for throughput-oriented workloads like Hadoop was generally strong. Most cloud benchmarking I have seen sticks to micro-benchmarks and single-node benchmarks, so they miss this.

deleted 8y ago

[deleted]

slackoverflower 8y ago

Just makes the eventual Google acquisition one bit easier.

mobilekid7 8y ago
That crossed my mind the minute I read the headline too. I can imagine folks at Google getting excited about this first step...
StudentStuff 8y ago
With Twitter's current state and Google's moves to kill their social side, this is unlikely. Why buy a moderate sized social network when you just killed your similarly sized social network?
- chipperyman573 8y ago
  If you're referring to G+, I don't think G+ was a "moderate sized social network" any more than twitter (although in opposite directions). Also, imo, calling them "similarly sized" would be incorrect, except perhaps in signed up users (from which google got a ton from the "login with google and your account is already setup", not daily active users)
  - StudentStuff 8y ago
    300 million monthly active users on G+ to Twitter's 335 million monthly active users, they were nearly identical in size. https://en.wikipedia.org/wiki/Google%2B#Growth https://en.wikipedia.org/wiki/Google%2B#Growth https://en.wikipedia.org/wiki/Twitter https://en.wikipedia.org/wiki/Twitter (see sidebar)
    - chipperyman573 8y ago
      Wow, really?? In that case, what caused G+ to fail? I was always under the impression that it was due to (low) DAUs.

echelon 8y ago

Correct me if I'm wrong, but this doesn't say what Twitter moved to the cloud. It could be literally anything, and may not in fact be core tweet or user data. This strikes me as a team (or teams) outgrowing the storage options that were offered internally and choosing to outsource to fit their needs. Isolated use for one business case, and not indicative of a broad movement to the cloud on the part of the whole org. I could be wrong, of course. I'd like to know more. And I'd certainly like to know what the engineers at Twitter think.

sidcool 8y ago
You are not wrong. The message here is that whatever data was transferred, Google was able to handle hundreds of petabytes of data. It could just be billions of photos of cats
- londons_explore 8y ago
  'cold storage' It's going to be logs data, backups, and users original uploaded data (rather than reencoded versions) that they hope will be valuable one day.
  - fhoffa 8y ago
    you missed this part "and our flexible compute Hadoop clusters"
- anvuong 8y ago
  billions photos of cats is invaluable data, mind you
  - inferiorhuman 8y ago
    Well Twitter is now part of the official White House record, isn't it?
andrewguenther 8y ago
The post on Twitter's side goes into more detail: https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/a-new-collaboration-with-google-cloud.html https://blog.twitter.com/engineering/en_us/topics/infrastruc... Relevant technical bits: > Today, we are excited to announce that we are working with Google Cloud to move cold data storage and our flexible compute Hadoop clusters to Google Cloud Platform. This will enable us to enhance the experience and productivity of our engineering teams working with our data platform. > Twitter runs multiple large Hadoop clusters that are among the biggest in the world. In fact, our Hadoop file systems host more than 300PB of data across tens of thousands of servers.

andrewguenther 8y ago

Is there anything new here? Twitter announced this move back in May of last year: https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/a-new-collaboration-with-google-cloud.html https://blog.twitter.com/engineering/en_us/topics/infrastruc... The video linked on the page is also from last August: https://www.youtube.com/watch?v=T1zjmNAuMjs https://www.youtube.com/watch?v=T1zjmNAuMjs Throw a 2018 tag on this?

boulos 8y ago

Disclosure: I work at Google Cloud (and directly with Derek and the Twitter team). I’ll try to edit this page tomorrow when I’m at a computer, but there’s much more information in Derek’s talk at NEXT [1]. They (rightfully) didn’t want to get into a detailed “this is what we saw on <X>”, but Derek alludes to their careful benchmarking across providers. While you should always assume smart people make economically reasonable decisions, Derek’s point about savings is about list price differences that result from total system performance (and not any sort of special discounting). I’m hoping a follow-up talk will let us say more about how the migration is going, while this talk was focused on the decision to move part of (!) their Hadoop environment to GCP. [1] https://m.youtube.com/watch?v=4FLFcWgZdo4 https://m.youtube.com/watch?v=4FLFcWgZdo4

fhoffa 8y ago
From the Twitter Hadoop to GCP talk that Boulos linked: - >500k cores - >300PB storage - >12,500 cluster size - >1T messages per day And there's also this other talk, "How Twitter Migrated its On-Prem Analytics to Google Cloud" - focused on their migration to BigQuery: - https://www.youtube.com/watch?v=sitnQxyejUg https://www.youtube.com/watch?v=sitnQxyejUg - 20 TB/ day of raw log data, >100k events/sec - Loading ~1TB/hour into BigQuery. - Serving 5,000+ complex queries / second. p99 ~300ms Disclosure: I'm Felipe Hoffa and I work for Google Cloud https://twitter.com/felipehoffa https://twitter.com/felipehoffa.
- opportune 8y ago
  At this scale I'm a little surprised Twitter saved money on the migration. My assumptions were that at this scale the upfront cost of migration trumped any kind of difference in annual maintenance, esp. considering you still do need an SRE team anyway who knows big data well enough to solve any and all software related problems
  - Aeolun 8y ago
    At that scale the team itself likely costs peanuts.

ranman 8y ago

I have had a hard time dealing with twitter engineering since the account activity API debacle.

sudhirj 8y ago

One thing to remember with the benchmarks is that when evaluating a cloud provider, the benchmarks are useless if talked about with absolute numbers or with regards to hardware specs. Because they're so horizontally scalable, it's more a question of average cost per compute operation or cost per (peta) byte. Both AWS and Google clouds are perfectly capable of running Twitter, and running any software that Twitter uses, even if the number of machines is different. The only "benchmark" applicable is actually the total negotiated cost of the machines required to get the job done. So it's not about being able to do things on Google cloud with fewer processors or less RAM or faster hard drives - Google was willing to give them a lower total cost of ownership for reasons known only to Google. Do not assume that Google (or anybody else) will give you a similar preferential deal. Ignore the "benchmarks".

sudhirj 8y ago
If you want to check which cloud is best for you, run your application on all of them, measure your average+median+p95 cost per whatever (tweet/post/reader/dollar of revenue/ticket/?). Then factor in which platform is easier to work with, has better tooling and community support, because until you hit a certain scale your time will always be more expensive. After you do all that, just start with Heroku and Postgres with some AWS Lambda, then move to ALB + AWS Fargate + RDS|Aurora / DynamoDB as you get bigger, then to NLB + ECS with a cluster of 20-80 On Demand and Spot Instances, then to a cluster of ARM instances on Spot. If you need to build your own datacenter at that point, you'll know. And you would have built a 12Factor app to make all of the above work, so migrating will be easy.
sciurus 8y ago
I don't think it's purely a measure of cost per compute operation. That assumes you would run the same software on any cloud using plain VMs or a service that slightly abstracts them. That may be true in this case since Twitter will continue to use Hadoop/Spark, but sometimes you can get a real advantage from switching to a service that only one cloud provider offers. For instance, someone in these comments pointed out that Twitter already migrated their anayltics workload to BigQuery. Evaluating BigQuery vs Redshift vs something else is not as clear cut as cost per compute operation.

gigatexal 8y ago

Huge win for Google. I hope this helps Twitter focus on features and not firefighting infrastructure issues.

SmellyGeekBoy 8y ago
Does Twitter need more features? Perhaps they could focus more on customer support and moderation.
- gigatexal 8y ago
  A downvote button would be nice. The ability to edit a tweet would also be nice. It’s 2019 after all, and we can edit things on other platforms. But yes, moderation would be great. Maybe they can leverage Google’s machine learning hardware and smarts.

rapsey 8y ago

Well hopefully google does not decide to abandon google cloud :)

advisedwang 8y ago
Hopefully Google doesn't abandon search!
- rapsey 8y ago
  They are king of search and distant third in the cloud game.
- samfisher83 8y ago
  Search+Ads:Google Windows+Office:microsoft Those are the sacred cows.
  - ocdtrekkie 8y ago
    Microsoft has been hacking off bits of dependency on Windows these days in order to push and support their cloud business, to the point I would argue Microsoft is aggressively trying to push people off of Windows+Office and onto Azure+365. Microsoft would far rather you use Linux on their cloud platform over having you use Windows literally anywhere else. I don't think Windows counts as a sacred cow anymore at Microsoft.
- halfjoking 8y ago
  Hopefully Google doesn't abandon human civilization, and choose intelligent machines over biological ones!

synaesthesisx 8y ago

[Removed]

jcmi 8y ago
LOL

temp231239 8y ago

Hun. I was under impression that at higher scale of operations building your own cloud is more economical. I guess money is not an issue for Twitter.

rjf72 8y ago
This thread, and site, tells the tale. Google is using this for marketing purposes. They already have some trite 2 minute marketing ad running as well. The deal that Twitter received is going to come with extensive incentives unavailable to typical operations. Twitter also was likely able to play the typical providers against each other. 'Oh, well AWS is offering me [x,y,z]. Can you do better?'
- rapsey 8y ago
  Well yeah but that is the case for anyone large enough.
pas 8y ago
They are moving batch jobs to GCP. Because there probably the occasional burst loads and the often underutilized capacity makes using a flexible compute fabric (aka cloud provider) a better choice. If they are moving everything, well, Netflix did that too. Maybe opportunity cost, maybe having your engineers work on your core product instead of running VMs is cheaper altogether.
- tybit 8y ago
  Much of Netflix’s spend would presumably be on CDNs, which they run themselves not on AWS.
  - opportune 8y ago
    uh correct me if I'm wrong but all CDN's that Netflix purchases also need to have a large storage cache backing them right? Meaning each CDN Netflix uses for local caching also requires a colocated datastore to circumvent their centralized-bandwidth issue.

tanilama 8y ago

I think by this announcement, it is probably safe to say, even at the size of Twitter, it is probably still more economical to handle your operations to bigger players. Hosting your own infrastructure makes less and less sense going forward, the cloud is the new king now.

kortilla 8y ago
Probably not safe to conclude anything. Twitter has very different requirements than something like a bank, an insurance company, an auto manufacturer, etc.
inapis 8y ago
I dont believe that this becomes a nail in the coffin for hosting your own infrastructure. This should still be evaluated on a case by case basis.

pizza 8y ago

The scale of these megacorps is crazy! I wonder how you even plan to move that much data. I know that Amazon has a service called Snowmobile for stuff like this. They probably did something similar here. "You can transfer up to 100PB per Snowmobile, a 45-foot long rugged sized shipping container, pulled by a semi-trailer truck." https://www.youtube.com/watch?v=8vQmTZTq7nw https://www.youtube.com/watch?v=8vQmTZTq7nw

fro0116 8y ago
> Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. https://what-if.xkcd.com/31/ https://what-if.xkcd.com/31/
- BerislavLopac 8y ago
  The mention of "SneakerNet" reminded me of old times when we had a bunch of old Macintoshes (pre-OSX) connected with a proprietary local network called LocalTalk. It was so unreliable that we had a couple of 3.5" floppy disks that we used to transfer files when it didn't work; the disks were labelled "LocalWalk". :D
pojzon 8y ago
Migrations like these take time and happen step by step. First you can try to replicate some data and move read instances. After that step by step the rest. Its really not possible to do that overnight or months even. Fyi: im currently migrating twice the Twitter size one in the world infrastructure from aws to azure now.
- tpetry 8y ago
  Isn‘t exporting 600TB of data from aws ridicously expensive?
  - deleted 8y ago
    [deleted]
  - 8y ago

dustinmoris 8y ago

Reading most comments here I feel like nobody wants to give a tiny bit of credit to either Twitter or Google, which I find disenchanting. It is true that any cloud provider would get good publicity by being able to say that Twitter runs on their cloud, but that is precisely the same reason why we shouldn't just shrug this off as a publicity/business decision, because just like Google might have tried to persuade Twitter other cloud providers will most certainly have tried their best too. If Twitter still went with GCP then I think it is only fair to assume that there must have been some real advantage in going with GCP. I don't think anyone who says this must have been a pure business decision is entirely honest with themselves, because a cloud which cannot handle Twitter's volume is no good even if it was entirely for free. I have nothing to do with Twitter (actually don't even like Twitter because it is such a toxic social media platform) and I don't like many things which Google does, but as an engineer myself I can only say that after using Azure, AWS and GCP for many years in a commercial setting that GCP is indeed years ahead. The quality, speed and reliability of GCP is second to none from my experience and I honestly couldn't say the same about AWS and especially not at all about Azure.

notyourwork 8y ago
> The quality, speed and reliability of GCP is second to none from my experience Can you be more specific on what reliability GCP is years ahead of the others? I'm guessing numbers will be pretty comparable on all platforms so if we want to declare outright winners it would be useful to cite a number to indicate where you experienced this.
privateSFacct 8y ago
the quality, speed and reliability of GCP is second to none? I hope this is a joke. AS someone who started on GCP, and got tired of the TERRIBLE quality and reliability of their docs among other issues saying that they are ahead is an absolute joke. I've found some corner cases with AWS, but got quick resolution even on things they have in "beta" - and consistency from documentation through to system is high. Secondarily, AWS seems to support even old tech FOREVER. I have a simpledb based app. That tech is 9 years old now. Still ticking. When you are building stuff up over time and can't afford to rebuild on the new hotness every other year, this is nice. Anyone know the revenue AWS and GCP generate? Seriously hard to believe GCP is so many years ahead in this space from my own experience.

cyberferret 8y ago

I've scoured the comments and article, but I might have missed it... What did Twitter migrate from?? Did they have their own data centre before moving to Google Cloud?

joatmon-snoo 8y ago
Yep. https://www.datacenterknowledge.com/archives/2015/07/13/custom-servers-process-tweets-in-twitter-data-centers https://www.datacenterknowledge.com/archives/2015/07/13/cust...
- userbinator 8y ago
  Another step backward for the decentralisation of the Internet... not that Twitter was decentralised in the first place, but it's unsettling to see Google taking over more and more of the "big pieces" of Internet.
  - zymhan 8y ago
    Funnily enough, Twitter and Google were/are neighbors in a QTS datacenter in Atlanta.
    - forgot-my-pw 8y ago
      Just moving across the street then? It's probably the fastest data transfer ever by physically moving all the hard drives!

Sample17 8y ago

<a href="https://luckypatchers.xyz/">lucky https://luckypatchers.xyz/">lucky patcher app</a>

thisgoodlife 8y ago

How are they gonna move all that 300 pb of data to Google's data center? A fleet of semitrailer trucks?

aboutruby 8y ago
For instance AWS has 50 Terabytes devices that they ship from your own data center to their data center: https://aws.amazon.com/snowball/ https://aws.amazon.com/snowball/
- kondro 8y ago
  They also have the 100PB Snowmobile. https://aws.amazon.com/snowmobile/ https://aws.amazon.com/snowmobile/
boulos 8y ago
Disclosure: I work on Google Cloud (with Twitter). Pipes! That data isn't static either. As Derek said in a sibling comment, they setup hundreds of Gbps of peering. Each 100 Gbps gets you a Petabyte per day (roughly). Planning the move and doing the tooling to keep stuff synchronized takes longer than the actual transfer once you've got say 800 Gbps (just 30 days ish). And again, you're going to have lots of data churn per month, so you want pipes anyway.

prepend 8y ago

This really makes me wonder if Twitter finally made up their mind who they will merge with.

partiallypro 8y ago
I too was thinking this, does this mean a future acquisition is not out of the question? Frankly, I find a Google owned Twitter to be a terrifying prospect. Not that Jack is doing a great job there...but I'd much rather they be owned by multiple companies in a partnership than by one company. Twitter, imo, is far far more powerful than Facebook in driving world events given the leadership and journalists on the platform.

egorfine 8y ago

The next global Twitter downtime? Their credit card failed at Google Payments, acc gets marked for fraud and no one in support can help them recover. Twitter is gone for good.

jasonvorhe 8y ago
This has long been fixed.
- marcinzm 8y ago
  Every few weeks there's a story about something in Google's automated machinery failing a customer with no recourse except to post an angry blog/twitter/hackernews note and hope it get's noticed. The core issue (bad and powerless customer service) isn't fixed and all that changes is the particular way it may bite you.
  - munchbunny 8y ago
    When you're Twitter sized, that was never a problem to begin with. For mere mortals, that continues to be a problem.
  - el_isma 8y ago
    Where's twitter gonna twit if they go down?
- wil421 8y ago
  6-7 months isn’t too long[1]. If this is the fix you were taking about it happened last year. According to Wikipedia GCP started in 2008 and generally available in 2011 as App engine. It took them 7-8 years to solve billing? [1]https://medium.com/@serverpunch/why-you-should-not-use-google-cloud-75ea2aec00de https://medium.com/@serverpunch/why-you-should-not-use-googl...
- egorfine 8y ago
  Not even close.
- 8y ago

opportune 8y ago

side note, if you want to know why distributed infrastructure engineering has been the place to be recently, think about the fact that engineers making $200-500k/year are making decisions that can save big corporations $1-5mm per year in compute/storage costs

tome 8y ago

I'm curious what happens to everyone at Twitter who was employed in managing their old data centre.

thecanman 8y ago
Well I'm sure they still have amazing career prospects just based on their resumes.
- tome 8y ago
  Sure, but what's the actual process for dealing with them? Will they be let go in small batches as the infrastructure moves over to GCP, or what?
Rafuino 8y ago
This isn't all of Twitter's workloads moving to GCP. Seems like just cold storage and some aspects of their Hadoop workloads.

maloneyg 8y ago

More beneficial for Google I believe.

patrickg_zill 8y ago

What Twitter is paying for and the level of service they receive from Google, is very different from the typical GCP pricing and service level that would be available for say, your company... Realistically, it can't be considered as the same service, can it?

yani 8y ago

This feels like marketing material. Twitter jumped over to GCC because of the offer they received.

SketchySeaBeast 8y ago
The first clue was that the news was being hosted on https://cloud.google.com/ https://cloud.google.com/

kerng 8y ago

The issue with GCP and with Google in general is support. I'm certain that's not an issue for Twitter as there are probably multiple people on call just for the Twitter deal. But it might mislead the average company down a bad path believing there is decent support with GCP. Any custom issues (like billing) and it's a headache dealing with Google.

keithnoizu 8y ago

GCE blows, good luck with that.

iheartpotatoes 8y ago

wow. talk about "loss of service challenges".

dugluak 8y ago

Next step - acquire twitter?

smt88 8y ago
I definitely wonder if Twitter's management considered this. Any price for Twitter goes down by at least a few million if no major cloud migration is necessary. That said, a few million is a rounding error on a price Google would pay for Twitter.

cr0sh 8y ago

I know my comment won't move this discussion forward in any fashion, but I have to ask anyway. What was this data, and why did they feel they needed to keep it? I mean - 300PB is nothing to sneeze at. I can understand wanting to keep important data around (IP, for instance, or recent server logs for the last 30 days or even a year) - but this amount seems to go well beyond that. In fact, it almost seems like they have stored every single tweet ever written. In addition to probably all of their logs, and who knows what else. Why are they keeping all of that information? If it is all the tweets ever written - they don't have copyright to it, so such information wouldn't belong to them... I'm not a big twitter user - can I search for a tweet on their system that I (or someone else) made say...5 years ago or longer? If this is a bunch of tweets - is that why they keep them around? Are people in general aware of this? That is - if they are tweets - that Twitter has stored all of them, forever and ever, and that they are all still accessible (if not by the general public, then by law enforcement at the very least)? If they are tweets - do users have any recourse at being able to remove those tweets (aside from those that they likely are legally bound to keep - I am thinking of the current POTUS's tweets, for instance)? What is the ultimate purpose behind keeping all of that data? Even if it wasn't a bunch of tweets, whatever it is can't really be that useful otherwise, except for maybe the most recent stuff created in the past year, at best. Things like that - logs, etc - might be destined for a rotated "cold storage" and would ultimately "age out". Accounting records (financial data, etc) probably isn't that great a percentage of the data (if at all?) - but whatever was there, too, would likely need to be saved for a greater range - but even there maybe only 10-15 years worth (and it certainly wouldn't even comprise a fraction of the bulk). So what is it? Why is it being kept? To what purpose? While 300PB isn't that great of an amount in physical terms (that is, the storage medium probably doesn't take up a great amount of space in a datacenter), it still isn't anything tiny, either (especially depending on what medium was used - I mean, I doubt they are storing it all on a ton of 256GB microSD cards - though that does bring up an interesting idea/thought to mind - but I digress). I'm of the thought that companies shouldn't just "throw away" data - but it should be tempered by "importance to humanity over time". Not everything should be kept - but, for instance, it would have been nice if we (humanity) had kept around the original transmissions from the surface of the moon, or the blueprints to the rocket that took humans there. But we probably don't need all the various interoffice memoranda and fluff at the various NASA contractors (we probably don't need the accounting records - but BOMs might be useful). That's just a couple of examples I can think of off the top of my head, but history is rife with instances of companies imploding or being bought out or restructured in such a manner that data is just "destroyed" instead of kept. Or lost, or otherwise made unretrievable to humanity. Here with Twitter - an active company to be sure - we have the seemingly exact opposite - almost like r/datahoarders were in charge. ...and that should concern people a bit, regardless of which company it is - but especially a company like Twitter.

yeukhon 8y ago

So what will happen to the small Cloud team in NY? I thought they were developing their own dc.