3 ms·

> In other words, in all likelihood, Twitter is ahead of the norm, ahead of the average, just not up to the same standard set by the leaders in tech. Really? I

by chx 4y ago

> In other words, in all likelihood, Twitter is ahead of the norm, ahead of the average, just not up to the same standard set by the leaders in tech. Really? It came out Twitter doesn't have a dev/staging environment. Source: https://twitter.com/lauren_feiner/status/1569695337190944775 https://twitter.com/lauren_feiner/status/1569695337190944775 in the same thread, "Twitter can't assure regulators that it's able to delete all data at a user's request because it's unsure where all that data lives" Earlier we have learned > “Even a temporary but overlapping outage of a small number of datacenters would likely result in the service [Twitter] going offline for weeks, months, or permanently,” according to Zatko’s whistleblower disclosure. (Twitter has criticized Zatko and broadly defended itself against the allegations, saying the disclosure paints a “false narrative” of the company.) Quoted in https://www.cnn.com/2022/09/12/tech/twitter-data-center-california-heat-wave/index.html https://www.cnn.com/2022/09/12/tech/twitter-data-center-cali... just yesterday. All in all, there's a picture here which says Twitter simply doesn't have a handle on its own infra any more, it can't replicate it for dev/staging purposes, it doesn't fully know where the data goes and if it crashed they couldn't rebuild it. You call that ahead of the norm?

itsdrewmiller 4y ago
Do either of these allegations pass the smell test? They seem pretty wild.
pkphilip 4y ago
Facebook is making very similar claims. Seems like Big Tech has decided the way to deal with data privacy complaints is to pretend that they don't even know what data they are storing and where. https://www.vice.com/en/article/akvmke/facebook-doesnt-know-what-it-does-with-your-data-or-where-it-goes https://www.vice.com/en/article/akvmke/facebook-doesnt-know-... I wonder if ordinary citizens can make that same claim when it comes to their stuff - "No, officer, I don't even know whether there are any drugs in my possession because I don't even know what is in my house and where"
ralph84 4y ago
> It came out Twitter doesn't have a dev/staging environment. And? Dev/staging is fine for enterprises that don't have to operate at Internet scale, think having a build script means they have CI/CD, and are doing waterfall development even if they call it agile. For truly Internet-scale services the cost/benefit of trying to build a staging environment that would have any usefulness doesn't pencil out. Canary deploys with well thought-out rollback plans works fine.
williamsmj 4y ago
Re: test environments, of course Twitter has test environments, but if you're asking does Twitter have "staging.twitter.com" that matches the load, configuration and topolgy of prod, the answer is probably no because, like many companies at this scale they choose to handle moving stuff from dev to production with feature flags and incremental rollout techniques instead of the dev/staging/prod promotion process that is conventional wisdom at smaller scales. This choice increases velocity (especially if you have the platform team to support incremental rollouts and rollbacks) and building (or simulating) test environments that match prod would be insanely expensive, if it were even possible. Like monorepos, this approach to ops might seem strange if you've only worked at small companies, and even then it's something reasonable people can disagree about. But I'm not convinced it's a useful discussion to have with an 88 year old senator. And it's not a sign that a company doesn't take engineering or security seriously, or that it has no idea what it's doing. Re: resiliency, one of Twitter's datacenters is offline right now and twitter.com is up (https://www.cnn.com/2022/09/12/tech/twitter-data-center-california-heat-wave/index.html https://www.cnn.com/2022/09/12/tech/twitter-data-center-cali.... So clearly the "small number" mudge is referring to is greater than one. Beyond that, yes, I'm absolutely sure that the loss of a "small number" of datacenters could take twitter offline and even result in permanant data loss (the data is in datacenters too!), because that statement is true of literally all tech companies. Twitter's expected uptime is a bunch of 9s. It's not 100.0%. The fact that it's not 100% is not evidence it "simply doesn't have a handle on its own infra any more". And FWIW, resiliency appears to be pretty high for Twitter relative to the rest of the industry. They were down for 40 minutes early this summer, and haven't had an outage > 1 hour for 6 years (https://www.theguardian.com/technology/2022/jul/14/twitter-experiences-longest-global-outage-in-years https://www.theguardian.com/technology/2022/jul/14/twitter-e...). Facebook (which I think most of us think of as pretty good at engineering, relative to the clown show most people seem to think Twitter is) was offline for 5.5 hours last year (https://en.wikipedia.org/wiki/2021_Facebook_outage https://en.wikipedia.org/wiki/2021_Facebook_outage).

by chx 4y ago

itsdrewmiller 4y ago

Do either of these allegations pass the smell test? They seem pretty wild.

pkphilip 4y ago

Facebook is making very similar claims. Seems like Big Tech has decided the way to deal with data privacy complaints is to pretend that they don't even know what data they are storing and where. https://www.vice.com/en/article/akvmke/facebook-doesnt-know-what-it-does-with-your-data-or-where-it-goes https://www.vice.com/en/article/akvmke/facebook-doesnt-know-... I wonder if ordinary citizens can make that same claim when it comes to their stuff - "No, officer, I don't even know whether there are any drugs in my possession because I don't even know what is in my house and where"

ralph84 4y ago

> It came out Twitter doesn't have a dev/staging environment. And? Dev/staging is fine for enterprises that don't have to operate at Internet scale, think having a build script means they have CI/CD, and are doing waterfall development even if they call it agile. For truly Internet-scale services the cost/benefit of trying to build a staging environment that would have any usefulness doesn't pencil out. Canary deploys with well thought-out rollback plans works fine.

williamsmj 4y ago

Re: test environments, of course Twitter has test environments, but if you're asking does Twitter have "staging.twitter.com" that matches the load, configuration and topolgy of prod, the answer is probably no because, like many companies at this scale they choose to handle moving stuff from dev to production with feature flags and incremental rollout techniques instead of the dev/staging/prod promotion process that is conventional wisdom at smaller scales. This choice increases velocity (especially if you have the platform team to support incremental rollouts and rollbacks) and building (or simulating) test environments that match prod would be insanely expensive, if it were even possible. Like monorepos, this approach to ops might seem strange if you've only worked at small companies, and even then it's something reasonable people can disagree about. But I'm not convinced it's a useful discussion to have with an 88 year old senator. And it's not a sign that a company doesn't take engineering or security seriously, or that it has no idea what it's doing. Re: resiliency, one of Twitter's datacenters is offline right now and twitter.com is up (https://www.cnn.com/2022/09/12/tech/twitter-data-center-california-heat-wave/index.html https://www.cnn.com/2022/09/12/tech/twitter-data-center-cali.... So clearly the "small number" mudge is referring to is greater than one. Beyond that, yes, I'm absolutely sure that the loss of a "small number" of datacenters could take twitter offline and even result in permanant data loss (the data is in datacenters too!), because that statement is true of literally all tech companies. Twitter's expected uptime is a bunch of 9s. It's not 100.0%. The fact that it's not 100% is not evidence it "simply doesn't have a handle on its own infra any more". And FWIW, resiliency appears to be pretty high for Twitter relative to the rest of the industry. They were down for 40 minutes early this summer, and haven't had an outage > 1 hour for 6 years (https://www.theguardian.com/technology/2022/jul/14/twitter-experiences-longest-global-outage-in-years https://www.theguardian.com/technology/2022/jul/14/twitter-e...). Facebook (which I think most of us think of as pretty good at engineering, relative to the clown show most people seem to think Twitter is) was offline for 5.5 hours last year (https://en.wikipedia.org/wiki/2021_Facebook_outage https://en.wikipedia.org/wiki/2021_Facebook_outage).