4 ms·

> as I have seen no real change in the service in the past ~5 years I've used it. You underestimate the effort it takes to keep a service as large as Twitter r

> as I have seen no real change in the service in the past ~5 years I've used it. You underestimate the effort it takes to keep a service as large as Twitter running. Constant software upgrades to maintain high standard of security, inevitable technological break-age, deprecated dependencies, operations, root-causing failures, migrations (you can't avoid hardware migrations; hardware will die out), planning fallacy, etc, etc.

killdozer 4y ago
Yes it's incredible to see so many people on _hacker news_ who don't understand how much work it is to keep even moderately large (low millions of users) services running.
- wink 4y ago
  Yes, and repeatedly people point out that high dozens or low hundreds should be enough but they have thousands (allegedly). I think the point you are missing that especially tech people might have been using twitter for 10 years like a public IRC client. Open App/Browser, read messages, write messages, complain about lack of linear timeline, close app. Yes, it has grown, but they added a million features that probably none of the people here asked for. We don't necessarily follow Justin Bieber where the system needs to be able to handle 2m users connected, we follow a kernel developer with 2000 followers. I guess it's a marvel of engineering that's mostly needed for whatever method they found to make money off, I'm still asking myself this after these years.
shaklee3 4y ago
I thought Twitter was mostly on AWS?
- boardwaalk 4y ago
  I don't think most of the issues mentioned are mitigated by AWS, sadly.
- everfrustrated 4y ago
  According to whistleblower Peiter Zatko they had two datacentres. He reports that they were worried that if a DC went offline they might not be able to bring it back up again! Their servers in their DCs were running unpatched operating systems with no disk encryption! I highly recommend reading his full report. The level of disfunction he reports is insane.
anthropodie 4y ago
Here's story for you A team of 30 works at zerodha for serving 6.2 million users. They are largest trading broker in India. There story is worth a read https://zerodha.tech/blog/hello-world/ https://zerodha.tech/blog/hello-world/ Edit: Apparently they have 1200 employees for operations and everything else. Thanks to child comment for pointing that out.
- kingofpandora 4y ago
  30 member tech team.
- sangnoir 4y ago
  So what are the other 1170 people doing? 10 managers to each person in tech? Sounds like bloat /s
  - anthropodie 4y ago
    No they dont have managers or a separte QA team. 30 is their entire tech team which handles everything. They are largest broker so others might be sales, legal, marketting, hr guys
- FartyMcFarter 4y ago
  > They are largest trading broker in India. Trading brokers don't need to make spam-filtering tools, CSAM filters, ads tools, handling huge video/image data, or handling a global audience. I suspect most brokers's tech is way simpler than Twitter's.
concinds 4y ago
> Constant software upgrades to maintain high standard of security https://www.washingtonpost.com/technology/interactive/2022/twitter-whistleblower-sec-spam/ https://www.washingtonpost.com/technology/interactive/2022/t... "Zatko’s complaint alleges he had warned colleagues that half the company’s servers were running out-of-date and vulnerable software and that executives withheld dire facts about the number of breaches and lack of protection for user data, instead presenting directors with rosy charts measuring unimportant changes." Firing incompetent people might improve things, not make them worse.
- sangnoir 4y ago
  One of the reasons I was looking forward to the trial was to hear what (if any) communication between Mudge and Elon('s team) had been dredged up in discovery.
hdjjhhvvhga 4y ago
That's why WhatsApp had 50 employees before the acquisition - it's very difficult to do these things if you have only 3-5 people.
Kye 4y ago
People also underestimate what it takes to keep advertisers happy. It might take a whole team just to manage one big account, and that account will cover the cost of the staff. With Elon threatening to fire half the company, it's no wonder brands are pulling out/pausing with no way to know who will be handling their accounts. A General Mills or Audi wants to keep working with the people it's come to trust over the years, not wonder if this new CEO is going to hand them off to some intern who just graduated with a major in advertising.

killdozer 4y ago

Yes it's incredible to see so many people on _hacker news_ who don't understand how much work it is to keep even moderately large (low millions of users) services running.

wink 4y ago
Yes, and repeatedly people point out that high dozens or low hundreds should be enough but they have thousands (allegedly). I think the point you are missing that especially tech people might have been using twitter for 10 years like a public IRC client. Open App/Browser, read messages, write messages, complain about lack of linear timeline, close app. Yes, it has grown, but they added a million features that probably none of the people here asked for. We don't necessarily follow Justin Bieber where the system needs to be able to handle 2m users connected, we follow a kernel developer with 2000 followers. I guess it's a marvel of engineering that's mostly needed for whatever method they found to make money off, I'm still asking myself this after these years.

shaklee3 4y ago

I thought Twitter was mostly on AWS?

boardwaalk 4y ago
I don't think most of the issues mentioned are mitigated by AWS, sadly.
everfrustrated 4y ago
According to whistleblower Peiter Zatko they had two datacentres. He reports that they were worried that if a DC went offline they might not be able to bring it back up again! Their servers in their DCs were running unpatched operating systems with no disk encryption! I highly recommend reading his full report. The level of disfunction he reports is insane.

anthropodie 4y ago

Here's story for you A team of 30 works at zerodha for serving 6.2 million users. They are largest trading broker in India. There story is worth a read https://zerodha.tech/blog/hello-world/ https://zerodha.tech/blog/hello-world/ Edit: Apparently they have 1200 employees for operations and everything else. Thanks to child comment for pointing that out.

kingofpandora 4y ago
30 member tech team.
sangnoir 4y ago
So what are the other 1170 people doing? 10 managers to each person in tech? Sounds like bloat /s
- anthropodie 4y ago
  No they dont have managers or a separte QA team. 30 is their entire tech team which handles everything. They are largest broker so others might be sales, legal, marketting, hr guys
FartyMcFarter 4y ago
> They are largest trading broker in India. Trading brokers don't need to make spam-filtering tools, CSAM filters, ads tools, handling huge video/image data, or handling a global audience. I suspect most brokers's tech is way simpler than Twitter's.

concinds 4y ago

> Constant software upgrades to maintain high standard of security https://www.washingtonpost.com/technology/interactive/2022/twitter-whistleblower-sec-spam/ https://www.washingtonpost.com/technology/interactive/2022/t... "Zatko’s complaint alleges he had warned colleagues that half the company’s servers were running out-of-date and vulnerable software and that executives withheld dire facts about the number of breaches and lack of protection for user data, instead presenting directors with rosy charts measuring unimportant changes." Firing incompetent people might improve things, not make them worse.

sangnoir 4y ago
One of the reasons I was looking forward to the trial was to hear what (if any) communication between Mudge and Elon('s team) had been dredged up in discovery.

hdjjhhvvhga 4y ago

That's why WhatsApp had 50 employees before the acquisition - it's very difficult to do these things if you have only 3-5 people.

Kye 4y ago

People also underestimate what it takes to keep advertisers happy. It might take a whole team just to manage one big account, and that account will cover the cost of the staff. With Elon threatening to fire half the company, it's no wonder brands are pulling out/pausing with no way to know who will be handling their accounts. A General Mills or Audi wants to keep working with the people it's come to trust over the years, not wonder if this new CEO is going to hand them off to some intern who just graduated with a major in advertising.