19 ms·

More than 1k people at Twitter had ability to aid hack of accounts

382 points by theBashShell 6y ago

pmiller2 6y ago
This is why internal tools that can modify account settings and such need to have audit trails.
- raz32dust 6y ago
  I would be really surprised if they did not have audit trails. What gives you the impression they did not? The suspicion is that the credentials were stolen via social engineering. I wonder if employees needed 2FA to log in to these tools.
  - hombre_fatal 6y ago
    > The attackers successfully manipulated a small number of employees and used their credentials to access Twitter’s internal systems, including getting through our two-factor protections. https://blog.twitter.com/en_us/topics/company/2020/an-update-on-our-security-incident.html https://blog.twitter.com/en_us/topics/company/2020/an-update...
- ceejayoz 6y ago
  It probably does, and it probably wouldn't have stopped this.
  - bawolff 6y ago
    > probably wouldn't have stopped this. Uh yes, that is how audit trails work
    - deleted 6y ago
      [deleted]
    - wil421 6y ago
      How does auditing itself prevent a present or future attack? Auditing and what you fix during audits are reactive.
      - halfjew22 6y ago
        It's like saying a boat's wake slows down the boat. Sometimes you've just got to wonder what people are thinking
      - calvinmorrison 6y ago
        100%. My work has really great auditing tools. I use them often to understand actions by other that are routine. It still doesn't prevent a employee emailing a datacenter to rack a malicious device or give someone service without paying. Record trails are not auditing. They are records. Auditing, post mortems, whatever diagnose the situation afterwards. At the end of the day Uber can't stop a driver from kidnapping people, but it can provide documentation and gps coordinates to police. My point is companies need reasonable records and audit policies and when _really bad stuff happens_ you call in the big guns for the arm of the law. At some point you also need to trust staff and weigh that against mistakes and malicious intent. In short, security remains an imperfect balance of practicality
    - hombre_fatal 6y ago
      Can we do without the condescending "Uh" and "Um" on HN? An audit trail would tell you who was social-engineered, but it wouldn't have prevented the attack in the same way Wikipedia's revision history doesn't keep you from vandalizing it.
      - TwoBit 6y ago
        It wouldn't even necessarily tell you who was social-engineered. My understanding of this case was that CS tool credentials were posted globally in their Slack.
        bawolff 6y ago
        In fairness, i don't think its a real audit trail if the tool doesn't have an effective per-user password.
    - halfjew22 6y ago
      And in today's "Pompous Commenter That Didn't Read the Article News": >But while logging helps with investigations, only alarms or constant reviews can turn logs into something that can prevent breaches.
    - manigandham 6y ago
      Auditing tells you what happened, it doesn't prevent it from happening. If they have logs then they can use it in the future (and it seems they do) to design better protections but only active alarms and security controls can prevent something happening in real-time. However that does raise the question of why Twitter ever needs such access to someone's account in the first place, especially without a combination of approvals to get that access.
      - suizi 6y ago
        "I forgot my password, please reset it for me."
akersten 6y ago
Kind of sensationalist. There's thousands of people that have the ability to drain your bank account right now. Your average call center employee wields immense power. The real story here is Twitter's lack of spear-phishing training for their support staff, not support employees have access to support tools.
- hombre_fatal 6y ago
  Education programs only move the needle, not something you do instead of minimizing the number of attack vectors. For example, education programs do squat against me attacking the employees directly (targeted malware, getting on their computer somehow, offering each of the thousands of employees $10,000 for temp access to their account). And each additional employee only strengthens my attack.
- dguido 6y ago
  It's not sensationalist when you realize it directly contradicts Twitter's prior statements from just last year about it: > Twitter, in a statement, said it is aware that "bad actors" will try to undermine its service and that the company "limits access to sensitive account information to a limited group of trained and vetted employees." https://www.npr.org/2019/11/06/777098293/2-former-twitter-employees-charged-with-spying-for-saudi-arabia https://www.npr.org/2019/11/06/777098293/2-former-twitter-em... 1,000 people, including contractors outside the company, is not a "limited group of trained and vetted employees." It's news because they misled people about their security, again.
  - ceejayoz 6y ago
    > 1,000 people, including contractors outside the company, is not a "limited group of trained and vetted employees." That's not necessarily true. 20% of the company could fairly reasonably be deemed "limited", and there being a thousand of them doesn't mean they're not trained on their tasks.
    - monsieurbanana 6y ago
      We'll have to agree to disagree on what we consider fairly reasonable to call "limited".
jbob2000 6y ago
I now understand why the bank I work for creates the separation of duties; the person who builds the system has no access to it, and the person with access has no idea how it works. As a developer, it frustrates the shit of out me because I can’t deploy fixes quickly or easily diagnose issues. But yep, there are 3 people that have access to the production databases that hold account info and they aren’t developers, just managers with no clue what to do once they log in. I also worked for a company that sold software to lawyers. We had a feature that would alert the client any time a member of our company accessed their data. I think we called the feature something like “fire call”, because if you tripped it without informing the client, you’d get a call informing you that you’d been fired.
- lowdose 6y ago
  > there are 3 people that have access to the production databases that hold account info and they aren’t developers, just managers with no clue what to do once they log in. Just for my curiosity is this your observation or is this a company assumption?
  - jbob2000 6y ago
    Hmm I think it’s just our group, we have a Production support team that holds the keys, and there’s only 3 of them that can access my app. For example, if I want to change an environment variable, I can’t just log into the cloud console or run a cli command. God no. That would be too easy. I have to write a script for this team to run. This script is entered into an authorization app where a few parties “sign off”, at which point the prod support team can log in to the authorization app and click Deploy. This app then runs my deployment script against our app container to update the env variable. Accessing and doing DB work follows a similar process.
    - ladberg 6y ago
      Reading this makes me happy! Always good to see people taking security seriously.
dzonga 6y ago
twitter, seems to have a cowboy engineering culture. that's why one of their exec's blamed rails for their failure to combat harassment[0]. n I bet now, if they still ran rails, it would've been blamed lol. [0]: https://char.gd/recharged/daily/twitter-blames-ruby-on-rails-for-harassment https://char.gd/recharged/daily/twitter-blames-ruby-on-rails...
- DetroitThrow 6y ago
  "...a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem" To me, this is analogous to the perhaps undeserved "the internet is a series of tubes" lampooning, but I'm still chuckling how they managed to word that so poorly.
  - spoopyskelly 6y ago
    It sounds like a perfect answer to those claiming "harassement" is a technology problem.
- suizi 6y ago
  Emails are overrated as an anti-troll measure. Anyone can make a trash mail in seconds.
laughinghan 6y ago
accounts with more than 10,000 followers should at least need two people to change key settings For accounts that could start a war this might be necessary, but for celebrities with >10K followers this sounds expensive and unnecessary to me. To me, it seems like you could instead ensure the admin view of every account has a timestamped log of recent settings changes, including changes done by admins, with a link to the profile of the admin responsible, and a button to suspend that admin account with one click. This way, the security team could've seen that Elon Musk's account had just been reset by J. Random Employee minutes before tweeting the suspicious bitcoin tweet, messaged J. on Slack to be like "hey did you do that?", and suspended the compromised admin account within minutes. Sure, some accounts might be briefly compromised initially, but it would be resolved in minutes and not the hours that it took Twitter, right? That seems fine for what should be a relatively low-likelihood, high-expense attack like compromised admin account (of course, you have to ensure that is the case).
- foota 6y ago
  I think the long tail was in undoing the actions made by the attackers. Resetting passwords, emails, etc.,.
  - laughinghan 6y ago
    No, according to The Block, @elonmusk repeatedly tweeted the scam at 4:17pm, 5:19pm, and 5:32pm, a span of 90 minutes, and the final scam tweet was at 6:05pm from @KimKardashian. An hour after @elonmusk's first scam tweet, 7 celebrity or corporate accounts had tweeted the scam, all with the same Bitcoin address. With the two-click system I described, how many compromised admin accounts would you expect the security team to have been able to suspend by then? 8 more celebrity accounts went on to tweet the scam, plus @elonmusk and @kanyewest repeating the scam tweets. https://www.theblockcrypto.com/post/71906/twitter-account-hacks-timeline https://www.theblockcrypto.com/post/71906/twitter-account-ha...
    - londons_explore 6y ago
      If your database system doesn't have a complete audit log of all fields (most databases have this capability, but more often than not it's disabled), it's possible that the mere act of reverting account ownership might remove data needed for tracing down what happened. Sure, it's a sucky position to be in, but I can see why they might have been hesitant to dive right in and start trying to undo damage before understanding what had happened.
sloshnmosh 6y ago
Ha! Did you see in that article that the head of cyber security for AT&T added his two cents in shaming Twitter? AT&T was just in the news recently where employees were accepting bribes that allowed criminals to swap SIMs steal bitcoins from AT&T customers. Unbelievable.
- manquer 6y ago
  Not just bitcoins, SIM swapping is how @jack twitter account got comprised as well.
- onetimemanytime 6y ago
  Thousands of employees, each with their own financial problems and dreams...you're bound to find a taker. Money moves mountains
  - hn_throwaway_99 6y ago
    Which is why once your company is big enough, you should need 2 employees who are unfamiliar to each other to sign off on high value operations.
    - onetimemanytime 6y ago
      Bingo. Corrupting two is much less likely.
      - tialaramex 6y ago
        The key trick isn't so much the two as that they're randomly selected. I moved a large amount of money a few years back to buy my home (I do not like debt, so I saved up until I could afford somewhere to live, then I bought it) The bank's web site lets you type in any amount of money but then it says politely that you can't do this from the web site, please call the bank. I called the bank (they always pick up in 2-3 rings, I've worked with one of their founders, ensuring this was one of the key ideas behind the bank) and explained what I wanted to do. The nice lady took down all the details and then she explained that now one of her colleagues would be randomly selected to call me back and confirm everything and we hung up. Sure enough, less than a minute later another of the people from the bank called (with the agreed password for when the bank calls me) and had me read out all the transaction details again, at which point the transaction was confirmed. Think about that scenario as a bad guy trying to corrupt it. You bribe one employee to pretend someone called and authorised a huge transfer. OK. But then a different random employee has to confirm it. How do you bribe them? You have no way to know who it will be! Do you try just bribing every single employee who works the phones? Not very practical. The other thing banks do is they background check employees. You can't test for "willing to take bribes" but you can weed out potential hires with previous convictions for financial crime, or debt problems. I've had checks like that for jobs touching sensitive personal information.
wiradikusuma 6y ago
Anyone watched Westworld? The whole enterprise is destroyed (almost) by 2 low level employees. It's either a complete blooper in the script or --after I read this article-- reflective of the real world that I don't know about. Your take?
throwaway220720 6y ago
For comparison, at Google in 2011, I was one of ~10 or so engineers that had the ability to view private Gmail or Gplus data (access that was heavily documented and audited). That being said, Google did have to go through it's own public humiliation [1] to put a system like that in place. https://gawker.com/5637234/gcreep-google-engineer-stalked-teens-spied-on-chats https://gawker.com/5637234/gcreep-google-engineer-stalked-te...
- necovek 6y ago
  Don't all engineers working on Gmail theoretically have the same access by conspiring with a code reviewer or two? It ultimately comes down to the person involved and I do not believe anyone can control the human factor.
  - throwaway220720 6y ago
    They can easily build and view their own versions of the gmail stack, but they would not be able to generate auth tokens to decode the private data of accounts they did not have passwords for.
    - necovek 6y ago
      I was more thinking of deploying trojan-code into the production service (as a trivial example, allow a special password to access any account): it can't be practical to vet every production service change through too many people. You seem to suggest that you are using an encryption key based on the password or oauth token on login, which is great to hear, which stops the simpler forms of trojans like the example above. That makes it much more involved to achieve the same (and login from new computer reports make it harder too), especially undetected (because it has to happen over a short period) but not impossible (thinking of cases like just making a new API endpoint or perusing an existing one to store actual content in an often unlooked at log file/service).
- koheripbal 6y ago
  I almost wonder if government officials should be outright banned from using any private messaging platform that isn't hosted by the government itself. There is just too much power in information.
imvetri 6y ago
Title corrected : More than 1k people at Twitter had ability to aid hack and chose not to.
- deleted 6y ago
  [deleted]
  - deleted 6y ago
    [deleted]
- amf12 6y ago
  > Title corrected : More than 1k people at Twitter had ability to aid hack and chose not to. This is a stupid way of looking at it. Similarly: - X number of people owned guns but they chose not to go do a mass shooting. - X number of cops could kill a black person, they chose not to. While it's a good thing that majority of the people know right from wrong, morality, etc, we still need to ensure one person can't do significant damage. The fact that there are 1000s of individuals that could have hacked is not a good thing.
KingOfCoders 6y ago
There is all this talk from those successful companies about security and what you should do with your keys and they open source hardware secret stores and brag about it and they fail at the most basic security operations.
sunilkumarc 6y ago
On a different note, online presence is becoming very important and with remote work culture gaining traction, having a good online presence has become a must have asset. I bought a course on building Twitter audience and been able to improve my following significantly from past 2 months. Twitter link: https://twitter.com/sunilc_ https://twitter.com/sunilc_ If you're looking to increase your social presence too, here's the course that I found very useful: https://gumroad.com/a/238777459/PBkrO https://gumroad.com/a/238777459/PBkrO
- accurateappL 6y ago
  Spam motivational quotes and hope people retweet and like?
alpb 6y ago
Worth mentioning only 5,000 people work at Twitter.
- Jaruzel 6y ago
  There are ~330 million active twitter users, which means 330,000 users per employee with access to admin accounts. That ratio is massively high compared to a large corporate (i.e, a global bank). In a typical global bank lets says there are 100,000 employees, with about 25-50 IT people with the rights to admin accounts (from first line support to third line engineers) that's only 2,000-4,000 users per IT admin person. Based on that, I'm surprised that it's only 1,000 staff members in Twitter with admin access, and not the whole company.
  - amf12 6y ago
    > There are ~330 million active twitter users, which means 330,000 users per employee with access to admin accounts. I think we should look at how many daily requests they get to reset account access settings (that cannot be done automatically - via some system rather than through these 1k users).
  - alpb 6y ago
    I am not sure why this bank example keeps coming up. Almost no twitter user tries to contact support like they contact their teller for their bank. It’s really bad that 20% of the company had access to user data. No wonder it was abused in the past. https://www.buzzfeednews.com/article/alexkantrowitz/how-saudi-arabia-infiltrated-twitter https://www.buzzfeednews.com/article/alexkantrowitz/how-saud...
- thinkingemote 6y ago
  Twitter is an advertisement company so most of the employees will be in sales and marketing. Those probably don't need admin access.
Laforet 6y ago
Right, thousands of people with admin access and nobody could help me reinstating my API access....
- OffensiveTomato 6y ago
  Apparently you gotta have that coveted Verified badge or be an influencer of some sort
austincheney 6y ago
This should be a wake up call. Thank god the malicious messaging was only limited to a tiny Bitcoin scam. Imagine if they had pulled this off on the accounts of national leaders to stir hostilities or violence. What is the recourse for this kind of failure? I suspect there is none. Twitter is shielded from lawsuits for its content. If this is provably negligent behavior and resulted in actual physical harm it are we supposed to do nothing and simply hope it never happens again? I cannot fathom what I would do if I were in the position of Timothy Klausutis: https://www.washingtonpost.com/politics/widower-of-late-joe-scarborough-staffer-seeks-removal-of-trump-tweets-that-promote-baseless-conspiracy-theory/2020/05/26/cf06257a-9f45-11ea-b5c9-570a91917d8d_story.html https://www.washingtonpost.com/politics/widower-of-late-joe-...
- suizi 6y ago
  GDPR is "supposed" to hold companies responsible for breaches and bad access controls. This is a different beast than liability protection for content.
dreen 6y ago
I remember during my time with a large mobile carrier in UK I was told of a person in the company who could in theory read any SMS on the network. Mind you this was literally one person for over 30 million customers. He had a high security clearance, extensive security training and the powers vested in him were used mainly to identify scammers and other criminals. Pretty sure this was a requirement set by law - we need someone to be able to do this, but lets make sure they know what they're doing. It is very weird we dont place the same requirements on social networks.
- ummwhat 6y ago
  Social networks were never supposed to be important or serious in the same way as phone networks. I would argue they still aren't. At the bottom, they are just time waster websites. You wouldn't demand that level of security of a php forum would you?
  - deadalus 6y ago
    Social networs including Twitter host tax-payer supported institutions such as USGS and NASA where they post updates.
  - dreen 6y ago
    At a certain threshold yes I would, if it served millions of people. A small ISP can get away with terrible security but once they start having millions of customers someone is going to sound an alarm. A forum, written in any language, should be no different. I realise there are challenges in making this happen but they are not unrealistic.
  - antihero 6y ago
    Except that huge public figures tweeting can actually affect real life a lot more than a bunch of SMS messages to yer nan....
  - sharken 6y ago
    You’re right about the first part, but large global social networks are quite close to phone networks in importance now. Though cases like the recent bipolar tweets from Kanye West on Twitter does seem to support your point. You could have said Wordpress forums, why take it out on PHP, though I get the message ;)
vlqubed 6y ago
I wonder if they automatically turned off log in with twitter to other websites. Seems like the bigger hole is that they can use these credentials for any people using twitter to log in using oauth.
atum47 6y ago
well, I worked on a software house that makes software for industry automation. each user of the software has all their actions logged and time-stamped. if you edit something, give a big discount, granted permission, deleted something... it all goes into a different DB filled with just the logs. why doesn't Twitter have something like this? am I missing something?
- suizi 6y ago
  If you're banning thousands of people, deleting thousands of tweets and updating thousands of people as routine (customer support), it might take time for someone to review what you're doing.
nextlevelwizard 6y ago
> implication that a hostile government might be able to cause even greater havoc. it is stuff like this that make me question the whole article. like yes, obviously this was no "hostile" government since they were just scamming for some pocket change. but also how exactly would this hostile government create havoc with twitter?
- M2Ys4U 6y ago
  There are so many government officials on Twitter, and causing any number of them to tweet something plausible but untrue could be a big deal - from moving markets to moving troops. Just imagine if Donald Trump's account tweeted that Antifa should be shot on sight. I'm certain people would die because of that. Or, perhaps slightly less plausibly, that Boris Johnson tweeted that he's had enough and is abandoning negotiations with the EU. That would cause a frantic reaction from the markets before any official statement could be put out.
  - nextlevelwizard 6y ago
    If people are taking Tweets as actual government announcements then they have too many screws loose. Dumb people do dumb shit, what else is new?
uallo 6y ago
I created a Twitter account close to a month ago and it was immediately suspended because it "appears to have exhibited automated behavior that violates the Twitter Rules". Well it did not really do anything yet, even less so anything against their rules. The account is still suspended despite multiple appeals and messages. At the same time, dozens (hundreds?) of verified accounts get taken over. I think their fraud detection systems are total crap.
- madeofpalk 6y ago
  Was it actually suspended, or did they just need you to verify with phone number?
  - uallo 6y ago
    It says "Your account is suspended and is not permitted to send Tweets". Why would they need to verify a phone number? I did not give them one.
    - madeofpalk 6y ago
      In an attempt to reduce sock puppet accounts often used to harass others, Twitter can very quickly "require" you enter a and verify a phone number when you make an account. Unsure what the criteria is.
- ecmascript 6y ago
  They do this for all new accounts. It's a way to harvest phone numbers from unsuspecting victims of this surveillance. It doesn't matter from what ip, machine or whatever you register. It will automatically get suspended because I think they've realized it's easier to force people to enter their phone numbers in "protection" after they just created an account rather than to just ask for it during signup. Less questions are asked. I wrote a blog post on my now deleted blog, but discussions on HN was here: https://news.ycombinator.com/item?id=19487304 https://news.ycombinator.com/item?id=19487304
  - 6y ago
anonunivgrad 6y ago
Should there be citizenship requirements for access to customer data at that scale? Background checks? Security clearances?[1] When you have so much private data and the ability to put words into people’s mouths, aren’t you a national security asset at that point? Today it’s some bitcoin scammers, tomorrow it’s Russian or Chinese intelligence. If I was in charge of Russian or Chinese intelligence, I’d make sure that my citizens working inside these companies are using that data to my advantage, or are at least positioned to should an opportunity arise. There is already tons of evidence of Chinese nationals coming to the US to work at these companies with the express purpose of stealing trade secrets and sending them back to China. Why would the Chinese government stop there? How about your personal emails, your Twitter DMs, etc.? Citizenship is loyalty. That is what it means legally and what it has meant in practice. Especially if your family is still in your country of citizenship. Yes, this would mean the international segmenting of the internet, at least in terms of which websites you plug your personal data into vs. “just browse”. This strikes us nerds as awful. But perhaps anything else was just a naive fantasy. The last decade should have shattered our innocence. What happens online matters for great power politics, and great power politics matters a lot for ordinary people. [1] The current security clearance process is at least partly a jobs programs for people with boring, unadventurous youths. I’m not advocating for that, just the principle of a security clearance.
- scohesc 6y ago
  You know, I used to think that locking down certain websites to citizens of the country the website resides in was a bad thing. Now with the advent of all these apparent "bots", "state actors", etc. etc. I'm starting to think it might not be a bad idea. There's a bunch of "what-ifs" however like "what if the government starts removing content it doesn't like", "should you be able to be banned from the platform?", etc.
  - anonunivgrad 6y ago
    At least within the US, I think sufficiently large platforms should not be allowed to censor on the basis of viewpoint. But that is exactly the kind of political question that nation states, not international forces, should be answering.
OfficialMuffin 6y ago
Interesting
flingo 6y ago
"Only two people can launch a nuke, the president, and the engineer who installed the system."

382 points by theBashShell 6y ago

pmiller2 6y ago

This is why internal tools that can modify account settings and such need to have audit trails.

raz32dust 6y ago
I would be really surprised if they did not have audit trails. What gives you the impression they did not? The suspicion is that the credentials were stolen via social engineering. I wonder if employees needed 2FA to log in to these tools.
- hombre_fatal 6y ago
  > The attackers successfully manipulated a small number of employees and used their credentials to access Twitter’s internal systems, including getting through our two-factor protections. https://blog.twitter.com/en_us/topics/company/2020/an-update-on-our-security-incident.html https://blog.twitter.com/en_us/topics/company/2020/an-update...
ceejayoz 6y ago
It probably does, and it probably wouldn't have stopped this.
- bawolff 6y ago
  > probably wouldn't have stopped this. Uh yes, that is how audit trails work
  - deleted 6y ago
    [deleted]
  - wil421 6y ago
    How does auditing itself prevent a present or future attack? Auditing and what you fix during audits are reactive.
    - halfjew22 6y ago
      It's like saying a boat's wake slows down the boat. Sometimes you've just got to wonder what people are thinking
    - calvinmorrison 6y ago
      100%. My work has really great auditing tools. I use them often to understand actions by other that are routine. It still doesn't prevent a employee emailing a datacenter to rack a malicious device or give someone service without paying. Record trails are not auditing. They are records. Auditing, post mortems, whatever diagnose the situation afterwards. At the end of the day Uber can't stop a driver from kidnapping people, but it can provide documentation and gps coordinates to police. My point is companies need reasonable records and audit policies and when _really bad stuff happens_ you call in the big guns for the arm of the law. At some point you also need to trust staff and weigh that against mistakes and malicious intent. In short, security remains an imperfect balance of practicality
  - hombre_fatal 6y ago
    Can we do without the condescending "Uh" and "Um" on HN? An audit trail would tell you who was social-engineered, but it wouldn't have prevented the attack in the same way Wikipedia's revision history doesn't keep you from vandalizing it.
    - TwoBit 6y ago
      It wouldn't even necessarily tell you who was social-engineered. My understanding of this case was that CS tool credentials were posted globally in their Slack.
      - bawolff 6y ago
        In fairness, i don't think its a real audit trail if the tool doesn't have an effective per-user password.
  - halfjew22 6y ago
    And in today's "Pompous Commenter That Didn't Read the Article News": >But while logging helps with investigations, only alarms or constant reviews can turn logs into something that can prevent breaches.
  - manigandham 6y ago
    Auditing tells you what happened, it doesn't prevent it from happening. If they have logs then they can use it in the future (and it seems they do) to design better protections but only active alarms and security controls can prevent something happening in real-time. However that does raise the question of why Twitter ever needs such access to someone's account in the first place, especially without a combination of approvals to get that access.
    - suizi 6y ago
      "I forgot my password, please reset it for me."

akersten 6y ago

Kind of sensationalist. There's thousands of people that have the ability to drain your bank account right now. Your average call center employee wields immense power. The real story here is Twitter's lack of spear-phishing training for their support staff, not support employees have access to support tools.

hombre_fatal 6y ago
Education programs only move the needle, not something you do instead of minimizing the number of attack vectors. For example, education programs do squat against me attacking the employees directly (targeted malware, getting on their computer somehow, offering each of the thousands of employees $10,000 for temp access to their account). And each additional employee only strengthens my attack.
dguido 6y ago
It's not sensationalist when you realize it directly contradicts Twitter's prior statements from just last year about it: > Twitter, in a statement, said it is aware that "bad actors" will try to undermine its service and that the company "limits access to sensitive account information to a limited group of trained and vetted employees." https://www.npr.org/2019/11/06/777098293/2-former-twitter-employees-charged-with-spying-for-saudi-arabia https://www.npr.org/2019/11/06/777098293/2-former-twitter-em... 1,000 people, including contractors outside the company, is not a "limited group of trained and vetted employees." It's news because they misled people about their security, again.
- ceejayoz 6y ago
  > 1,000 people, including contractors outside the company, is not a "limited group of trained and vetted employees." That's not necessarily true. 20% of the company could fairly reasonably be deemed "limited", and there being a thousand of them doesn't mean they're not trained on their tasks.
  - monsieurbanana 6y ago
    We'll have to agree to disagree on what we consider fairly reasonable to call "limited".

jbob2000 6y ago

I now understand why the bank I work for creates the separation of duties; the person who builds the system has no access to it, and the person with access has no idea how it works. As a developer, it frustrates the shit of out me because I can’t deploy fixes quickly or easily diagnose issues. But yep, there are 3 people that have access to the production databases that hold account info and they aren’t developers, just managers with no clue what to do once they log in. I also worked for a company that sold software to lawyers. We had a feature that would alert the client any time a member of our company accessed their data. I think we called the feature something like “fire call”, because if you tripped it without informing the client, you’d get a call informing you that you’d been fired.

lowdose 6y ago
> there are 3 people that have access to the production databases that hold account info and they aren’t developers, just managers with no clue what to do once they log in. Just for my curiosity is this your observation or is this a company assumption?
- jbob2000 6y ago
  Hmm I think it’s just our group, we have a Production support team that holds the keys, and there’s only 3 of them that can access my app. For example, if I want to change an environment variable, I can’t just log into the cloud console or run a cli command. God no. That would be too easy. I have to write a script for this team to run. This script is entered into an authorization app where a few parties “sign off”, at which point the prod support team can log in to the authorization app and click Deploy. This app then runs my deployment script against our app container to update the env variable. Accessing and doing DB work follows a similar process.
  - ladberg 6y ago
    Reading this makes me happy! Always good to see people taking security seriously.

dzonga 6y ago

twitter, seems to have a cowboy engineering culture. that's why one of their exec's blamed rails for their failure to combat harassment[0]. n I bet now, if they still ran rails, it would've been blamed lol. [0]: https://char.gd/recharged/daily/twitter-blames-ruby-on-rails-for-harassment https://char.gd/recharged/daily/twitter-blames-ruby-on-rails...

DetroitThrow 6y ago
"...a rudimentary web-application framework that made it nearly impossible to find a technical solution to the harassment problem" To me, this is analogous to the perhaps undeserved "the internet is a series of tubes" lampooning, but I'm still chuckling how they managed to word that so poorly.
- spoopyskelly 6y ago
  It sounds like a perfect answer to those claiming "harassement" is a technology problem.
suizi 6y ago
Emails are overrated as an anti-troll measure. Anyone can make a trash mail in seconds.

laughinghan 6y ago

accounts with more than 10,000 followers should at least need two people to change key settings For accounts that could start a war this might be necessary, but for celebrities with >10K followers this sounds expensive and unnecessary to me. To me, it seems like you could instead ensure the admin view of every account has a timestamped log of recent settings changes, including changes done by admins, with a link to the profile of the admin responsible, and a button to suspend that admin account with one click. This way, the security team could've seen that Elon Musk's account had just been reset by J. Random Employee minutes before tweeting the suspicious bitcoin tweet, messaged J. on Slack to be like "hey did you do that?", and suspended the compromised admin account within minutes. Sure, some accounts might be briefly compromised initially, but it would be resolved in minutes and not the hours that it took Twitter, right? That seems fine for what should be a relatively low-likelihood, high-expense attack like compromised admin account (of course, you have to ensure that is the case).

foota 6y ago
I think the long tail was in undoing the actions made by the attackers. Resetting passwords, emails, etc.,.
- laughinghan 6y ago
  No, according to The Block, @elonmusk repeatedly tweeted the scam at 4:17pm, 5:19pm, and 5:32pm, a span of 90 minutes, and the final scam tweet was at 6:05pm from @KimKardashian. An hour after @elonmusk's first scam tweet, 7 celebrity or corporate accounts had tweeted the scam, all with the same Bitcoin address. With the two-click system I described, how many compromised admin accounts would you expect the security team to have been able to suspend by then? 8 more celebrity accounts went on to tweet the scam, plus @elonmusk and @kanyewest repeating the scam tweets. https://www.theblockcrypto.com/post/71906/twitter-account-hacks-timeline https://www.theblockcrypto.com/post/71906/twitter-account-ha...
  - londons_explore 6y ago
    If your database system doesn't have a complete audit log of all fields (most databases have this capability, but more often than not it's disabled), it's possible that the mere act of reverting account ownership might remove data needed for tracing down what happened. Sure, it's a sucky position to be in, but I can see why they might have been hesitant to dive right in and start trying to undo damage before understanding what had happened.

sloshnmosh 6y ago

Ha! Did you see in that article that the head of cyber security for AT&T added his two cents in shaming Twitter? AT&T was just in the news recently where employees were accepting bribes that allowed criminals to swap SIMs steal bitcoins from AT&T customers. Unbelievable.

manquer 6y ago
Not just bitcoins, SIM swapping is how @jack twitter account got comprised as well.
onetimemanytime 6y ago
Thousands of employees, each with their own financial problems and dreams...you're bound to find a taker. Money moves mountains
- hn_throwaway_99 6y ago
  Which is why once your company is big enough, you should need 2 employees who are unfamiliar to each other to sign off on high value operations.
  - onetimemanytime 6y ago
    Bingo. Corrupting two is much less likely.
    - tialaramex 6y ago
      The key trick isn't so much the two as that they're randomly selected. I moved a large amount of money a few years back to buy my home (I do not like debt, so I saved up until I could afford somewhere to live, then I bought it) The bank's web site lets you type in any amount of money but then it says politely that you can't do this from the web site, please call the bank. I called the bank (they always pick up in 2-3 rings, I've worked with one of their founders, ensuring this was one of the key ideas behind the bank) and explained what I wanted to do. The nice lady took down all the details and then she explained that now one of her colleagues would be randomly selected to call me back and confirm everything and we hung up. Sure enough, less than a minute later another of the people from the bank called (with the agreed password for when the bank calls me) and had me read out all the transaction details again, at which point the transaction was confirmed. Think about that scenario as a bad guy trying to corrupt it. You bribe one employee to pretend someone called and authorised a huge transfer. OK. But then a different random employee has to confirm it. How do you bribe them? You have no way to know who it will be! Do you try just bribing every single employee who works the phones? Not very practical. The other thing banks do is they background check employees. You can't test for "willing to take bribes" but you can weed out potential hires with previous convictions for financial crime, or debt problems. I've had checks like that for jobs touching sensitive personal information.

wiradikusuma 6y ago

Anyone watched Westworld? The whole enterprise is destroyed (almost) by 2 low level employees. It's either a complete blooper in the script or --after I read this article-- reflective of the real world that I don't know about. Your take?

throwaway220720 6y ago

For comparison, at Google in 2011, I was one of ~10 or so engineers that had the ability to view private Gmail or Gplus data (access that was heavily documented and audited). That being said, Google did have to go through it's own public humiliation [1] to put a system like that in place. https://gawker.com/5637234/gcreep-google-engineer-stalked-teens-spied-on-chats https://gawker.com/5637234/gcreep-google-engineer-stalked-te...

necovek 6y ago
Don't all engineers working on Gmail theoretically have the same access by conspiring with a code reviewer or two? It ultimately comes down to the person involved and I do not believe anyone can control the human factor.
- throwaway220720 6y ago
  They can easily build and view their own versions of the gmail stack, but they would not be able to generate auth tokens to decode the private data of accounts they did not have passwords for.
  - necovek 6y ago
    I was more thinking of deploying trojan-code into the production service (as a trivial example, allow a special password to access any account): it can't be practical to vet every production service change through too many people. You seem to suggest that you are using an encryption key based on the password or oauth token on login, which is great to hear, which stops the simpler forms of trojans like the example above. That makes it much more involved to achieve the same (and login from new computer reports make it harder too), especially undetected (because it has to happen over a short period) but not impossible (thinking of cases like just making a new API endpoint or perusing an existing one to store actual content in an often unlooked at log file/service).
koheripbal 6y ago
I almost wonder if government officials should be outright banned from using any private messaging platform that isn't hosted by the government itself. There is just too much power in information.

imvetri 6y ago

Title corrected : More than 1k people at Twitter had ability to aid hack and chose not to.

deleted 6y ago
[deleted]
- deleted 6y ago
  [deleted]
amf12 6y ago
> Title corrected : More than 1k people at Twitter had ability to aid hack and chose not to. This is a stupid way of looking at it. Similarly: - X number of people owned guns but they chose not to go do a mass shooting. - X number of cops could kill a black person, they chose not to. While it's a good thing that majority of the people know right from wrong, morality, etc, we still need to ensure one person can't do significant damage. The fact that there are 1000s of individuals that could have hacked is not a good thing.

KingOfCoders 6y ago

There is all this talk from those successful companies about security and what you should do with your keys and they open source hardware secret stores and brag about it and they fail at the most basic security operations.

sunilkumarc 6y ago

On a different note, online presence is becoming very important and with remote work culture gaining traction, having a good online presence has become a must have asset. I bought a course on building Twitter audience and been able to improve my following significantly from past 2 months. Twitter link: https://twitter.com/sunilc_ https://twitter.com/sunilc_ If you're looking to increase your social presence too, here's the course that I found very useful: https://gumroad.com/a/238777459/PBkrO https://gumroad.com/a/238777459/PBkrO

accurateappL 6y ago
Spam motivational quotes and hope people retweet and like?

alpb 6y ago

Worth mentioning only 5,000 people work at Twitter.

Jaruzel 6y ago
There are ~330 million active twitter users, which means 330,000 users per employee with access to admin accounts. That ratio is massively high compared to a large corporate (i.e, a global bank). In a typical global bank lets says there are 100,000 employees, with about 25-50 IT people with the rights to admin accounts (from first line support to third line engineers) that's only 2,000-4,000 users per IT admin person. Based on that, I'm surprised that it's only 1,000 staff members in Twitter with admin access, and not the whole company.
- amf12 6y ago
  > There are ~330 million active twitter users, which means 330,000 users per employee with access to admin accounts. I think we should look at how many daily requests they get to reset account access settings (that cannot be done automatically - via some system rather than through these 1k users).
- alpb 6y ago
  I am not sure why this bank example keeps coming up. Almost no twitter user tries to contact support like they contact their teller for their bank. It’s really bad that 20% of the company had access to user data. No wonder it was abused in the past. https://www.buzzfeednews.com/article/alexkantrowitz/how-saudi-arabia-infiltrated-twitter https://www.buzzfeednews.com/article/alexkantrowitz/how-saud...
thinkingemote 6y ago
Twitter is an advertisement company so most of the employees will be in sales and marketing. Those probably don't need admin access.

Laforet 6y ago

Right, thousands of people with admin access and nobody could help me reinstating my API access....

OffensiveTomato 6y ago
Apparently you gotta have that coveted Verified badge or be an influencer of some sort

austincheney 6y ago

This should be a wake up call. Thank god the malicious messaging was only limited to a tiny Bitcoin scam. Imagine if they had pulled this off on the accounts of national leaders to stir hostilities or violence. What is the recourse for this kind of failure? I suspect there is none. Twitter is shielded from lawsuits for its content. If this is provably negligent behavior and resulted in actual physical harm it are we supposed to do nothing and simply hope it never happens again? I cannot fathom what I would do if I were in the position of Timothy Klausutis: https://www.washingtonpost.com/politics/widower-of-late-joe-scarborough-staffer-seeks-removal-of-trump-tweets-that-promote-baseless-conspiracy-theory/2020/05/26/cf06257a-9f45-11ea-b5c9-570a91917d8d_story.html https://www.washingtonpost.com/politics/widower-of-late-joe-...

suizi 6y ago
GDPR is "supposed" to hold companies responsible for breaches and bad access controls. This is a different beast than liability protection for content.

dreen 6y ago

I remember during my time with a large mobile carrier in UK I was told of a person in the company who could in theory read any SMS on the network. Mind you this was literally one person for over 30 million customers. He had a high security clearance, extensive security training and the powers vested in him were used mainly to identify scammers and other criminals. Pretty sure this was a requirement set by law - we need someone to be able to do this, but lets make sure they know what they're doing. It is very weird we dont place the same requirements on social networks.

ummwhat 6y ago
Social networks were never supposed to be important or serious in the same way as phone networks. I would argue they still aren't. At the bottom, they are just time waster websites. You wouldn't demand that level of security of a php forum would you?
- deadalus 6y ago
  Social networs including Twitter host tax-payer supported institutions such as USGS and NASA where they post updates.
- dreen 6y ago
  At a certain threshold yes I would, if it served millions of people. A small ISP can get away with terrible security but once they start having millions of customers someone is going to sound an alarm. A forum, written in any language, should be no different. I realise there are challenges in making this happen but they are not unrealistic.
- antihero 6y ago
  Except that huge public figures tweeting can actually affect real life a lot more than a bunch of SMS messages to yer nan....
- sharken 6y ago
  You’re right about the first part, but large global social networks are quite close to phone networks in importance now. Though cases like the recent bipolar tweets from Kanye West on Twitter does seem to support your point. You could have said Wordpress forums, why take it out on PHP, though I get the message ;)

vlqubed 6y ago

I wonder if they automatically turned off log in with twitter to other websites. Seems like the bigger hole is that they can use these credentials for any people using twitter to log in using oauth.

atum47 6y ago

well, I worked on a software house that makes software for industry automation. each user of the software has all their actions logged and time-stamped. if you edit something, give a big discount, granted permission, deleted something... it all goes into a different DB filled with just the logs. why doesn't Twitter have something like this? am I missing something?

suizi 6y ago
If you're banning thousands of people, deleting thousands of tweets and updating thousands of people as routine (customer support), it might take time for someone to review what you're doing.

nextlevelwizard 6y ago

> implication that a hostile government might be able to cause even greater havoc. it is stuff like this that make me question the whole article. like yes, obviously this was no "hostile" government since they were just scamming for some pocket change. but also how exactly would this hostile government create havoc with twitter?

M2Ys4U 6y ago
There are so many government officials on Twitter, and causing any number of them to tweet something plausible but untrue could be a big deal - from moving markets to moving troops. Just imagine if Donald Trump's account tweeted that Antifa should be shot on sight. I'm certain people would die because of that. Or, perhaps slightly less plausibly, that Boris Johnson tweeted that he's had enough and is abandoning negotiations with the EU. That would cause a frantic reaction from the markets before any official statement could be put out.
- nextlevelwizard 6y ago
  If people are taking Tweets as actual government announcements then they have too many screws loose. Dumb people do dumb shit, what else is new?

uallo 6y ago

I created a Twitter account close to a month ago and it was immediately suspended because it "appears to have exhibited automated behavior that violates the Twitter Rules". Well it did not really do anything yet, even less so anything against their rules. The account is still suspended despite multiple appeals and messages. At the same time, dozens (hundreds?) of verified accounts get taken over. I think their fraud detection systems are total crap.

madeofpalk 6y ago
Was it actually suspended, or did they just need you to verify with phone number?
- uallo 6y ago
  It says "Your account is suspended and is not permitted to send Tweets". Why would they need to verify a phone number? I did not give them one.
  - madeofpalk 6y ago
    In an attempt to reduce sock puppet accounts often used to harass others, Twitter can very quickly "require" you enter a and verify a phone number when you make an account. Unsure what the criteria is.
ecmascript 6y ago
They do this for all new accounts. It's a way to harvest phone numbers from unsuspecting victims of this surveillance. It doesn't matter from what ip, machine or whatever you register. It will automatically get suspended because I think they've realized it's easier to force people to enter their phone numbers in "protection" after they just created an account rather than to just ask for it during signup. Less questions are asked. I wrote a blog post on my now deleted blog, but discussions on HN was here: https://news.ycombinator.com/item?id=19487304 https://news.ycombinator.com/item?id=19487304
- 6y ago

anonunivgrad 6y ago

Should there be citizenship requirements for access to customer data at that scale? Background checks? Security clearances?[1] When you have so much private data and the ability to put words into people’s mouths, aren’t you a national security asset at that point? Today it’s some bitcoin scammers, tomorrow it’s Russian or Chinese intelligence. If I was in charge of Russian or Chinese intelligence, I’d make sure that my citizens working inside these companies are using that data to my advantage, or are at least positioned to should an opportunity arise. There is already tons of evidence of Chinese nationals coming to the US to work at these companies with the express purpose of stealing trade secrets and sending them back to China. Why would the Chinese government stop there? How about your personal emails, your Twitter DMs, etc.? Citizenship is loyalty. That is what it means legally and what it has meant in practice. Especially if your family is still in your country of citizenship. Yes, this would mean the international segmenting of the internet, at least in terms of which websites you plug your personal data into vs. “just browse”. This strikes us nerds as awful. But perhaps anything else was just a naive fantasy. The last decade should have shattered our innocence. What happens online matters for great power politics, and great power politics matters a lot for ordinary people. [1] The current security clearance process is at least partly a jobs programs for people with boring, unadventurous youths. I’m not advocating for that, just the principle of a security clearance.

scohesc 6y ago
You know, I used to think that locking down certain websites to citizens of the country the website resides in was a bad thing. Now with the advent of all these apparent "bots", "state actors", etc. etc. I'm starting to think it might not be a bad idea. There's a bunch of "what-ifs" however like "what if the government starts removing content it doesn't like", "should you be able to be banned from the platform?", etc.
- anonunivgrad 6y ago
  At least within the US, I think sufficiently large platforms should not be allowed to censor on the basis of viewpoint. But that is exactly the kind of political question that nation states, not international forces, should be answering.

OfficialMuffin 6y ago

Interesting

flingo 6y ago

"Only two people can launch a nuke, the president, and the engineer who installed the system."