12 ms·
Incident report: Employee and customer account compromise
- MattGaiser 4y agoThis is why I have always wondered about the secondary domains companies have for major incidents or auth or internal employees. If you train people to think that several can exist, few will question another appearing. Shouldn’t having multiple domains for different use cases be considered a security risk going forward? Is it already considered a bad practice?
- lbriner 4y agoGood idea. Seems like you could use DNS or .well-known to provide canonical domains that could be checked by devices/email clients.
- MBCook 4y agoGiven Twilio is an SMS company it seems like they would/should have an “official phone number” that any company related SMSes would come from. That would make this kind of attack harder. (Maybe they already have this, the article doesn’t say) Very unfortunate incident, and rather scary if this is targeting multiple companies.
- toomuchtodo 4y agoOnly trust corp comms for corp comms (Gmail/Outlook, Slack, Teams). No SMS! (Twilio also owns Authy for 2FA)
- tauntz 4y agoThe sender number/name for SMS can be easily changed and should not be relied on for _anything_. Twilio even offers that as a service (as do most bulk-SMS providers): https://support.twilio.com/hc/en-us/articles/223181348-Alphanumeric-Sender-ID-for-Twilio-Programmable-SMS https://support.twilio.com/hc/en-us/articles/223181348-Alpha...
- toast0 4y agoThat greatly depends on telco policies at the destination. In the US, alpha senders aren't allowed. Shortcode (5 or 6 digit numeric senders) spoofing is highly frowned upon to the point where you can't send from the same shortcode with multiple aggregators. Other countries vary between anything goes, aggregator takes anything in the sender field, some high profile names are checked, or all sender names need to be run through the telecoms regulator and validated by the aggregator and cell carrier. Of course, there's (usually?) no customer id transparency between the carrier and the aggregator, so if there's a misconfiguration at the aggregator, one customer may be able to send with a registered sender of another customer.
- kelnos 4y agoTwilio does not use SMS 2FA for access to internal systems. It's all either Yubikey or TOTP apps. I believe SMS 2FA was disabled several years ago. So, really, Twilio employees should be trained that the company will never send SMSes to employees for these sorts of purposes. But it only takes a few people to get fooled when their guard is down. That's the difficult thing about defending against security-related attacks: the defender needs to be perfect 100% of the time, but the attacker only needs to get lucky once. (Disclosure: I'm a former employee who has been getting these phishing texts, the most recent of which was yesterday... not that I have access to any systems anymore. I guess they are going off leaked employee lists that are at least 5 months out of date.)
- mcculley 4y agoOrganizations should definitely think very carefully about creating domains they expect others to trust. @SwiftOnSecurity used [0] a turn of phrase recently that I had not yet encountered and I think it is a great way to put it: "strong DNS governance". 0: https://twitter.com/SwiftOnSecurity/status/1553538867919261697 https://twitter.com/SwiftOnSecurity/status/15535388679192616...
- deleted 4y ago[deleted]
- alberth 4y agoI wouldn't be surprised if the fraudster used Twilio's own platform to send the phishing SMS to its own employees. If this is in fact the case, I wonder if Twilio becomes even more liability for such customer damages. Note: additionally, Twilio uses it's own Authy for MFA.
- kelnos 4y agoThis isn't the case. I'm a former employee, and received these phishing emails after I left the company (I just got two on Sunday, and I've been gone over 5 months). I used Twilio's number lookup API[0], and they're not Twilio numbers. They were both T-Mobile numbers, and one of them even had a caller name that looked like the number was associated with a retail consumer SIM card. Possibly a cloned or stolen SIM? Twilio has really stepped up KYC efforts over the past couple years. I don't think an anonymous attacker could send this volume of messages through the platform without getting noticed. And if they were using Twilio to send these messages, it would have been trivial to block them. [0] https://www.twilio.com/docs/lookup/api https://www.twilio.com/docs/lookup/api
- hatware 4y agoI can't be the only one laughing that Twilio employees were phished with SMS messages.
- Justin_K 4y agoIt's funny because they open by calling it a "sophisticated" attack... just not true.
- Raed667 4y agoAn attack is sophisticated when you don't want to get sued.
- hn_throwaway_99 4y ago> Additionally, the threat actors seemed to have sophisticated abilities to match employee names from sources with their phone numbers. Exactly, that is absolutely not sophisticated - it's often as simple as "scraping LinkedIn".
- karamanolev 4y agoI guess BigCorp will call anything that compromises them "sophisticated". They wouldn't admit being compromised by a technologically simple decades-old attack, would they?
- mattbee 4y agoKudos to them for showing the kind of messages that caught their own employees out - though to me they seem jarring and unsophisticated. So what other texts must Twilio send their own staff that make these ones feel legit? And what systems do they have online allowing for stolen employee credentials to be tested and used?
- madeofpalk 4y agoSend thousands of these out. You just need one person to be on auto-pilot mode to fall for it.
- pete5x5 4y agoA month ago I had a call with Twilio sales / onboarding to consider switching to them from our current IP phone provider. Ironically, I was unable to complete the process because my current work number, which is IP based, would not pass their "put in your number so we can make sure you're a human" verification test because IP numbers are not supported. I do [edit: NOT] use my personal cell number for anything for security reasons, so even after their insistence that it was safe to use I refused and therefore I was unable to get past the first step of the signup process and went with another provider. After reading this I am feeling validated that I didn't cave.
- daniel-cussen 4y agoYou mean you do not use your personal cell for those things. Good for you. Your refusal and my refusal protect one another, refusing collectively is stronger than refusing alone.
- pete5x5 4y agoGood catch, edited in the *not*. I agree, if everyone refused then it would change. I asked them what would happen if I tried to use a Twilio number to verify, but they did not seem amused by the irony.
- b112 4y agoI refuse to give my mobile number, or use my mobile number for anything. SMS auth, main contact, anything. My SMS spam is almost non-existent, compared to others. But one thing, beyond my desire to not give out my number... it is pointless regardless. When I worked in office, I had mobile access. At home, a bit rural, my access is nil via 4g/5g. I just have no access. My mobile forwards after 6 rings to my voip, so that works well. But for SMS auth? Hello! I cannot do that! I have been an ebay customer for 21, yes 21 years. I can no longer log in, as they now insist I enter a mobile number to continue. Gee thanks ebay. (No, SMS won't work via voip, they check numbers, even ported numbers) 21 years. Years of thousands, even >10k spent reliably per year. Gone as a customer. Calling paypal support, results in people literally unable to understand ... anything. They repeat a mantra off their screen without deviation. Many cuatomer support people I spoke to, were barely paying attention. I really don't get it. SMS is barely secure to begin with. They are willing to throw away accounts, just for pennies on tracking. 21 years.
- codegeek 4y agotl;dr: Social Engineering that sent SMS with links to Twilio employees and a few fell for it. Time for more security training to employees ? Weird that this happened to a company like Twilio which sells SMS API.
- geofft 4y agoI don't think "more security training" will get us anywhere - we've been trying it for a long while. This is a technical problem and admits technical solutions. The most obvious, simple, and straightforward solution here is logins with security keys (WebAuthn/FIDO), which cannot be phished. Perhaps Twilio fell for this precisely because they don't want to admit to themselves that the SMS verification product they sell (https://www.twilio.com/verify https://www.twilio.com/verify) is an obsolete joke compared to security keys. If you don't want to use that, for some reason, or if you want additional protection, other solutions include using dedicated machines (not personal cell phones and certainly not whatever devices the attacker was using) ideally with hardware-locked keys (TPMs etc.) to access the corporate network or at least to access sensitive systems like customer data, or giving people separate privileged accounts that they don't use for day-to-day access, or establishing a two-person rule for logging into sensitive systems (so two people with the same access need to get successfully phished at the same time), or setting up some real-time auditing of changes (e.g., a Slack channel gets notified when people manually log into prod).
- rvz 4y agoLet me guess. Phone numbers also compromised? I won't be surprised about that if it was the case. Either way, not only there is poor security at Twilio, but I would expect that they will be fined in the multi-millions of dollars for this breach.
- kelnos 4y ago> I would expect that they will be fined in the multi-millions of dollars for this breach. By whom? Most breaches do not involve fines of any sort.
- bearjaws 4y agoThe sophistication of attacks is rising exponentially. "Spear phishing" has pretty much become the norm, all our new hires get an email from our "CEO" pretending they have to sign paper work for their equity... Its part of our onboarding now to warn them that if they join our LinkedIn they will get multiple phishing attempts from the "founders".
- moepstar 4y ago> Its part of our onboarding now to warn them that if they join our LinkedIn they will get multiple phishing attempts from the "founders". Hm, i wonder what the point is to warn them beforehand?
- whylo 4y agoI think OP means that new hires are receiving actual spearphishing emails from attackers outside of the company, not that they're testing them by sending fake spearphising emails. (I misread it as the latter at first too)
- moepstar 4y agoHa - thanks for clarifying - now that i read it over again... you're probably right and it even makes sense to do so...
- haswell 4y agoI'm not sure if they ever did this during onboarding, but my former employer would regularly run fake spearphishing campaigns to raise awareness about spearphishing. The number of people who regularly fell for it was worrisome. Falling for it meant auto-enrollment in a mandatory security awareness training. Failing to take the training would result in deactivation of the individual's network credentials. I don't know if these campaigns are actually effective at changing people's behavior, but they certainly revealed how effective spearphishing is.
- bearjaws 4y agoThe main problem is they are real attacks that 100% happen as soon as they update their LinkedIn. They aren't tests like a Knowb4.
- deleted 4y ago[deleted]
- status200 4y agoThis is the exact same phishing attack that i see on many retail stores that i consult, and i have yet to see any of them fall for it. Smaller surface, sure, but even people loosely familiar with computers are skeptical of those emails/SMSs
- tgv 4y agoIs your company's information that interesting? We had to prepare for an audit, and concluded that our info is simply not worth the effort. People get training in noticing phishing emails, but the only one we've ever had was a cheap shot at getting someone to transfer money. We may have been wrong though, and overlooked something.
- kelnos 4y agoI'm a former Twilio employee, and even I've been getting slammed by these phishing attempts over the past couple months. Remember that this is a company of more than 8,000 people: it only takes one person to let down their guard. Maybe they're tired, maybe it's been a long day, maybe they're just the weakest link.
- Vaslo 4y agoHow many "hacks" are really computer hacks at all?
- SQueeeeeL 4y agoMost aren't, we basically figured out a fair bit of security in the 90s (barring increasing of computational power, which really has just required increasing the number of bits). It's almost always been implementation problems.
- bierjunge 4y agoBad ideas can be replaced with better concepts. Implementation flaws can be patched. But the biggest problem is still the human factor aka "you can't patch stupidity".
- madeofpalk 4y agoI guess that's why the title or article never mentions the word "hack".
- psanford 4y agoHey Twilio, maybe now you'll implement webauthn for both your internal systems and for your customers? Its pretty frustrating that there's a well known technical solution that prevents this type of attack from being feasible and companies like twilio simply have not prioritized it yet. Replacing SMS and Push based 2fa with webauthn is the best bang for your buck security upgrade available to companies right now. EDIT a bit later: > We have reemphasized our security training to ensure employees are on high alert for social engineering attacks, and have issued security advisories on the specific tactics being utilized by malicious actors since they first started to appear several weeks ago. We have also instituted additional mandatory awareness training on social engineering attacks in recent weeks. Separately, we are examining additional technical precautions as the investigation progresses. Security training is ineffective against phishing attacks like this. Please stop wasting your money and your employees time; implement the technical solution that nullifies the attack.
- Kerrick 4y agoIsn’t SMS based 2FA responsible for a lot of Twilio’s revenue?
- reaperducer 4y agoI guess dogfooding isn't a great idea if the kibble is rancid.
- Jenk 4y agoSMS campaigns, both target and blanket, simply must dwarf any 2fa usage. In just one role I saw Sms-shots with 10s of thousands of recipients. SMS is also fairly heavily used for non-secure notifications (you bill is due / payment received / your password changed, etc.) quite heavily IME.
- powerhour 4y agoI wonder how often SMS 2FA serves as the entry point for more business. Like, devs choosing Twilio to try it out for low volume work before expanding into large campaigns.
- gz5 4y agoTwilio must use MFA (Authy)? If so, the "sophisticated" part of this attack may have included a breach of the MFA or a way to get around it?
- mr-ron 4y agoAbsolutely wild that apparently the Twilio team doesnt require any sort of MFA to access internal data, and then even wilder that this incident report doesnt even mention this.
- sofixa 4y agoDepending on the type of MFA (sms, push, TOTP), phishing can get around it.
- jeromeparadis 4y agoWho says an attacker auth Web site cannot ask for the MFA code behind the scenes and supply it? Problem is no one, especially Twilio employees shouldn't never click and never trust any link they receive from trusted or untrusted source. They should use the links they already have bookmarked.
- PeterisP 4y agoFor properly implemented MFA (FIDO/U2F tokens) an attacker-spoofed website can't ask for the code behind the scenes - i.e. they can ask, but they'll get a code that won't work on the proper site.
- jeromeparadis 4y agoNot sure about MFA with a USB key but for the sake of the argument, if they are using App-based MFA as their own Authy, I would think a headless browser in the backend of the fake site accessing the proper site on behalf of the real user would do the trick. It asks the code for the user on the real site and the user replies on the fake site and the fake site supplied the real code to the real site. The only thing needed is that the user gets and supply the code that was asked on their behalf to the fake site.
- PeterisP 4y agoNo, MITM does not circumvent that, unless you can MITM the TLS connection and convince the browser (not the user) that you're actually connecting to the proper domain, e.g. hacked private keys or malicious CA issuing fake certs, which is quite rare. For U2F, there is no possibility for a user mistakenly approving one site's challenge on another site, if the challenge request is coming from (and the response would be sent to) https://badsite.com https://badsite.com, then any challenge that's not for https://badsite.com https://badsite.com would be automatically rejected by the browser even before asking the user anything. (This is the type that is usually implemented through a USB key.)
- sltkr 4y agoSo I guess the real story is that Twilio does not require second-factor authentication for employees that have access to user data? It seems like hardware-based 2FA with something like FIDO/U2F would have easily stopped this attack and generally make stealing credentials really difficult (it's still technically possible if you're able to spoof the target domain, but that's much harder than registering a fake domain). Kind of weird that they only mention "security training" as a remedy, which is obviously never going to be 100% effective, instead of a more effective technical solution.
- Operyl 4y agoIt seems likely to me that they do, they use Authy, and the employees that fell for the attack just supplied the codes they needed. Each time you get in, you can muddle around and figure out what is needed for the next time you send out an attack until you get where you need to. The real story is that Twilio, likely due to one of their largest products being Authy, still refuses to move away from SMS 2FA. They need to finally join the rest of the industry and move on to webauthn, internally and externally for their customers.
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- phkahler 4y ago>> Additionally, the threat actors seemed to have sophisticated abilities to match employee names from sources with their phone numbers. Is this another case of caller-ID spoofing? That needs to be illegal/impossible.
- MBCook 4y agoNo I believe what they meant was the attackers knew that if they wanted to spear-phish Joe Smith from IT they had a way to correctly get Joe Smith’s phone number to send a text to him, so things could be personalized and seem more legit.
- dopylitty 4y agoA recent article[0] on HN about on-call culture called out Twilio specifically for having overworked and under compensated employees as well as the subsequent attrition those cause. I wonder how those factors impact the likelihood that an employee might be susceptible to phishing. 0: https://news.ycombinator.com/item?id=32378752 https://news.ycombinator.com/item?id=32378752
- i67vw3 4y agoSo what exactly was breached related to customers.... Even in the Techcrunch article, they have not specified anything. Are these customers other businesses or regular users using Twilio apps like Authy? If the 'hackers' got too deep into the systems they might create a big mess. Authy 2fa tokens are backed up on Twilio servers (opt-in) unlike Aegis/andOTP. If you lose access to your offline 2fa tokens (phone stolen/lost), Authy can re-send these token from their servers. Users need to wait 24-48 hours for the whole recovery process to be over. https://support.authy.com/hc/en-us/articles/115012672088-Restoring-Authy-Access-on-a-New-Lost-or-Inaccessible-Phone https://support.authy.com/hc/en-us/articles/115012672088-Res... https://authy.com/phones/reset/?proceed=true https://authy.com/phones/reset/?proceed=true
- Deathmax 4y agoThe actual TOTP secrets are (or should be) encrypted with the backup password that the user chose, so access to the backups wouldn't automatically result in compromised TOTP secrets unless the backup password was weak.
- i67vw3 4y agoAs far as I know, the backup password is not needed for recovery using this method. When you have 'Multi-device' enabled in settings and install Authy on second device then the backup password is used/useful. But for this method, your phone number linked to Authy Account, email Id linked to Authy Account are needed. The Process is Started by old-school SMS based OTP sent to the linked number. You then have to Cancel it via a email sent to you, if you think some is doing is maliciously without your consent.
- itsdrewmiller 4y ago"How I used Twilio to hack Twilio"
- awinder 4y agoThese screenshots don’t look very sophisticated. Does okta even have a scheduling solution? Who was clicking these links, engineers? Is it normal for Twilio to send corporate communications like this over text (at least without a good reason & an email warning it’s coming and verifiable details)? I’m also curious how much customer data an attacker should have been able to get into from outside. Is there no 2fa-auth vpn needed to get in? Or is there just lots of customer data hanging out in email/whatever?
- sofixa 4y agoVPNs are so 2010s. They just add friction and a false sense of security (the way they're traditionally implemented - VPN is "trusted" and everything behind is open access), and an amazing point of entry for attackers. A "zero trust" approach where nothing is trusted and you pass through an SSO with MFA like Okta is better in that Okta do a better job in security than "random network team running an unpatched firewall from vendor X" do on average. And each service has an auth layer and doesn't implicitly trust some IPs are good. In any case, a phishing that captures login+password+mfa means game over in both scenarios.
- exabrial 4y ago100000x Incorrect. VPNs add a layer of security at the network level. You do not rely SOLELY on a VPN. You add it to drop network traffic that does not belong, decreasing the noise level in other logs, highlighting threats elsewhere. This is one of the worst pieces of advice to be repeated ad deafenun.
- deleted 4y ago[deleted]
- stefan_ 4y ago> Additionally, the threat actors seemed to have sophisticated abilities to match employee names from sources with their phone numbers. Wow, what amazing technology could this be! Maybe it's one of those shitty websites or apps that collect your entire phone book and leave it on some Mongo NoSQL public database? Maybe its Twitter that pinkie-promised to not store phone numbers, then stored and lost them?
- exabrial 4y agoDear Twilio: I wrote you an email a year ago about how the Authy app is dumb and SMS "2fA" is NOT a second factor! To re-iterate: SMS is not authenticated, it can be spoofed (using your platform), there are no delivery guarantees, no read receipts, it's not encrypted... and most importantly: It defers security to Phone Carriers, who have the security posture of an air seal made of swiss cheese. Once again: Dump the Authy App. Promote actual 2fA technology like U2F, now WebAuthn, hardware keys, OR _actual_ TOTP which cannot be reset using an sms. My 67 year old Mother, grandmother of 12, knows how to use hardware keys and TOTP. I'm embarrassed for you guys that you still can't figure it out. Sincerely, -The I told you so department
- paxys 4y agoWhat's wrong with the Authy app?
- exabrial 4y agoYour security tokens can be reset using SMS.
- haswell 4y agoI have only ever used the Authy app as a generic place to centralize my TOTP secrets from various services. When you say tokens can be reset using SMS, which tokens are you referring to? Is this reset something specific to Twilio? I can't imagine a scenario where SMS would enable the resetting of other 3rd party TOTP secrets, but I may not be understanding your comment.
- exabrial 4y agoYour TOTP accounts can be moved between devices using Authy, correct? That means Authy has access to the plain text secrets at some point. So mistake #1: your "secrets" aren't secret, they are merely held in escrow. If they "were evil", or perhaps if a state actor, or if you pissed a rouge employee off at Authy, they could absolutely leak your credentials. #2: The authy app lets you recover your account using SMS. So anyone that wants to pull off a simjack attack on you can login to your Authy as you and obtain the keys to the kingdom. The entire point of TOTP is the "Secret" is held locally in an oracle. If you break that constraint, you've broken the security of the protocol.
- deleted 4y ago[deleted]
- londons_explore 4y ago> If you are not contacted by Twilio, then it means we have no evidence that your account was impacted by this attack. aka... There were a bunch of ways employees could access the customer data unaudited, so we don't have evidence of that. Rather than say "we don't believe you were impacted in this attack", we're using weasel words to set your mind at rest, when we really have no idea.
- londons_explore 4y agoIf your company is hit by something like this, how sure are you that you'll find out? I suspect most companies would never know that someone was sharing an employees login and downloading data... Well if you want slightly more confidence, use my tool to catch thieves...[1] Turns out most thieves are rather tempted by some cryptocurrency that appears discarded and up for grabs. [1]: https://serverthiefbait.com/ https://serverthiefbait.com/
- Nextgrid 4y agoIt seems like a cryptocurrency wallet would only attract a very specific kind of thieves and would only apply to cases where you'd normally expect crypto wallets to be present. Given the "sophistication" of some of these attacks I would be surprised if the people behind it would even recognize a crypto wallet file name, let alone know how to import it and actually steal the coins. I remember seeing another service that provides usernames, emails or links to seed into your DB and will alert when the link is requested or spam starts coming into that. I think that will be more effective.
- londons_explore 4y agoI send a survey to each user who has seen their wallet drained. Many are just testing the service (ie. they drain the wallet themselves), but of the rest, most are unaware via other methods that their data has been accessed. The usual caveats of data from surveys applies...
- ChrisArchitect 4y agoThis isn't just Twilio-proper right? It would affect Sendgrid and whatever else they own?
- AtNightWeCode 4y agoI don’t get it. Have credentials been entered on third-party sites that then been used to logon from elsewhere? Would it be stupid to force mTLS for employees?
- tgsovlerkhgsel 4y agoI find it concerning that after getting pwned through a relatively lazy phish, their reaction is to double down on training instead of implementing unphishable second factor auth like they should have from the beginning. For everyone: If you have more than a handful of employees, one of them is guaranteed to be phished. If that can result in a meaningful compromise, work on your security until it doesn't. Then work on securing against attackers tricking that employee into downloading and running malware, because guess what, that's also going to happen.