19 ms·

Twilio: Someone broke into our AWS S3 silo, added 'non-malicious' code to JS SDK

194 points by LinuxBender 6y ago

madmulita 6y ago
English is not my main language, but "broke into" doesn't implicate "breaking" something to get "into" something?
- fokinsean 6y ago
  Yeah, in reality its like complaining someone "broke" into your care when you left all of your doors open.
  - JadeNB 6y ago
    > Yeah, in reality its like complaining someone "broke" into your care when you left all of your doors open. And you wouldn't get much sympathy, but this is a legally viable complaint. When my car was stolen, the thief was charged for theft, but his girlfriend who was joyriding with him was charged for criminal trespass. Just being in the car when you don't have permission is illegal.
- k1t 6y ago
  In the physical world (think "breaking and entering"), any amount of force (e.g. Pushing an already partially open door) counts as "breaking". It isn't necessary to actually "break" anything. Edit: Of course, it's still not a great metaphor because it implies there was at least some kind of barrier to entry that had to be pushed aside - but from the sounds of it, there wasn't.
- wolf550e 6y ago
  "break into" means "gained unauthorized entry" or more simply "got into without permission".
orenfromberg 6y ago
would using the `integrity` attribute in the `<script>` tag mitigate this attack?
- pquerna 6y ago
  Yes! Subresource Integrity is the exact counter-measure to this kind of attack: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres... However, on Twilio's documentation site, they do not include the integrity attribute in their examples: https://www.twilio.com/docs/taskrouter/js-sdk/workspace/taskqueue https://www.twilio.com/docs/taskrouter/js-sdk/workspace/task...
  - untog 6y ago
    It's difficult. I like the idea of subresource integrity a lot, but a counter example: Twilio discovers a bug in their JS SDK. Without subresource integrity they can push up a fix and propagate it to all clients immediately. With subresource integrity they would need every client to update their web site to change the integrity attribute. If it was a serious bug then that could be a huge issue. Even putting the security arguments aside you can bet a lot of customers would be really irritated having to constantly update this stuff on their site. If a competitor didn't have that restriction it would become a selling point. IMO the ideal is that Twilio provides both options: a "sdk-latest.js" that's a moving target, as well as "sdk-v1.3.js" that is frozen and can be locked with subresource integrity.
    - erichurkman 6y ago
      What about an alternate for these 'platform script includes' like Twilio and Stripe: <script src="..." report-integrity="https://cdn.stripe.com/report"> in which clients compute the integrity and asynchronously ping a lightweight payload of the integrity to the third party? You would then seed your report service with a list of known hashes and set up alerting for when new hashes were observed.
      - 6y ago
_alex_ 6y ago
If it’s world-writable, then is it really a break-in?
- TheSpiciestDev 6y ago
  "In local news, stranger off the street walks through residence's open door and moves around furniture."
  - ljoshua 6y ago
    Well, even in such a world, that would still be considered trespassing.
    - TheSpiciestDev 6y ago
      For sure, and I believe Twilio did not intend to "have their door open" and that whoever modified their code knew that they were "trespassing" similarly.
    - jfoutz 6y ago
      I am not a lawyer, I'm definitely not your lawyer. Trespassing and possibly theft, I think. Theft includes moving stuff without the owners permission. so if I ,say, have a tow truck and move a car across the street without the owners consent, I'm a thief. I only know this because a friend talked about a case where they rotated a car in-place, and there was a question about is that theft? its center of mass is still where they left it, so it wound up not being theft. Moving furniture in a house is a weird one. Probably leans on ill intent, moving a chair to block a door seems like it's pointed to giving the owner a hard time. moving a chair to perform cpr has much purer motives. So anyway, there's my ill-informed view.
      - benmmurphy 6y ago
        I thought in common law countries theft required you to have the intent to permanently deprive the victim of the property. this is why there are separate statutes to cover stealing cars because people just claimed they were taking it temporarily.
rafaelturk 6y ago
S3 security is a pain. S3 options are tricky to master, docs are beyond confusing. A common S3 use case is: Use a Bucket for Read-only static content, (js, html, imgs). From a dev perspective ideally it will work like a protected folder + Web Server, whereas the webserver will have read-only privileges, but reality is far more complicated. I can understand why was easy for Twillio to have this infosec issue, since we've done it ourselves many times when using S3.
- ed25519FUUU 6y ago
  Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a single role, and you don’t have to worry about rogue roles exposing the bucket. Perhaps some of the issue is that permission can be managed on both IAM and a bucket policy?
  - deleted 6y ago
    [deleted]
  - rafaelturk 6y ago
    This looks safe, and most importantly correct setup, but looks like it will not support Twillo use case
  - throwlogon 6y ago
    What you described sounds like a pain.
    - ed25519FUUU 6y ago
      What’s the pain? You literally don’t do anything to the bucket policy. It’s intentionally blank. You already have a role in AWS that you use. Go into the console and say “this role can do S3 to this bucket” and you’re done.
nwsm 6y ago
Wow. They were serving one of their SDKs out of this unsecured bucket and someone was able to upload a new version and it was distributed to customers.
bzb3 6y ago
How did they track if the modification worked? By calling home? Because then they leaked PII
- aeyes 6y ago
  This is the Javascript SDK so checking any website which uses it would be enough to see if the modification is present.
  - bzb3 6y ago
    Twilio says the only modification done to the script was to check if the modification could be done. If the hackers modified the script so it would call their own servers to check for that, they now have a list of IP addresses and timestamps.
coding123 6y ago
Aws should start taking blame for this shit, then maybe they would start auditing their customers setups...
- _vertigo 6y ago
  * how should AWS know when its customers want the permissions they choose and when they don't? * the cost of auditing would be spread across all customers equally, punishing competent customers and benefitting incompetent ones..
- deleted 6y ago
  [deleted]
- ceejayoz 6y ago
  AWS does audit this. You get a huge warning when you make a bucket public in this way, and they send you emails about it.
- l33tman 6y ago
  There are big signs on world-accessible buckets when you go into the S3 console. It's a fairly new thing but they are (slowly) trying to help auditing and finding these things I guess. However I'm not sure if there are any built-in tools to find weird nested policies attached that could cause a similar permission to be granted to all..
- x86_64Ubuntu 6y ago
  What does AWS have to do with this? Sure, the technical debt of S3 being one of the first services is clear to see. But it was a world writable bucket. What is AWS going to do about that? Eventually, you have to know and care about what the hell you are doing.
- aka1234 6y ago
  AWS's shared responsibility model is clear. AWS is responsible for security of the cloud. The customer is responsible for the security in the cloud (i.e. the customer's resources). By the way, enterprise support customers do get access to well-architected reviews by AWS. If you ask for help from AWS, AWS will provide it. It may not be free, but it's available. Even if AWS were to start proactively auditing customer setups, how in the world is AWS supposed to know what a customer's usecase is? Nevermind the fact it's a breach of the customer's privacy to just go rooting around in the customer's account without permission. But let's assume AWS is going to take responsibility for customers' configuration decisions and violate customers' privacy by proactively auditing their accounts. Would AWS auditing Twilio's configuration here work? The default is for S3 buckets to be private. The customer has to take specific, affirmative steps to give s3 buckets public access. You really have to jump through hoops to make a bucket public accessible. Since the Twilio chose to make their bucket public, AWS auditing Twilio's setup wouldn't be helpful. AWS would just assume the Twilio knows what they're doing. How is AWS supposed to know there is a misconfiguration? Because Twilio clearly decided to make their S3 bucket publicly available.
ebg13 6y ago
> someone was able to get into Twilio's Amazon Web Services S3 bucket, which was left unprotected and world-writable "Someone" was able? If the bucket was unprotected and world-writable, then _everyone_ was able. > 'non-malicious' > And judging from the URL involved, it appears to be an attempt to install a payment-card skimmer – RiskIQ has spotted the same URL in other S3 buckets targeted by miscreants. In what world is that non-malicious? Non-immediately-damaging _maybe_, but it seems clear that there was a malicious chain of reasoning involved. Twilio got super lucky if this is all that happened.
carterklein13 6y ago
One of the most shocking things I realized when I graduated college and started working as a software engineer is how insecure, or really just improperly-configured, large enterprise systems are. This is anecdotal, of course, but it seems to me that any company that didn't start cloud-native is going to have flaws like this, and it's often not at the fault of developers but at the fault of top-down initiatives to "get to the cloud" ASAP instead of correctly.
whoisjuan 6y ago
Having an S3 Bucket with writing permissions to anyone is a major fuck-up. I know many people open buckets to the world at the read-level for whatever reason. I do that myself for static websites. But also writing permissions? This requires a Bucket Policy that is written explicitly to allow this. I don't want to make assumptions because of course we don't know all the information and I personally know very little about infosec, but that sounds very strange. Why would you do that? I assume the SDK gets uploaded from a secure environment that has the credentials to upload objects, so there's no legitimate reason to have writing permissions open to the world.
- corytheboyd 6y ago
  Yeah this is absolutely inexcusable for a public tech giant. I wonder how incompetently secured the rest of their backend is. This is terrifying, the amount of PII flowing through Twilio is insane, there really needs to be a way to bite companies that completely ignore all of these security compliance checks they say they pass...
  - kyboren 6y ago
    To add another data point, consider this email Twilio sent, announcing that they are now forcing all users to use 2FA by October 12th: > [...] a 2FA verification code sent to your phone (via a call, SMS message, or an authentication app like Authy). PSTN-based 2FA in 2020 is not a vote of confidence in the security team's competence.
    - u801e 6y ago
      > PSTN-based 2FA in 2020 is not a vote of confidence in the security team's competence. A lot of companies go with email and/or PSTN based 2FA for customer convenience and to get them to use 2FA for their accounts. I try to avoid using 2FA for accounts that only offer PSTN or email based 2FA because it's less secure than my randomly generated long password. It would be nice if more services offered 2FA based on client side TLS certificates.
fmakunbound 6y ago
Don't they sign their code so that package managers can verify it's really from Twilio?
tyingq 6y ago
Doesn't sound so non-malicious to me: "Specifically, the modification added code to the end of the TaskRouter.js v1.20 SDK that made an HTTP GET request to hxxps://gold.platinumus.top/track/awswrite?q=dmn and followed the URL returned in the HTML by that request."
- oefrha 6y ago
  > followed the URL returned in the HTML by that request Translated: we can't possibly know it's non-malicious. TFA even has this to say: > And judging from the URL involved, it appears to be an attempt to install a payment-card skimmer – RiskIQ has spotted the same URL in other S3 buckets targeted by miscreants. Details in the linked blog post[0]. So, very much malicious. Why the hell they included "non-malicious" in the title (granted, in quotes), I don't know. Readers could have easily dismissed this as actually non-malicious. [0] https://www.riskiq.com/blog/labs/misconfigured-s3-buckets/ https://www.riskiq.com/blog/labs/misconfigured-s3-buckets/
  - FDSGSG 6y ago
    FWIW, that's a redirector and not a skimmer.
    - dsl 6y ago
      Magecart is a well known framework in the underground for stealing credit card numbers from websites (if they can get the JavaScript running on a checkout page). If you hit it directly they will redirect to a random "Spin the wheel win a prize" type site in an attempt to hide the malicious nature of the payload.
      - ryanlol 6y ago
        This is not magecart.
        heavenlyblue 6y ago
paulie_a 6y ago
Non malicious? Does the person. That wrote that bit of bullshit even believe it. They might as well have said "we are not smart enough to figure out the nature of this code. In unrelated news we are hiring senior JavaScript engineers, experience in opsec is a bonus"
natpalmer1776 6y ago
I was curious about the URL that they indicated was non-malicious and so looked it up on google. I found this any.run report: Report: https://any.run/report/e8c581d52ca3527b61c7ddd9aba5dfeaf944069b625577889b83e425d54f887a/a5f1e345-5559-4955-960d-752bf877e028 https://any.run/report/e8c581d52ca3527b61c7ddd9aba5dfeaf9440... Original URL: h t t p s : / / gold.platinumus.top/track/awswrite?q=dmn
- jlgaddis 6y ago
  Nice find. That certainly indicates the exact opposite of "non-malicious".
cj 6y ago
Quick shout out to GuardScript which exists to help catch malicious 3rd party (or 1st party) javascript changes: https://www.guardscript.com/ https://www.guardscript.com/ It's a really simple service that fetches a list of script URLs that you provide, and notifies you with the before/after diff whenever the file changes. I found GuardScript through a Show HN a while ago (https://news.ycombinator.com/item?id=20265141 https://news.ycombinator.com/item?id=20265141) and have been very pleased with it.
- jszymborski 6y ago
  Isn't this kinda solved by Subresource Integrity (SRI)? https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
  - lozenge 6y ago
    Some sites use ad networks that prevent them from setting good SRI policies.
    - jaywalk 6y ago
      So you don't set the SRI on those scripts. Nothing about the Twilio script referenced by this article would prevent SRI from being applied.
  - cj 6y ago
    Not really. For example, here is Stripe's script tag: <script src="https://js.stripe.com/v3/"></script> https://js.stripe.com/v3/"></script> SRI only works for static files whose contents never change. In this example, it would mean that Stripe could never push minor bug fixes to the SDK. The workaround would be to use static versioned files instead of a generic script URL (or self-host the script), so that the value of the SRI checksum never changes. Edit: In the case of Twilio, their docs say "By linking directly to our CDN in production, any patch-level (non-breaking) changes made to the SDK will automatically be applied [...]" meaning that if you link to Twilio's CDN, you can't expect the file to be static, and therefore can't use SRI.
kirillzubovsky 6y ago
I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably to provide the piping, and leave the rest of the end user, but at this point it's not enough. In my opinion, they need to heavily invest into the customer-facing layer of their applications that shows all the status, gaps and holes, or these things keep happening. Then again, they can continue as is, and the incidents will keep as is, and as long as nothing implodes and no big customer leaves, financially it makes sense to do nothing. Just thinking out loud.
- MrBuddyCasino 6y ago
  I am still amazed some people want to do devops. Write yaml, run script, wait 15 minutes, oh no a typo lets retry. The developer experience is worse than nearly everything that came before it, and I would include ancient PHP and callback-hell nodejs in that. The AWS web UI isn’t much better. Looking forward to finally try Pulumi in a project.
  - marcocampos 6y ago
    How will Pulumi fix the issues you mentioned?
    - MrBuddyCasino 6y ago
      By doing what XML-with-schema did 15 years ago: give compile-time type checking, making sure at least the syntax and types are valid. Ever had CloudFormation yell at you because some function like "Sub!" only accepts alphanumeric characters? Created complicated constructs because there are no variables, and very limited conditionals? Copy-pasted templates like an insane person because there is no way to abstract anything? Not my idea of fun. All of this is solved by using a proper programming language.
  - 6y ago
_josh_ 6y ago
My immediate thought was about their Authy Desktop app - looks and feels like a web app in a container. If someone gets access to that then boom, all your 2FA tokens are stolen. Time to find an alternative.
Someone1234 6y ago
Twilio's examples[0] for taskrouter.min.js, even now, don't include Subresource Integrity hashes. Subresource Integrity would have stopped the modified script from being loaded, and removed a single point of failure. Subresource Integrity is supported by every evergreen browser and 80%+ of market share (no IE11/pre-Chromium Edge). [0] https://www.twilio.com/docs/taskrouter/js-sdk/workspace/worker https://www.twilio.com/docs/taskrouter/js-sdk/workspace/work...
donor20 6y ago
One thing I wish AWS would do. Allow admins to "down feature" the account. Ie, simple mode. Basically, allow admins to toggle OFF S3 ACL's and even S3 Bucket Policies - zero those out. Then let users use IAM policies for access which may be a more familiar / more used mental model.
- codeduck 6y ago
  A better approach is simply not to grant normal users modify access to IAM, VPC, or anything else potentially destructive, and force them to make all changes via a ci/cd pipeline with ops/sre oversight and review.
kolencherry 6y ago
Hey folks, we've published a post with more details on the incident here: https://www.twilio.com/blog/incident-report-taskrouter-js-sdk-july-2020 https://www.twilio.com/blog/incident-report-taskrouter-js-sd... (I work for Twilio)
- aka1234 6y ago
  Thanks for posting this. I'm really impressed with the transparency Twilio showed in actually admitting to having such a silly, silly bucket policy. Not impressed that it was there in the first place; but that should go without saying. This incident report should really put to bed all of the "It's AWS's fault for making things so complex" complaints. (To be clear, it won't... but it should.) Even a cursory look at that bucket policy should tell you something named "Allow Public Read" should NOT be associated with anything named 'Put'. This takes 0 AWS knowledge to figure out.
  - oefrha 6y ago
    Really not impressed with the obligatory "really impressed with transparency" pat-on-the-back under every incident report for a big corp screw-up that provides any details at all. And stating to the press the clearly malicious payload is "non-malicious" (assuming TFA didn't lie about Twilio's statement)? That's ridiculous.
    - ficklepickle 6y ago
      Even if the payload was not malicious when they looked, it could change at any time. I don't see how that can be confidently labeled non-malicious.
    - aka1234 6y ago
      When talking about screw-ups on AWS, public incident reports try to obfuscate and spin the Hell out of issues that boil down to "really, really stupid configuration issue". They owned it. That is more than can be said about other large incident reports that I've seen regarding AWS.
zelly 6y ago
This wouldn't have happened if they just used an old school nginx file server. It has sane defaults like never letting random people edit your files through an API. Oh and it also would have been cheaper than S3. Just sayin.
Schnitz 6y ago
Is there a Twilio page that's confirming this?
dbielik 6y ago
Doesn't look isolated to Twilio - seems to be on 85 websites via other affected 3rd party scripts: https://www.nerdydata.com/reports/gold-platinumus-top-track/c5bb39e3-baa4-40ec-9dcb-f53876a59471 https://www.nerdydata.com/reports/gold-platinumus-top-track/...

194 points by LinuxBender 6y ago

madmulita 6y ago

English is not my main language, but "broke into" doesn't implicate "breaking" something to get "into" something?

fokinsean 6y ago
Yeah, in reality its like complaining someone "broke" into your care when you left all of your doors open.
- JadeNB 6y ago
  > Yeah, in reality its like complaining someone "broke" into your care when you left all of your doors open. And you wouldn't get much sympathy, but this is a legally viable complaint. When my car was stolen, the thief was charged for theft, but his girlfriend who was joyriding with him was charged for criminal trespass. Just being in the car when you don't have permission is illegal.
k1t 6y ago
In the physical world (think "breaking and entering"), any amount of force (e.g. Pushing an already partially open door) counts as "breaking". It isn't necessary to actually "break" anything. Edit: Of course, it's still not a great metaphor because it implies there was at least some kind of barrier to entry that had to be pushed aside - but from the sounds of it, there wasn't.
wolf550e 6y ago
"break into" means "gained unauthorized entry" or more simply "got into without permission".

orenfromberg 6y ago

would using the `integrity` attribute in the `<script>` tag mitigate this attack?

pquerna 6y ago
Yes! Subresource Integrity is the exact counter-measure to this kind of attack: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres... However, on Twilio's documentation site, they do not include the integrity attribute in their examples: https://www.twilio.com/docs/taskrouter/js-sdk/workspace/taskqueue https://www.twilio.com/docs/taskrouter/js-sdk/workspace/task...
- untog 6y ago
  It's difficult. I like the idea of subresource integrity a lot, but a counter example: Twilio discovers a bug in their JS SDK. Without subresource integrity they can push up a fix and propagate it to all clients immediately. With subresource integrity they would need every client to update their web site to change the integrity attribute. If it was a serious bug then that could be a huge issue. Even putting the security arguments aside you can bet a lot of customers would be really irritated having to constantly update this stuff on their site. If a competitor didn't have that restriction it would become a selling point. IMO the ideal is that Twilio provides both options: a "sdk-latest.js" that's a moving target, as well as "sdk-v1.3.js" that is frozen and can be locked with subresource integrity.
  - erichurkman 6y ago
    What about an alternate for these 'platform script includes' like Twilio and Stripe: <script src="..." report-integrity="https://cdn.stripe.com/report"> in which clients compute the integrity and asynchronously ping a lightweight payload of the integrity to the third party? You would then seed your report service with a list of known hashes and set up alerting for when new hashes were observed.
    - 6y ago

_alex_ 6y ago

If it’s world-writable, then is it really a break-in?

TheSpiciestDev 6y ago
"In local news, stranger off the street walks through residence's open door and moves around furniture."
- ljoshua 6y ago
  Well, even in such a world, that would still be considered trespassing.
  - TheSpiciestDev 6y ago
    For sure, and I believe Twilio did not intend to "have their door open" and that whoever modified their code knew that they were "trespassing" similarly.
  - jfoutz 6y ago
    I am not a lawyer, I'm definitely not your lawyer. Trespassing and possibly theft, I think. Theft includes moving stuff without the owners permission. so if I ,say, have a tow truck and move a car across the street without the owners consent, I'm a thief. I only know this because a friend talked about a case where they rotated a car in-place, and there was a question about is that theft? its center of mass is still where they left it, so it wound up not being theft. Moving furniture in a house is a weird one. Probably leans on ill intent, moving a chair to block a door seems like it's pointed to giving the owner a hard time. moving a chair to perform cpr has much purer motives. So anyway, there's my ill-informed view.
    - benmmurphy 6y ago
      I thought in common law countries theft required you to have the intent to permanently deprive the victim of the property. this is why there are separate statutes to cover stealing cars because people just claimed they were taking it temporarily.

rafaelturk 6y ago

S3 security is a pain. S3 options are tricky to master, docs are beyond confusing. A common S3 use case is: Use a Bucket for Read-only static content, (js, html, imgs). From a dev perspective ideally it will work like a protected folder + Web Server, whereas the webserver will have read-only privileges, but reality is far more complicated. I can understand why was easy for Twillio to have this infosec issue, since we've done it ourselves many times when using S3.

ed25519FUUU 6y ago
Is it really a pain though? Maybe because I’ve been working with it since the beginning. Bucket ACL with NO permissions, then manage all permissions on an IAM role on the account. If it’s cross account then allow assuming to other accounts, but no reason to bother with the bucket ACL. Leave the bucket policy blank and you’ll never have to worry about an open bucket. Better yet make a deny rule to everything but a single role, and you don’t have to worry about rogue roles exposing the bucket. Perhaps some of the issue is that permission can be managed on both IAM and a bucket policy?
- deleted 6y ago
  [deleted]
- rafaelturk 6y ago
  This looks safe, and most importantly correct setup, but looks like it will not support Twillo use case
- throwlogon 6y ago
  What you described sounds like a pain.
  - ed25519FUUU 6y ago
    What’s the pain? You literally don’t do anything to the bucket policy. It’s intentionally blank. You already have a role in AWS that you use. Go into the console and say “this role can do S3 to this bucket” and you’re done.

nwsm 6y ago

Wow. They were serving one of their SDKs out of this unsecured bucket and someone was able to upload a new version and it was distributed to customers.

bzb3 6y ago

How did they track if the modification worked? By calling home? Because then they leaked PII

aeyes 6y ago
This is the Javascript SDK so checking any website which uses it would be enough to see if the modification is present.
- bzb3 6y ago
  Twilio says the only modification done to the script was to check if the modification could be done. If the hackers modified the script so it would call their own servers to check for that, they now have a list of IP addresses and timestamps.

coding123 6y ago

Aws should start taking blame for this shit, then maybe they would start auditing their customers setups...

_vertigo 6y ago
* how should AWS know when its customers want the permissions they choose and when they don't? * the cost of auditing would be spread across all customers equally, punishing competent customers and benefitting incompetent ones..
deleted 6y ago
[deleted]
ceejayoz 6y ago
AWS does audit this. You get a huge warning when you make a bucket public in this way, and they send you emails about it.
l33tman 6y ago
There are big signs on world-accessible buckets when you go into the S3 console. It's a fairly new thing but they are (slowly) trying to help auditing and finding these things I guess. However I'm not sure if there are any built-in tools to find weird nested policies attached that could cause a similar permission to be granted to all..
x86_64Ubuntu 6y ago
What does AWS have to do with this? Sure, the technical debt of S3 being one of the first services is clear to see. But it was a world writable bucket. What is AWS going to do about that? Eventually, you have to know and care about what the hell you are doing.
aka1234 6y ago
AWS's shared responsibility model is clear. AWS is responsible for security of the cloud. The customer is responsible for the security in the cloud (i.e. the customer's resources). By the way, enterprise support customers do get access to well-architected reviews by AWS. If you ask for help from AWS, AWS will provide it. It may not be free, but it's available. Even if AWS were to start proactively auditing customer setups, how in the world is AWS supposed to know what a customer's usecase is? Nevermind the fact it's a breach of the customer's privacy to just go rooting around in the customer's account without permission. But let's assume AWS is going to take responsibility for customers' configuration decisions and violate customers' privacy by proactively auditing their accounts. Would AWS auditing Twilio's configuration here work? The default is for S3 buckets to be private. The customer has to take specific, affirmative steps to give s3 buckets public access. You really have to jump through hoops to make a bucket public accessible. Since the Twilio chose to make their bucket public, AWS auditing Twilio's setup wouldn't be helpful. AWS would just assume the Twilio knows what they're doing. How is AWS supposed to know there is a misconfiguration? Because Twilio clearly decided to make their S3 bucket publicly available.

ebg13 6y ago

> someone was able to get into Twilio's Amazon Web Services S3 bucket, which was left unprotected and world-writable "Someone" was able? If the bucket was unprotected and world-writable, then _everyone_ was able. > 'non-malicious' > And judging from the URL involved, it appears to be an attempt to install a payment-card skimmer – RiskIQ has spotted the same URL in other S3 buckets targeted by miscreants. In what world is that non-malicious? Non-immediately-damaging _maybe_, but it seems clear that there was a malicious chain of reasoning involved. Twilio got super lucky if this is all that happened.

carterklein13 6y ago

One of the most shocking things I realized when I graduated college and started working as a software engineer is how insecure, or really just improperly-configured, large enterprise systems are. This is anecdotal, of course, but it seems to me that any company that didn't start cloud-native is going to have flaws like this, and it's often not at the fault of developers but at the fault of top-down initiatives to "get to the cloud" ASAP instead of correctly.

whoisjuan 6y ago

Having an S3 Bucket with writing permissions to anyone is a major fuck-up. I know many people open buckets to the world at the read-level for whatever reason. I do that myself for static websites. But also writing permissions? This requires a Bucket Policy that is written explicitly to allow this. I don't want to make assumptions because of course we don't know all the information and I personally know very little about infosec, but that sounds very strange. Why would you do that? I assume the SDK gets uploaded from a secure environment that has the credentials to upload objects, so there's no legitimate reason to have writing permissions open to the world.

corytheboyd 6y ago
Yeah this is absolutely inexcusable for a public tech giant. I wonder how incompetently secured the rest of their backend is. This is terrifying, the amount of PII flowing through Twilio is insane, there really needs to be a way to bite companies that completely ignore all of these security compliance checks they say they pass...
- kyboren 6y ago
  To add another data point, consider this email Twilio sent, announcing that they are now forcing all users to use 2FA by October 12th: > [...] a 2FA verification code sent to your phone (via a call, SMS message, or an authentication app like Authy). PSTN-based 2FA in 2020 is not a vote of confidence in the security team's competence.
  - u801e 6y ago
    > PSTN-based 2FA in 2020 is not a vote of confidence in the security team's competence. A lot of companies go with email and/or PSTN based 2FA for customer convenience and to get them to use 2FA for their accounts. I try to avoid using 2FA for accounts that only offer PSTN or email based 2FA because it's less secure than my randomly generated long password. It would be nice if more services offered 2FA based on client side TLS certificates.

fmakunbound 6y ago

Don't they sign their code so that package managers can verify it's really from Twilio?

tyingq 6y ago

Doesn't sound so non-malicious to me: "Specifically, the modification added code to the end of the TaskRouter.js v1.20 SDK that made an HTTP GET request to hxxps://gold.platinumus.top/track/awswrite?q=dmn and followed the URL returned in the HTML by that request."

oefrha 6y ago
> followed the URL returned in the HTML by that request Translated: we can't possibly know it's non-malicious. TFA even has this to say: > And judging from the URL involved, it appears to be an attempt to install a payment-card skimmer – RiskIQ has spotted the same URL in other S3 buckets targeted by miscreants. Details in the linked blog post[0]. So, very much malicious. Why the hell they included "non-malicious" in the title (granted, in quotes), I don't know. Readers could have easily dismissed this as actually non-malicious. [0] https://www.riskiq.com/blog/labs/misconfigured-s3-buckets/ https://www.riskiq.com/blog/labs/misconfigured-s3-buckets/
- FDSGSG 6y ago
  FWIW, that's a redirector and not a skimmer.
  - dsl 6y ago
    Magecart is a well known framework in the underground for stealing credit card numbers from websites (if they can get the JavaScript running on a checkout page). If you hit it directly they will redirect to a random "Spin the wheel win a prize" type site in an attempt to hide the malicious nature of the payload.
    - ryanlol 6y ago
      This is not magecart.
      - heavenlyblue 6y ago

paulie_a 6y ago

Non malicious? Does the person. That wrote that bit of bullshit even believe it. They might as well have said "we are not smart enough to figure out the nature of this code. In unrelated news we are hiring senior JavaScript engineers, experience in opsec is a bonus"

natpalmer1776 6y ago

I was curious about the URL that they indicated was non-malicious and so looked it up on google. I found this any.run report: Report: https://any.run/report/e8c581d52ca3527b61c7ddd9aba5dfeaf944069b625577889b83e425d54f887a/a5f1e345-5559-4955-960d-752bf877e028 https://any.run/report/e8c581d52ca3527b61c7ddd9aba5dfeaf9440... Original URL: h t t p s : / / gold.platinumus.top/track/awswrite?q=dmn

jlgaddis 6y ago
Nice find. That certainly indicates the exact opposite of "non-malicious".

cj 6y ago

Quick shout out to GuardScript which exists to help catch malicious 3rd party (or 1st party) javascript changes: https://www.guardscript.com/ https://www.guardscript.com/ It's a really simple service that fetches a list of script URLs that you provide, and notifies you with the before/after diff whenever the file changes. I found GuardScript through a Show HN a while ago (https://news.ycombinator.com/item?id=20265141 https://news.ycombinator.com/item?id=20265141) and have been very pleased with it.

jszymborski 6y ago
Isn't this kinda solved by Subresource Integrity (SRI)? https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
- lozenge 6y ago
  Some sites use ad networks that prevent them from setting good SRI policies.
  - jaywalk 6y ago
    So you don't set the SRI on those scripts. Nothing about the Twilio script referenced by this article would prevent SRI from being applied.
- cj 6y ago
  Not really. For example, here is Stripe's script tag: <script src="https://js.stripe.com/v3/"></script> https://js.stripe.com/v3/"></script> SRI only works for static files whose contents never change. In this example, it would mean that Stripe could never push minor bug fixes to the SDK. The workaround would be to use static versioned files instead of a generic script URL (or self-host the script), so that the value of the SRI checksum never changes. Edit: In the case of Twilio, their docs say "By linking directly to our CDN in production, any patch-level (non-breaking) changes made to the SDK will automatically be applied [...]" meaning that if you link to Twilio's CDN, you can't expect the file to be static, and therefore can't use SRI.

kirillzubovsky 6y ago

I love AWS as any other developer, but this is their fault. They've invested all their resources into building out the infrastructure, and have spent very little if any time on building the UX around it. You have to be fairly technical, never mind painstakingly detailed-oriented, in order to manage their services, a big chunk of which is hidden behind black screens and various control nobs. Amazon's logic is probably to provide the piping, and leave the rest of the end user, but at this point it's not enough. In my opinion, they need to heavily invest into the customer-facing layer of their applications that shows all the status, gaps and holes, or these things keep happening. Then again, they can continue as is, and the incidents will keep as is, and as long as nothing implodes and no big customer leaves, financially it makes sense to do nothing. Just thinking out loud.

MrBuddyCasino 6y ago
I am still amazed some people want to do devops. Write yaml, run script, wait 15 minutes, oh no a typo lets retry. The developer experience is worse than nearly everything that came before it, and I would include ancient PHP and callback-hell nodejs in that. The AWS web UI isn’t much better. Looking forward to finally try Pulumi in a project.
- marcocampos 6y ago
  How will Pulumi fix the issues you mentioned?
  - MrBuddyCasino 6y ago
    By doing what XML-with-schema did 15 years ago: give compile-time type checking, making sure at least the syntax and types are valid. Ever had CloudFormation yell at you because some function like "Sub!" only accepts alphanumeric characters? Created complicated constructs because there are no variables, and very limited conditionals? Copy-pasted templates like an insane person because there is no way to abstract anything? Not my idea of fun. All of this is solved by using a proper programming language.
- 6y ago

_josh_ 6y ago

My immediate thought was about their Authy Desktop app - looks and feels like a web app in a container. If someone gets access to that then boom, all your 2FA tokens are stolen. Time to find an alternative.

Someone1234 6y ago

Twilio's examples[0] for taskrouter.min.js, even now, don't include Subresource Integrity hashes. Subresource Integrity would have stopped the modified script from being loaded, and removed a single point of failure. Subresource Integrity is supported by every evergreen browser and 80%+ of market share (no IE11/pre-Chromium Edge). [0] https://www.twilio.com/docs/taskrouter/js-sdk/workspace/worker https://www.twilio.com/docs/taskrouter/js-sdk/workspace/work...

donor20 6y ago

One thing I wish AWS would do. Allow admins to "down feature" the account. Ie, simple mode. Basically, allow admins to toggle OFF S3 ACL's and even S3 Bucket Policies - zero those out. Then let users use IAM policies for access which may be a more familiar / more used mental model.

codeduck 6y ago
A better approach is simply not to grant normal users modify access to IAM, VPC, or anything else potentially destructive, and force them to make all changes via a ci/cd pipeline with ops/sre oversight and review.

kolencherry 6y ago

Hey folks, we've published a post with more details on the incident here: https://www.twilio.com/blog/incident-report-taskrouter-js-sdk-july-2020 https://www.twilio.com/blog/incident-report-taskrouter-js-sd... (I work for Twilio)

aka1234 6y ago
Thanks for posting this. I'm really impressed with the transparency Twilio showed in actually admitting to having such a silly, silly bucket policy. Not impressed that it was there in the first place; but that should go without saying. This incident report should really put to bed all of the "It's AWS's fault for making things so complex" complaints. (To be clear, it won't... but it should.) Even a cursory look at that bucket policy should tell you something named "Allow Public Read" should NOT be associated with anything named 'Put'. This takes 0 AWS knowledge to figure out.
- oefrha 6y ago
  Really not impressed with the obligatory "really impressed with transparency" pat-on-the-back under every incident report for a big corp screw-up that provides any details at all. And stating to the press the clearly malicious payload is "non-malicious" (assuming TFA didn't lie about Twilio's statement)? That's ridiculous.
  - ficklepickle 6y ago
    Even if the payload was not malicious when they looked, it could change at any time. I don't see how that can be confidently labeled non-malicious.
  - aka1234 6y ago
    When talking about screw-ups on AWS, public incident reports try to obfuscate and spin the Hell out of issues that boil down to "really, really stupid configuration issue". They owned it. That is more than can be said about other large incident reports that I've seen regarding AWS.

zelly 6y ago

This wouldn't have happened if they just used an old school nginx file server. It has sane defaults like never letting random people edit your files through an API. Oh and it also would have been cheaper than S3. Just sayin.

Schnitz 6y ago

Is there a Twilio page that's confirming this?

dbielik 6y ago

Doesn't look isolated to Twilio - seems to be on 85 websites via other affected 3rd party scripts: https://www.nerdydata.com/reports/gold-platinumus-top-track/c5bb39e3-baa4-40ec-9dcb-f53876a59471 https://www.nerdydata.com/reports/gold-platinumus-top-track/...