50 ms·

CircleCI trusts 8 analytics companies with your source code and API tokens

508 points by tomwas54 9y ago

CaliforniaKarl 9y ago
To me, this seems like a vulnerability report. The following line is what really made it stand out for me as being a vulnerability report: >Why don't you include a POC? I have one that demonstrates this attack, but I don't want to show script kiddies how. That leads me to this question: Did the author reach out to CircleCI to report this issue, before publishing their blog post?
- kevinburke 9y ago
  No, I didn't. If one of the JavaScript sources was immediately compromisable, I would have.
  - kierenj 9y ago
    Why post a big public dig? If there's a problem, make the world a better place by helping to solve it maybe? Just having a bad day?
    - kevinburke 9y ago
      1) they know exactly what they are doing; 2) arbitrary 3rd party JS running in privileged contexts (dashboards) is a wider problem I would like to draw attention to; 3) It's not immediately exploitable; 4) It takes five seconds to understand what the problem is; 5) I did not want to wait 90 days to be told "we are not going to fix this;" 6) my history of reporting problems (not security related) to the company in question is that they are not very responsive to fixing them.
      - wersow 9y ago
        Not all of the companies you listed are analytics company, just FYI.
      - kierenj 9y ago
        Umm, either its a problem and worth reporting responsibly, or not a problem and so not worth a hit piece on your blog?
        tombrossman 9y ago
        This comment highlights the problem with calling disclosure 'responsible', because the word is used subjectively. What you as a commenter, or a company (usually prefers to quietly fix this), or the wider user base (usually prefers a 'heads up' immediately), all have different opinions about what they see as responsible. Also, it doesn't seem fair to dismiss this disclosure as a 'hit piece' unless it is factually incorrect.
        icebraining 9y ago
        It's only worth reporting "responsibly" if waiting might prevent users from being hurt. The way that concept has been manipulated into implying that the companies are owed an early warning is crap. The author doesn't owe CircleCI anything.
        kevinburke 9y ago
        Precisely
      - j_s 9y ago
        I was with you until I saw this on your consulting pitch: Write. I have written at least ten posts that made the front page of Hacker News / Programming Reddit (nb - Those aren't the best proxies for quality or popularity, but they are well known proxies). I can help kickstart your company's engineering blog, or work with your team on story/content ideas. Now I'm not sure how much making the front page motivates your writing process. It is an interesting topic but I feel it would be a stronger case to demonstrate the issue across products from multiple vendors, though less incendiary/front-page-y. PS. Props for not being the one to submit this particular article!
    - idunno246 9y ago
      Because it's a rampant problem not limited to them, and not obvious, and nobody seems to care? This is a good example for getting the target market to understand the dangers of just dropping in the latest js tool that marketing wants. I used to be in ad tech, it always baffled me that we had first party js on every single customer page. Conceivable if we were breached, someone could inject code to read every login token or cc# or anything and send it to their servers.
sytse 9y ago
At GitLab we separated the infrastructure between about.gitlab.com (our marketing site, lots of third party javascript) and gitlab.com (our application, zero third party javascript). We recently deprecated the Piwik instance we ran in house for gitlab.com because it slowed the page load time. Please let me know if there is anything we can do better, these things are complex.
- pg_is_a_butt 9y ago
  why are you still using the same top level domain? sure there are XSS protections on subdomains, but it's still 1 exploit away from the top level domain. google uses googlehosted.com for stuff that might include 3rd party stuff.
- BoorishBears 9y ago
  I feel this is off topic.
  - davnicwil 9y ago
    I'm not sure if you're aware, but GitLab has integrated CI available on gitlab.com, which is a like-for-like alternative to CircleCI - so in that sense this comment is very much on topic, in my opinion.
    - mr337 9y ago
      I agree too. When reading about CircleCI my first thought is, "wonder why my CI is doing?". Now gotta check the requests to validate their claim.
    - BoorishBears 9y ago
      I'm aware, to me it feels like a plug. I've seen a nearly identical comment on Gitlab from a member of their team on other occasions
      - BoorishBears 9y ago
deleted 9y ago
[deleted]
koolba 9y ago
> Send an email any time an API access token is created. Add a setting to allow org-wide disabling of API token creation. I wish more apps supported this. Also on my list is a "Don't ever let me disable 2FA" setting. I'm more worried about malicoius resets than I am about ever losing my 2FA device and backup codes. > Add an option to delete old logs. If you have ever dumped env vars to the log file, an attacker can export these. I surprised that secrets make it into the logs at all. I would have expected them to filter anything that's in an env secret from the log output. Pretty sure Travis does this. > Enable subresource integrity, or serve JS from each of these companies from the CircleCI domain. That's not much of an option for this type of thing as the third parties would then have a PITA time dealing with upgrades (so they wouldn't support it). Using <script src="hxxps://tracker.example.com/path/to/script.js"></script> (rather than a fixed version) allows for live upgrades as they're in control of the resource that is loaded. And you can't load the script from your domain as at the end of the day it's got to get instructions and upload data to the third party. At best you'd be loading a shim that does the same thing as the script tag.
- orginal__idear 9y ago
  > "Don't ever let me disable 2FA" setting. I like this idea. I'd only want to use this though if my Google 2FA backup codes are backed up via Authy or a similar app. I don't trust codes I've printed out and have always lost.
  - rhizome 9y ago
    Have you considered getting a safe-deposit box?
  - dabernathy89 9y ago
    I tend to just stick all my 2FA backup codes into "secure notes" in LastPass.
    - ihattendorf 9y ago
      That sort of defeats the purpose of 2FA though. Now if your LastPass database is compromised, they have access to your 2FA codes as well.
jbg_ 9y ago
uMatrix.
- fluxsauce 9y ago
  That treats the symptom, not the problem.
  - beagle3 9y ago
    uMatrix is defense in depth. They could solve the problem today, and tomorrow a new CircleCI programmer/marketer/CEO would decide that, in fact, they do care more about analytics than about user's secrecy, and revert this change. By installing uMatrix and not approving these things, you are (a) reducing the probability that a mistake, misaligned incentives, etc, would harm you, and (b) get a notification that there's something you wish to block. uMatrix IS part of the real solution. The other part is avoiding providers whose security standards are not up to yours.
    - 45h34jh53k4j 9y ago
      Its a real solution, for YOU personally. I use and love umatrix (whitelisting ALL js/content by hand on all domains). We need to consider the good of the many, and client side solutions like umatrix are not a solution for the many.
  - dom0 9y ago
    Good security does generally not rely on a single measure. The service provider not doing this in the first place is good, but the user (also) having a second line of defense is objectively better, although it may incentivize poor behaviour of SPs.
kevan 9y ago
At my last company we had a similar issue with third party scripts on credit application forms. Analytics is critical for ecommerce sites but it's way too easy to accidentally log sensitive data across dozens of third party services.
- bm1362 9y ago
  There are self-hosted analytics (like Piwik) that you could self-host for sensitive data to solve this generally. I'd like to think that conversion of credit applications isn't something a marketer is trying to gamify though, that's probably wishful thinking.
  - kevan 9y ago
    >I'd like to think that conversion of credit applications isn't something a marketer is trying to gamify though If it's in the overall sales conversion funnel it will be optimized. Credit application flow was scrutinized just as much as adding to cart and checking out.
nexfitter 9y ago
I use CircleCI because it allows 1 free private repo. Personally though I do not like Circle too much, it has a lot of cool features but the UI loads so slowly because they have so much stuff going on. They use the UI skeletons technique to make the load time appear to be faster but it is still very apparent how slow the load time is. I have hit numerous bugs with their website as well where stuff doesn't load, builds kick off into infinity, when I transferred a repo everything broke, ack! Furthermore, I am frustrated with their pricing, they go from free to $50/mo for the next upgrade tier, that seems like a crazy jump to me. I would gladly pay $10/mo or so for another container or 2x parallelization. The issue is being a single developer I rarely would save any time and the $50/mo is just too steep. Finally, when I tried to build a justification case for my manager to pay the $50/mo, I wanted to use data from their CircleCI Insights which shows how long your builds are queued on average and some other important data points. But you cannot access these insights from a free account. Seems like that info should be available and prominent to help people understand the cost-savings they might receive by upgrading. I emailed support and asked for a one-time data point for that statistic to build the case as I was considering upgrading and they said sorry, nothing we can do for you. Is their goal to make money and have happy customers? If so they aren't doing a great job of it. Overall a lot of frustrations with the platform and this just adds more fuel to the fire.
- daxorid 9y ago
  Serious question: why CircleCI over, say, Jenkins or custom_build_script.sh running on a VPS that costs much less, and that you control?
  - rhizome 9y ago
    Nobody cultivates sysadmin skills anymore. Jenkins and buildfoo.sh aren't turnkey.
    - SomeCallMeTim 9y ago
      Jenkins is also a bloody pain in the ass. Just saying.
coding123 9y ago
If npm or even a single popular npm package is compromised, almost every website will be compromised.
- lettucecarrot 9y ago
  While the possibility of these 8 sites getting hacked I think is low, and then targeting CircleCI, the NPM issue is a huge issue that has happened. Isn't yarn addressing this however?
ivanfon 9y ago
Does anyone know if Travis CI does something like this.
- matthewcford 9y ago
  pretty sure they also use pusher
richardknop 9y ago
Wouldn't similar vulnerability be present in most SAAS platforms? All of them include tons of analytics scripts on their website. If you are a logged in user, your session presumably includes a token (encrypted cookie?) to use the SAAS API. Therefor in theory other scripts loaded on the page could grab the token and make authenticated API calls as well? Although I guess this is mitigated by verifying script integrity when loading scripts from CDN (e.g. integrity attribute of script tag)?
- kevinburke 9y ago
  There are a few approaches, but yes, one thing I hope to do by publishing this is to draw attention to the problem of third party Javacript scripts running in a privileged environment and not on e.g the marketing page.
Posibyte 9y ago
> Why don't you include a POC? I have one that demonstrates this attack, but I don't want to show script kiddies how. If you can't figure out how to construct it by reading the above description and the network traffic, you don't deserve to know how. I feel this is a bad way to approach it and a bad attitude in general. Showing people how this can be exploited (after responsible disclosure) is a great opportunity to spread awareness of potentially their own issues in the same space and create points of discussion. If a user feels compelled to do so, they can seek out the info. But to show disdain to the reader for their lack of knowledge comes off as unprofessional in spirit.
- kevinburke 9y ago
  I disagree, and we'll just have to leave it at that. Anyone who knows how to construct an AJAX request and look at the Network tab can figure this out.
  - lloeki 9y ago
    While I understand the point you made in that last sentence, the very last part of it sounded needlessly loaded even though I can easily figure such things out. Replacing "you don't deserve to know how" by, say, "you're not the target audience" or anything to that effect goes a long way to not detracting from getting your real point across. BTW this last easily misunderstood sentence sits just above your "I'm available for hire" link, which is not exactly painting you in a bright light for recruitment.
    - rhizome 9y ago
      You're defending a made-up (mis)interpretation.
      - Posibyte 9y ago
        Saying "You don't deserve to know how" is not a friendly and constructive message to send to your audience or potential employers. There's no mis-interpretation here, it's in bad spirit.
danpalmer 9y ago
I feel calling all 8 of these companies "analytics" is a little unfair. Pusher is a SaaS product for websockets and push notifications. I think it's reasonable that a company like CircleCI might outsource this instead of maintaining the infrastructure for it in-house. Launch Darkly is a feature flagging service. While I feel the value is much less than Pusher (and I'd be inclined to just build an in-house solution), I think it's still a relatively reasonable tool to use to create a better user experience. Intercom when used for customer communication is also something that I feel I'd want as a user, although this could perhaps only be loaded when requested by the user.
- tptacek 9y ago
  Why does this matter? Most 3rd party services are "reasonable" in some setting. When you build a highly-sensitive application, the bar for reasonableness changes.
  - 45h34jh53k4j 9y ago
    This is a world where internet banking and healthcare, arguably the canonical 'highly-sensitive applications', are using 3rd party js services like this. I think we need to redefine 'reasonable' as the industry clearly isn't acting reasonably (IMHO).
    - xorcist 9y ago
      There is no reason to accept that your bank runs javascript from third parties. Complain and use only in an environment where you can block them. While it may seem futile to report these transgressions to big institutions, the fact that your complaints are logged helps those fighting these stupidities from the inside.
    - avip 9y ago
      The overwhelming majority of online financial services would self-host everything. It's not strictly a PCI-DSS compliance requirement, but it is common sense and industry standard.
motdiem 9y ago
I think these kind of issues will become more and more important, both from a security standpoint, and from a privacy standpoint as well. GPDR might apply to api access from 3rd party tools (not just analytics btw) - it’ll be up to them to prove they don’t, and I’m curious to see how it’ll be regulated. I’ve tried to explore the privacy aspects here [1] (like many, we completely separated libraries for the public site and libraries for the apps) I think we need a framework when thinking about which companies to pick when adding 3rd party features to our products - and probably a system to trust such 3rd party with their data processing. I’d be interested to know how companies make these decisions today... [1] https://blog.getkumbu.com/posts/building-a-privacy-respectful-startup https://blog.getkumbu.com/posts/building-a-privacy-respectfu...
raesene6 9y ago
Loading third party JS is increasingly common for a lot of sites, and I tend to raise it when doing security reviews, for this reason, you're trusting the security of those 3rd parties. There are some defenses that can be put in place. The first one is kind of awkward in many cases which is to host the JS on your own domain. There's still the risk of course that it will go off and get additional code from the 3rd party source to execute, but that can be reviewed for. The other option is to use sub-resource integrity (https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...) to ensure that only scripts you've reviewed are used. Of course you need then to make sure you're notified before the 3rd party makes changes that would break the signature.
- rhizome 9y ago
  Loading third party JS is increasingly common for a lot of sites, and I tend to raise it when doing security reviews What kind of pushback do you get and how do you handle it?
  - raesene6 9y ago
    To be honest I'm a external security assessor/pentester and I've not had much pushback from clients on this. That said I don't always get visibility of whether they implement our recommendations or not :) To me, it's not really a debatable point that loading JS from a source you don't control implies trust in that source and therefore a risk that if they are compromised it affects your site. Whether that risk is ok for a business depends on a number of factors like :- - How trustworthy are the sources they're loading from? - What reviews have they completed on the security of those sources? - Do they have contracts in place with those sources that cover the requirement for security?
sbr464 9y ago
I noticed that Galaxy - Meteor Development Group’s hosting solution for Meteor apps does similar. I checked the console and saw multiple analytics, even some that seem to monitor your visual usage/screenshots etc, which would seemingly capture screens that have portions of environment variables / settings etc. So basically potentially a lower level marketing employee who has an infected laptop and is reviewing UX patterns etc could expose sensitive data on their MacBook Air that could potentially be infected. They might not expose that data. But I had the same thoughts as this article points out.
Sir_Cmpwn 9y ago
The web is a disaster. Why do people insist on adding this bullshit to their pages?
- richardknop 9y ago
  Marketing and sales people usually ask to have tons of third party scripts loaded everywhere and devs don't have power to turn them down. They need analytics for their work. It's a bad idea from security standpoint for sure.
  - soared 9y ago
    Yeah, management and marketing need web analytics. If you don't want 3rd party scripts you'd have to, what, roll your own analytics software? There is some open source analytics tools you could self host, but they're garbage compared to google/adobe/etc.
    - kevin_thibedeau 9y ago
      > Yeah, management and marketing ... should get off their duff and analyze their own internal server logs.
      - flukus 9y ago
        That's when you discover what they really want it flashy graphs and charts, they don't actually care about the data.
      - manigandham 9y ago
        Server logs dont have much useful information, especially for SaaS interfaces. You need to have the events from the actual UI tracked, which requires a lot more work and is the reason why these analytics tools are used.
epanastasi 9y ago
Thinking about mitigation here... It appears that some of the included scripts have crossorigin="anonymous" script tags. Wouldn't this prevent authenticated access to to the circleCI domain, aka preventing the creation of api tokens or access to the API using the logged in browser context for any script loaded off the circleci domain? Also, not that they do so, but would Access-Control-Allow-Origin set to something other than * prevent 3rd party requests to the API for scripts loaded from 3rd party domains. Also curious if anyone has written a JS library that patches XMLHttpRequest.prototype to audit exfiltration of data in the DOM.
- noway421 9y ago
  usually blockers like ghostly and ublock/adblock are doing that, but modifying xhr in runtime is interesting idea!
Animats 9y ago
Google's policies are a big part of this problem. Ad code and tracking code ought to be in an iframe, where the code can't snoop on the surrounding page context. Google insists that their ad code must not be in an iframe. That gives the lesser players an opening to insist that they, too, should't be constrained. A few big site operators should push back against Google on this. The New York Times and CBS, for example. They use Google Publisher Tags, but no Google ads, so they don't really need Google on their pages.
- deleted 9y ago
  [deleted]
- icebraining 9y ago
  I'm getting securepubads.g.doubleclick.net being loaded from the NYTimes homepage.
  - Animats 9y ago
    Ah. I had Google Publisher Tags blocked, so loading didn't get far enough let DoubleClick try to load.
- manigandham 9y ago
  Most ad code does run in iframes and the industry standard is to use "friendly iframes" which offer security but also access to the parent window. And even if you use iframes, something has to create that iframe initially and publisher devs are not going to do that manually, which is why there are top level script tags. All tag managers and adservers today default to using iframes for 3rd party tags.
  - Animats 9y ago
    Google AdSense FAQ: Q: Is it violating program policy if I place ads on iframe webpages in my software? A: Yes, it does violate our policies. Firstly, you’re not allowed to place ads in a frame within another page. Exceptions to our policies are permitted only with authorization from Google for the valid use of iframes.[1] [1] https://support.google.com/adsense/answer/3394713?hl=en https://support.google.com/adsense/answer/3394713?hl=en
grandalf 9y ago
Also, Github and Bitbucket should allow much more granular ACLs such as single repo, single repo shallow clone only. Ideally they would also support some form of steganographic tagging of repo contents as well, in case of a breach.
z00b 9y ago
Rob Zuber, CTO of CircleCI here. We take security very seriously and are taking a deep look at the issues Kevin raised. We'll save additional commenting until we have gathered more information. In the meantime, our security policy and steps for reporting issues are here: https://circleci.com/security/ https://circleci.com/security/ and we'd like to ask the community to please use our outlined methods for reporting potential security issues so we can keep CircleCI as safe as possible for everyone.
- caust1c 9y ago
  Who's Ryan?
  - z00b 9y ago
    Welp. Wanted to write quickly and made a stupid mistake. Sorry about that Kevin!
- kevinburke 9y ago
  OP here. Thanks for responding promptly. To be clear, letting third party JS run in a trusted environment like a dashboard is an industry wide problem. If we assume CircleCI is the only bad actor we're kind of missing the point of the exercise. CircleCI is a notable example, due to the fact it hosts source code and secrets for so many different companies and loads so many third party scripts in its dashboard context. But they're not unique in this regard. I hope everyone reading this is reconsidering the scripts that have access to their dashboard and pushing for changes at their company. If CircleCI was immediately vulnerable to injection via the scripts above I would have used the private disclosure route and I encourage others to as well. But I don't know that there is a "vulnerability" here so much as a discussion that we need to have about what and how much third party code we let run in a trusted context. I wrote a little more in the thread below, https://news.ycombinator.com/item?id=15442988 https://news.ycombinator.com/item?id=15442988.
  - z00b 9y ago
    Thanks Kevin. We really do appreciate the discussion and, as you would imagine, are rethinking some of our approaches in line with the issues you've raised.
alpb 9y ago
I'm trying to understand why "CircleCI browser context has full access to the CircleCI API, which is hosted on the same domain". Doesn't API have seperate authentication credentials (API key/tokens) than the web UI (i.e. cookies)? I understand these loaded 3rd party scripts can scrape what's rendered on the UI, but making calls to the API??? I wonder how that's possible in CircleCI case.
- avitzurel 9y ago
  That's the first thing that triggered me as well. The only thing that 3rd party JS will have access to is everything on the page. Now, the things on the page are sensitive (the secrets from the env variables are dumped in the UI). Secrets in the secrets page are not exposed at any time, even when you go to edit. Only in the build screen, it's exposed to the output. Those 3rd party libs DO NOT have access to your source code at any time. The only time they will have access to your code is if they gain SSH access to the machines that are running the build. And that's not trivial. Even if your public key is exposed, they don't have access to checkout github with that. So yeah, it's 3rd party libs in a "private" context but not more than that IMHO.
  - detaro 9y ago
    From searching for discussions around CircleCI API and cookies it seems like at least some API endpoints might accept session cookies for authentication. I guess we'll have to wait for a CircleCI statement to address this in detail (or someone testing it).
nathan_f77 9y ago
No one has mentioned this, but I'm really curious about the script from Quora. Why is Circle CI loading JS from Quora? Or is that a browser extension that the OP forgot about?
- kevinburke 9y ago
  I don't run browser extensions
- efrafa 9y ago
  Conversion pixel
deleted 9y ago
[deleted]
jeffnappi 9y ago
As someone who has been a web developer since the 90's, this topic brings me back to ~2005 when Google Analytics launched. I thought it was absolutely insane that people would just stick someone else's JavaScript in their pages (and especially on e-commerce sites!). Today it's perfectly normal. I don't have exact numbers but at least 50% of all sites pull in third party code. It really is bizarre to me that this became the norm.

508 points by tomwas54 9y ago

To me, this seems like a vulnerability report. The following line is what really made it stand out for me as being a vulnerability report: >Why don't you include a POC? I have one that demonstrates this attack, but I don't want to show script kiddies how. That leads me to this question: Did the author reach out to CircleCI to report this issue, before publishing their blog post?

kevinburke 9y ago
No, I didn't. If one of the JavaScript sources was immediately compromisable, I would have.
- kierenj 9y ago
  Why post a big public dig? If there's a problem, make the world a better place by helping to solve it maybe? Just having a bad day?
  - kevinburke 9y ago
    1) they know exactly what they are doing; 2) arbitrary 3rd party JS running in privileged contexts (dashboards) is a wider problem I would like to draw attention to; 3) It's not immediately exploitable; 4) It takes five seconds to understand what the problem is; 5) I did not want to wait 90 days to be told "we are not going to fix this;" 6) my history of reporting problems (not security related) to the company in question is that they are not very responsive to fixing them.
    - wersow 9y ago
      Not all of the companies you listed are analytics company, just FYI.
    - kierenj 9y ago
      Umm, either its a problem and worth reporting responsibly, or not a problem and so not worth a hit piece on your blog?
      - tombrossman 9y ago
        This comment highlights the problem with calling disclosure 'responsible', because the word is used subjectively. What you as a commenter, or a company (usually prefers to quietly fix this), or the wider user base (usually prefers a 'heads up' immediately), all have different opinions about what they see as responsible. Also, it doesn't seem fair to dismiss this disclosure as a 'hit piece' unless it is factually incorrect.
      - icebraining 9y ago
        It's only worth reporting "responsibly" if waiting might prevent users from being hurt. The way that concept has been manipulated into implying that the companies are owed an early warning is crap. The author doesn't owe CircleCI anything.
        kevinburke 9y ago
        Precisely
    - j_s 9y ago
      I was with you until I saw this on your consulting pitch: Write. I have written at least ten posts that made the front page of Hacker News / Programming Reddit (nb - Those aren't the best proxies for quality or popularity, but they are well known proxies). I can help kickstart your company's engineering blog, or work with your team on story/content ideas. Now I'm not sure how much making the front page motivates your writing process. It is an interesting topic but I feel it would be a stronger case to demonstrate the issue across products from multiple vendors, though less incendiary/front-page-y. PS. Props for not being the one to submit this particular article!
  - idunno246 9y ago
    Because it's a rampant problem not limited to them, and not obvious, and nobody seems to care? This is a good example for getting the target market to understand the dangers of just dropping in the latest js tool that marketing wants. I used to be in ad tech, it always baffled me that we had first party js on every single customer page. Conceivable if we were breached, someone could inject code to read every login token or cc# or anything and send it to their servers.

sytse 9y ago

At GitLab we separated the infrastructure between about.gitlab.com (our marketing site, lots of third party javascript) and gitlab.com (our application, zero third party javascript). We recently deprecated the Piwik instance we ran in house for gitlab.com because it slowed the page load time. Please let me know if there is anything we can do better, these things are complex.

pg_is_a_butt 9y ago
why are you still using the same top level domain? sure there are XSS protections on subdomains, but it's still 1 exploit away from the top level domain. google uses googlehosted.com for stuff that might include 3rd party stuff.
BoorishBears 9y ago
I feel this is off topic.
- davnicwil 9y ago
  I'm not sure if you're aware, but GitLab has integrated CI available on gitlab.com, which is a like-for-like alternative to CircleCI - so in that sense this comment is very much on topic, in my opinion.
  - mr337 9y ago
    I agree too. When reading about CircleCI my first thought is, "wonder why my CI is doing?". Now gotta check the requests to validate their claim.
  - BoorishBears 9y ago
    I'm aware, to me it feels like a plug. I've seen a nearly identical comment on Gitlab from a member of their team on other occasions
    - BoorishBears 9y ago

deleted 9y ago

[deleted]

koolba 9y ago

> Send an email any time an API access token is created. Add a setting to allow org-wide disabling of API token creation. I wish more apps supported this. Also on my list is a "Don't ever let me disable 2FA" setting. I'm more worried about malicoius resets than I am about ever losing my 2FA device and backup codes. > Add an option to delete old logs. If you have ever dumped env vars to the log file, an attacker can export these. I surprised that secrets make it into the logs at all. I would have expected them to filter anything that's in an env secret from the log output. Pretty sure Travis does this. > Enable subresource integrity, or serve JS from each of these companies from the CircleCI domain. That's not much of an option for this type of thing as the third parties would then have a PITA time dealing with upgrades (so they wouldn't support it). Using <script src="hxxps://tracker.example.com/path/to/script.js"></script> (rather than a fixed version) allows for live upgrades as they're in control of the resource that is loaded. And you can't load the script from your domain as at the end of the day it's got to get instructions and upload data to the third party. At best you'd be loading a shim that does the same thing as the script tag.

orginal__idear 9y ago
> "Don't ever let me disable 2FA" setting. I like this idea. I'd only want to use this though if my Google 2FA backup codes are backed up via Authy or a similar app. I don't trust codes I've printed out and have always lost.
- rhizome 9y ago
  Have you considered getting a safe-deposit box?
- dabernathy89 9y ago
  I tend to just stick all my 2FA backup codes into "secure notes" in LastPass.
  - ihattendorf 9y ago
    That sort of defeats the purpose of 2FA though. Now if your LastPass database is compromised, they have access to your 2FA codes as well.

jbg_ 9y ago

uMatrix.

fluxsauce 9y ago
That treats the symptom, not the problem.
- beagle3 9y ago
  uMatrix is defense in depth. They could solve the problem today, and tomorrow a new CircleCI programmer/marketer/CEO would decide that, in fact, they do care more about analytics than about user's secrecy, and revert this change. By installing uMatrix and not approving these things, you are (a) reducing the probability that a mistake, misaligned incentives, etc, would harm you, and (b) get a notification that there's something you wish to block. uMatrix IS part of the real solution. The other part is avoiding providers whose security standards are not up to yours.
  - 45h34jh53k4j 9y ago
    Its a real solution, for YOU personally. I use and love umatrix (whitelisting ALL js/content by hand on all domains). We need to consider the good of the many, and client side solutions like umatrix are not a solution for the many.
- dom0 9y ago
  Good security does generally not rely on a single measure. The service provider not doing this in the first place is good, but the user (also) having a second line of defense is objectively better, although it may incentivize poor behaviour of SPs.

kevan 9y ago

At my last company we had a similar issue with third party scripts on credit application forms. Analytics is critical for ecommerce sites but it's way too easy to accidentally log sensitive data across dozens of third party services.

bm1362 9y ago
There are self-hosted analytics (like Piwik) that you could self-host for sensitive data to solve this generally. I'd like to think that conversion of credit applications isn't something a marketer is trying to gamify though, that's probably wishful thinking.
- kevan 9y ago
  >I'd like to think that conversion of credit applications isn't something a marketer is trying to gamify though If it's in the overall sales conversion funnel it will be optimized. Credit application flow was scrutinized just as much as adding to cart and checking out.

nexfitter 9y ago

I use CircleCI because it allows 1 free private repo. Personally though I do not like Circle too much, it has a lot of cool features but the UI loads so slowly because they have so much stuff going on. They use the UI skeletons technique to make the load time appear to be faster but it is still very apparent how slow the load time is. I have hit numerous bugs with their website as well where stuff doesn't load, builds kick off into infinity, when I transferred a repo everything broke, ack! Furthermore, I am frustrated with their pricing, they go from free to $50/mo for the next upgrade tier, that seems like a crazy jump to me. I would gladly pay $10/mo or so for another container or 2x parallelization. The issue is being a single developer I rarely would save any time and the $50/mo is just too steep. Finally, when I tried to build a justification case for my manager to pay the $50/mo, I wanted to use data from their CircleCI Insights which shows how long your builds are queued on average and some other important data points. But you cannot access these insights from a free account. Seems like that info should be available and prominent to help people understand the cost-savings they might receive by upgrading. I emailed support and asked for a one-time data point for that statistic to build the case as I was considering upgrading and they said sorry, nothing we can do for you. Is their goal to make money and have happy customers? If so they aren't doing a great job of it. Overall a lot of frustrations with the platform and this just adds more fuel to the fire.

daxorid 9y ago
Serious question: why CircleCI over, say, Jenkins or custom_build_script.sh running on a VPS that costs much less, and that you control?
- rhizome 9y ago
  Nobody cultivates sysadmin skills anymore. Jenkins and buildfoo.sh aren't turnkey.
  - SomeCallMeTim 9y ago
    Jenkins is also a bloody pain in the ass. Just saying.

coding123 9y ago

If npm or even a single popular npm package is compromised, almost every website will be compromised.

lettucecarrot 9y ago
While the possibility of these 8 sites getting hacked I think is low, and then targeting CircleCI, the NPM issue is a huge issue that has happened. Isn't yarn addressing this however?

ivanfon 9y ago

Does anyone know if Travis CI does something like this.

matthewcford 9y ago
pretty sure they also use pusher

richardknop 9y ago

Wouldn't similar vulnerability be present in most SAAS platforms? All of them include tons of analytics scripts on their website. If you are a logged in user, your session presumably includes a token (encrypted cookie?) to use the SAAS API. Therefor in theory other scripts loaded on the page could grab the token and make authenticated API calls as well? Although I guess this is mitigated by verifying script integrity when loading scripts from CDN (e.g. integrity attribute of script tag)?

kevinburke 9y ago
There are a few approaches, but yes, one thing I hope to do by publishing this is to draw attention to the problem of third party Javacript scripts running in a privileged environment and not on e.g the marketing page.

Posibyte 9y ago

> Why don't you include a POC? I have one that demonstrates this attack, but I don't want to show script kiddies how. If you can't figure out how to construct it by reading the above description and the network traffic, you don't deserve to know how. I feel this is a bad way to approach it and a bad attitude in general. Showing people how this can be exploited (after responsible disclosure) is a great opportunity to spread awareness of potentially their own issues in the same space and create points of discussion. If a user feels compelled to do so, they can seek out the info. But to show disdain to the reader for their lack of knowledge comes off as unprofessional in spirit.

kevinburke 9y ago
I disagree, and we'll just have to leave it at that. Anyone who knows how to construct an AJAX request and look at the Network tab can figure this out.
- lloeki 9y ago
  While I understand the point you made in that last sentence, the very last part of it sounded needlessly loaded even though I can easily figure such things out. Replacing "you don't deserve to know how" by, say, "you're not the target audience" or anything to that effect goes a long way to not detracting from getting your real point across. BTW this last easily misunderstood sentence sits just above your "I'm available for hire" link, which is not exactly painting you in a bright light for recruitment.
  - rhizome 9y ago
    You're defending a made-up (mis)interpretation.
    - Posibyte 9y ago
      Saying "You don't deserve to know how" is not a friendly and constructive message to send to your audience or potential employers. There's no mis-interpretation here, it's in bad spirit.

danpalmer 9y ago

I feel calling all 8 of these companies "analytics" is a little unfair. Pusher is a SaaS product for websockets and push notifications. I think it's reasonable that a company like CircleCI might outsource this instead of maintaining the infrastructure for it in-house. Launch Darkly is a feature flagging service. While I feel the value is much less than Pusher (and I'd be inclined to just build an in-house solution), I think it's still a relatively reasonable tool to use to create a better user experience. Intercom when used for customer communication is also something that I feel I'd want as a user, although this could perhaps only be loaded when requested by the user.

tptacek 9y ago
Why does this matter? Most 3rd party services are "reasonable" in some setting. When you build a highly-sensitive application, the bar for reasonableness changes.
- 45h34jh53k4j 9y ago
  This is a world where internet banking and healthcare, arguably the canonical 'highly-sensitive applications', are using 3rd party js services like this. I think we need to redefine 'reasonable' as the industry clearly isn't acting reasonably (IMHO).
  - xorcist 9y ago
    There is no reason to accept that your bank runs javascript from third parties. Complain and use only in an environment where you can block them. While it may seem futile to report these transgressions to big institutions, the fact that your complaints are logged helps those fighting these stupidities from the inside.
  - avip 9y ago
    The overwhelming majority of online financial services would self-host everything. It's not strictly a PCI-DSS compliance requirement, but it is common sense and industry standard.

motdiem 9y ago

I think these kind of issues will become more and more important, both from a security standpoint, and from a privacy standpoint as well. GPDR might apply to api access from 3rd party tools (not just analytics btw) - it’ll be up to them to prove they don’t, and I’m curious to see how it’ll be regulated. I’ve tried to explore the privacy aspects here [1] (like many, we completely separated libraries for the public site and libraries for the apps) I think we need a framework when thinking about which companies to pick when adding 3rd party features to our products - and probably a system to trust such 3rd party with their data processing. I’d be interested to know how companies make these decisions today... [1] https://blog.getkumbu.com/posts/building-a-privacy-respectful-startup https://blog.getkumbu.com/posts/building-a-privacy-respectfu...

raesene6 9y ago

Loading third party JS is increasingly common for a lot of sites, and I tend to raise it when doing security reviews, for this reason, you're trusting the security of those 3rd parties. There are some defenses that can be put in place. The first one is kind of awkward in many cases which is to host the JS on your own domain. There's still the risk of course that it will go off and get additional code from the 3rd party source to execute, but that can be reviewed for. The other option is to use sub-resource integrity (https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...) to ensure that only scripts you've reviewed are used. Of course you need then to make sure you're notified before the 3rd party makes changes that would break the signature.

rhizome 9y ago
Loading third party JS is increasingly common for a lot of sites, and I tend to raise it when doing security reviews What kind of pushback do you get and how do you handle it?
- raesene6 9y ago
  To be honest I'm a external security assessor/pentester and I've not had much pushback from clients on this. That said I don't always get visibility of whether they implement our recommendations or not :) To me, it's not really a debatable point that loading JS from a source you don't control implies trust in that source and therefore a risk that if they are compromised it affects your site. Whether that risk is ok for a business depends on a number of factors like :- - How trustworthy are the sources they're loading from? - What reviews have they completed on the security of those sources? - Do they have contracts in place with those sources that cover the requirement for security?

sbr464 9y ago

I noticed that Galaxy - Meteor Development Group’s hosting solution for Meteor apps does similar. I checked the console and saw multiple analytics, even some that seem to monitor your visual usage/screenshots etc, which would seemingly capture screens that have portions of environment variables / settings etc. So basically potentially a lower level marketing employee who has an infected laptop and is reviewing UX patterns etc could expose sensitive data on their MacBook Air that could potentially be infected. They might not expose that data. But I had the same thoughts as this article points out.

Sir_Cmpwn 9y ago

The web is a disaster. Why do people insist on adding this bullshit to their pages?

richardknop 9y ago
Marketing and sales people usually ask to have tons of third party scripts loaded everywhere and devs don't have power to turn them down. They need analytics for their work. It's a bad idea from security standpoint for sure.
- soared 9y ago
  Yeah, management and marketing need web analytics. If you don't want 3rd party scripts you'd have to, what, roll your own analytics software? There is some open source analytics tools you could self host, but they're garbage compared to google/adobe/etc.
  - kevin_thibedeau 9y ago
    > Yeah, management and marketing ... should get off their duff and analyze their own internal server logs.
    - flukus 9y ago
      That's when you discover what they really want it flashy graphs and charts, they don't actually care about the data.
    - manigandham 9y ago
      Server logs dont have much useful information, especially for SaaS interfaces. You need to have the events from the actual UI tracked, which requires a lot more work and is the reason why these analytics tools are used.

epanastasi 9y ago

Thinking about mitigation here... It appears that some of the included scripts have crossorigin="anonymous" script tags. Wouldn't this prevent authenticated access to to the circleCI domain, aka preventing the creation of api tokens or access to the API using the logged in browser context for any script loaded off the circleci domain? Also, not that they do so, but would Access-Control-Allow-Origin set to something other than * prevent 3rd party requests to the API for scripts loaded from 3rd party domains. Also curious if anyone has written a JS library that patches XMLHttpRequest.prototype to audit exfiltration of data in the DOM.

noway421 9y ago
usually blockers like ghostly and ublock/adblock are doing that, but modifying xhr in runtime is interesting idea!

Animats 9y ago

Google's policies are a big part of this problem. Ad code and tracking code ought to be in an iframe, where the code can't snoop on the surrounding page context. Google insists that their ad code must not be in an iframe. That gives the lesser players an opening to insist that they, too, should't be constrained. A few big site operators should push back against Google on this. The New York Times and CBS, for example. They use Google Publisher Tags, but no Google ads, so they don't really need Google on their pages.

deleted 9y ago
[deleted]
icebraining 9y ago
I'm getting securepubads.g.doubleclick.net being loaded from the NYTimes homepage.
- Animats 9y ago
  Ah. I had Google Publisher Tags blocked, so loading didn't get far enough let DoubleClick try to load.
manigandham 9y ago
Most ad code does run in iframes and the industry standard is to use "friendly iframes" which offer security but also access to the parent window. And even if you use iframes, something has to create that iframe initially and publisher devs are not going to do that manually, which is why there are top level script tags. All tag managers and adservers today default to using iframes for 3rd party tags.
- Animats 9y ago
  Google AdSense FAQ: Q: Is it violating program policy if I place ads on iframe webpages in my software? A: Yes, it does violate our policies. Firstly, you’re not allowed to place ads in a frame within another page. Exceptions to our policies are permitted only with authorization from Google for the valid use of iframes.[1] [1] https://support.google.com/adsense/answer/3394713?hl=en https://support.google.com/adsense/answer/3394713?hl=en

grandalf 9y ago

Also, Github and Bitbucket should allow much more granular ACLs such as single repo, single repo shallow clone only. Ideally they would also support some form of steganographic tagging of repo contents as well, in case of a breach.

z00b 9y ago

Rob Zuber, CTO of CircleCI here. We take security very seriously and are taking a deep look at the issues Kevin raised. We'll save additional commenting until we have gathered more information. In the meantime, our security policy and steps for reporting issues are here: https://circleci.com/security/ https://circleci.com/security/ and we'd like to ask the community to please use our outlined methods for reporting potential security issues so we can keep CircleCI as safe as possible for everyone.

caust1c 9y ago
Who's Ryan?
- z00b 9y ago
  Welp. Wanted to write quickly and made a stupid mistake. Sorry about that Kevin!
kevinburke 9y ago
OP here. Thanks for responding promptly. To be clear, letting third party JS run in a trusted environment like a dashboard is an industry wide problem. If we assume CircleCI is the only bad actor we're kind of missing the point of the exercise. CircleCI is a notable example, due to the fact it hosts source code and secrets for so many different companies and loads so many third party scripts in its dashboard context. But they're not unique in this regard. I hope everyone reading this is reconsidering the scripts that have access to their dashboard and pushing for changes at their company. If CircleCI was immediately vulnerable to injection via the scripts above I would have used the private disclosure route and I encourage others to as well. But I don't know that there is a "vulnerability" here so much as a discussion that we need to have about what and how much third party code we let run in a trusted context. I wrote a little more in the thread below, https://news.ycombinator.com/item?id=15442988 https://news.ycombinator.com/item?id=15442988.
- z00b 9y ago
  Thanks Kevin. We really do appreciate the discussion and, as you would imagine, are rethinking some of our approaches in line with the issues you've raised.

alpb 9y ago

I'm trying to understand why "CircleCI browser context has full access to the CircleCI API, which is hosted on the same domain". Doesn't API have seperate authentication credentials (API key/tokens) than the web UI (i.e. cookies)? I understand these loaded 3rd party scripts can scrape what's rendered on the UI, but making calls to the API??? I wonder how that's possible in CircleCI case.

avitzurel 9y ago
That's the first thing that triggered me as well. The only thing that 3rd party JS will have access to is everything on the page. Now, the things on the page are sensitive (the secrets from the env variables are dumped in the UI). Secrets in the secrets page are not exposed at any time, even when you go to edit. Only in the build screen, it's exposed to the output. Those 3rd party libs DO NOT have access to your source code at any time. The only time they will have access to your code is if they gain SSH access to the machines that are running the build. And that's not trivial. Even if your public key is exposed, they don't have access to checkout github with that. So yeah, it's 3rd party libs in a "private" context but not more than that IMHO.
- detaro 9y ago
  From searching for discussions around CircleCI API and cookies it seems like at least some API endpoints might accept session cookies for authentication. I guess we'll have to wait for a CircleCI statement to address this in detail (or someone testing it).

nathan_f77 9y ago

No one has mentioned this, but I'm really curious about the script from Quora. Why is Circle CI loading JS from Quora? Or is that a browser extension that the OP forgot about?

kevinburke 9y ago
I don't run browser extensions
efrafa 9y ago
Conversion pixel

deleted 9y ago

[deleted]

jeffnappi 9y ago

As someone who has been a web developer since the 90's, this topic brings me back to ~2005 when Google Analytics launched. I thought it was absolutely insane that people would just stick someone else's JavaScript in their pages (and especially on e-commerce sites!). Today it's perfectly normal. I don't have exact numbers but at least 50% of all sites pull in third party code. It really is bizarre to me that this became the norm.