5 ms·

I mean, it was preventable, you just had to not use the massive single point of failure that is a cloud CI app. But everything is "so much easier" and "so much

I mean, it was preventable, you just had to not use the massive single point of failure that is a cloud CI app. But everything is "so much easier" and "so much cheaper" with the cloud I guess!

zdragnar 4y ago
Having experienced cloud CI on several platforms as well as in-house installs of both Travis and Jenkins, I think the ratio of downtime between cloud and local makes the cloud a far, far better pick. That said, I am currently very glad I'm not running anything on circle ci...
- BarryMilo 4y ago
  Hindsight is 20/20 for sure, but for this use case confidentiality seems more important than availability.It is something of an impossible choice, but I've come to think that every service big or small gets hacked sooner or later. I'm thinking being a smaller target is better in that case.
  - chii 4y ago
    But a smaller target might mean less funds for security, as it's basically a constant overhead.
    - Xylakant 4y ago
      I’m fairly confident that in the same breach scenario (laptop with admin permissions taken over), most small orgs would fare worse. The CI system would likely be behind a VPN, but the laptop would likely have those credentials, so it would not stop an attacker. A small, 20 person org has maybe 2 people assigned to ops, so monitoring and breach detection is likely worse. Now, a small org may be a less attractive target and some orgs can have top notch security people, but on average, the trade-off is likely not in favor of hosting your own.
- danw1979 4y ago
  I had the opposite experience with a large UK corp when moving from self-hosted Gitlab to gitlab.com. Runners were still self hosted, but if the thing controlling them is just giving 500s all day and you’ve no influence on fixing it, then your jobs aren’t being run and your developers are sitting somewhat idle. Github Actions has been better, but not perfect… but you can still fully self-host this if you think it’s worth it !
scarface74 4y ago
Yes because every company that is not using the cloud for CI/CD have much better security. I’ve seen internal hosted Jenkins servers give up the ghost more times than I care to mention.
- xiwenc 4y ago
  The difference, hopefully, is that private Jenkins installation is behind corporate network. So for attackers to reach it, either they need to breach the network first or it’s an internal employee. Personally i’m a big proponent of both cloud services and privately managed services. CI is one of those i think are better kept private due to its sensitivity.
  - scarface74 4y ago
    You really haven’t seen all of the reports of internal git repos getting breached?
  - sofixa 4y ago
    > So for attackers to reach it, either they need to breach the network first or it’s an internal employee So for instance an employee getting their laptop infected with malware, or through phishing, or through one of the many vulnerabilities discovered regularly on enterprise VPN software? There is a reason that many organisations are going away with VPNs and "corporate networks" all together - it gives a false sense of security that stuff behind it is protected by the VPN and leads to poor security practices inside like obsolete Jenkins installs. Disclaimer: I work at a company that sells Zero Trust as a concept and associated software, but I've had this opinion since before joining (I've seen enough of 'there's a VPN and then lots of apps/services/servers with very poor auth practices because it's behind the VPN, why bother?')
    - xiwenc 4y ago
      As someone from the security domain you should know there is not one single tool/service that will solve all your security challenges. I did not claim VPN will. Your example of employee laptop getting infected with malware should be remediated by proper device management. Which should include policies like forced updates, prevents software install, uses corporate firewall, up to date antivirus, etc. Since you mentioned you work for a company that sells zero trust, your response sounds like your solution would replace VPN. Don’t get me wrong, but feels like you are wearing the sales hat now. Yes, old fashion VPN appliances will be probably seize to exist. Replaced by modern equivalent like tailscale. Layered security is the answer.
kayodelycaon 4y ago
I think you’re over-estimating the amount of resources and expertise maintaining your own CI infrastructure requires. The places I typically work for don’t have the operations capacity necessary. Nor do the small development teams have anyone other than me familiar enough with system administration to set things up in a reasonable amount of time. A lot of junior developers have little to no Linux server experience. And I don’t really count following a step-by-step tutorial as experience. I have that experience and server problems can easily eat days of time the team can’t afford. :(
- borplk 4y ago
  I think you meant that they are under-estimating

I mean, it was preventable, you just had to not use the massive single point of failure that is a cloud CI app. But everything is "so much easier" and "so much cheaper" with the cloud I guess!

zdragnar 4y ago

Having experienced cloud CI on several platforms as well as in-house installs of both Travis and Jenkins, I think the ratio of downtime between cloud and local makes the cloud a far, far better pick. That said, I am currently very glad I'm not running anything on circle ci...

BarryMilo 4y ago
Hindsight is 20/20 for sure, but for this use case confidentiality seems more important than availability.It is something of an impossible choice, but I've come to think that every service big or small gets hacked sooner or later. I'm thinking being a smaller target is better in that case.
- chii 4y ago
  But a smaller target might mean less funds for security, as it's basically a constant overhead.
  - Xylakant 4y ago
    I’m fairly confident that in the same breach scenario (laptop with admin permissions taken over), most small orgs would fare worse. The CI system would likely be behind a VPN, but the laptop would likely have those credentials, so it would not stop an attacker. A small, 20 person org has maybe 2 people assigned to ops, so monitoring and breach detection is likely worse. Now, a small org may be a less attractive target and some orgs can have top notch security people, but on average, the trade-off is likely not in favor of hosting your own.
danw1979 4y ago
I had the opposite experience with a large UK corp when moving from self-hosted Gitlab to gitlab.com. Runners were still self hosted, but if the thing controlling them is just giving 500s all day and you’ve no influence on fixing it, then your jobs aren’t being run and your developers are sitting somewhat idle. Github Actions has been better, but not perfect… but you can still fully self-host this if you think it’s worth it !

scarface74 4y ago

Yes because every company that is not using the cloud for CI/CD have much better security. I’ve seen internal hosted Jenkins servers give up the ghost more times than I care to mention.

xiwenc 4y ago
The difference, hopefully, is that private Jenkins installation is behind corporate network. So for attackers to reach it, either they need to breach the network first or it’s an internal employee. Personally i’m a big proponent of both cloud services and privately managed services. CI is one of those i think are better kept private due to its sensitivity.
- scarface74 4y ago
  You really haven’t seen all of the reports of internal git repos getting breached?
- sofixa 4y ago
  > So for attackers to reach it, either they need to breach the network first or it’s an internal employee So for instance an employee getting their laptop infected with malware, or through phishing, or through one of the many vulnerabilities discovered regularly on enterprise VPN software? There is a reason that many organisations are going away with VPNs and "corporate networks" all together - it gives a false sense of security that stuff behind it is protected by the VPN and leads to poor security practices inside like obsolete Jenkins installs. Disclaimer: I work at a company that sells Zero Trust as a concept and associated software, but I've had this opinion since before joining (I've seen enough of 'there's a VPN and then lots of apps/services/servers with very poor auth practices because it's behind the VPN, why bother?')
  - xiwenc 4y ago
    As someone from the security domain you should know there is not one single tool/service that will solve all your security challenges. I did not claim VPN will. Your example of employee laptop getting infected with malware should be remediated by proper device management. Which should include policies like forced updates, prevents software install, uses corporate firewall, up to date antivirus, etc. Since you mentioned you work for a company that sells zero trust, your response sounds like your solution would replace VPN. Don’t get me wrong, but feels like you are wearing the sales hat now. Yes, old fashion VPN appliances will be probably seize to exist. Replaced by modern equivalent like tailscale. Layered security is the answer.

kayodelycaon 4y ago

I think you’re over-estimating the amount of resources and expertise maintaining your own CI infrastructure requires. The places I typically work for don’t have the operations capacity necessary. Nor do the small development teams have anyone other than me familiar enough with system administration to set things up in a reasonable amount of time. A lot of junior developers have little to no Linux server experience. And I don’t really count following a step-by-step tutorial as experience. I have that experience and server problems can easily eat days of time the team can’t afford. :(

borplk 4y ago
I think you meant that they are under-estimating