6 ms·

When I decided to move away from self-hosted git ages ago and then from Jenkins to GA two years ago, reliability was a huge factor in my decision because Github

by pilif 4y ago

When I decided to move away from self-hosted git ages ago and then from Jenkins to GA two years ago, reliability was a huge factor in my decision because Github, I supposed, would be much better at keeping their infrastructure running than I am at keeping ours. Turns out the uptime of both our git server and even the Jenkins instance beat GitHub by far and while the former only cost a marginal amount of CPU time on infrastructure I was running anyways, GitHub is a noticeable expense for us. Of course it still saves me from the panic attacks every time I'm compelled to press the "Update now" button in Jenkins because either I do nothing and get my instance RCEd or I do press the button and who knows what plugin update will break which part of our setup, but while that was a constant fear in my mind, the amount of downtime caused by Jenkins plugin updates was zero whereas what GitHub is doing lately is way, way, way worse than zero. I'm starting to get frustrated and like I presume many other paying users, I think I'm at a point where I feel like we should get partial refunds of our subscription money given the very spotty uptime all year now.

iso1631 4y ago
> When I decided to move away from self-hosted git ages ago and then from Jenkins to GA two years ago, reliability was a huge factor in my decision because Github, I supposed, would be much better at keeping their infrastructure running than I am at keeping ours. Oh you sweet summer child
_joel 4y ago
PSA: Never expose Jenkins to the public internet, make sure it's via VPN. If you need webhooks, there are services for that which allow you to broker webhooks whilst calling in from the Jenkins side (i.e. not exposing ports). Even so if you do have to use native webhooks, at least lock it down to the upstream's IP range(s). Ideally have a dev jenkins to test all the things first before hitting upgrade on your prod instance and killing some plugins (hell even better if it's all IaaC and can just spin up a jenkins host per env, but ££$$$££/Time etc)
- NhanH 4y ago
  Nowadays, tailscale or cloudflare access + tunnel works amazing well for private service that you might need access on untrusted network. So the needs for keeping them up to can be delayed a lot more (of course, jenkins is a special case since it might be pulling and executing untrusted code, but I think that is something you need to care even without security issue specific to jenkins itself).
  - _joel 4y ago
    Yea, quite right, also use runners on ephemeral build targets as that can reduce some of the attack surface for running untrusted code, don't do it all in the box.
AtNightWeCode 4y ago
In general, don't use Jenkins. It is mediocre piece of software with an even worse ecosystem. It self-implodes every now and then. Big tech companies build a lot of custom stuff to get it to work correctly. Once that is done sure, it works. I am not a fan of Github actions but to host GIT and Jenkins is not really a serious option. Pick another CI platform in that case.
- blown_gasket 4y ago
  This comment makes a lot of assertions without any backing data. How is it mediocre? Is it because of the CVEs that have been released in the prior years? I recall GitLab also having quite a bad week of CVEs in February[1]. How is it a bad ecosystem? If this is about plugins in order to do things, I actually like this framework - it lets there be specific owners for portions of the open source development. Self-implodes? This seems like it would be tracked as a bug. I've encountered an instance where Jenkins wouldn't start due to a crypto issue but that was due to a bug and all I needed to do was install a patch. I think that using Jenkins can be a thought of a serious option if like anything else, you follow security protocols ie: don't allow public access, maintain RBAC standards, have a maintenance schedule. [1]https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/ https://about.gitlab.com/releases/2022/02/25/critical-securi...
  - AtNightWeCode 4y ago
    I have stolen access to production systems though Jenkins. Cause the scrambled pw is always sent to the Jenkins client. And then by default anybody could decode it from the console… They fixed it in 2.0 I believe. The problem with dependencies is that different plugins need different versions but there is no solution to that. Even more. You update Jenkins. Plugins break. Data gets corrupted. You try to align the plugins to some dependent version. It fails, cause most of the ecosystem is a hobby-project that never gets updated. Jenkins has absolutely nothing to do on the public Internet as most of these tools. There are so many CI/CD services. Most are very cheap. You really need to love yak shaving to pick Jenkins over something from the shelf. No major cloud offers Jenkins Saas. Wonder why…
fartcannon 4y ago
Also: "you grant to Microsoft a worldwide and royalty-free intellectual property license to use Your Content, for example, to make copies of, retain, transmit, reformat, display, and distribute via communication tools Your Content on the Services." https://www.microsoft.com/en-ca/servicesagreement/upcoming.aspx https://www.microsoft.com/en-ca/servicesagreement/upcoming.a...
- Pryde 4y ago
  Am I missing something, or is GitHub distinctly not listed in the Covered Services section of that services agreement?
  - fartcannon 4y ago
    Nope, looks like I missed that. My mistake. So what part of the github terms gives them permission to sell my IP back to me through copilot?
- mey 4y ago
  https://docs.github.com/en/site-policy/github-terms/github-terms-of-service#d-user-generated-content https://docs.github.com/en/site-policy/github-terms/github-t... GitHub has a separate policy...
  - fartcannon 4y ago
    Thank you, I missed that part. As I asked elsewhere, where is the part of these agreements that permits them to profit off my IP through copilot?
    - vel0city 4y ago
      If your code was in a private repo they supposedly didn't use it for training. I think it was also mentioned they tried to limit it to only repos where they could determine some kind of license they thought would permit such usage. If you're publishing your IP to GitHub in a public repo and using a widely permissive license, they didn't need an agreement to use that IP to train copilot.
tapoxi 4y ago
GitHub is running a complex, planet-scale product. I think w they've crossed the threshold where doing it yourself is more likely to be reliable for some use cases. We've been running GitLab on GKE for the past three years, no problems outside of initial migration pains.
sascha_sl 4y ago
We have open tickets about the SLA being broken for Q1 and Q2. They've been open for a while, and we're out of ways to escalate them (despite enterprise). And GitHub's SLA is not great to begin with: 10% of spend refund at 3 nines (99,9%) and 25% of spend at 2 nines (99%).

by pilif 4y ago

iso1631 4y ago

> When I decided to move away from self-hosted git ages ago and then from Jenkins to GA two years ago, reliability was a huge factor in my decision because Github, I supposed, would be much better at keeping their infrastructure running than I am at keeping ours. Oh you sweet summer child

_joel 4y ago

PSA: Never expose Jenkins to the public internet, make sure it's via VPN. If you need webhooks, there are services for that which allow you to broker webhooks whilst calling in from the Jenkins side (i.e. not exposing ports). Even so if you do have to use native webhooks, at least lock it down to the upstream's IP range(s). Ideally have a dev jenkins to test all the things first before hitting upgrade on your prod instance and killing some plugins (hell even better if it's all IaaC and can just spin up a jenkins host per env, but ££$$$££/Time etc)

NhanH 4y ago
Nowadays, tailscale or cloudflare access + tunnel works amazing well for private service that you might need access on untrusted network. So the needs for keeping them up to can be delayed a lot more (of course, jenkins is a special case since it might be pulling and executing untrusted code, but I think that is something you need to care even without security issue specific to jenkins itself).
- _joel 4y ago
  Yea, quite right, also use runners on ephemeral build targets as that can reduce some of the attack surface for running untrusted code, don't do it all in the box.

AtNightWeCode 4y ago

In general, don't use Jenkins. It is mediocre piece of software with an even worse ecosystem. It self-implodes every now and then. Big tech companies build a lot of custom stuff to get it to work correctly. Once that is done sure, it works. I am not a fan of Github actions but to host GIT and Jenkins is not really a serious option. Pick another CI platform in that case.

blown_gasket 4y ago
This comment makes a lot of assertions without any backing data. How is it mediocre? Is it because of the CVEs that have been released in the prior years? I recall GitLab also having quite a bad week of CVEs in February[1]. How is it a bad ecosystem? If this is about plugins in order to do things, I actually like this framework - it lets there be specific owners for portions of the open source development. Self-implodes? This seems like it would be tracked as a bug. I've encountered an instance where Jenkins wouldn't start due to a crypto issue but that was due to a bug and all I needed to do was install a patch. I think that using Jenkins can be a thought of a serious option if like anything else, you follow security protocols ie: don't allow public access, maintain RBAC standards, have a maintenance schedule. [1]https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/ https://about.gitlab.com/releases/2022/02/25/critical-securi...
- AtNightWeCode 4y ago
  I have stolen access to production systems though Jenkins. Cause the scrambled pw is always sent to the Jenkins client. And then by default anybody could decode it from the console… They fixed it in 2.0 I believe. The problem with dependencies is that different plugins need different versions but there is no solution to that. Even more. You update Jenkins. Plugins break. Data gets corrupted. You try to align the plugins to some dependent version. It fails, cause most of the ecosystem is a hobby-project that never gets updated. Jenkins has absolutely nothing to do on the public Internet as most of these tools. There are so many CI/CD services. Most are very cheap. You really need to love yak shaving to pick Jenkins over something from the shelf. No major cloud offers Jenkins Saas. Wonder why…

fartcannon 4y ago

Also: "you grant to Microsoft a worldwide and royalty-free intellectual property license to use Your Content, for example, to make copies of, retain, transmit, reformat, display, and distribute via communication tools Your Content on the Services." https://www.microsoft.com/en-ca/servicesagreement/upcoming.aspx https://www.microsoft.com/en-ca/servicesagreement/upcoming.a...

Pryde 4y ago
Am I missing something, or is GitHub distinctly not listed in the Covered Services section of that services agreement?
- fartcannon 4y ago
  Nope, looks like I missed that. My mistake. So what part of the github terms gives them permission to sell my IP back to me through copilot?
mey 4y ago
https://docs.github.com/en/site-policy/github-terms/github-terms-of-service#d-user-generated-content https://docs.github.com/en/site-policy/github-terms/github-t... GitHub has a separate policy...
- fartcannon 4y ago
  Thank you, I missed that part. As I asked elsewhere, where is the part of these agreements that permits them to profit off my IP through copilot?
  - vel0city 4y ago
    If your code was in a private repo they supposedly didn't use it for training. I think it was also mentioned they tried to limit it to only repos where they could determine some kind of license they thought would permit such usage. If you're publishing your IP to GitHub in a public repo and using a widely permissive license, they didn't need an agreement to use that IP to train copilot.

tapoxi 4y ago

GitHub is running a complex, planet-scale product. I think w they've crossed the threshold where doing it yourself is more likely to be reliable for some use cases. We've been running GitLab on GKE for the past three years, no problems outside of initial migration pains.

sascha_sl 4y ago

We have open tickets about the SLA being broken for Q1 and Q2. They've been open for a while, and we're out of ways to escalate them (despite enterprise). And GitHub's SLA is not great to begin with: 10% of spend refund at 3 nines (99,9%) and 25% of spend at 2 nines (99%).