4 ms·

I'm trying to set up a deployment pipeline, quite from scratch. That also includes picking source repository software. Self-hosted is a must for every component

I'm trying to set up a deployment pipeline, quite from scratch. That also includes picking source repository software. Self-hosted is a must for every component. So I chose gitlab and after that I dived into it. Oh boy was I in for a surprise. For CI/CD one needs docker, kubernetes and whatnot: https://docs.gitlab.com/ee/topics/autodevops/ https://docs.gitlab.com/ee/topics/autodevops/ But the alternatives don't look better either. So, fellow hn'ians. My requirements ATM are quite simple: Submit code via git, run automated test, if all good, put the files into the live system. Pull-requests are must and everything must be self-hosted because cloud is just another persons computer. Code is written in python. What would you recommend? Edit: grammar.

bthornbury 8y ago
Sounds like you can get away with a bash script that calls rsync, and restarts the code on the server with ssh. (I deploy one of my websites this way, simple & reliable) If you're looking for something more legit: Ansible
te_chris 8y ago
Heroku. It is everything you want.
jl-gitlab 8y ago
Hey there, I'm the product manager for CI/CD at GitLab. AutoDevOps is one of our more advanced features to get up and running in a k8s environment quickly, but is not required if you just want basic CI. Our documentation on getting started with CI/CD is here: https://docs.gitlab.com/ee/ci/ https://docs.gitlab.com/ee/ci/
- ratiolat 8y ago
  Thank You! I will look into that. OTOH this is how I came to conclusion that I must go the AutoDevOps way: This is my first installation of gitlab ever. So naturally, after somehow successfully installing it on the 2nd try, everything seems to work. So - lets explore configuration options. For example we should disable registration since, well, we don't want to be a public gitlab host. After a while, after the Pages section, we come to a section named "Continuous Integration and Deployment". And there's a checkbox named "Default to Auto DevOps pipeline for all projects". And there does not seem to be any other kind of "Continuous Integration and Deployment" available so one naturally comes to a conclusion that the only way seems to be Auto DevOps. My bad.
  - jl-gitlab 8y ago
    No worries, I understand how you could come to that conclusion. I've opened up an issue to see if we can make that more clear: https://gitlab.com/gitlab-org/gitlab-ce/issues/52144 https://gitlab.com/gitlab-org/gitlab-ce/issues/52144
  - dgruesso 8y ago
    Hi ratiolat, another GitLab PM here. Thanks for the feedback, as jl-gitlab mentioned, we'll work to make that clearer. You can think of Auto DevOps as a CI template that has a job for all devops stages built-in. If you want to experiment with kubernetes first you can easily add a kubernetes cluster from GitLab and experiment using with CI with or without auto devops. More info here: https://docs.gitlab.com/ee/user/project/clusters/ https://docs.gitlab.com/ee/user/project/clusters/
- donmcronald 8y ago
  I might be wrong about this, so feel free to correct me. Since you're here... AutoDevOps has a good example of GitLab licensing that frustrates me. It's the "Support for multiple Kubernetes clusters" feature. Running development and production in separate environments seems like a best practice. A mistake that brings down a cluster is really the type of thing you'd want to catch in development or testing, right? However, IMO, if you're a small developer using Core or Starter, GitLab is forcing you to do something subpar by using the same cluster for development and production. I think "Environment-specific variables" is the same situation where Core and Starter users are lead into bad practices. For example, ASP.NET Core uses ASPNETCORE_ENVIRONMENT which (unfortunately) defaults to "production". The lack of environment specific varaibles means people will put that in the job config of .gitlab-ci.yml. deploy_dev: variables: ASPNETCORE_ENVIRONMENT: "Development" It also means protected pipelines will be given all the variables needed for deploying to all environments. Ex: DEVELOPMENT_DB,PRODUCTION_DB. I'm sure you can see where this is going, but, if you're using something like EF Core migrations, all it takes is for someone to mess up the (ex:) "deploy_dev" job's ASPNETCORE_ENVIRONMENT variable on protected branch and the CI build is going to end up in "production" mode with access to all the variables needed to break a production DB. GitLab Flow has a diagram showing master, which is a protected branch, deploying to staging with another branch used for production. I like that workflow, but it means all deployable branches end up being protected branches (I think that's the intent ?) which means production credentials end up exposed everywhere but feature branches. I don't know how protected variables interact with environment specific variables, but, ideally, I'd want to have ASPNETCORE_ENVIRONMENT be an environment specific variable and PRODUCTION_DB be both environment specific AND protected. The more safeguards the better and navigating into the "production" environment variables page is a good way for me to get into "be super careful" mode. I think small developers will tend to be the ones where people merge their own changes, so they're especially vulnerable to the type of mistake I just described.
  - dgruesso 8y ago
    Thanks for the feedback Don. For your particular use case, you could use the same cluster for multiple environments if you have separate namespaces and use separate service accounts for each. Using RBAC will further ensure you don't have any collisions, this way if there's an error in development, the production namespace will remain untouched. When deciding which features are paid vs free we ask ourselves if the particular feature may be more relevant to larger organizations; that doesn't necessarily mean there won't be a use case at smaller companies, but just wanted to let you know our reasoning. You can read more about it here: https://about.gitlab.com/stewardship#what-features-are-paid-only https://about.gitlab.com/stewardship#what-features-are-paid-....
tnolet 8y ago
I don’t mean this snarky at all, but I’m stunned by the fact that you think a distributed container orchestration platform (which Kubernetes is) is needed to do CI/CD, even when self hosted. It shows I guess how hazy these topics still are. Have a look at Jenkins or Gitlab or Bamboo or any of the other dozen of well known CI server out there. They will pull, test and build your code. Then look at something like dokku or Capistrano or even Ansible to take care of putting builds on servers. With a little bash scripting you have a full pipeline. No Kubernetes needed at all.
- ratiolat 8y ago
  Believe me I don't wan't to dive into Kubernetes and learn a beast like that for CI/CD but it says so in the GitLab documentation referenced in my original comment.
  - jl-gitlab 8y ago
    The documentation link you shared was for AutoDevOps, a different (but related) feature from CI/CD. It's a build and deployment solution specifically for Kubernetes/Containers, which is why it has those requirements. GitLab CI/CD documentation can be found at the link that I shared, and it can build and deploy truly all kinds of software, and does not require k8s.
deleted 8y ago
[deleted]
gizzlon 8y ago
there are _many_ self hosted alternatives to gitlab. Nothing against them, just saying :) https://gogs.io/ https://gogs.io/ https://gitea.io/en-us/ https://gitea.io/en-us/ <= The golang ones are deployed with: Copy the binary, run the binary.
hesarenu 8y ago
app engine + cloudbuild.yaml?
ptman 8y ago
I'm quite happy with gitea + drone currently.

bthornbury 8y ago

Sounds like you can get away with a bash script that calls rsync, and restarts the code on the server with ssh. (I deploy one of my websites this way, simple & reliable) If you're looking for something more legit: Ansible

te_chris 8y ago

Heroku. It is everything you want.

jl-gitlab 8y ago

Hey there, I'm the product manager for CI/CD at GitLab. AutoDevOps is one of our more advanced features to get up and running in a k8s environment quickly, but is not required if you just want basic CI. Our documentation on getting started with CI/CD is here: https://docs.gitlab.com/ee/ci/ https://docs.gitlab.com/ee/ci/

ratiolat 8y ago
Thank You! I will look into that. OTOH this is how I came to conclusion that I must go the AutoDevOps way: This is my first installation of gitlab ever. So naturally, after somehow successfully installing it on the 2nd try, everything seems to work. So - lets explore configuration options. For example we should disable registration since, well, we don't want to be a public gitlab host. After a while, after the Pages section, we come to a section named "Continuous Integration and Deployment". And there's a checkbox named "Default to Auto DevOps pipeline for all projects". And there does not seem to be any other kind of "Continuous Integration and Deployment" available so one naturally comes to a conclusion that the only way seems to be Auto DevOps. My bad.
- jl-gitlab 8y ago
  No worries, I understand how you could come to that conclusion. I've opened up an issue to see if we can make that more clear: https://gitlab.com/gitlab-org/gitlab-ce/issues/52144 https://gitlab.com/gitlab-org/gitlab-ce/issues/52144
- dgruesso 8y ago
  Hi ratiolat, another GitLab PM here. Thanks for the feedback, as jl-gitlab mentioned, we'll work to make that clearer. You can think of Auto DevOps as a CI template that has a job for all devops stages built-in. If you want to experiment with kubernetes first you can easily add a kubernetes cluster from GitLab and experiment using with CI with or without auto devops. More info here: https://docs.gitlab.com/ee/user/project/clusters/ https://docs.gitlab.com/ee/user/project/clusters/
donmcronald 8y ago
I might be wrong about this, so feel free to correct me. Since you're here... AutoDevOps has a good example of GitLab licensing that frustrates me. It's the "Support for multiple Kubernetes clusters" feature. Running development and production in separate environments seems like a best practice. A mistake that brings down a cluster is really the type of thing you'd want to catch in development or testing, right? However, IMO, if you're a small developer using Core or Starter, GitLab is forcing you to do something subpar by using the same cluster for development and production. I think "Environment-specific variables" is the same situation where Core and Starter users are lead into bad practices. For example, ASP.NET Core uses ASPNETCORE_ENVIRONMENT which (unfortunately) defaults to "production". The lack of environment specific varaibles means people will put that in the job config of .gitlab-ci.yml. deploy_dev: variables: ASPNETCORE_ENVIRONMENT: "Development" It also means protected pipelines will be given all the variables needed for deploying to all environments. Ex: DEVELOPMENT_DB,PRODUCTION_DB. I'm sure you can see where this is going, but, if you're using something like EF Core migrations, all it takes is for someone to mess up the (ex:) "deploy_dev" job's ASPNETCORE_ENVIRONMENT variable on protected branch and the CI build is going to end up in "production" mode with access to all the variables needed to break a production DB. GitLab Flow has a diagram showing master, which is a protected branch, deploying to staging with another branch used for production. I like that workflow, but it means all deployable branches end up being protected branches (I think that's the intent ?) which means production credentials end up exposed everywhere but feature branches. I don't know how protected variables interact with environment specific variables, but, ideally, I'd want to have ASPNETCORE_ENVIRONMENT be an environment specific variable and PRODUCTION_DB be both environment specific AND protected. The more safeguards the better and navigating into the "production" environment variables page is a good way for me to get into "be super careful" mode. I think small developers will tend to be the ones where people merge their own changes, so they're especially vulnerable to the type of mistake I just described.
- dgruesso 8y ago
  Thanks for the feedback Don. For your particular use case, you could use the same cluster for multiple environments if you have separate namespaces and use separate service accounts for each. Using RBAC will further ensure you don't have any collisions, this way if there's an error in development, the production namespace will remain untouched. When deciding which features are paid vs free we ask ourselves if the particular feature may be more relevant to larger organizations; that doesn't necessarily mean there won't be a use case at smaller companies, but just wanted to let you know our reasoning. You can read more about it here: https://about.gitlab.com/stewardship#what-features-are-paid-only https://about.gitlab.com/stewardship#what-features-are-paid-....

tnolet 8y ago

I don’t mean this snarky at all, but I’m stunned by the fact that you think a distributed container orchestration platform (which Kubernetes is) is needed to do CI/CD, even when self hosted. It shows I guess how hazy these topics still are. Have a look at Jenkins or Gitlab or Bamboo or any of the other dozen of well known CI server out there. They will pull, test and build your code. Then look at something like dokku or Capistrano or even Ansible to take care of putting builds on servers. With a little bash scripting you have a full pipeline. No Kubernetes needed at all.

ratiolat 8y ago
Believe me I don't wan't to dive into Kubernetes and learn a beast like that for CI/CD but it says so in the GitLab documentation referenced in my original comment.
- jl-gitlab 8y ago
  The documentation link you shared was for AutoDevOps, a different (but related) feature from CI/CD. It's a build and deployment solution specifically for Kubernetes/Containers, which is why it has those requirements. GitLab CI/CD documentation can be found at the link that I shared, and it can build and deploy truly all kinds of software, and does not require k8s.

deleted 8y ago

[deleted]

gizzlon 8y ago

there are _many_ self hosted alternatives to gitlab. Nothing against them, just saying :) https://gogs.io/ https://gogs.io/ https://gitea.io/en-us/ https://gitea.io/en-us/ <= The golang ones are deployed with: Copy the binary, run the binary.

hesarenu 8y ago

app engine + cloudbuild.yaml?

ptman 8y ago

I'm quite happy with gitea + drone currently.