8 ms·

From Kubernetes to a complete application development tool [video]

151 points by sytse 10y ago

sytse 10y ago
I'm very excited about this. Especially about the terminal we show at https://www.youtube.com/watch?v=a7OIQfOJO-0&t=7m11s https://www.youtube.com/watch?v=a7OIQfOJO-0&t=7m11s AMA
- annerajb 10y ago
  looks really good. How is the terminal being done websockets?
  - sytse 10y ago
    It does use websockets, see https://gitlab.com/gitlab-org/gitlab-ce/issues/22864 https://gitlab.com/gitlab-org/gitlab-ce/issues/22864 for more information.
    - annerajb 10y ago
      Thanks
- nstart 10y ago
  Super quick one. One of the things I'm working on at work is a staging env per branch system. We use k8s too. Biggest consideration here is individual environment variables per branch. Eg. For testing we have a separate db to work on so we have test data and a db where we can perform migrations if needed. Another example would be adding a new feature that requires a different env variable for another api key. Curious how or if these features support it. Super sorry if it's in the docs. Since you were doing ama I thought I might ask it here :)
  - sytse 10y ago
    We're not doing environment variables per environment yet but we would love to, see https://gitlab.com/gitlab-org/gitlab-ce/issues/20367 https://gitlab.com/gitlab-org/gitlab-ce/issues/20367
kemenaran 10y ago
I wasn't sure what to thing about Cycle Analytics, and the metrics it provides ("time from thoughts to issue", "time from issue to code", "time spent reviewing"). IMHO these numbers may be too synthetic to give meaningful information. Plus, once we have this kind of dashboard, there is a risk of starting to optimize for the numbers (instead of optimizing the reality underneath). That said, I see with this demo how these metrics could be useful to track, well, when reviews are taking too much time (needs more people? better repartition of reviewers?) – or when maintenance tasks are slowing new features (needs better tests? more maintainers?). I guess I'll have to try this out :)
- sytse 10y ago
  For more detail about the numbers see https://gitlab.com/help/user/project/cycle_analytics#how-the-data-is-measured https://gitlab.com/help/user/project/cycle_analytics#how-the... Do you mean that number are to coarse to indicate a specific problem? Of course there is the problem of gaming the numbers. But we do think that compared to many other ways of measuring productivity (for example the number of issues solved) this is relatively robust against manipulation. Getting something out sooner is better most of the time. But thanks for the thoughts and I hope you try it out soon.
  - jacques_chester 10y ago
    > relatively robust against manipulation My own view is that robustness to manipulation can't come from metric design, it can only come from culture. If metrics are tied to reward or punishment, then no matter how clever they are, they will be gamed.
    - sytse 10y ago
      I agree that everything will be gamed and that culture is the best protection. I do think that some metrics are easier to manipulate (cyclomatic complexity comes to mind) and some metrics don't correspond with effectiveness (lines of code written).
- jacques_chester 10y ago
dcgoss 10y ago
Very often on HN I see links describing the latest and greatest GitLab feature. It really seems like they've been working hard and doing a good job, and they seem to have a clear vision ("master plan", in their terms). Where is GitHub in all of this? Aren't they concerned about being outinnovated?
- Untit1ed 10y ago
  Github is so ubiquitous that they can't afford to move fast and break things like Gitlab can - another team at work considered moving to Gitlab to use all these new features, found bugs everywhere and eventually went back to Github.
  - Perihelion 10y ago
    Thanks for the feedback -- it sucks that they found so many bugs in the product. Any chance you remember what specifically they ran into?
    - Untit1ed 10y ago
      Mainly intermittent errors clicking around the interface, particularly 404s when clicking on files that existed - they're not sure whether that was a problem in the web interface or the files actually temporarily went missing.
      - sytse 10y ago
        That sounds more like a problem in the installation than a bug in GitLab. Was this self-hosted or on GitLab.com?
    - theptip 10y ago
      Perf seems to be a recurring issue, e.g. see https://gitlab.com/gitlab-com/infrastructure/issues/675 https://gitlab.com/gitlab-com/infrastructure/issues/675 That said, I find the openness with which core architectural issues are discussed to be refreshing. (e.g. see the very detailed writeup explaining the root cause of the above issue, here https://gitlab.com/gitlab-com/infrastructure/issues/677 https://gitlab.com/gitlab-com/infrastructure/issues/677)
no_protocol 10y ago
This video is really neat and it seems like the promised plan is almost done. Can you clarify at all what the plan is for supporting other container manager systems? I looked at Openshift and it doesn't look like something you can run on your own hardware. Maybe I am misunderstanding this though. -- EDIT: I had spent my time looking at openshift.com and didn't notice that there is an open source version located at openshift.org. Sorry for the confusion. -- I'm interested in using all the functionality shown in the video, but for a small team and on our own hardware. Maybe it doesn't make sense that way, any clarification would help!
- jacques_chester 10y ago
  > I looked at Openshift and it doesn't look like something you can run on your own hardware. I work on a competing platform, Cloud Foundry. But I'm pretty darn sure OpenShift can be installed on your own platforms. Red Hat know a thing or two about Linux, after all. Speaking of Red Hat, they have a dog in this fight in Fabric8. Disclosure: I work for Pivotal, we're the majority donors of engineering to Cloud Foundry.
  - jobvandervoort 10y ago
    For sake of being complete: GitLab offers a paid version for Pivotal Cloud Foundry: https://about.gitlab.com/products/ https://about.gitlab.com/products/
    - jacques_chester 10y ago
      I saw it on PivNet today!
- sytse 10y ago
  Thanks for your kind words. We would love supporting other container management systems. From the OP: "We believe container schedulers such as Kubernetes are the future of application lifecycle management and are working on Mesosphere support. We would love it if people would contribute support for other container schedulers such as Docker Swarm and for other Kubernetes providers such as Tectonic." In the demo we use our own Openshift Origin installation on our own cloud servers, for more information please see https://www.openshift.org/ https://www.openshift.org/
wmf 10y ago
What is OpenShift providing here that isn't in upstream Kubernetes?
- sytse 10y ago
  Nothing, we just had to pick a quick way to install it and Openshift looked nice. All the features are intended to work with all Kubernetes installations down the road.
  - deleted 10y ago
    [deleted]
dmourati 10y ago
The paragraph on "move to the left" does not make any sense to me.
- sytse 10y ago
  I'm sorry to hear that. I should have probably included a drawing. Can others maybe paraphrase what they think it means or does it not make sense to anyone?
  - wmf 10y ago
    You're saying that people used to edit locally, build locally, test locally, then upload to staging/the cloud; now you upload to the cloud first, then build/test/etc. (Of course, people did it this way with Heroku for years because it was harder to build locally, then the pendulum swung back with Docker...)
    - sytse 10y ago
      Exactly, thanks!
- atsaloli 10y ago
  The usual pipeline looks something like this: Dev -> QA -> Ops or Idea -> Design -> Build -> Test -> Deploy -> Maintain For example, Infrastructure Operations used to be responsible for providing a Test environment, now developers can test without Ops because tools like GitLab automatically provision test environments on demand (and then destroy them when they are no longer needed). You still need Ops to take care of that CI system, but you don't need as many personnel.
gtaylor 10y ago
I've got a gitlab-ce helm chart in review in the helm charts repo: https://github.com/kubernetes/charts/pull/202 https://github.com/kubernetes/charts/pull/202 While it's still in the process of working towards a merge, it will make for a very easy installation process: https://github.com/gtaylor/charts/blob/gitlab-ce/stable/gitlab-ce/README.md https://github.com/gtaylor/charts/blob/gitlab-ce/stable/gitl... This is probably most interesting to those who don't want to use OpenShift. There's some more fleshing out to do re: CI runners, but the basics are there.
- sytse 10y ago
  Cool to see this progressing!
nathan_f77 10y ago
What's the minimum monthly cost for running this setup? Can it be done on a single server? If the minimum setup requires a lot of redundancy, then I think this will still be a blocker for many side projects and small startups. Does gitlab.com offer hosting for containers?
- jobvandervoort 10y ago
  I can't comment on the monthly cost. You should be able to run everything on a single server, but I suspect that if everything is under load, that would need to be a decent server. You can use our Docker registry on GitLab.com for free.
Nux 10y ago
That looks pretty darn neat! As a sysadmin though, I do wonder how maintainable and especially upgradable is that thing without breaking half the world.
- jobvandervoort 10y ago
  We release all of GitLabs components at the same time, guaranteeing that they work together. Every month on the 22nd. Updating GitLab is no more work than 'apt-get install'.
  - Nux 10y ago
    That's great. I was referring to the whole stack. Should I upgrade all my hosts, docker, kubernetes, openshift, gitlab, mattermost ... what would be the chances of something to break to the point of ruining my day/week. Not everything is merely "apt-get upgrade"-able alas.

151 points by sytse 10y ago

sytse 10y ago

I'm very excited about this. Especially about the terminal we show at https://www.youtube.com/watch?v=a7OIQfOJO-0&t=7m11s https://www.youtube.com/watch?v=a7OIQfOJO-0&t=7m11s AMA

annerajb 10y ago
looks really good. How is the terminal being done websockets?
- sytse 10y ago
  It does use websockets, see https://gitlab.com/gitlab-org/gitlab-ce/issues/22864 https://gitlab.com/gitlab-org/gitlab-ce/issues/22864 for more information.
  - annerajb 10y ago
    Thanks
nstart 10y ago
Super quick one. One of the things I'm working on at work is a staging env per branch system. We use k8s too. Biggest consideration here is individual environment variables per branch. Eg. For testing we have a separate db to work on so we have test data and a db where we can perform migrations if needed. Another example would be adding a new feature that requires a different env variable for another api key. Curious how or if these features support it. Super sorry if it's in the docs. Since you were doing ama I thought I might ask it here :)
- sytse 10y ago
  We're not doing environment variables per environment yet but we would love to, see https://gitlab.com/gitlab-org/gitlab-ce/issues/20367 https://gitlab.com/gitlab-org/gitlab-ce/issues/20367

kemenaran 10y ago

I wasn't sure what to thing about Cycle Analytics, and the metrics it provides ("time from thoughts to issue", "time from issue to code", "time spent reviewing"). IMHO these numbers may be too synthetic to give meaningful information. Plus, once we have this kind of dashboard, there is a risk of starting to optimize for the numbers (instead of optimizing the reality underneath). That said, I see with this demo how these metrics could be useful to track, well, when reviews are taking too much time (needs more people? better repartition of reviewers?) – or when maintenance tasks are slowing new features (needs better tests? more maintainers?). I guess I'll have to try this out :)

sytse 10y ago
For more detail about the numbers see https://gitlab.com/help/user/project/cycle_analytics#how-the-data-is-measured https://gitlab.com/help/user/project/cycle_analytics#how-the... Do you mean that number are to coarse to indicate a specific problem? Of course there is the problem of gaming the numbers. But we do think that compared to many other ways of measuring productivity (for example the number of issues solved) this is relatively robust against manipulation. Getting something out sooner is better most of the time. But thanks for the thoughts and I hope you try it out soon.
- jacques_chester 10y ago
  > relatively robust against manipulation My own view is that robustness to manipulation can't come from metric design, it can only come from culture. If metrics are tied to reward or punishment, then no matter how clever they are, they will be gamed.
  - sytse 10y ago
    I agree that everything will be gamed and that culture is the best protection. I do think that some metrics are easier to manipulate (cyclomatic complexity comes to mind) and some metrics don't correspond with effectiveness (lines of code written).
jacques_chester 10y ago

dcgoss 10y ago

Very often on HN I see links describing the latest and greatest GitLab feature. It really seems like they've been working hard and doing a good job, and they seem to have a clear vision ("master plan", in their terms). Where is GitHub in all of this? Aren't they concerned about being outinnovated?

Untit1ed 10y ago
Github is so ubiquitous that they can't afford to move fast and break things like Gitlab can - another team at work considered moving to Gitlab to use all these new features, found bugs everywhere and eventually went back to Github.
- Perihelion 10y ago
  Thanks for the feedback -- it sucks that they found so many bugs in the product. Any chance you remember what specifically they ran into?
  - Untit1ed 10y ago
    Mainly intermittent errors clicking around the interface, particularly 404s when clicking on files that existed - they're not sure whether that was a problem in the web interface or the files actually temporarily went missing.
    - sytse 10y ago
      That sounds more like a problem in the installation than a bug in GitLab. Was this self-hosted or on GitLab.com?
  - theptip 10y ago
    Perf seems to be a recurring issue, e.g. see https://gitlab.com/gitlab-com/infrastructure/issues/675 https://gitlab.com/gitlab-com/infrastructure/issues/675 That said, I find the openness with which core architectural issues are discussed to be refreshing. (e.g. see the very detailed writeup explaining the root cause of the above issue, here https://gitlab.com/gitlab-com/infrastructure/issues/677 https://gitlab.com/gitlab-com/infrastructure/issues/677)

no_protocol 10y ago

This video is really neat and it seems like the promised plan is almost done. Can you clarify at all what the plan is for supporting other container manager systems? I looked at Openshift and it doesn't look like something you can run on your own hardware. Maybe I am misunderstanding this though. -- EDIT: I had spent my time looking at openshift.com and didn't notice that there is an open source version located at openshift.org. Sorry for the confusion. -- I'm interested in using all the functionality shown in the video, but for a small team and on our own hardware. Maybe it doesn't make sense that way, any clarification would help!

jacques_chester 10y ago
> I looked at Openshift and it doesn't look like something you can run on your own hardware. I work on a competing platform, Cloud Foundry. But I'm pretty darn sure OpenShift can be installed on your own platforms. Red Hat know a thing or two about Linux, after all. Speaking of Red Hat, they have a dog in this fight in Fabric8. Disclosure: I work for Pivotal, we're the majority donors of engineering to Cloud Foundry.
- jobvandervoort 10y ago
  For sake of being complete: GitLab offers a paid version for Pivotal Cloud Foundry: https://about.gitlab.com/products/ https://about.gitlab.com/products/
  - jacques_chester 10y ago
    I saw it on PivNet today!
sytse 10y ago
Thanks for your kind words. We would love supporting other container management systems. From the OP: "We believe container schedulers such as Kubernetes are the future of application lifecycle management and are working on Mesosphere support. We would love it if people would contribute support for other container schedulers such as Docker Swarm and for other Kubernetes providers such as Tectonic." In the demo we use our own Openshift Origin installation on our own cloud servers, for more information please see https://www.openshift.org/ https://www.openshift.org/

wmf 10y ago

What is OpenShift providing here that isn't in upstream Kubernetes?

sytse 10y ago
Nothing, we just had to pick a quick way to install it and Openshift looked nice. All the features are intended to work with all Kubernetes installations down the road.
- deleted 10y ago
  [deleted]

dmourati 10y ago

The paragraph on "move to the left" does not make any sense to me.

sytse 10y ago
I'm sorry to hear that. I should have probably included a drawing. Can others maybe paraphrase what they think it means or does it not make sense to anyone?
- wmf 10y ago
  You're saying that people used to edit locally, build locally, test locally, then upload to staging/the cloud; now you upload to the cloud first, then build/test/etc. (Of course, people did it this way with Heroku for years because it was harder to build locally, then the pendulum swung back with Docker...)
  - sytse 10y ago
    Exactly, thanks!
atsaloli 10y ago
The usual pipeline looks something like this: Dev -> QA -> Ops or Idea -> Design -> Build -> Test -> Deploy -> Maintain For example, Infrastructure Operations used to be responsible for providing a Test environment, now developers can test without Ops because tools like GitLab automatically provision test environments on demand (and then destroy them when they are no longer needed). You still need Ops to take care of that CI system, but you don't need as many personnel.

gtaylor 10y ago

I've got a gitlab-ce helm chart in review in the helm charts repo: https://github.com/kubernetes/charts/pull/202 https://github.com/kubernetes/charts/pull/202 While it's still in the process of working towards a merge, it will make for a very easy installation process: https://github.com/gtaylor/charts/blob/gitlab-ce/stable/gitlab-ce/README.md https://github.com/gtaylor/charts/blob/gitlab-ce/stable/gitl... This is probably most interesting to those who don't want to use OpenShift. There's some more fleshing out to do re: CI runners, but the basics are there.

sytse 10y ago
Cool to see this progressing!

nathan_f77 10y ago

What's the minimum monthly cost for running this setup? Can it be done on a single server? If the minimum setup requires a lot of redundancy, then I think this will still be a blocker for many side projects and small startups. Does gitlab.com offer hosting for containers?

jobvandervoort 10y ago
I can't comment on the monthly cost. You should be able to run everything on a single server, but I suspect that if everything is under load, that would need to be a decent server. You can use our Docker registry on GitLab.com for free.

Nux 10y ago

That looks pretty darn neat! As a sysadmin though, I do wonder how maintainable and especially upgradable is that thing without breaking half the world.

jobvandervoort 10y ago
We release all of GitLabs components at the same time, guaranteeing that they work together. Every month on the 22nd. Updating GitLab is no more work than 'apt-get install'.
- Nux 10y ago
  That's great. I was referring to the whole stack. Should I upgrade all my hosts, docker, kubernetes, openshift, gitlab, mattermost ... what would be the chances of something to break to the point of ruining my day/week. Not everything is merely "apt-get upgrade"-able alas.