6 ms·

Gravity: Upstream Kubernetes packaging tools

98 points by gk1 7y ago

alexk 7y ago
Hey all, I'm part of the team at Gravitational - a company behind this effort. Will be happy to answer any questions about this project.
- richardwhiuk 7y ago
  This seems to assume that each app will run on a separate Kubernetes cluster. Have you given any thought to packaging apps which will run on a single Kubernetes cluster?
  - alexk 7y ago
    Yep, check out integration with helm for multiple apps: https://gravitational.com/gravity/docs/catalog/#publish-an-application-image https://gravitational.com/gravity/docs/catalog/#publish-an-a... It's something we support right now, but still polishing UX, the idea is to add more support for Helm 3.0 in the future in addition to Helm 2.0 we already support.
    - llama052 7y ago
      Looks like that functionality is locked behind the Enterprise edition correct?
      - alexk 7y ago
        Seems like we have to fix the docs a bit, `gravity app` is supposed to work with any helm registry or even without one: tele build helm-package ->> tarball gravity app install tarball
  - closeparen 7y ago
    Is cluster-per-app a thing people really do? Seems to defeat the purpose. We use Mesos rather than Kubernetes, but the point is that a central infrastructure team (maybe 15 people) manages the cluster itself. It provides service owners (several thousand people) with a small and straightfoward abstraction to provision, upgrade, and decommission their particular apps. If each service team had to deal with its own cluster scheduling, these systems would usually be massive timewasters relative to the old ways (Puppet, shell scripts, etc).
    - jacques_chester 7y ago
      > Is cluster-per-app a thing people really do? Seems to defeat the purpose. Yes it is and yes, it does. Kubernetes has a fairly soft tenancy model, so in practice, multi-cluster is becoming a big deal, despite the cost in utilisation efficiency. Various attempts are being made to recover the utilisation efficiencies, such as Virtual Kubelet or Project Pacific. I think that the original sin of Kubernetes was the lack of a firm, first-class, top-down, mandatory access control model of tenancy. Disclosure: I work for VMware, which is responsible for Project Pacific.
      - alexk 7y ago
        If you think of an application as composed of hundreds of micro services, than it makes perfect sense. It's a multi-tenant system, it's just each tenant is a part of the same app.
        jacques_chester 7y ago
        I'm not sure I follow you, could you elaborate?
    - alexk 7y ago
      > Is cluster-per-app a thing people really do? Gravity enables a somewhat unique scenario of companies taking their complex micro-services stack and delivering it as an installer for software application. From the end users' perspective they don't even know that it's a k8s cluster, they consume an application that consists of multiple components. However Gravity supports the scenario when multiple applications are installed in the cluster as well.
- chucky_z 7y ago
  Would Gravitational ever think about creating a more generic layer on this, e.g.: to manage Mesos, Nomad, or something else entirely like Corosync via some kind of driver system? I love Teleport but don't use k8s anywhere.
  - alexk 7y ago
    This is one of those ideas we always wanted to implement, but in reality, once we took on deploying K8s it completely consumed all the teams' resources :D
    - chucky_z 7y ago
      Anyway, just a 1-person note, but "here's an open source bottom layer for you to build (x) on top with the ux of teleport/gravity" would be incredible.
- ablekh 7y ago
  1. Is Gravity a fully open source software (with commercial support) or an Open Core software? If the latter, please point to a feature matrix or relevant part of documentation. 2. How Gravity solution (and/or approach) is different from a similar offering by Replicated? EDIT: Oops, sorry, just found the answer to my Q1: about halfway down on the following page: https://gravitational.com/gravity https://gravitational.com/gravity. However, some information on pricing structure and approximate numbers would be appreciated. Replicated is more transparent in this regard. :-) Q2 still stands. Looking forward to hearing from you.
- mwcampbell 7y ago
  I wonder if you've considered building a complete VM image instead of a tarball. A minimal, immutable VM image could be more finely tuned and hardened than a tarball installed on top of a general-purpose distro. You could take inspiration from CoreOS or LinuxKit here. Or do you find that the sysadmins in the kind of organization that install these on-prem packages really want to install on top of their favorite distro?
  - alexwilliamsca 7y ago
    A minimal immutable VM image which is finely tuned and hardened is the exact approach we're taking with https://on-premises.com https://on-premises.com - except we haven't focused on k8s workloads. We've found that customers would much rather import a VM than "install" something, however there are valid use-cases which require special monitoring tools and other customizations which are not possible on an immutable system. Gravity is interesting and seems to meet that demand.
    - moondev 7y ago
      This look interesting, is there a way to try out meta without being funneled into the sales pipeline? What is the workflow for baking a machine? Are you using packer under the covers or some other tooling? What on-prem machine image formats are supported?
      - alexwilliamsca 7y ago
        There's still a bit of "manual" work to get someone up and running on our Meta appliance, so unfortunately you would have to go through our sales process. However if you just want a video or screen recording of how it works then I can put something online in the next hour or two. For baking the machine, we use Ansible under the covers and have a set of Lisp scripts to manage everything. As for image formats: qcow2, raw, vhd (and vmdk in the .ova file). Not trying to hijack Gravitational's thread, please contact me (email in profile, or 'aw-' on FreeNode) if you want to discuss more.
  - alexk 7y ago
catern 7y ago
I find each new development in the field of deploying Kubernetes to be grimly humorous. We had lots of techniques to package and deploy things before Kubernetes, but they were complex and inadequate in some ways, and Kubernetes fixes some of those issues. But it turns out Kubernetes itself is substantially more complicated to package and deploy than the old solutions, if you're deploying it yourself rather than using some proprietary cloud. Oops! What I just can't decide is whether we'll successfully put another layer of container-container-containers on top of Kubernetes, or whether the whole effort will eventually collapse and we'll extract the good parts out into something simpler.
- alexk 7y ago
  Kubernetes cluster is hard to operate indeed. Techniques that we always had to deploy before Kubernetes, for example package managers, don't always translate to operating a distributed system on a hundreds or thousands of servers. Some of this complexity is definitely incidental, not all components are not always coordinating well with each other, e.g. docker and API server, networking layer during upgrades for example. On the other hand lots of this complexity is essential - K8s is a distributed system with database, network and container orchestration layer solving a hard problem.
- cdoxsey 7y ago
  > But it turns out Kubernetes itself is substantially more complicated to package and deploy than the old solutions Not really. It's pretty much the same level of complication if you used the same components. You could use all the same tools: chef, puppet, ansible, etc. Once you have it available though other applications are easier to deploy. At any rate, this tool provides something entirely different. It lets you image the entire data center and reproduce it somewhere else. Not sure how you would've done that before.
  - deleted 7y ago
    [deleted]
- eeZah7Ux 7y ago
  > We had lots of techniques to package and deploy things before Kubernetes, but they were complex and inadequate in some ways They worked very well for 30 years. And the new container-based tools inevitably end up doing the same mistakes, rediscovering the same solutions and we'll go back to the beginning. > turns out Kubernetes itself is substantially more complicated to package and deploy than the old solutions Exactly, and you can't solve a problem by making it more complex.
redwood 7y ago
Hard not to wonder if k8s is the new hadoop. "If you build it they will come" platform team thinking.
- Proven 7y ago
  Haha, the new Hadoop - that's funny! Upvote from me!
- Thaxll 7y ago
  Hadoop did not event reach 1/10 of Kubernetes popularity.
  - pas 7y ago
    That's very hard to eyeball in my opinion. Hadoop and its ecosystem fascinated a big chunk of the dev (and ops) world. Everybody and their dog wanted to be "into" Big Data. Data Lake. Realtime feeds, just add nodes (Cassandra), CQRS, and of course when Facebook said they are using HBase for Messenger, it meant that HBase is the new MySQL/Mongo/sliced-bread. Then there was YARN, then Tez, Spark, Flink, and Drill, and various other projects that added to the hype (Aerospike, RamSQL, Kafka + Storm). And ever new system had to be built like it'll be web scale from day one. Instagram was acquired in 2012, just 18 months after launch, and everyone knew that meant every new even barely "social" thing will blow up even faster than that. So you absolutely need to plan ahead, scale scale scale. Compared to that people seem to be a bit more wary of k8s, especially because it's targeted at ops folks, and they are naturally predisposed to oppose changes they don't understand. But that's just my - probably ridiculously non-representative - take on this :)
  - redwood 7y ago
    https://trends.google.com/trends/explore?date=all&geo=US&q=Hadoop,Kubernetes https://trends.google.com/trends/explore?date=all&geo=US&q=H...
mwcampbell 7y ago
Requiring a dedicated volume with at least 50 GB of space and 1500 privisioned IOPS just for etcd [1] seems excessive to me. How big a cluster, e.g. how many nodes and pods, is this for? [1]: https://gravitational.com/gravity/docs/ver/6.x/requirements/#etcd-disk https://gravitational.com/gravity/docs/ver/6.x/requirements/...
- alexk 7y ago
  We are staying on the safe side here on our recommendations. We have seen many scenarios when high latency on volumes on AWS creates a lot of problems even for small 3 node clusters.
- AaronFriel 7y ago
  Even small clusters with K8s operators doing complex work (e.g.: autoscaling) can generate a lot of etcd traffic updating state. I'm not sure how (etcd) stateless Istio and other meshes are, too.
skube 7y ago
There's going to be a lot of consultants making a bunch of cash deplatforming all these companies once the k8s bubble pops.
moondev 7y ago
I played with this briefly last year but couldn't get it going in the time frame so I gave up. I wanted to like it but it was a bit unwieldy getting started, and I was not a fan of needing helm to bootstrap an installation. There also was a quirky installation workflow, could not set a key/password successfully for the life of me and it was unclear on if the cli needed a browser? May have to check this out again, hopefully the quck-start experience has improved.
- kevin_nisbet 7y ago
  Sorry to hear about your experience, the getting started experience is an ongoing challenge, there have been improvements in the last year, but we still have a long way to go as well. Sorry if this wasn't clear, but helm actually isn't required at all, it's just a majority of our examples are written to use helm due helms popularity. The installation hooks really just boil down to kubernetes jobs, so anything that can be represented as a kubernetes job can be used for any of the hooks. This can be a simple script, a helm command, or a complicated custom built application. The only feature really tied specifically to helm is the catalog feature, which is for building additional applications to be installed on top of an existing gravity cluster. That feature was built around a helm chart as a building block. The cli should only need to invoke a browser when doing third party authentication flows, ie to use github for login. Using the gravity users invite will also generate a link to send to the enrolling user, so they can set their own password, setup 2fa, etc through the web interface. We've also been trying to use our community site https://community.gravitational.com/ https://community.gravitational.com/ as a resource for being able to search and ask questions. Disclaimer: I'm a developer on gravity.

98 points by gk1 7y ago

Hey all, I'm part of the team at Gravitational - a company behind this effort. Will be happy to answer any questions about this project.

richardwhiuk 7y ago
This seems to assume that each app will run on a separate Kubernetes cluster. Have you given any thought to packaging apps which will run on a single Kubernetes cluster?
- alexk 7y ago
  Yep, check out integration with helm for multiple apps: https://gravitational.com/gravity/docs/catalog/#publish-an-application-image https://gravitational.com/gravity/docs/catalog/#publish-an-a... It's something we support right now, but still polishing UX, the idea is to add more support for Helm 3.0 in the future in addition to Helm 2.0 we already support.
  - llama052 7y ago
    Looks like that functionality is locked behind the Enterprise edition correct?
    - alexk 7y ago
      Seems like we have to fix the docs a bit, `gravity app` is supposed to work with any helm registry or even without one: tele build helm-package ->> tarball gravity app install tarball
- closeparen 7y ago
  Is cluster-per-app a thing people really do? Seems to defeat the purpose. We use Mesos rather than Kubernetes, but the point is that a central infrastructure team (maybe 15 people) manages the cluster itself. It provides service owners (several thousand people) with a small and straightfoward abstraction to provision, upgrade, and decommission their particular apps. If each service team had to deal with its own cluster scheduling, these systems would usually be massive timewasters relative to the old ways (Puppet, shell scripts, etc).
  - jacques_chester 7y ago
    > Is cluster-per-app a thing people really do? Seems to defeat the purpose. Yes it is and yes, it does. Kubernetes has a fairly soft tenancy model, so in practice, multi-cluster is becoming a big deal, despite the cost in utilisation efficiency. Various attempts are being made to recover the utilisation efficiencies, such as Virtual Kubelet or Project Pacific. I think that the original sin of Kubernetes was the lack of a firm, first-class, top-down, mandatory access control model of tenancy. Disclosure: I work for VMware, which is responsible for Project Pacific.
    - alexk 7y ago
      If you think of an application as composed of hundreds of micro services, than it makes perfect sense. It's a multi-tenant system, it's just each tenant is a part of the same app.
      - jacques_chester 7y ago
        I'm not sure I follow you, could you elaborate?
  - alexk 7y ago
    > Is cluster-per-app a thing people really do? Gravity enables a somewhat unique scenario of companies taking their complex micro-services stack and delivering it as an installer for software application. From the end users' perspective they don't even know that it's a k8s cluster, they consume an application that consists of multiple components. However Gravity supports the scenario when multiple applications are installed in the cluster as well.
chucky_z 7y ago
Would Gravitational ever think about creating a more generic layer on this, e.g.: to manage Mesos, Nomad, or something else entirely like Corosync via some kind of driver system? I love Teleport but don't use k8s anywhere.
- alexk 7y ago
  This is one of those ideas we always wanted to implement, but in reality, once we took on deploying K8s it completely consumed all the teams' resources :D
  - chucky_z 7y ago
    Anyway, just a 1-person note, but "here's an open source bottom layer for you to build (x) on top with the ux of teleport/gravity" would be incredible.
ablekh 7y ago
1. Is Gravity a fully open source software (with commercial support) or an Open Core software? If the latter, please point to a feature matrix or relevant part of documentation. 2. How Gravity solution (and/or approach) is different from a similar offering by Replicated? EDIT: Oops, sorry, just found the answer to my Q1: about halfway down on the following page: https://gravitational.com/gravity https://gravitational.com/gravity. However, some information on pricing structure and approximate numbers would be appreciated. Replicated is more transparent in this regard. :-) Q2 still stands. Looking forward to hearing from you.
mwcampbell 7y ago
I wonder if you've considered building a complete VM image instead of a tarball. A minimal, immutable VM image could be more finely tuned and hardened than a tarball installed on top of a general-purpose distro. You could take inspiration from CoreOS or LinuxKit here. Or do you find that the sysadmins in the kind of organization that install these on-prem packages really want to install on top of their favorite distro?
- alexwilliamsca 7y ago
  A minimal immutable VM image which is finely tuned and hardened is the exact approach we're taking with https://on-premises.com https://on-premises.com - except we haven't focused on k8s workloads. We've found that customers would much rather import a VM than "install" something, however there are valid use-cases which require special monitoring tools and other customizations which are not possible on an immutable system. Gravity is interesting and seems to meet that demand.
  - moondev 7y ago
    This look interesting, is there a way to try out meta without being funneled into the sales pipeline? What is the workflow for baking a machine? Are you using packer under the covers or some other tooling? What on-prem machine image formats are supported?
    - alexwilliamsca 7y ago
      There's still a bit of "manual" work to get someone up and running on our Meta appliance, so unfortunately you would have to go through our sales process. However if you just want a video or screen recording of how it works then I can put something online in the next hour or two. For baking the machine, we use Ansible under the covers and have a set of Lisp scripts to manage everything. As for image formats: qcow2, raw, vhd (and vmdk in the .ova file). Not trying to hijack Gravitational's thread, please contact me (email in profile, or 'aw-' on FreeNode) if you want to discuss more.
- alexk 7y ago

catern 7y ago

I find each new development in the field of deploying Kubernetes to be grimly humorous. We had lots of techniques to package and deploy things before Kubernetes, but they were complex and inadequate in some ways, and Kubernetes fixes some of those issues. But it turns out Kubernetes itself is substantially more complicated to package and deploy than the old solutions, if you're deploying it yourself rather than using some proprietary cloud. Oops! What I just can't decide is whether we'll successfully put another layer of container-container-containers on top of Kubernetes, or whether the whole effort will eventually collapse and we'll extract the good parts out into something simpler.

alexk 7y ago
Kubernetes cluster is hard to operate indeed. Techniques that we always had to deploy before Kubernetes, for example package managers, don't always translate to operating a distributed system on a hundreds or thousands of servers. Some of this complexity is definitely incidental, not all components are not always coordinating well with each other, e.g. docker and API server, networking layer during upgrades for example. On the other hand lots of this complexity is essential - K8s is a distributed system with database, network and container orchestration layer solving a hard problem.
cdoxsey 7y ago
> But it turns out Kubernetes itself is substantially more complicated to package and deploy than the old solutions Not really. It's pretty much the same level of complication if you used the same components. You could use all the same tools: chef, puppet, ansible, etc. Once you have it available though other applications are easier to deploy. At any rate, this tool provides something entirely different. It lets you image the entire data center and reproduce it somewhere else. Not sure how you would've done that before.
- deleted 7y ago
  [deleted]
eeZah7Ux 7y ago
> We had lots of techniques to package and deploy things before Kubernetes, but they were complex and inadequate in some ways They worked very well for 30 years. And the new container-based tools inevitably end up doing the same mistakes, rediscovering the same solutions and we'll go back to the beginning. > turns out Kubernetes itself is substantially more complicated to package and deploy than the old solutions Exactly, and you can't solve a problem by making it more complex.

redwood 7y ago

Hard not to wonder if k8s is the new hadoop. "If you build it they will come" platform team thinking.

Proven 7y ago
Haha, the new Hadoop - that's funny! Upvote from me!
Thaxll 7y ago
Hadoop did not event reach 1/10 of Kubernetes popularity.
- pas 7y ago
  That's very hard to eyeball in my opinion. Hadoop and its ecosystem fascinated a big chunk of the dev (and ops) world. Everybody and their dog wanted to be "into" Big Data. Data Lake. Realtime feeds, just add nodes (Cassandra), CQRS, and of course when Facebook said they are using HBase for Messenger, it meant that HBase is the new MySQL/Mongo/sliced-bread. Then there was YARN, then Tez, Spark, Flink, and Drill, and various other projects that added to the hype (Aerospike, RamSQL, Kafka + Storm). And ever new system had to be built like it'll be web scale from day one. Instagram was acquired in 2012, just 18 months after launch, and everyone knew that meant every new even barely "social" thing will blow up even faster than that. So you absolutely need to plan ahead, scale scale scale. Compared to that people seem to be a bit more wary of k8s, especially because it's targeted at ops folks, and they are naturally predisposed to oppose changes they don't understand. But that's just my - probably ridiculously non-representative - take on this :)
- redwood 7y ago
  https://trends.google.com/trends/explore?date=all&geo=US&q=Hadoop,Kubernetes https://trends.google.com/trends/explore?date=all&geo=US&q=H...

mwcampbell 7y ago

Requiring a dedicated volume with at least 50 GB of space and 1500 privisioned IOPS just for etcd [1] seems excessive to me. How big a cluster, e.g. how many nodes and pods, is this for? [1]: https://gravitational.com/gravity/docs/ver/6.x/requirements/#etcd-disk https://gravitational.com/gravity/docs/ver/6.x/requirements/...

alexk 7y ago
We are staying on the safe side here on our recommendations. We have seen many scenarios when high latency on volumes on AWS creates a lot of problems even for small 3 node clusters.
AaronFriel 7y ago
Even small clusters with K8s operators doing complex work (e.g.: autoscaling) can generate a lot of etcd traffic updating state. I'm not sure how (etcd) stateless Istio and other meshes are, too.

skube 7y ago

There's going to be a lot of consultants making a bunch of cash deplatforming all these companies once the k8s bubble pops.

moondev 7y ago

I played with this briefly last year but couldn't get it going in the time frame so I gave up. I wanted to like it but it was a bit unwieldy getting started, and I was not a fan of needing helm to bootstrap an installation. There also was a quirky installation workflow, could not set a key/password successfully for the life of me and it was unclear on if the cli needed a browser? May have to check this out again, hopefully the quck-start experience has improved.

kevin_nisbet 7y ago
Sorry to hear about your experience, the getting started experience is an ongoing challenge, there have been improvements in the last year, but we still have a long way to go as well. Sorry if this wasn't clear, but helm actually isn't required at all, it's just a majority of our examples are written to use helm due helms popularity. The installation hooks really just boil down to kubernetes jobs, so anything that can be represented as a kubernetes job can be used for any of the hooks. This can be a simple script, a helm command, or a complicated custom built application. The only feature really tied specifically to helm is the catalog feature, which is for building additional applications to be installed on top of an existing gravity cluster. That feature was built around a helm chart as a building block. The cli should only need to invoke a browser when doing third party authentication flows, ie to use github for login. Using the gravity users invite will also generate a link to send to the enrolling user, so they can set their own password, setup 2fa, etc through the web interface. We've also been trying to use our community site https://community.gravitational.com/ https://community.gravitational.com/ as a resource for being able to search and ask questions. Disclaimer: I'm a developer on gravity.