5 ms·

I never saw a compan benefitting from openstack.

karlkatzke 10y ago
I'm at one of the only companies I know of outside Rackspace that's running it in production. We're doing a very poor job of it; we never hit production with any effort to upgrade the cluster in the past three years, so we're still running Havana with Nova Network. The most likely upgrade path for us right now is VMWare 6.5.
- kefka 10y ago
  So are we. And so is another department in our org. Frankly, we're doing a piss poor job of it. Migrations don't happen even if manually ran, the compute nodes are already at capacity, even with the meagre VMs we use. I have kept pushing for Mesos myself. I see it as a good way forward. Openstack may be a technical good body of code... But in the end its too convoluted codebase and many moving parts to be effective in most deployments.
  - benley 10y ago
    Mesos isn't really a direct replacement for Openstack, but you're probably still right to be pushing for it. If you roll with Mesos you will still need to solve a bunch of lower-level infrastructure challenges: machine provisioning, basic system config management, storage management (if you want persistent volumes), fancy networking (if you want ip-per-container or any sort of network isolation), and so on. Like Openstack, Mesos brings its own set of new complications. Unlike Openstack, Mesos almost always increases efficiency of operations once it's in use. Openstack always seems to result in more and more effort spent on just operating Openstack...
    - kefka 10y ago
      Too true that Mesos isn't a drop-in replacement. Far from it. There is quite a lot of complexity as well in Mesos.. But looking at it, there are also less moving parts with Mesos compared to OpenStack. As a test, I set up a Mesos Cluster in 3 VM's. It wasn't terribly hard at all. But at work, we had an alarm thrown in nagios about critical ram consumption on a node. So I go and do the migrate. It works.... And it doesn't. It migrated the VM to the same node. I had to do 2 extra flags to tell OpenStack to migrate the node properly. That's just ridiculous, and dumb. And not to mention, but Mesos supports proper handling of VMs and migration. It frustrates me to no end the amount of hand-holding I have to do with anything OpenStack based. Whereas Mesos was "Install, configure, run".. With a large amount of work on Configure.
- SEJeff 10y ago
  Do you work at eBay, because they run kubernetes ontop of openstack, which sounds a bit awful if you really think about it all.
  - kefka 10y ago
    Not really. Kubernetes is stupidly unstable. When I was testing it a while back, the master processes kept dying for no good reason and leaving half-on processes with no networking. And then you get to restart everything. Yay. Whereas if Kubernetes is on something else (OpenStack/Mesos) you at least have control to restart Kubernetes when it shits its brick.
    - specialp 10y ago
      The "a while back" qualifier here is important. Both Mesos and Kubernetes are relatively new projects in a rapidly moving space. FWIW we are running Kubernetes 1.4x with multiple redundant masters in production with no problems. We are also running Mesos/Marathon internally. These projects did have major issues early on but now they are becoming much more robust. For control we are using CoreOS with kube-aws. It actually works quite nicely as we can scale and upgrade running clusters. The only issue with it right now is upgrading and managing etcd. I think the whole issue with OpenStack is the overambitious goal to be a one stop from bare metal servers and switches to applications. Kubernetes with Flannel or Calico can take care of the software level networking and provisioning, while there are a lot of other ways to provision VMs for running Mesos or Kubernetes depending on where you are deploying.
      - traf68 10y ago
        Kubernetes and the push for it is religious. Get out of here with that. Not all of us have been industry < 10 years
        SEJeff 10y ago
        I've been in "the industry" professionally since ~2005, and with Linux since 1998 or so. I have a business need for kubernetes and there is nothing at all religious about it. It is simply that we have lots of new applications that need to be spun up and deployed. They might need to scale up or scale down, and we have gobs and gobs of on-premise (read 0 cloud) new hardware. "API Driven" infrastructure is the best way to achieve a better story for the various ops teams, and so we're building out some kubernetes clusters globally for them. In other news: http://i0.kym-cdn.com/photos/images/facebook/001/044/247/297.png http://i0.kym-cdn.com/photos/images/facebook/001/044/247/297...
- praneshp 10y ago
  Huh, it's that bad for you? I used to work in the Openstack team at Yahoo, and thanks to the efforts of my colleagues, the first thing we got out of the way was upgrades. One of my friends made Anvil[0] We stayed with the community till the SVP decided it was time to move all of Yahoo to AWS. I always wonder if one of Openstack's weaknesses was also the same thing that was it's strength, it's openness and need to support everything, as well as remaining agnostic to users. Nearly every company I know needs to register VMs on a local DB of some sort, and has patches on Openstack to do that. But every time I went on IRC and asked if we should think about a nicer way of providing this hook, I'd either be redirected to a different project, or get some rude replies. I also wish there were some more modern companies behind the project; a lot of the developers were probably under pressure from penny pinchers in the management. That said, it was a terrific project to start my career on. [0]: https://github.com/openstack/anvil https://github.com/openstack/anvil
- justinhensley 10y ago
  Are you using TripleO/Red Hat Platform Director? Can you share the size of your OpenStack environment? When you say VMware 6.5, are you referring to ESXi aka vSphere Hypervisor? That is not a 1 for 1 replacement of OpenStack.
  - karlkatzke 10y ago
    No, we're not. Our environment is small; only about 100 VMs. We're a Debian shop, so no redhat. No, it's not a 1:1 replacement. It's a migration. It would also be a migration if we decided to upgrade OpenStack, because we'd be going Havana -> Ocata. However, there are defined upgrade paths for ESX, we already have ESXi/vSphere in our environment, and we can hire someone relatively inexpensively to help us with vSphere if we need to augment our staff. Folks who understand Openstack are not anywhere near as easy to find and pay.
- cr0sh 10y ago
  I worked for a company (Nobis Technology Group) that was acquired by LeaseWeb USA; I was a part of the "server automation" team and was somewhat reluctantly "forced out" of the team after the transition (they basically offered to pay me a smaller salary to work for them - do I look stupid?). The team was essentially dissolved several months later, as I knew it would be. But for the short period I was there (about 2 years), it was a great place to work, and a high point in my total career. Prior to that position, I had been doing web development in PHP pretty much exclusively. Doing server automation was a completely new space to me. Soon after starting, our team was tasked with a migration to OpenStack. Since our current infrastructure front-end was already based on PHP, I got tasked with looking into how or if we could use PHP OpenCloud to work with OpenStack. It seemed workable to me; I was able to extend the classes in such a way using namespaces and other techniques so that we could add additional capabilities to the interface that weren't already supported (and there were a lot of holes to fill!), but wouldn't break things if/when we had to upgrade OpenCloud (this was ultimately tested a couple times I was there, and the changes proved flawless - at least on the PHP side). Ultimately, we had a nice stable front-end that worked well with both our original system (some VM architecture that I forget) and the new "cloud" infrastructure based around OpenStack. In effect, a user could deploy either an actual server (if available), provision a VM (if a server was available), or build a cloud server "system" from a myriad of parts (we tried to support and provide as much access to the OpenStack stuff as we could). We also had a RESTful API for clients to also used (some of our clients resold our services under their own names). Some of the backend stuff was a bit "messy" in how it worked (I won't go into details, but I "authored" a fake "O'Reilly" "book" (really just a front "cover" mainly) whose mascot was dickbutt) - but despite the mess, overall it worked well, considering all the moving parts (where it would tend to fall down - not always, but enough - was when an upgrade to OpenStack was performed). In short - we were also one of the few companies running OpenStack in production. Our owners ended up selling to LeaseWeb, I left - but the idea was that LeaseWeb wanted to transition things to their API and system, and I honestly don't know what happened with all the work and such I was involved in on the PHP side of things (there was also a point where me and a coworker had to quickly ramp up and learn GoLang to make an interface from Rancher/Docker over to the Nobis API - that was a fun and interesting experience). I imagine that some portion is still running, but who knows. I personally think that in the right hands and with the right infrastructure OpenStack can be a very workable and working technology. It seemed to work well for the systems we used while I was at Nobis. I don't know honestly whether you could use it to scale up to anything like AWS or Google's offerings, but I think for medium-sized stuff like we were doing (or like Digital Ocean does - who at the time was our direct competition), it can work well - at least as I experienced things.
SSLy 10y ago
att runs it in-house and they seem to like it.
- praneshp 10y ago
  I was at their keynote in one of the Openstack summit. I can guarantee that at least the VP that spoke about it knows nothing about openstack, and were just using Mirantis to deploy. They just threw Buzzwords around, one of which was Fuel[0] [0]: https://www.mirantis.com/software/openstack/fuel/ https://www.mirantis.com/software/openstack/fuel/
  - AndyNemmity 10y ago
    That's VPs for you, I wouldn't want our understanding of openstack to be judged by VPs either...
    - praneshp 10y ago
      Oh, to be clear, his knowledge of Openstack was thin, even for a vp
p932 10y ago
Apparently RedHat does: http://www.serverwatch.com/server-news/red-hat-growing-million-dollar-openstack-deals.html http://www.serverwatch.com/server-news/red-hat-growing-milli... https://investors.redhat.com/~/media/Files/R/Red-Hat-IR/documents/events/q4-2016-earnings-call-prep-remarks.pdf https://investors.redhat.com/~/media/Files/R/Red-Hat-IR/docu...
AndyNemmity 10y ago
We're running Openstack with Kubernetes and it's working quite well. Lots of challenges though, and we're just exiting a preview phase into production.
- kiranmova 10y ago
  How are you deploying? Are you using Tectonic? https://coreos.com/openstack/ https://coreos.com/openstack/
justinhensley 10y ago
I have seen several successful implementations in the 18 months I've been working with OpenStack. There are definitely gotchas and low level (driver, hardware)issues you have to deal with. I never thought I would be so familiar with IPMI. That said, OpenStack is an ambitious product. It essentially seeks to give you a lot of the functionality of AWS, except with on-premise infrastructure. That is a tall order. AWS is a multi-billion dollar effort that I assume has has far more engineers than the OpenStack project. OpenStack is a complex solution to a very complex problem. Having worked with other commercial IaaS solutions, I will say I hands down prefer OpenStack to the others I've worked with. If there is a private cloud solution that delivers the same/similar functionality of OpenStack with less admin overhead and more simplicity, then I want to hear about it.

I'm at one of the only companies I know of outside Rackspace that's running it in production. We're doing a very poor job of it; we never hit production with any effort to upgrade the cluster in the past three years, so we're still running Havana with Nova Network. The most likely upgrade path for us right now is VMWare 6.5.

kefka 10y ago
So are we. And so is another department in our org. Frankly, we're doing a piss poor job of it. Migrations don't happen even if manually ran, the compute nodes are already at capacity, even with the meagre VMs we use. I have kept pushing for Mesos myself. I see it as a good way forward. Openstack may be a technical good body of code... But in the end its too convoluted codebase and many moving parts to be effective in most deployments.
- benley 10y ago
  Mesos isn't really a direct replacement for Openstack, but you're probably still right to be pushing for it. If you roll with Mesos you will still need to solve a bunch of lower-level infrastructure challenges: machine provisioning, basic system config management, storage management (if you want persistent volumes), fancy networking (if you want ip-per-container or any sort of network isolation), and so on. Like Openstack, Mesos brings its own set of new complications. Unlike Openstack, Mesos almost always increases efficiency of operations once it's in use. Openstack always seems to result in more and more effort spent on just operating Openstack...
  - kefka 10y ago
    Too true that Mesos isn't a drop-in replacement. Far from it. There is quite a lot of complexity as well in Mesos.. But looking at it, there are also less moving parts with Mesos compared to OpenStack. As a test, I set up a Mesos Cluster in 3 VM's. It wasn't terribly hard at all. But at work, we had an alarm thrown in nagios about critical ram consumption on a node. So I go and do the migrate. It works.... And it doesn't. It migrated the VM to the same node. I had to do 2 extra flags to tell OpenStack to migrate the node properly. That's just ridiculous, and dumb. And not to mention, but Mesos supports proper handling of VMs and migration. It frustrates me to no end the amount of hand-holding I have to do with anything OpenStack based. Whereas Mesos was "Install, configure, run".. With a large amount of work on Configure.
SEJeff 10y ago
Do you work at eBay, because they run kubernetes ontop of openstack, which sounds a bit awful if you really think about it all.
- kefka 10y ago
  Not really. Kubernetes is stupidly unstable. When I was testing it a while back, the master processes kept dying for no good reason and leaving half-on processes with no networking. And then you get to restart everything. Yay. Whereas if Kubernetes is on something else (OpenStack/Mesos) you at least have control to restart Kubernetes when it shits its brick.
  - specialp 10y ago
    The "a while back" qualifier here is important. Both Mesos and Kubernetes are relatively new projects in a rapidly moving space. FWIW we are running Kubernetes 1.4x with multiple redundant masters in production with no problems. We are also running Mesos/Marathon internally. These projects did have major issues early on but now they are becoming much more robust. For control we are using CoreOS with kube-aws. It actually works quite nicely as we can scale and upgrade running clusters. The only issue with it right now is upgrading and managing etcd. I think the whole issue with OpenStack is the overambitious goal to be a one stop from bare metal servers and switches to applications. Kubernetes with Flannel or Calico can take care of the software level networking and provisioning, while there are a lot of other ways to provision VMs for running Mesos or Kubernetes depending on where you are deploying.
    - traf68 10y ago
      Kubernetes and the push for it is religious. Get out of here with that. Not all of us have been industry < 10 years
      - SEJeff 10y ago
        I've been in "the industry" professionally since ~2005, and with Linux since 1998 or so. I have a business need for kubernetes and there is nothing at all religious about it. It is simply that we have lots of new applications that need to be spun up and deployed. They might need to scale up or scale down, and we have gobs and gobs of on-premise (read 0 cloud) new hardware. "API Driven" infrastructure is the best way to achieve a better story for the various ops teams, and so we're building out some kubernetes clusters globally for them. In other news: http://i0.kym-cdn.com/photos/images/facebook/001/044/247/297.png http://i0.kym-cdn.com/photos/images/facebook/001/044/247/297...
praneshp 10y ago
Huh, it's that bad for you? I used to work in the Openstack team at Yahoo, and thanks to the efforts of my colleagues, the first thing we got out of the way was upgrades. One of my friends made Anvil[0] We stayed with the community till the SVP decided it was time to move all of Yahoo to AWS. I always wonder if one of Openstack's weaknesses was also the same thing that was it's strength, it's openness and need to support everything, as well as remaining agnostic to users. Nearly every company I know needs to register VMs on a local DB of some sort, and has patches on Openstack to do that. But every time I went on IRC and asked if we should think about a nicer way of providing this hook, I'd either be redirected to a different project, or get some rude replies. I also wish there were some more modern companies behind the project; a lot of the developers were probably under pressure from penny pinchers in the management. That said, it was a terrific project to start my career on. [0]: https://github.com/openstack/anvil https://github.com/openstack/anvil
justinhensley 10y ago
Are you using TripleO/Red Hat Platform Director? Can you share the size of your OpenStack environment? When you say VMware 6.5, are you referring to ESXi aka vSphere Hypervisor? That is not a 1 for 1 replacement of OpenStack.
- karlkatzke 10y ago
  No, we're not. Our environment is small; only about 100 VMs. We're a Debian shop, so no redhat. No, it's not a 1:1 replacement. It's a migration. It would also be a migration if we decided to upgrade OpenStack, because we'd be going Havana -> Ocata. However, there are defined upgrade paths for ESX, we already have ESXi/vSphere in our environment, and we can hire someone relatively inexpensively to help us with vSphere if we need to augment our staff. Folks who understand Openstack are not anywhere near as easy to find and pay.
cr0sh 10y ago
I worked for a company (Nobis Technology Group) that was acquired by LeaseWeb USA; I was a part of the "server automation" team and was somewhat reluctantly "forced out" of the team after the transition (they basically offered to pay me a smaller salary to work for them - do I look stupid?). The team was essentially dissolved several months later, as I knew it would be. But for the short period I was there (about 2 years), it was a great place to work, and a high point in my total career. Prior to that position, I had been doing web development in PHP pretty much exclusively. Doing server automation was a completely new space to me. Soon after starting, our team was tasked with a migration to OpenStack. Since our current infrastructure front-end was already based on PHP, I got tasked with looking into how or if we could use PHP OpenCloud to work with OpenStack. It seemed workable to me; I was able to extend the classes in such a way using namespaces and other techniques so that we could add additional capabilities to the interface that weren't already supported (and there were a lot of holes to fill!), but wouldn't break things if/when we had to upgrade OpenCloud (this was ultimately tested a couple times I was there, and the changes proved flawless - at least on the PHP side). Ultimately, we had a nice stable front-end that worked well with both our original system (some VM architecture that I forget) and the new "cloud" infrastructure based around OpenStack. In effect, a user could deploy either an actual server (if available), provision a VM (if a server was available), or build a cloud server "system" from a myriad of parts (we tried to support and provide as much access to the OpenStack stuff as we could). We also had a RESTful API for clients to also used (some of our clients resold our services under their own names). Some of the backend stuff was a bit "messy" in how it worked (I won't go into details, but I "authored" a fake "O'Reilly" "book" (really just a front "cover" mainly) whose mascot was dickbutt) - but despite the mess, overall it worked well, considering all the moving parts (where it would tend to fall down - not always, but enough - was when an upgrade to OpenStack was performed). In short - we were also one of the few companies running OpenStack in production. Our owners ended up selling to LeaseWeb, I left - but the idea was that LeaseWeb wanted to transition things to their API and system, and I honestly don't know what happened with all the work and such I was involved in on the PHP side of things (there was also a point where me and a coworker had to quickly ramp up and learn GoLang to make an interface from Rancher/Docker over to the Nobis API - that was a fun and interesting experience). I imagine that some portion is still running, but who knows. I personally think that in the right hands and with the right infrastructure OpenStack can be a very workable and working technology. It seemed to work well for the systems we used while I was at Nobis. I don't know honestly whether you could use it to scale up to anything like AWS or Google's offerings, but I think for medium-sized stuff like we were doing (or like Digital Ocean does - who at the time was our direct competition), it can work well - at least as I experienced things.

SSLy 10y ago

att runs it in-house and they seem to like it.

praneshp 10y ago
I was at their keynote in one of the Openstack summit. I can guarantee that at least the VP that spoke about it knows nothing about openstack, and were just using Mirantis to deploy. They just threw Buzzwords around, one of which was Fuel[0] [0]: https://www.mirantis.com/software/openstack/fuel/ https://www.mirantis.com/software/openstack/fuel/
- AndyNemmity 10y ago
  That's VPs for you, I wouldn't want our understanding of openstack to be judged by VPs either...
  - praneshp 10y ago
    Oh, to be clear, his knowledge of Openstack was thin, even for a vp

p932 10y ago

Apparently RedHat does: http://www.serverwatch.com/server-news/red-hat-growing-million-dollar-openstack-deals.html http://www.serverwatch.com/server-news/red-hat-growing-milli... https://investors.redhat.com/~/media/Files/R/Red-Hat-IR/documents/events/q4-2016-earnings-call-prep-remarks.pdf https://investors.redhat.com/~/media/Files/R/Red-Hat-IR/docu...

AndyNemmity 10y ago

We're running Openstack with Kubernetes and it's working quite well. Lots of challenges though, and we're just exiting a preview phase into production.

kiranmova 10y ago
How are you deploying? Are you using Tectonic? https://coreos.com/openstack/ https://coreos.com/openstack/

justinhensley 10y ago

I have seen several successful implementations in the 18 months I've been working with OpenStack. There are definitely gotchas and low level (driver, hardware)issues you have to deal with. I never thought I would be so familiar with IPMI. That said, OpenStack is an ambitious product. It essentially seeks to give you a lot of the functionality of AWS, except with on-premise infrastructure. That is a tall order. AWS is a multi-billion dollar effort that I assume has has far more engineers than the OpenStack project. OpenStack is a complex solution to a very complex problem. Having worked with other commercial IaaS solutions, I will say I hands down prefer OpenStack to the others I've worked with. If there is a private cloud solution that delivers the same/similar functionality of OpenStack with less admin overhead and more simplicity, then I want to hear about it.