7 ms·

> The Guardian newspaper tried building their own private cloud and failed What I see often is that it's hard and costly to replicate the whole cloud experienc

by krab 5y ago

> The Guardian newspaper tried building their own private cloud and failed What I see often is that it's hard and costly to replicate the whole cloud experience in-house. OpenStack is a behemoth to operate. However, if you build something more specific to your needs (and often mix in selected public cloud services), you'll get a much cheaper stack. Small businesses can often run very efficiently with a few rented physical servers in a colo, G Suite and AWS S3. Larger ones can provide a platform specific to the technologies used in-house and combine with public cloud for any other tech stacks.

Spivak 5y ago
Why are you recommending S3 when Ceph exists? You almost literally just throw hard drives at it and it gives you remote block devices for VMs, a NFS style shared FS, and an S3 client compatible object store.
- Eikon 5y ago
  What? Operating a small ceph cluster is almost a full time job, at least if you care about your data...
  - Spivak 5y ago
    People always think Ceph is super complicated but I promise you it isn't. I worked at a place where my (3 person) team and I operated a 1 PB Ceph cluster replicated across two datacenters along with literally every other service needed to run a production web app. It was one of the lowest maintenance services we deployed -- Kafka and our RabbitMQ clusters were the things that always gave us trouble.
    - krab 5y ago
      I was on a team that had a tiny (a few GBs) Ceph cluster because it was the easiest option for high availability of these files that we really wanted to have on premise. There were never any issues so I can believe what you're saying. Still, for the amount of data I work with, the cloud storage usually wins.
- krab 5y ago
  Others may have other preferences. This is what worked for me. I just find S3 quite cheap for storing a few (tens of) terabytes plus some glacier archives. For data I often want multiple locations redundancy and encryption. That gets more complicated than what I can save. I guess the equation changes when you have more data or looser requirements. That's actually my point. It's very hard to recreate the whole AWS experience, but it's quite possible to pick any piece and do it much cheaper.
laura2013 5y ago
what about openstack makes it a behemoth to operate?
- Spivak 5y ago
  It's basically consultingware for IBM/Redhat. It tries to be everything you could ever possibly need, the interactions between the components are brittle, and it has all the complexity of a huge k8s deployment except your ability to affect what's going on underneath is like trying to control an angry beehive with a 20 ft pole. Think Urbit but for sysadmins and you pretty much get the vibe.
- krab 5y ago
  It tries to do pretty much everything. Just my opinion.
- redwood 5y ago
  Where to even begin? It's like assuming a car is easy to operate that you have to build from parts
- mschuster91 5y ago
  Ever tried installing it? It's an incredibly complex piece of work with boatloads of services to configure and debug... and upgrade. At the time I last tried it (some two or three years ago): - most of the documentation was written assuming an rpm-based distro and not Ubuntu or Debian (I remember especially having had trouble with subtle differences in iptables/nftables between the distributions) - on top of that it was right in the middle of a python2 => 3 migration with the result of shit breaking left and right - it was/still is written in Python which means a second or more for every CLI command to initialize, much less actually do something - half the instructions in the documentation seem to never have been actually tested, which combined with the two points above made for a really punishing "experience", especially with more rarely-used features such as k8s-in-openstack - the documentation itself still is (just looked it up) badly organized, with you needing to read four guides (Install, User, Configuration, Ops/Administration) per OpenStack component. Ideally, there should only be one guide for Installation/Upgrade (including Configuration and everything from Ops/Admin) that has everything in it to get all supported parts of the component up and running, one User's Guide that shows how the component is used and what the best practices are, and one Troubleshooting guide. If you can set it up into a working configuration, it's a nice project for sure. But the road to that point is rough, and you will likely not even want to think about an upgrade simply because of how complex it is.
tootie 5y ago
I worked at a place that had a large, underutilized data center. The DevOps team was stuck in the past and operated too slowly to provision capacity when it was needed so a lot of it was just fallow. They had spent months and months working with a vendor to get these into a k8s fleet, but I never saw the outcome. The devs teams in my office were just throwing shit up on Azure or even Heroku and just not asking permission. We got substantial increases in revenue and the incremental cost of hosting was negligible.
- krab 5y ago
  I've been there as well (modulo cloud brand to which we moved). I understand this reason for moving to cloud but it makes me sad. It worked like this - you had to fill a form to request a VM. After a few emails (and days because time zones), you got a few years old RHEL VM without root access, but also without any real support from the ops organization. They tried to provide the generic cloud experience with VMs and failed.
hodgesrm 5y ago
Most companies that I see operating private clouds are using Kubernetes. It's much more flexible than OpenStack. It also has the advantage that apps are more or less portable across private/public cloud infrastructure.
AtlasBarfed 5y ago
The comments here are largely unquantified opinions that haven't really changed in 10 years or so, more that there are far more people that have those opinions. Ten years ago I loved the cloud because it broke the monopoly / organizational dictatorship of IT in large organizations, always a wellspring of "daily wtfs". But now the cloud is becoming too powerful, and too small in number. We NEED a competitive-ish on-prems approach... but... what is it? Does anyone have a template to do so? They aren't well advertised that's for sure. AWS is NOT CHEAP. It may be slightly cheaper than the old monolith IT orgs' DCs, and far more flexible, but it's not cheap. There has to be a way to accept less flexibility for substantial savings over reserved instances and negotiated discounts. For christsakes, a Raspberry Pi would make a perfectly good server for massive ranges of use cases. 20$ and ultra low power.
- originalvichy 5y ago
  The reason there are many unattractive alternatives that are underused is the fact that they are unattractive. Nothing has probably changed in years in the traditional server renting space.

by krab 5y ago

Spivak 5y ago

Why are you recommending S3 when Ceph exists? You almost literally just throw hard drives at it and it gives you remote block devices for VMs, a NFS style shared FS, and an S3 client compatible object store.

Eikon 5y ago
What? Operating a small ceph cluster is almost a full time job, at least if you care about your data...
- Spivak 5y ago
  People always think Ceph is super complicated but I promise you it isn't. I worked at a place where my (3 person) team and I operated a 1 PB Ceph cluster replicated across two datacenters along with literally every other service needed to run a production web app. It was one of the lowest maintenance services we deployed -- Kafka and our RabbitMQ clusters were the things that always gave us trouble.
  - krab 5y ago
    I was on a team that had a tiny (a few GBs) Ceph cluster because it was the easiest option for high availability of these files that we really wanted to have on premise. There were never any issues so I can believe what you're saying. Still, for the amount of data I work with, the cloud storage usually wins.
krab 5y ago
Others may have other preferences. This is what worked for me. I just find S3 quite cheap for storing a few (tens of) terabytes plus some glacier archives. For data I often want multiple locations redundancy and encryption. That gets more complicated than what I can save. I guess the equation changes when you have more data or looser requirements. That's actually my point. It's very hard to recreate the whole AWS experience, but it's quite possible to pick any piece and do it much cheaper.

laura2013 5y ago

what about openstack makes it a behemoth to operate?

Spivak 5y ago
It's basically consultingware for IBM/Redhat. It tries to be everything you could ever possibly need, the interactions between the components are brittle, and it has all the complexity of a huge k8s deployment except your ability to affect what's going on underneath is like trying to control an angry beehive with a 20 ft pole. Think Urbit but for sysadmins and you pretty much get the vibe.
krab 5y ago
It tries to do pretty much everything. Just my opinion.
redwood 5y ago
Where to even begin? It's like assuming a car is easy to operate that you have to build from parts
mschuster91 5y ago
Ever tried installing it? It's an incredibly complex piece of work with boatloads of services to configure and debug... and upgrade. At the time I last tried it (some two or three years ago): - most of the documentation was written assuming an rpm-based distro and not Ubuntu or Debian (I remember especially having had trouble with subtle differences in iptables/nftables between the distributions) - on top of that it was right in the middle of a python2 => 3 migration with the result of shit breaking left and right - it was/still is written in Python which means a second or more for every CLI command to initialize, much less actually do something - half the instructions in the documentation seem to never have been actually tested, which combined with the two points above made for a really punishing "experience", especially with more rarely-used features such as k8s-in-openstack - the documentation itself still is (just looked it up) badly organized, with you needing to read four guides (Install, User, Configuration, Ops/Administration) per OpenStack component. Ideally, there should only be one guide for Installation/Upgrade (including Configuration and everything from Ops/Admin) that has everything in it to get all supported parts of the component up and running, one User's Guide that shows how the component is used and what the best practices are, and one Troubleshooting guide. If you can set it up into a working configuration, it's a nice project for sure. But the road to that point is rough, and you will likely not even want to think about an upgrade simply because of how complex it is.

tootie 5y ago

I worked at a place that had a large, underutilized data center. The DevOps team was stuck in the past and operated too slowly to provision capacity when it was needed so a lot of it was just fallow. They had spent months and months working with a vendor to get these into a k8s fleet, but I never saw the outcome. The devs teams in my office were just throwing shit up on Azure or even Heroku and just not asking permission. We got substantial increases in revenue and the incremental cost of hosting was negligible.

krab 5y ago
I've been there as well (modulo cloud brand to which we moved). I understand this reason for moving to cloud but it makes me sad. It worked like this - you had to fill a form to request a VM. After a few emails (and days because time zones), you got a few years old RHEL VM without root access, but also without any real support from the ops organization. They tried to provide the generic cloud experience with VMs and failed.

hodgesrm 5y ago

Most companies that I see operating private clouds are using Kubernetes. It's much more flexible than OpenStack. It also has the advantage that apps are more or less portable across private/public cloud infrastructure.

AtlasBarfed 5y ago

The comments here are largely unquantified opinions that haven't really changed in 10 years or so, more that there are far more people that have those opinions. Ten years ago I loved the cloud because it broke the monopoly / organizational dictatorship of IT in large organizations, always a wellspring of "daily wtfs". But now the cloud is becoming too powerful, and too small in number. We NEED a competitive-ish on-prems approach... but... what is it? Does anyone have a template to do so? They aren't well advertised that's for sure. AWS is NOT CHEAP. It may be slightly cheaper than the old monolith IT orgs' DCs, and far more flexible, but it's not cheap. There has to be a way to accept less flexibility for substantial savings over reserved instances and negotiated discounts. For christsakes, a Raspberry Pi would make a perfectly good server for massive ranges of use cases. 20$ and ultra low power.

originalvichy 5y ago
The reason there are many unattractive alternatives that are underused is the fact that they are unattractive. Nothing has probably changed in years in the traditional server renting space.