3 ms·

I hope this is a signal that a third cloud option, BYOC (build your own cloud), is finally becoming practical. Yes, the physical management of racks is a massiv

by jmward01 11mo ago

I hope this is a signal that a third cloud option, BYOC (build your own cloud), is finally becoming practical. Yes, the physical management of racks is a massive part of managing a cloud but the software stack is honestly why AWS and the like are winning much of the time, at least for the small use cases I have been a part of. I priced out some medium servers and the cost of buying enough for load plus extras for fail over, and host them, was -way- under AWS and other cloud vendors (these were GPU loads) but the management of them was the issue. 'just spin up an instance...' is such a massive enabler for ideas. Something that gives me a viable software stack to build my own cloud on easily is a huge win for abandoning the major cloud vendors. Keep it coming!

sekh60 11mo ago
What about OpenStack, or even CloudStack?
- written-beyond 11mo ago
  PLEASE DON'T DOWN VOTE ME TO HELL THIS IS A DISCLAIMER I AM JUST SHARING WHAT I'VE READ I AM NOT CLAIMING THEM AS FACTS. ...ahem... When I was researching about this a few years ago I read some really long in-depth scathing posts about Open stack. One of them explicitly called it a childish set of glued together python scripts that fall apart very quickly when you get off the happy path. OTH opinions on Proxmox were very measured.
  - 0x457 11mo ago
    > When I was researching about this a few years ago I read some really long in-depth scathing posts about Open stack. One of them explicitly called it a childish set of glued together python scripts that fall apart very quickly when you get off the happy path. And according to every ex-Amazoner I've ment: the core of AWS is a bunch of Perl scripts glued together
    - apple4ever 11mo ago
      That explains a lot
    - ikiris 11mo ago
      It doesn't matter when there's an entire amazon staff keeping it running.
      - estimator7292 11mo ago
        I think you know as well as I do that it very much does matter. Even if you have an army of engineers around to fix things when they break, things still break.
        Spivak 11mo ago
        I think the point is that for Amazon it's their own code and they pay full time staff to be familiar with the codebase, make improvements, and fix bugs. OpenStack is a product. The people deploying it are expected to be knowledgeable about it as users / "system integrators" but not developers. So when the abstraction leaks, and for OpenStack the pipe has all but burst, it becomes a mess. It's not expected that they'll be digging around in the internals and have 5 other projects to work on.
  - MrDarcy 11mo ago
    This matches my personal experience having worked with OpenStack.
  - dangus 11mo ago
    Yeah, I think that makes solutions like Proxmox better is that there’s no reason to try and copy Amazon’s public cloud on your own could. I find that the main paradigms are: 1. Run something in a VM 2. Run something on in a container (docker compose on portainer or something similar) 3. Run a Kubernetes cluster. Then if you need something that Amazon offers you don’t implement it like open stack, you just run that specific service on options #1-3.
    - written-beyond 11mo ago
      I think the utility really comes from getting an accessible control plane over your company's data centers/server rack. Kubernetes clusters doesn't really solve the storage plane issue, or a unified dashboard for users to interact with it easily. Something like harvester is pretty close IMO to getting a kubernetes alternative to Proxmox/open cloud.
  - throw0101a 11mo ago
    > One of them explicitly called it a childish set of glued together python scripts that fall apart very quickly when you get off the happy path. A 'childish set scripts' that manages (as of 2020) a few hundreds of thousands of cores, 7,700 hypervisors, and 54,000 VMs at CERN: * https://superuser.openinfra.org/articles/cern-openstack-update/ https://superuser.openinfra.org/articles/cern-openstack-upda... The Proxmox folks themselves know (as of 2023) of Proxmox clusters as large as 51 nodes: * https://forum.proxmox.com/threads/the-maximum-number-of-nodes-supported-by-the-pve-host-cluster.123507/ https://forum.proxmox.com/threads/the-maximum-number-of-node... So what scale do you need?
    - lucyjojo 11mo ago
      vast vast (vaaast) majority of businesses are in that 1-100 nodes range.
      - throw0101d 11mo ago
        > vast vast (vaaast) majority of businesses are in that 1-100 nodes range. Yes, but even the Proxmox folks themselves say the most they've seen is 51: * https://forum.proxmox.com/threads/the-maximum-number-of-nodes-supported-by-the-pve-host-cluster.123507/ https://forum.proxmox.com/threads/the-maximum-number-of-node... I'm happily running some Proxmox now, and wouldn't want to got more than a dozen hypervisor or so. At least not in one cluster: that's partially what PDM 1.0 is probably about. I have run OpenStack with many dozens of hypervisors (plus dedicated, non-hyperconverged Ceph servers) though.
    - pezezin 11mo ago
      CERN is the biggest scientific facility in the world, with a huge IT group and their own IXP. Most places are not like that. Heck, I work at a much smaller particle accelerator (https://ifmif.org https://ifmif.org) and have met the CERN guys, and they were the first to say that for our needs, OpenStack is absolutely overkill.
      - throw0101d 11mo ago
  - SEJeff 11mo ago
    The reason there were so many commercial distributions of open stack was because setting it up reliably end to end was nearly impossible for most mere mortals. Company’s like meta cloud or mirantis made a ton of money with little more than openstack installers and a good out of the box default config with some solid monitoring and management tooling
- chrisandchris 11mo ago
  I think the main selling point for SME (wtih a small IT team) is that Proxmox is very easy to setup (download iso, install debian, ready to go). CloudStack seems to require a lot of work just to get it running: https://docs.cloudstack.apache.org/en/latest/quickinstallationguide/qig.html https://docs.cloudstack.apache.org/en/latest/quickinstallati... Maybe I'm wrong - but where I am from, companies with less than 500 employees are like 95% of the workforce of the country. That's big enough for a small cluster (in-house/colocation), but to small for something bigger.
  - jmward01 11mo ago
    Yeah. The keys here are 'easy' and 'I can play with it at home first'. Let's be honest, being able to throw together a bunch of old dead boxes and put proxmox on them in a weekend is a game changer for a learning curve.
    - doubled112 11mo ago
      The main reason I never tried OpenStack was that the official requirements were more than I had in my home VM host, and I couldn't figure out if the hardware requirements were real or suggested. Proxmox has very little overhead. I've since moved to Incus. There are some really decent options out there, although Incus still has some gaps in the functionality Proxmox fills out of the box.

sekh60 11mo ago

What about OpenStack, or even CloudStack?

written-beyond 11mo ago
PLEASE DON'T DOWN VOTE ME TO HELL THIS IS A DISCLAIMER I AM JUST SHARING WHAT I'VE READ I AM NOT CLAIMING THEM AS FACTS. ...ahem... When I was researching about this a few years ago I read some really long in-depth scathing posts about Open stack. One of them explicitly called it a childish set of glued together python scripts that fall apart very quickly when you get off the happy path. OTH opinions on Proxmox were very measured.
- 0x457 11mo ago
  > When I was researching about this a few years ago I read some really long in-depth scathing posts about Open stack. One of them explicitly called it a childish set of glued together python scripts that fall apart very quickly when you get off the happy path. And according to every ex-Amazoner I've ment: the core of AWS is a bunch of Perl scripts glued together
  - apple4ever 11mo ago
    That explains a lot
  - ikiris 11mo ago
    It doesn't matter when there's an entire amazon staff keeping it running.
    - estimator7292 11mo ago
      I think you know as well as I do that it very much does matter. Even if you have an army of engineers around to fix things when they break, things still break.
      - Spivak 11mo ago
        I think the point is that for Amazon it's their own code and they pay full time staff to be familiar with the codebase, make improvements, and fix bugs. OpenStack is a product. The people deploying it are expected to be knowledgeable about it as users / "system integrators" but not developers. So when the abstraction leaks, and for OpenStack the pipe has all but burst, it becomes a mess. It's not expected that they'll be digging around in the internals and have 5 other projects to work on.
- MrDarcy 11mo ago
  This matches my personal experience having worked with OpenStack.
- dangus 11mo ago
  Yeah, I think that makes solutions like Proxmox better is that there’s no reason to try and copy Amazon’s public cloud on your own could. I find that the main paradigms are: 1. Run something in a VM 2. Run something on in a container (docker compose on portainer or something similar) 3. Run a Kubernetes cluster. Then if you need something that Amazon offers you don’t implement it like open stack, you just run that specific service on options #1-3.
  - written-beyond 11mo ago
    I think the utility really comes from getting an accessible control plane over your company's data centers/server rack. Kubernetes clusters doesn't really solve the storage plane issue, or a unified dashboard for users to interact with it easily. Something like harvester is pretty close IMO to getting a kubernetes alternative to Proxmox/open cloud.
- throw0101a 11mo ago
  > One of them explicitly called it a childish set of glued together python scripts that fall apart very quickly when you get off the happy path. A 'childish set scripts' that manages (as of 2020) a few hundreds of thousands of cores, 7,700 hypervisors, and 54,000 VMs at CERN: * https://superuser.openinfra.org/articles/cern-openstack-update/ https://superuser.openinfra.org/articles/cern-openstack-upda... The Proxmox folks themselves know (as of 2023) of Proxmox clusters as large as 51 nodes: * https://forum.proxmox.com/threads/the-maximum-number-of-nodes-supported-by-the-pve-host-cluster.123507/ https://forum.proxmox.com/threads/the-maximum-number-of-node... So what scale do you need?
  - lucyjojo 11mo ago
    vast vast (vaaast) majority of businesses are in that 1-100 nodes range.
    - throw0101d 11mo ago
      > vast vast (vaaast) majority of businesses are in that 1-100 nodes range. Yes, but even the Proxmox folks themselves say the most they've seen is 51: * https://forum.proxmox.com/threads/the-maximum-number-of-nodes-supported-by-the-pve-host-cluster.123507/ https://forum.proxmox.com/threads/the-maximum-number-of-node... I'm happily running some Proxmox now, and wouldn't want to got more than a dozen hypervisor or so. At least not in one cluster: that's partially what PDM 1.0 is probably about. I have run OpenStack with many dozens of hypervisors (plus dedicated, non-hyperconverged Ceph servers) though.
  - pezezin 11mo ago
    CERN is the biggest scientific facility in the world, with a huge IT group and their own IXP. Most places are not like that. Heck, I work at a much smaller particle accelerator (https://ifmif.org https://ifmif.org) and have met the CERN guys, and they were the first to say that for our needs, OpenStack is absolutely overkill.
    - throw0101d 11mo ago
- SEJeff 11mo ago
  The reason there were so many commercial distributions of open stack was because setting it up reliably end to end was nearly impossible for most mere mortals. Company’s like meta cloud or mirantis made a ton of money with little more than openstack installers and a good out of the box default config with some solid monitoring and management tooling
chrisandchris 11mo ago
I think the main selling point for SME (wtih a small IT team) is that Proxmox is very easy to setup (download iso, install debian, ready to go). CloudStack seems to require a lot of work just to get it running: https://docs.cloudstack.apache.org/en/latest/quickinstallationguide/qig.html https://docs.cloudstack.apache.org/en/latest/quickinstallati... Maybe I'm wrong - but where I am from, companies with less than 500 employees are like 95% of the workforce of the country. That's big enough for a small cluster (in-house/colocation), but to small for something bigger.
- jmward01 11mo ago
  Yeah. The keys here are 'easy' and 'I can play with it at home first'. Let's be honest, being able to throw together a bunch of old dead boxes and put proxmox on them in a weekend is a game changer for a learning curve.
  - doubled112 11mo ago
    The main reason I never tried OpenStack was that the official requirements were more than I had in my home VM host, and I couldn't figure out if the hardware requirements were real or suggested. Proxmox has very little overhead. I've since moved to Incus. There are some really decent options out there, although Incus still has some gaps in the functionality Proxmox fills out of the box.