3 ms·

I was recently going to set up Proxmox as well, since I had used it ~7 years ago and it was working well. But I ran into an expectation that I have a drive ded

I was recently going to set up Proxmox as well, since I had used it ~7 years ago and it was working well. But I ran into an expectation that I have a drive dedicated to putting ISOs on, in addition to the OS install and the virtual machine storage. I was trying to set it up on a SFF machine and was limited in what I could put into it and I just kind of noped out of it. I couldn't install any VMs until I did this. I don't recall this being an issue before. I ended up re-installing TrueNAS Scale beta, which I had installed and tried, but ran into a problem. Ended up finding the solution on the forums, something related to bridged or VLAN interfaces not coming up. In the end I probably will just install Ubuntu and put Ganeti on these boxes, since that's been really reliable for me at work. I was actually hoping to try out Proxmox to possibly move to at work.

ylk 6y ago
What made you think that you need a separate disk just for ISOs? They can be uploaded to and stored on the default directory storage.
egberts1 6y ago
I snapped up an old Dell T710 2-socket, 12 CPU, 24-thread Xeon rack, PERc 6 RAID for $100, populated with cheap Seagate 2TB IronWolf HDs, leftover 1033Mz DDR3 to 96GB for 3 RAID HD groups (RAID 1 for OS, RAID5 for data, and JBOD for VM images.) Has a pfSense/FreeBSD VM for a gateway and a bunch of VMs on a detached bridge. Also have several more bridges for test VMs for real-world airgapped network setup. 8 different OS distros. Also am tweaking a MIPS and a M1 QEMUs outside of Proxmox . runs very quiet. so now it’s my home datacenter.
- joshspankit 6y ago
  I have something very similar, but an Optiplex. One thing I had to do recently though is swap pfsense for Untangle because of drastically better FQ-CoDel implementation in Untangle.
  - egberts1 6y ago
    I went over to Untangle website for any data sheet but was unable to get one immediately (have to fork over everything but your youngest child, so to speak in order to get it. Not sure why would one even do that. :-(
    - joshspankit 6y ago
      I don’t want to sidetrack this in to an Untangle conversation, but yes: it’s fundamentally not free as in speech while also being supported as though it’s just a thin shell over Debian. However, the YouTube breakdown by Battle(non)sense was conclusive (and is confirmed in my case in undeniable ways) so I switched.
m463 6y ago
(I think) I ran into the same or similar problem you did - there's a partition for the debian OS + iso images of a fixed size (df shows it as /dev/mapper/pce-root -> / ) and all the reallocatable storage used by the VMs comes out of the rest. Thing is it's hard to re-allocate this partition, and that's where all the iso images, and the backups go. I tried to migrate a vm and boom - it filled up. I tried to figure out how to reallocate it, but in the end, I found this (sort of hacky) solution documented: https://pve.proxmox.com/wiki/Logical_Volume_Manager_(LVM) https://pve.proxmox.com/wiki/Logical_Volume_Manager_(LVM) basically: # lvcreate -n vz -V 200G pve/data # mkfs.ext4 /dev/pve/vz # echo '/dev/pve/vz /var/lib/vz ext4 defaults 0 2' >> /etc/fstab you migrate the stuff from /var/lib/vz to the new location and you have 200G for iso images/etc Why this has to fight for storage from the root system, or why it's not easily expandable is a head-scratcher. I think it might be better to have a dedicated boot debian partition, a resizable/easily growable pve partition, then storage that can be allocated to vms/containers. but yeah - that stopped me in my tracks, and when I'm doing it on a home machine you just don't have the time to chase this stuff down.
- egberts1 6y ago
  Yep. so glad that i experimented (over a week) with Proxmox install over an existing Denuvan (Debian SysV init) install. Made more sense to have a second drive or second but largest LVM to play with and store all things VM. So confident that I disabled the Proxmox 'local' drive.
aaronius 6y ago
If you install Proxmox VE the next time and are not happy with the default layout, check out the parameters for LVM [0]. I personally prefer to install on ZFS as all ZFS datasets share the same free space unlike with LVM. If you want full control of the disk layout and maybe want to do some more custom stuff, consider installing a base Debian and then install Proxmox VE on top of it [1]. The PVE installer makes it easy to get a system up and running quickly but is opinionated what the disk layout should be. That of course might not be working for everyone. [0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#advanced_lvm_options https://pve.proxmox.com/pve-docs/pve-admin-guide.html#advanc... [1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_install_proxmox_ve_on_debian https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_insta...

ylk 6y ago

What made you think that you need a separate disk just for ISOs? They can be uploaded to and stored on the default directory storage.

egberts1 6y ago

I snapped up an old Dell T710 2-socket, 12 CPU, 24-thread Xeon rack, PERc 6 RAID for $100, populated with cheap Seagate 2TB IronWolf HDs, leftover 1033Mz DDR3 to 96GB for 3 RAID HD groups (RAID 1 for OS, RAID5 for data, and JBOD for VM images.) Has a pfSense/FreeBSD VM for a gateway and a bunch of VMs on a detached bridge. Also have several more bridges for test VMs for real-world airgapped network setup. 8 different OS distros. Also am tweaking a MIPS and a M1 QEMUs outside of Proxmox . runs very quiet. so now it’s my home datacenter.

joshspankit 6y ago
I have something very similar, but an Optiplex. One thing I had to do recently though is swap pfsense for Untangle because of drastically better FQ-CoDel implementation in Untangle.
- egberts1 6y ago
  I went over to Untangle website for any data sheet but was unable to get one immediately (have to fork over everything but your youngest child, so to speak in order to get it. Not sure why would one even do that. :-(
  - joshspankit 6y ago
    I don’t want to sidetrack this in to an Untangle conversation, but yes: it’s fundamentally not free as in speech while also being supported as though it’s just a thin shell over Debian. However, the YouTube breakdown by Battle(non)sense was conclusive (and is confirmed in my case in undeniable ways) so I switched.

m463 6y ago

(I think) I ran into the same or similar problem you did - there's a partition for the debian OS + iso images of a fixed size (df shows it as /dev/mapper/pce-root -> / ) and all the reallocatable storage used by the VMs comes out of the rest. Thing is it's hard to re-allocate this partition, and that's where all the iso images, and the backups go. I tried to migrate a vm and boom - it filled up. I tried to figure out how to reallocate it, but in the end, I found this (sort of hacky) solution documented: https://pve.proxmox.com/wiki/Logical_Volume_Manager_(LVM) https://pve.proxmox.com/wiki/Logical_Volume_Manager_(LVM) basically: # lvcreate -n vz -V 200G pve/data # mkfs.ext4 /dev/pve/vz # echo '/dev/pve/vz /var/lib/vz ext4 defaults 0 2' >> /etc/fstab you migrate the stuff from /var/lib/vz to the new location and you have 200G for iso images/etc Why this has to fight for storage from the root system, or why it's not easily expandable is a head-scratcher. I think it might be better to have a dedicated boot debian partition, a resizable/easily growable pve partition, then storage that can be allocated to vms/containers. but yeah - that stopped me in my tracks, and when I'm doing it on a home machine you just don't have the time to chase this stuff down.

egberts1 6y ago
Yep. so glad that i experimented (over a week) with Proxmox install over an existing Denuvan (Debian SysV init) install. Made more sense to have a second drive or second but largest LVM to play with and store all things VM. So confident that I disabled the Proxmox 'local' drive.

aaronius 6y ago

If you install Proxmox VE the next time and are not happy with the default layout, check out the parameters for LVM [0]. I personally prefer to install on ZFS as all ZFS datasets share the same free space unlike with LVM. If you want full control of the disk layout and maybe want to do some more custom stuff, consider installing a base Debian and then install Proxmox VE on top of it [1]. The PVE installer makes it easy to get a system up and running quickly but is opinionated what the disk layout should be. That of course might not be working for everyone. [0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#advanced_lvm_options https://pve.proxmox.com/pve-docs/pve-admin-guide.html#advanc... [1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_install_proxmox_ve_on_debian https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_insta...