6 ms·

I can get a Xeon E5-2690V4 with 28 threads and 64GB of RAM for about $150. If you need cores and memory to make a lot of VMs you can do it extremely cheaply

nine_k 1y ago
It will probably consume $150 worth of electricity in less than a month, even sitting idle :-\
- blobbers 1y ago
  The internet says 100W idle, so maybe more like $40-50 electricity, depending on where you live could be cheaper could be more expensive. Makes me wonder if I should unplug more stuff when on vacation.
  - yjftsjthsd-h 1y ago
    > Makes me wonder if I should unplug more stuff when on vacation. What's the margin on unplugging vs just powering off?
    - dijit 1y ago
      By "off" you mean, functionally disabled but with whatever auto-update system in the background with all the radios on for "smart home" reasons - or, "off"?
    - Symbiote 1y ago
      That also depends on the country you live. The EU (and maybe China?) have been regulating standby power consumption, so most of my appliances either have a physical off switch (usually as the only switch) or should have very low standby power draw. I don't have the equipment to measure this myself.
  - nine_k 1y ago
    I was surprised to find out that my apartment pulls 80-100W when everything is seemingly down during the night. A tiny light here and there, several displays in sleep mode, a desktop idling (mere 15W, but), a laptop charging, several phones charging, etc, the fridge switches on for a short moment. The many small amounts add up to something considerable.
    - ToucanLoucan 1y ago
      I got out of the homelab game as I finished my transition from DevOps to Engineering Lead, and it was simply massively overbuilt for what I actually needed. I replaced an ancient Dell R700 series, R500 series, and a couple supermicros with 3 old desktop PCs in rack enclosures and cut my electric bill nearly $90/month. Fuckin nutty how much juice those things tear through.
    - amatecha 1y ago
      Yeah it kinda puts it all into perspective when you think of how every home used to use 60-watt light bulbs all throughout. Most people just leave lights on all over their home all day, probably using hundreds of watts of electricity. Makes me realize my 35-65w laptop is pretty damn efficient haha
  - rogerrogerr 1y ago
    100W over a month (rule of thumb 730 hours) is 73kWh. Which is $7.30 at my $0.10/kWh rate, or less than $25 at (what Google told me is) Cali’s average $0.30/kWh.
    - mercutio2 1y ago
      Your googling gave results that were likely accurate for California 4-5 years ago. My average cost per kWh is about 60 cents. Rates have gone up enormously because the cost of wildfires is falling on ratepayers, not the utility owners. Regulated monopolies are pretty great, aren’t they? Heads I win, tales you lose.
      - lukevp 1y ago
        60 cents per kWh? That’s shocking. Here in Oregon people complain about energy prices and my fully loaded cost (not the per kWh but including everything) is 19c. And I go over the limit for single family residential where I end up in a higher priced bracket. Thanks for making me feel better about my electricity rate. I’m sorry you have to deal with that. The utility companies should have to pay to cover those costs.
      - cogman10 1y ago
        Depends entirely on the utilities board doing the regulation. That said, I'm of the opinion that power/water/internet should all be state/county/city ran. I don't want my utilities companies to have profit motives. My water company just got bought up by a huge water company conglomerate and, you guessed it, immediate rate increases.
        SoftTalker 1y ago
        Most utilities, even if ostensibly privately-owned, are profit-limited and rates must be approved by a regulatory board. Some are organized as non-profits (rural water and electric co-ops, etc.) This is in exchange for the local monopoly. If your local regulators approved the merger and higher rates, your complaint is with them as much as the utility company. Not saying that some regulators are not basically rubber stamps or even corrupt.
  - titanomachy 1y ago
    100W continuous at 12¢/kWh (US average) is only ~$9 / month. Is your electricity 5x more expensive than the US average?
    - mercutio2 1y ago
      Not OP, but my California TOU rates are between a 40 and 70 cents per kWh. Still only $50/month, not $150, but I very much care about 100W loads doing no work.
      - cjbgkagh 1y ago
        Those kWh prices are insane, that’ll make industry move out of there.
        selkin 1y ago
        Industrial pays different rates than homes. That said, I am not sure those numbers are true. I am in California (PG&E with East Bay community generation), and my TOU rates are much lower than those.
        mrkstu 1y ago
        If he’s only paying $50 most of it is connection fees and low usage distorting his per kWh price way up.
        mercutio2 1y ago
        There are 3 different components of PG&E electricity bills, which makes the bill difficult to read. I am also in PG&E East Bay community generation, and when I look at all components, it’s: Minimum Delivery Charge (what’s paid monthly, which is largely irrelevant, before annual true-up of NEM charges): $11.69/month Actual charges, billed annually, per kWh: Peak NEM charge: $.62277 Off-Peak NEM charges: $.31026 Plus 3-20% extra (depending on the month) in “non-bypassable charges” (I haven’t figured out where these numbers come from), then a 7.5% local utility tax. Those rates do get a little lower in the winter (.30 to .48), and of course the very high rates benefit me when I generate more energy than I consume (which only happens when I’m on vacation). But the marginal all-in costs are just very high. That’s NEM2 + TOU-EV2A, specifically.
  - p12tic 1y ago
    Depends on a server. This test got 79W idle for _two socket_ E5 2690-V4 server. https://www.servethehome.com/lenovo-system-x3650-m5-workhorse-2u-server-review/ https://www.servethehome.com/lenovo-system-x3650-m5-workhors...
- kjkjadksj 1y ago
  So shut it off when you don’t need it.
- Almondsetat 1y ago
  Isn't your home lab supposed to make you learn stuff? Why would you leave it idle?
  - cjbgkagh 1y ago
    You wouldn’t, it’s given as a lower bound, it costs more than that when not idling
    - dijit 1y ago
      but then you’d turn it off, if you don’t then cloud is much more expensive too. Also $150 for 100w is crazy, thats like $1.70 per kWh; it would cost about $150 a year at the (high) rates of southern Sweden.
      - cjbgkagh 1y ago
        Im not the OP, don’t know how they arrived at that cost. Personally it’s cheaper to buy the hardware that does spend most of its time idling. Fast turnaround on very large private datasets being key.
- swiftcoder 1y ago
  Obviously the solution is to pickup another hobby, and enter the DIY solar game at the same time as your home lab obsession :D
  - bokohut 1y ago
    Interestingly enough it is often times a foundational change in one's 'normal' that inspires something 'new'. In this case that 'new' is energy efficient software down to the individual lines of code and what their energy cost is on certain hardware. Academics are publishing about it in niche corners of the web and some entrepreneurs are doing it but of course none of this is cool now so we remain a mockery for our objectives. In time this too will become a real thing as many now are just beginning to feel the ever rising costs of energy which is only just starting to increase from decisions made years ago. The worst is yet to come as seen and heard directly from every single expert that has testified in the last years before the Energy and Commerce committee however only the outside-the-boxers among us watch such educational content to better prepare for tomorrow. Electricity powers our world and nearly all take it for granted, time too will change this thinking. :D
sebastiansm 1y ago
On Aliexpress those Xeon+mobo+ram kits are really cheap.
- datadrivenangel 1y ago
  1. Not in the US with tariffs now. 2. I would not trust complicated electronics from Aliexpress from a safety and security perspective.
deleted 1y ago
[deleted]
kbenson 1y ago
Source? That seems like something I would want to take advantage if at the moment...
- kllrnohj 1y ago
  Note the E5-2690V4 is a 10 year old CPU, they are talking about used servers. You can find those on ebay or whatever as well as stores specializing in that. Depending on where you live, you might even find them free as they are often considered literal ewaste by the companies decommissioning them. It also means it performs like a 10 year old server CPU, so those 28 threads are not exactly worth a lot. The geekbench results, for whatever value those are worth, are very mediocre in the context of anything remotely modern: https://browser.geekbench.com/processors/intel-xeon-e5-2690-v4 https://browser.geekbench.com/processors/intel-xeon-e5-2690-... Like a modern 12-thread 9600x runs absolute circles around it https://browser.geekbench.com/processors/amd-ryzen-5-9600x https://browser.geekbench.com/processors/amd-ryzen-5-9600x
  - mattbillenstein 1y ago
    This is the correct analysis - there's a reason you see this stuff cheap or free. The homelab group on Reddit is full of people who don't understand any of this - they have full racks in their house that could be replaced with one high-end desktop.
    - kllrnohj 1y ago
      > The homelab group on Reddit is full of people who don't understand any of this - they have full racks in their house that could be replaced with one high-end desktop. A lot of that group is making use of the IO capabilities of these systems to run lots of PCI-E devices & hard drives. There's not exactly a cost-effective modern equivalent for that. If there were cost-effective ways to do something like take a PCI-E 5.0 x2 and turn it into a PCI-E 3.0 x8 that'd be incredible, but there isn't really. So raw PCI-E lane count is significant if you want cheap networking gear or HBAs or whatever, and raw PCI-E lane count is $$$$ if you're buying new. Also these old systems mean cheap RAM in large, large capacities. Like 128GB RAM to make ZFS or VMs purr is much cheaper to do on these used systems than anything modern.
mattbillenstein 1y ago
Power and noise - old server hardware is not something you want in your home. Commodity desktop cpus with 32 or 64GB RAM can do all of this in a low-power and quiet way without a lot more expense.
- p12tic 1y ago
  The problem is with the form factor, not the server hardware per-se. If one buys regular ATX motherboard that accepts server CPUs and fits it in regular ATX case, then there's lots of space for a relatively silent CPU air cooler. 2690 v4 idles at less than 40W which is not much more than a regular gaming desktop with a powerful GPU. The only problem in practice is that server CPUs don't support S3 suspend, so putting whole thing to sleep after finishing with it doesn't work.
semi-extrinsic 1y ago
For $3000 you can get 3x used Epyc servers with a total of 144 cores and 384 GB memory, with dual-port 25Gbe networking so you can run them in a fully connected cluster without a switch. It will have >20x better perf/$ and ~3x better perf/W. That combo gives you the better part of a gigabyte of L3 cache and an aggregate memory bandwidth of 600 GB/s, while still below 1000W total running at full speed. Plus your NICs are the fancy kind that let you play around with RoCEv2 and such nifty stuff. It would also be relevant to then also learn how to do stuff properly with SLURM and Warewulf etc. instead of a poor mans solution with Ansible playbooks like in these blog posts.
- Almondsetat 1y ago
  You are taking my reply completely out of context. If you want to learn clustering, you need a lot of cores and ram to run many VMs. You don't need them to be individually very powerful.
- p12tic 1y ago
  Better build a single workstation - less noise, less power usage and the form factor is way more convenient. A budget of $3000 can buy 128 cores with 512GB of RAM on a single regular EATX motherboard, a case, a power supply and other accessories. Power usage is ~550W at maximum utilization which not much more than a gaming rig with a powerful GPU.
Aurornis 1y ago
> I can get a Xeon E5-2690V4 with 28 threads and 64GB of RAM for about $150. If the goal is a lot of RAM and you don’t care about noise, power, or heat then these can be an okay deal. Don’t underestimate how far CPUs have come, though. That machine will be slower than AMD’s slowest entry-level CPU. Even an AMD 5800X will double its single core performance and even walk away from it on multithreaded tasks despite only having 8 cores. It will use less electricity and be quiet, too. More expensive, but if this is something you plan to leave running 24/7 the electricity costs over a few years might make the power hungry server more expensive over time.

nine_k 1y ago

It will probably consume $150 worth of electricity in less than a month, even sitting idle :-\

blobbers 1y ago
The internet says 100W idle, so maybe more like $40-50 electricity, depending on where you live could be cheaper could be more expensive. Makes me wonder if I should unplug more stuff when on vacation.
- yjftsjthsd-h 1y ago
  > Makes me wonder if I should unplug more stuff when on vacation. What's the margin on unplugging vs just powering off?
  - dijit 1y ago
    By "off" you mean, functionally disabled but with whatever auto-update system in the background with all the radios on for "smart home" reasons - or, "off"?
  - Symbiote 1y ago
    That also depends on the country you live. The EU (and maybe China?) have been regulating standby power consumption, so most of my appliances either have a physical off switch (usually as the only switch) or should have very low standby power draw. I don't have the equipment to measure this myself.
- nine_k 1y ago
  I was surprised to find out that my apartment pulls 80-100W when everything is seemingly down during the night. A tiny light here and there, several displays in sleep mode, a desktop idling (mere 15W, but), a laptop charging, several phones charging, etc, the fridge switches on for a short moment. The many small amounts add up to something considerable.
  - ToucanLoucan 1y ago
    I got out of the homelab game as I finished my transition from DevOps to Engineering Lead, and it was simply massively overbuilt for what I actually needed. I replaced an ancient Dell R700 series, R500 series, and a couple supermicros with 3 old desktop PCs in rack enclosures and cut my electric bill nearly $90/month. Fuckin nutty how much juice those things tear through.
  - amatecha 1y ago
    Yeah it kinda puts it all into perspective when you think of how every home used to use 60-watt light bulbs all throughout. Most people just leave lights on all over their home all day, probably using hundreds of watts of electricity. Makes me realize my 35-65w laptop is pretty damn efficient haha
- rogerrogerr 1y ago
  100W over a month (rule of thumb 730 hours) is 73kWh. Which is $7.30 at my $0.10/kWh rate, or less than $25 at (what Google told me is) Cali’s average $0.30/kWh.
  - mercutio2 1y ago
    Your googling gave results that were likely accurate for California 4-5 years ago. My average cost per kWh is about 60 cents. Rates have gone up enormously because the cost of wildfires is falling on ratepayers, not the utility owners. Regulated monopolies are pretty great, aren’t they? Heads I win, tales you lose.
    - lukevp 1y ago
      60 cents per kWh? That’s shocking. Here in Oregon people complain about energy prices and my fully loaded cost (not the per kWh but including everything) is 19c. And I go over the limit for single family residential where I end up in a higher priced bracket. Thanks for making me feel better about my electricity rate. I’m sorry you have to deal with that. The utility companies should have to pay to cover those costs.
    - cogman10 1y ago
      Depends entirely on the utilities board doing the regulation. That said, I'm of the opinion that power/water/internet should all be state/county/city ran. I don't want my utilities companies to have profit motives. My water company just got bought up by a huge water company conglomerate and, you guessed it, immediate rate increases.
      - SoftTalker 1y ago
        Most utilities, even if ostensibly privately-owned, are profit-limited and rates must be approved by a regulatory board. Some are organized as non-profits (rural water and electric co-ops, etc.) This is in exchange for the local monopoly. If your local regulators approved the merger and higher rates, your complaint is with them as much as the utility company. Not saying that some regulators are not basically rubber stamps or even corrupt.
- titanomachy 1y ago
  100W continuous at 12¢/kWh (US average) is only ~$9 / month. Is your electricity 5x more expensive than the US average?
  - mercutio2 1y ago
    Not OP, but my California TOU rates are between a 40 and 70 cents per kWh. Still only $50/month, not $150, but I very much care about 100W loads doing no work.
    - cjbgkagh 1y ago
      Those kWh prices are insane, that’ll make industry move out of there.
      - selkin 1y ago
        Industrial pays different rates than homes. That said, I am not sure those numbers are true. I am in California (PG&E with East Bay community generation), and my TOU rates are much lower than those.
        mrkstu 1y ago
        If he’s only paying $50 most of it is connection fees and low usage distorting his per kWh price way up.
        mercutio2 1y ago
        There are 3 different components of PG&E electricity bills, which makes the bill difficult to read. I am also in PG&E East Bay community generation, and when I look at all components, it’s: Minimum Delivery Charge (what’s paid monthly, which is largely irrelevant, before annual true-up of NEM charges): $11.69/month Actual charges, billed annually, per kWh: Peak NEM charge: $.62277 Off-Peak NEM charges: $.31026 Plus 3-20% extra (depending on the month) in “non-bypassable charges” (I haven’t figured out where these numbers come from), then a 7.5% local utility tax. Those rates do get a little lower in the winter (.30 to .48), and of course the very high rates benefit me when I generate more energy than I consume (which only happens when I’m on vacation). But the marginal all-in costs are just very high. That’s NEM2 + TOU-EV2A, specifically.
- p12tic 1y ago
  Depends on a server. This test got 79W idle for _two socket_ E5 2690-V4 server. https://www.servethehome.com/lenovo-system-x3650-m5-workhorse-2u-server-review/ https://www.servethehome.com/lenovo-system-x3650-m5-workhors...
kjkjadksj 1y ago
So shut it off when you don’t need it.
Almondsetat 1y ago
Isn't your home lab supposed to make you learn stuff? Why would you leave it idle?
- cjbgkagh 1y ago
  You wouldn’t, it’s given as a lower bound, it costs more than that when not idling
  - dijit 1y ago
    but then you’d turn it off, if you don’t then cloud is much more expensive too. Also $150 for 100w is crazy, thats like $1.70 per kWh; it would cost about $150 a year at the (high) rates of southern Sweden.
    - cjbgkagh 1y ago
      Im not the OP, don’t know how they arrived at that cost. Personally it’s cheaper to buy the hardware that does spend most of its time idling. Fast turnaround on very large private datasets being key.
swiftcoder 1y ago
Obviously the solution is to pickup another hobby, and enter the DIY solar game at the same time as your home lab obsession :D
- bokohut 1y ago
  Interestingly enough it is often times a foundational change in one's 'normal' that inspires something 'new'. In this case that 'new' is energy efficient software down to the individual lines of code and what their energy cost is on certain hardware. Academics are publishing about it in niche corners of the web and some entrepreneurs are doing it but of course none of this is cool now so we remain a mockery for our objectives. In time this too will become a real thing as many now are just beginning to feel the ever rising costs of energy which is only just starting to increase from decisions made years ago. The worst is yet to come as seen and heard directly from every single expert that has testified in the last years before the Energy and Commerce committee however only the outside-the-boxers among us watch such educational content to better prepare for tomorrow. Electricity powers our world and nearly all take it for granted, time too will change this thinking. :D

sebastiansm 1y ago

On Aliexpress those Xeon+mobo+ram kits are really cheap.

datadrivenangel 1y ago
1. Not in the US with tariffs now. 2. I would not trust complicated electronics from Aliexpress from a safety and security perspective.

deleted 1y ago

[deleted]

kbenson 1y ago

Source? That seems like something I would want to take advantage if at the moment...

kllrnohj 1y ago
Note the E5-2690V4 is a 10 year old CPU, they are talking about used servers. You can find those on ebay or whatever as well as stores specializing in that. Depending on where you live, you might even find them free as they are often considered literal ewaste by the companies decommissioning them. It also means it performs like a 10 year old server CPU, so those 28 threads are not exactly worth a lot. The geekbench results, for whatever value those are worth, are very mediocre in the context of anything remotely modern: https://browser.geekbench.com/processors/intel-xeon-e5-2690-v4 https://browser.geekbench.com/processors/intel-xeon-e5-2690-... Like a modern 12-thread 9600x runs absolute circles around it https://browser.geekbench.com/processors/amd-ryzen-5-9600x https://browser.geekbench.com/processors/amd-ryzen-5-9600x
- mattbillenstein 1y ago
  This is the correct analysis - there's a reason you see this stuff cheap or free. The homelab group on Reddit is full of people who don't understand any of this - they have full racks in their house that could be replaced with one high-end desktop.
  - kllrnohj 1y ago
    > The homelab group on Reddit is full of people who don't understand any of this - they have full racks in their house that could be replaced with one high-end desktop. A lot of that group is making use of the IO capabilities of these systems to run lots of PCI-E devices & hard drives. There's not exactly a cost-effective modern equivalent for that. If there were cost-effective ways to do something like take a PCI-E 5.0 x2 and turn it into a PCI-E 3.0 x8 that'd be incredible, but there isn't really. So raw PCI-E lane count is significant if you want cheap networking gear or HBAs or whatever, and raw PCI-E lane count is $$$$ if you're buying new. Also these old systems mean cheap RAM in large, large capacities. Like 128GB RAM to make ZFS or VMs purr is much cheaper to do on these used systems than anything modern.

mattbillenstein 1y ago

Power and noise - old server hardware is not something you want in your home. Commodity desktop cpus with 32 or 64GB RAM can do all of this in a low-power and quiet way without a lot more expense.

p12tic 1y ago
The problem is with the form factor, not the server hardware per-se. If one buys regular ATX motherboard that accepts server CPUs and fits it in regular ATX case, then there's lots of space for a relatively silent CPU air cooler. 2690 v4 idles at less than 40W which is not much more than a regular gaming desktop with a powerful GPU. The only problem in practice is that server CPUs don't support S3 suspend, so putting whole thing to sleep after finishing with it doesn't work.

semi-extrinsic 1y ago

For $3000 you can get 3x used Epyc servers with a total of 144 cores and 384 GB memory, with dual-port 25Gbe networking so you can run them in a fully connected cluster without a switch. It will have >20x better perf/$ and ~3x better perf/W. That combo gives you the better part of a gigabyte of L3 cache and an aggregate memory bandwidth of 600 GB/s, while still below 1000W total running at full speed. Plus your NICs are the fancy kind that let you play around with RoCEv2 and such nifty stuff. It would also be relevant to then also learn how to do stuff properly with SLURM and Warewulf etc. instead of a poor mans solution with Ansible playbooks like in these blog posts.

Almondsetat 1y ago
You are taking my reply completely out of context. If you want to learn clustering, you need a lot of cores and ram to run many VMs. You don't need them to be individually very powerful.
p12tic 1y ago
Better build a single workstation - less noise, less power usage and the form factor is way more convenient. A budget of $3000 can buy 128 cores with 512GB of RAM on a single regular EATX motherboard, a case, a power supply and other accessories. Power usage is ~550W at maximum utilization which not much more than a gaming rig with a powerful GPU.

Aurornis 1y ago

> I can get a Xeon E5-2690V4 with 28 threads and 64GB of RAM for about $150. If the goal is a lot of RAM and you don’t care about noise, power, or heat then these can be an okay deal. Don’t underestimate how far CPUs have come, though. That machine will be slower than AMD’s slowest entry-level CPU. Even an AMD 5800X will double its single core performance and even walk away from it on multithreaded tasks despite only having 8 cores. It will use less electricity and be quiet, too. More expensive, but if this is something you plan to leave running 24/7 the electricity costs over a few years might make the power hungry server more expensive over time.