7 ms·

I've found that the home lab is a great place for cast-off corporate equipment either bought from ebay or scrounged from data center upgrades. Mine, assembled a

I've found that the home lab is a great place for cast-off corporate equipment either bought from ebay or scrounged from data center upgrades. Mine, assembled at little to no cost, consists of: 42u IBM-branded cabinet Cisco ASA 5510 firewall Dell R610 running ESXi Old Dell 2950 full of 2TB drives running FreeNAS Cisco 'Small Business' 50 port managed switch A couple of 4U server cases containing the guts of obsolete gaming systems, repurposed as assorted servers If you've got the space and can deal with the noise, I can't say too many good things about the R610s - available dirt cheap, and though they're about four generations old at this point, still perform decently since Intel/AMD have, until recently, pretty much stagnated on CPU performance. Luckily I have a basement.

amenghra 9y ago
That’s really cool! I have a bunch of old hardware, unfortunately I can only bring them online when I want to play with them and don’t have any permanent space. Do you know how many watts you consume? Are alternatives to old enterprise gear from eBay going to be cheaper from an electricity point of view?
- ThrowawayR2 9y ago
  I'm kind of guessing here but: -Cisco ASA 5510 firewall - 100W -Dell R610 running ESXi - 400W -Old Dell 2950 full of 2TB drives running FreeNAS - 400W -Cisco 'Small Business' 50 port managed switch - 100W -A couple of 4U server cases containing the guts of obsolete gaming systems, repurposed as assorted -servers - 2 x 300W So, maybe 1600W under medium load? A 15A circuit can deliver 1800W max., so it would be occupying one whole circuit. Hardly worth it, considering the noise, heat, space, and power. Old enterprise hardware gets given away for a reason.
  - akeating 9y ago
    yeah, I considered holding onto a pair of old 2950s but ditched them considering my peak electricity rate is 43c kw/h. This setup makes me shudder.
gravypod 9y ago
How do you find data centers that trash gear? I want a setup like this eventually. There's a lot of cool things (in my opinion) I'd want to do if I had serious compute power.
- zhengyi13 9y ago
  Coming from an IT/Ops background, often the answer is, know the Ops/sysad folk in a small shop, and be around when they're upgrading. Apart from that, places like www.weirdstuff.com might exist in your area, and have stuff like this laying around for pennies on the dollar.
- Spooky23 9y ago
  Look for government auctions. I bought a pallet of LCD screens for my son’s school for like $75. A few years ago, they sold 4 pallets of Extreme 10/100 switches for $20! The catch is they sell lots and you have to pick them up at one time.
  - loeg 9y ago
    And even a few years ago, 100 mbit switches were obsolete.
mentos 9y ago
Someone shared this website with me a while back to find used servers on ebay https://www.labgopher.com/ https://www.labgopher.com/ What do you think of that? I am trying to put together a render farm where I need a lot of parallel CPUs to run Unreal Engine Swarm [0] but I just don't have the time or experience to execute on it. My goal is 16 Xeon chips across maybe 8 'boxes' to get to around 64/128 cores. Anyone know where I could look to find someone I could contract to put something like this together? [0] - https://docs.unrealengine.com/udk/Three/Swarm.html https://docs.unrealengine.com/udk/Three/Swarm.html
- JoshTriplett 9y ago
  > I am trying to put together a render farm where I need a lot of parallel CPUs to run Unreal Engine Swarm [0] but I just don't have the time or experience to execute on it. My goal is 16 Xeon chips across maybe 8 'boxes' to get to around 64/128 cores. These days, one dual-socket server can give you that, for a pretty reasonable cost compared to the time, expense, and power usage of running a rack full of systems.
  - mentos 9y ago
    Yea I think the cost savings when it comes to cores may not make sense in the long run when it comes to setup/teardown and power usage. If you had to put together a copy of the Amazon X1 instance https://aws.amazon.com/ec2/instance-types/x1/ https://aws.amazon.com/ec2/instance-types/x1/ any recommendation on a mobo/memory/case to use?
- cweagans 9y ago
  You might be better off using EC2 or similar. Do the work to configure everything once and package it as an AMI, then when you need your swarm, spin up as many instance as you need, render whatever needs rendered, and then turn off the instances. Should be reasonably cheap, and it'll let you decide on the speed vs cost tradeoff (i.e. more servers cost more, but will get your work done faster; fewer servers cost less, but will take longer). I don't think a rack full of hardware is a very good choice for this.
timc3 9y ago
This was more like the setup I was expecting when I clicked on the link, not some buying of UniFi and setting up a few VLANs and SSH Keys. I agree with you that they are great value but I find the noise and power consumption of that generation a bit much so just built my own boxes
z3t4 9y ago
Or you build a super quite desktop server, that will both have more CPU and use less power. The trick is to use large heat sinks with large silent fans. When the CPU's crank up to 100% all you'll hear is a subtle wind breeze.
- Ascetik 9y ago
  Sys Admin/Hardware dork here. Absolutely this. No need to run enterprise level hardware at home. You can easily run a huge environment with 128gb of ram and an 8-core Ryzen processor. If you want to get crazy you could even get an Epyc CPU with a SuperMicro motherboard, put it into a eATX whitebox case and run silent fans with a huge heatsink and still barely use any power. You can even run all the networking OS's in VM's, either with PFSense or some other vm template or use GNS3. No need to purchase a ton of hardware. It really makes no sense to me why people drop all this money on R710's and make their house sound like the Hartsfield-Jackson Delta Terminal.
  - HankB99 9y ago
    What about the benefit of using ECC RAM for a file server? I don't see that available in consumer grade H/W. (As far as network H/W I'm in full agreement.)
    - namibj 9y ago
      AMD h/w pretty much always supports ECC DIMMs, it just depends on whether the MoBo propagates that to the sockets, as it takes more traces.
    - san-fran 9y ago
      If you plan on running a ZFS file server at home, which enthusiasts prefer because it's one of the only filesystems that supports full file checksumming to prevent bit rot. Anyway, the ZFS approach is to inherently trust the data that's in RAM over the data on the drive if there's a discrepancy, so ECC RAM is required to maintain the integrity of data on RAM.
zippergz 9y ago
I used to have an ASA has my home firewall, but the problem was that its ethernet ports were 100 megabit, which meant that it eventually became the bottleneck in my internet connection. I ended up replacing it with a Ubiquiti router, which was faster, far cheaper (than even a used market ASA), still getting security updates (without a support contract), and a lot easier to manage (and I say this as someone with extensive experience with Cisco gear, dating back to the 90s). The Ubiquiti might not look as nice or be as cool of hardware, but in practice it's better in every way that matters to me.
chrissnell 9y ago
I'm not a fan of running a hardware firewall that's long since passed its end-of-life date. There have been a number of critical vulnerabilities on these ASAs in the past. I'd rather stick with software that's kept up to date, like PFSense or OpenBSD. Definitely agree on using other cast-off hardware, though.
Johnny555 9y ago
I've found that enterprise gear is not exactly power friendly -- the Cisco 5510 is rated at 150W steady state, if it uses half that much power in practice (75W), that's around $90/year where I live. In contrast, I paid around $120 for a PCEngines APU2 (running PFSense), and it uses around 8W of power (measured). Old servers are the same -- the amount of power they use for the performance makes them expensive to run 24x7.
- craftyguy 9y ago
  This is pretty much what I ended up doing. It's a collossal waste of power (and money) to run enterprise equipment in my area for a home network that is nowhere near enterprise-level in terms of utilization.
  - stock_toaster 9y ago
    I also used to have more gear, but hav really scaled back and consolidated. I now have one fairly beefy server (storage array, decent zeon, lots of ram), run a few vms (dev, plex, etc), and export storage. Then I have a pair of NUCs doing various things.
    - cm2187 9y ago
      Plus hard drive capacity seems to have gone back on a steep increasing curve with 12TB drives coming to the market, offering further opportunities for consolidation.
rayiner 9y ago
You can get some good deals here too: https://forums.servethehome.com/index.php?forums/for-sale-for-trade-want-to-buy.2 https://forums.servethehome.com/index.php?forums/for-sale-fo.... I've been wiring up 10G throughout the house, and managed to get a ER-8-XG (beta version) on this site for under $500. (8 10-gig SFP+). Wife made me get rid of it because it was too loud. :( Got a Dell T1700 (they're like giving them away on EBay). Quad core Xeon E3, plenty of headroom to do firewall/routing in software.
jdironman 9y ago
Can you tell me more about how you did the Dell 2950? I'm looking at NAS server options and would love to roll my own. I have 2-3 of these sitting around.

amenghra 9y ago

That’s really cool! I have a bunch of old hardware, unfortunately I can only bring them online when I want to play with them and don’t have any permanent space. Do you know how many watts you consume? Are alternatives to old enterprise gear from eBay going to be cheaper from an electricity point of view?

ThrowawayR2 9y ago
I'm kind of guessing here but: -Cisco ASA 5510 firewall - 100W -Dell R610 running ESXi - 400W -Old Dell 2950 full of 2TB drives running FreeNAS - 400W -Cisco 'Small Business' 50 port managed switch - 100W -A couple of 4U server cases containing the guts of obsolete gaming systems, repurposed as assorted -servers - 2 x 300W So, maybe 1600W under medium load? A 15A circuit can deliver 1800W max., so it would be occupying one whole circuit. Hardly worth it, considering the noise, heat, space, and power. Old enterprise hardware gets given away for a reason.
- akeating 9y ago
  yeah, I considered holding onto a pair of old 2950s but ditched them considering my peak electricity rate is 43c kw/h. This setup makes me shudder.

gravypod 9y ago

How do you find data centers that trash gear? I want a setup like this eventually. There's a lot of cool things (in my opinion) I'd want to do if I had serious compute power.

zhengyi13 9y ago
Coming from an IT/Ops background, often the answer is, know the Ops/sysad folk in a small shop, and be around when they're upgrading. Apart from that, places like www.weirdstuff.com might exist in your area, and have stuff like this laying around for pennies on the dollar.
Spooky23 9y ago
Look for government auctions. I bought a pallet of LCD screens for my son’s school for like $75. A few years ago, they sold 4 pallets of Extreme 10/100 switches for $20! The catch is they sell lots and you have to pick them up at one time.
- loeg 9y ago
  And even a few years ago, 100 mbit switches were obsolete.

mentos 9y ago

Someone shared this website with me a while back to find used servers on ebay https://www.labgopher.com/ https://www.labgopher.com/ What do you think of that? I am trying to put together a render farm where I need a lot of parallel CPUs to run Unreal Engine Swarm [0] but I just don't have the time or experience to execute on it. My goal is 16 Xeon chips across maybe 8 'boxes' to get to around 64/128 cores. Anyone know where I could look to find someone I could contract to put something like this together? [0] - https://docs.unrealengine.com/udk/Three/Swarm.html https://docs.unrealengine.com/udk/Three/Swarm.html

JoshTriplett 9y ago
> I am trying to put together a render farm where I need a lot of parallel CPUs to run Unreal Engine Swarm [0] but I just don't have the time or experience to execute on it. My goal is 16 Xeon chips across maybe 8 'boxes' to get to around 64/128 cores. These days, one dual-socket server can give you that, for a pretty reasonable cost compared to the time, expense, and power usage of running a rack full of systems.
- mentos 9y ago
  Yea I think the cost savings when it comes to cores may not make sense in the long run when it comes to setup/teardown and power usage. If you had to put together a copy of the Amazon X1 instance https://aws.amazon.com/ec2/instance-types/x1/ https://aws.amazon.com/ec2/instance-types/x1/ any recommendation on a mobo/memory/case to use?
cweagans 9y ago
You might be better off using EC2 or similar. Do the work to configure everything once and package it as an AMI, then when you need your swarm, spin up as many instance as you need, render whatever needs rendered, and then turn off the instances. Should be reasonably cheap, and it'll let you decide on the speed vs cost tradeoff (i.e. more servers cost more, but will get your work done faster; fewer servers cost less, but will take longer). I don't think a rack full of hardware is a very good choice for this.

timc3 9y ago

This was more like the setup I was expecting when I clicked on the link, not some buying of UniFi and setting up a few VLANs and SSH Keys. I agree with you that they are great value but I find the noise and power consumption of that generation a bit much so just built my own boxes

z3t4 9y ago

Or you build a super quite desktop server, that will both have more CPU and use less power. The trick is to use large heat sinks with large silent fans. When the CPU's crank up to 100% all you'll hear is a subtle wind breeze.

Ascetik 9y ago
Sys Admin/Hardware dork here. Absolutely this. No need to run enterprise level hardware at home. You can easily run a huge environment with 128gb of ram and an 8-core Ryzen processor. If you want to get crazy you could even get an Epyc CPU with a SuperMicro motherboard, put it into a eATX whitebox case and run silent fans with a huge heatsink and still barely use any power. You can even run all the networking OS's in VM's, either with PFSense or some other vm template or use GNS3. No need to purchase a ton of hardware. It really makes no sense to me why people drop all this money on R710's and make their house sound like the Hartsfield-Jackson Delta Terminal.
- HankB99 9y ago
  What about the benefit of using ECC RAM for a file server? I don't see that available in consumer grade H/W. (As far as network H/W I'm in full agreement.)
  - namibj 9y ago
    AMD h/w pretty much always supports ECC DIMMs, it just depends on whether the MoBo propagates that to the sockets, as it takes more traces.
  - san-fran 9y ago
    If you plan on running a ZFS file server at home, which enthusiasts prefer because it's one of the only filesystems that supports full file checksumming to prevent bit rot. Anyway, the ZFS approach is to inherently trust the data that's in RAM over the data on the drive if there's a discrepancy, so ECC RAM is required to maintain the integrity of data on RAM.

zippergz 9y ago

I used to have an ASA has my home firewall, but the problem was that its ethernet ports were 100 megabit, which meant that it eventually became the bottleneck in my internet connection. I ended up replacing it with a Ubiquiti router, which was faster, far cheaper (than even a used market ASA), still getting security updates (without a support contract), and a lot easier to manage (and I say this as someone with extensive experience with Cisco gear, dating back to the 90s). The Ubiquiti might not look as nice or be as cool of hardware, but in practice it's better in every way that matters to me.

chrissnell 9y ago

I'm not a fan of running a hardware firewall that's long since passed its end-of-life date. There have been a number of critical vulnerabilities on these ASAs in the past. I'd rather stick with software that's kept up to date, like PFSense or OpenBSD. Definitely agree on using other cast-off hardware, though.

Johnny555 9y ago

I've found that enterprise gear is not exactly power friendly -- the Cisco 5510 is rated at 150W steady state, if it uses half that much power in practice (75W), that's around $90/year where I live. In contrast, I paid around $120 for a PCEngines APU2 (running PFSense), and it uses around 8W of power (measured). Old servers are the same -- the amount of power they use for the performance makes them expensive to run 24x7.

craftyguy 9y ago
This is pretty much what I ended up doing. It's a collossal waste of power (and money) to run enterprise equipment in my area for a home network that is nowhere near enterprise-level in terms of utilization.
- stock_toaster 9y ago
  I also used to have more gear, but hav really scaled back and consolidated. I now have one fairly beefy server (storage array, decent zeon, lots of ram), run a few vms (dev, plex, etc), and export storage. Then I have a pair of NUCs doing various things.
  - cm2187 9y ago
    Plus hard drive capacity seems to have gone back on a steep increasing curve with 12TB drives coming to the market, offering further opportunities for consolidation.

rayiner 9y ago

You can get some good deals here too: https://forums.servethehome.com/index.php?forums/for-sale-for-trade-want-to-buy.2 https://forums.servethehome.com/index.php?forums/for-sale-fo.... I've been wiring up 10G throughout the house, and managed to get a ER-8-XG (beta version) on this site for under $500. (8 10-gig SFP+). Wife made me get rid of it because it was too loud. :( Got a Dell T1700 (they're like giving them away on EBay). Quad core Xeon E3, plenty of headroom to do firewall/routing in software.

jdironman 9y ago

Can you tell me more about how you did the Dell 2950? I'm looking at NAS server options and would love to roll my own. I have 2-3 of these sitting around.