3 ms·

> That said when I want to run a public server I just rent a VPS and run it on that I have a setup similar to the OP and there's a really good reason for it: c

> That said when I want to run a public server I just rent a VPS and run it on that I have a setup similar to the OP and there's a really good reason for it: cost. For $2k in hardware (one time) and a VPS + electricity at $20/mo, I can run a 64 core, 192Gb, 24TB server. Let's call that $4400 over the course of 10 years. You would burn through that budget in roughly 1 month to get the same specs on AWS. Obviously I'm neglecting the impact and cost of network, if it's a hobby you can just reuse your existing connection but you can always buy a dedicated line (and adjust the cost calculation accordingly) In terms of security, you can mitigate the surface area by running a reverse proxy on the VPS. I've got nginx on the front line which does TLS then proxy_passes to my basement server at its wireguard IP address. So it's strictly limited to http, no direct database or ssh access. I don't know if I'd ever run a "real" website this way but it's great for hobbies and side projects.

deleted 5y ago
[deleted]
fgonzag 5y ago
Explain how you got a 64 core 192 GB RAM 24 TB server or workstation for 2k please
- hobs 5y ago
  Same, that's going to be like 4x that price refurbished around the best case...
- dinosaurdynasty 5y ago
  Ebay I would guess. Requires a bit of luck though
- mardifoufs 5y ago
  Not ideal but you can go for used rack mounted servers. You can easily get even better specs for less than 2k with reasonably recent CPUs too (E5-xxxxv3 and up). At least that was the case when I set up my home server rack ~2 years ago. (Though 24TB might be too much since server grade SAS storage is very expensive even when it's used. So what I did was to buy servers with as little storage as possible and then just put SATA HDDs that I can get for dirt cheap.)
- r3trohack3r 5y ago
  Not OP, but I have an R620 running in my garage I use for self hosting. It has 12 cores (Intel Xeon E5-2665), 96GB of RAM, 10x2.5" drive bays (came empty). The SSDs I use I shuck out of old external USB drives. Together with a full standing rack on wheels, I bought them for $600 cash (plus U-Haul rental to get it home). My trick is craigslist in a city with datacenters (PHX). There are resellers here who buy used hardware from local datacenters and resell them to smaller businesses. I've had it running 24x7 for almost 3 years now and it is still going strong. https://www.reddit.com/r/homelab/ https://www.reddit.com/r/homelab/ has some good references for acquiring hardware for a home lab setup.
  - fgonzag 5y ago
    That makes sense. 2 x 6 core Xeons were cheap even back in the day. And it came without storage. Again, as someone who actually buys straight off FB and Google when they dispose servers (as the companies who you buy from off eBay), I'd really like to know where he got that specific 64 core (which would be a quad socket if it's older than 2015) 192 GB 24 TB server because I'd like to order more than a few. Your server is not even worth 1/20th of what his is worth. The closest I can get a disposed server with those specs is at least 8k
- washadjeffmad 5y ago
  LGA 2011, DDR3, Easystore shucks or data center decomms (8x3TB was $200). I got 48 vCPUs and 96GB DDR3 for <$300 in 2016 off eBay. It isn't the most powerful, but it was cheap.
  - capableweb 5y ago
    vCPUs is very different than dedicated cores. My 8 core machine could have 48 vCPUs as well, it doesn't say anything.
- deleted 5y ago
  [deleted]
- perrygeo 5y ago
  Refurbished Supermiro from The Server Store, there were plenty 2 years ago when I bought it - market has probably changed since then.
renonce 5y ago
Would be surprised if you would still be using that server actively after 10 years. I wonder how you build such a hardware with just $2k (3990x alone seems to cost $3.5k), but with €94/mon you can get 16C32T, 128Gb, 8TB server from Hetzner[1]. It has Zen 3 5950X which has better single core performance than your 64 core assuming that you refer to 3990X, and often single core perf is more important than total perf because you don't use all cores all the time. I estimate such a configuration to be $2.5k. With $2.5k you can keep the server for 2 years, with network and electricity provided by Hetzner. Also you can terminate it and rent another, better server at any time. [1]: https://www.hetzner.com/dedicated-rootserver/matrix-ax https://www.hetzner.com/dedicated-rootserver/matrix-ax
- zwayhowder 5y ago
  Most things don't need that much power. My homelab, NextCloud and family Plex server is a 10 year old i7 with 32gb ram. The only upgrade in 10 years was a PCIe NVME adapter so I could boot from faster storage.
- perrygeo 5y ago
  2 years in, it's working fine and already paid for itself. I hope it makes it 10 years. Why do you think it won't?
- jjav 5y ago
  > Would be surprised if you would still be using that server actively after 10 years. I thank the death of Moore's Law. Compute power obsoletes very slowly these days. Back in the 90s I was buying new equipment every year or two. My primary hobby server is now 12 years old and going strong. My youngest server is 8 years old. Every now and then I start shopping for an upgrade but eventually decide I'll spend the money later. It's still good enough for my needs and works well.
- marginalia_nu 5y ago
  My search engine is self-hosted off hardware with those rough storage specs (well, sans ECC). I do not regret one bit going for this set-up, although my next step toward scaling this up is indeed going to be some sort of hosted solution, the primary reason being the BTUs and decibels involved in running an actual server from my apartment. Running a search engine or not, anyone with a serious interest in programming should have a server. I really can't recommend it enough. The ability to run large jobs that chew through hundreds of gigabytes of data for weeks if need be, without considering the cost, that dramatically increases the level of ambition you can have in your projects. * I had an idea for a re-skinned wikipedia. So I processed all 40 gigabytes of wikipedia articles from an OpenZIM file. Took a week. No big deal. Later I used the same corpus to calculate language data models. That's another several days. * I had an idea a while back. It required me to screengrab 500,000 websites. So I wrote a python script. Takes anywhere between 0.5-120 seconds per site. It's still not finished by any stretch, but I'm up to about 322,000 screenshots now. It's just great.
riku_iki 5y ago
> I have a setup similar to the OP and there's a really good reason for it: cost. For $2k in hardware I think in you case colocation data center would be alternative solution.
- perrygeo 5y ago
  I would but my local colo offers max 25 Mbps (!) networking unless you get the enterprise plan. I'm guessing there's better colo options near a big city?
  - rosndo 5y ago
    You don’t have to colo locally.
Dracophoenix 5y ago
A machine like that is only 2k? Can you break down the cost into parts?I'm guessing the CPU is the Ryzen 3990X. Is the RAM ECC ? Are the drives in a RAID configuration? Is it kept in a server rack? What router, switch, and firewall are your using?
- perrygeo 5y ago
  It's a refurb supermicro 2U. DDR4 ECC Ram, 6 spinny disks in a zfs pool. It's older gen hardware and it's really loud. I put it on a makeshift rack and put it far away, heating the corner of my basement! I've got a consumer wifi router with DD-WRT, router has never been the bottleneck (that would be Comcast).
  - Dracophoenix 5y ago
    Are you running Linux, FreeBSD or Solaris?
dangus 5y ago
I wouldn’t necessarily argue that the cloud is cheaper as a rule, but I would say that this comparison misrepresents how you’re supposed to use cloud computing and what its strengths are. That 64 core server is probably sitting nearly idle most of the time. You don’t just spin up big systems on AWS and leave them idle. The whole point of cloud computing is the on-demand scalability. You can basically spend nothing until a request comes in, and most hobby projects are low traffic. For example, I found a project that runs plex media server encoding jobs as Kubernetes pods: https://github.com/munnerz/kube-plex https://github.com/munnerz/kube-plex When you’re not encoding anything, you’re not running much compute. I think I’ve heard of game servers that wait for players to attempt to connect before starting the instance. If all you’re doing is playing a multiplayer game with friends that instance is going to be off 20+ hours a day. Still, I haven’t done any napkin math on what applications represent cost savings.
- perrygeo 5y ago
  > The whole point of cloud computing is the on-demand scalability. True, my system is fairly static. But it scales when I need it. As mentioned, it's for my personal use. "on-demand scalability" is a bullshit buzzword in this context. My goal was to buy a machine capable of several specific tasks. > That 64 core server is probably sitting nearly idle most of the time > ... plex media server > ... encoding > ... game servers > ... that instance is going to be off 20+ hours a day. Not those tasks :-) I have no trouble maxxing out my disk IO and memory, slightly more challenging to keep my CPU busy but it's keeping an average load of 4-8. If you can't keep a machine busy, just turn it off. Simple as that. I would not have bought such a beast if I didn't have a plan keep it busy!

deleted 5y ago

[deleted]

fgonzag 5y ago

Explain how you got a 64 core 192 GB RAM 24 TB server or workstation for 2k please

hobs 5y ago
Same, that's going to be like 4x that price refurbished around the best case...
dinosaurdynasty 5y ago
Ebay I would guess. Requires a bit of luck though
mardifoufs 5y ago
Not ideal but you can go for used rack mounted servers. You can easily get even better specs for less than 2k with reasonably recent CPUs too (E5-xxxxv3 and up). At least that was the case when I set up my home server rack ~2 years ago. (Though 24TB might be too much since server grade SAS storage is very expensive even when it's used. So what I did was to buy servers with as little storage as possible and then just put SATA HDDs that I can get for dirt cheap.)
r3trohack3r 5y ago
Not OP, but I have an R620 running in my garage I use for self hosting. It has 12 cores (Intel Xeon E5-2665), 96GB of RAM, 10x2.5" drive bays (came empty). The SSDs I use I shuck out of old external USB drives. Together with a full standing rack on wheels, I bought them for $600 cash (plus U-Haul rental to get it home). My trick is craigslist in a city with datacenters (PHX). There are resellers here who buy used hardware from local datacenters and resell them to smaller businesses. I've had it running 24x7 for almost 3 years now and it is still going strong. https://www.reddit.com/r/homelab/ https://www.reddit.com/r/homelab/ has some good references for acquiring hardware for a home lab setup.
- fgonzag 5y ago
  That makes sense. 2 x 6 core Xeons were cheap even back in the day. And it came without storage. Again, as someone who actually buys straight off FB and Google when they dispose servers (as the companies who you buy from off eBay), I'd really like to know where he got that specific 64 core (which would be a quad socket if it's older than 2015) 192 GB 24 TB server because I'd like to order more than a few. Your server is not even worth 1/20th of what his is worth. The closest I can get a disposed server with those specs is at least 8k
washadjeffmad 5y ago
LGA 2011, DDR3, Easystore shucks or data center decomms (8x3TB was $200). I got 48 vCPUs and 96GB DDR3 for <$300 in 2016 off eBay. It isn't the most powerful, but it was cheap.
- capableweb 5y ago
  vCPUs is very different than dedicated cores. My 8 core machine could have 48 vCPUs as well, it doesn't say anything.
deleted 5y ago
[deleted]
perrygeo 5y ago
Refurbished Supermiro from The Server Store, there were plenty 2 years ago when I bought it - market has probably changed since then.

renonce 5y ago

Would be surprised if you would still be using that server actively after 10 years. I wonder how you build such a hardware with just $2k (3990x alone seems to cost $3.5k), but with €94/mon you can get 16C32T, 128Gb, 8TB server from Hetzner[1]. It has Zen 3 5950X which has better single core performance than your 64 core assuming that you refer to 3990X, and often single core perf is more important than total perf because you don't use all cores all the time. I estimate such a configuration to be $2.5k. With $2.5k you can keep the server for 2 years, with network and electricity provided by Hetzner. Also you can terminate it and rent another, better server at any time. [1]: https://www.hetzner.com/dedicated-rootserver/matrix-ax https://www.hetzner.com/dedicated-rootserver/matrix-ax

zwayhowder 5y ago
Most things don't need that much power. My homelab, NextCloud and family Plex server is a 10 year old i7 with 32gb ram. The only upgrade in 10 years was a PCIe NVME adapter so I could boot from faster storage.
perrygeo 5y ago
2 years in, it's working fine and already paid for itself. I hope it makes it 10 years. Why do you think it won't?
jjav 5y ago
> Would be surprised if you would still be using that server actively after 10 years. I thank the death of Moore's Law. Compute power obsoletes very slowly these days. Back in the 90s I was buying new equipment every year or two. My primary hobby server is now 12 years old and going strong. My youngest server is 8 years old. Every now and then I start shopping for an upgrade but eventually decide I'll spend the money later. It's still good enough for my needs and works well.
marginalia_nu 5y ago
My search engine is self-hosted off hardware with those rough storage specs (well, sans ECC). I do not regret one bit going for this set-up, although my next step toward scaling this up is indeed going to be some sort of hosted solution, the primary reason being the BTUs and decibels involved in running an actual server from my apartment. Running a search engine or not, anyone with a serious interest in programming should have a server. I really can't recommend it enough. The ability to run large jobs that chew through hundreds of gigabytes of data for weeks if need be, without considering the cost, that dramatically increases the level of ambition you can have in your projects. * I had an idea for a re-skinned wikipedia. So I processed all 40 gigabytes of wikipedia articles from an OpenZIM file. Took a week. No big deal. Later I used the same corpus to calculate language data models. That's another several days. * I had an idea a while back. It required me to screengrab 500,000 websites. So I wrote a python script. Takes anywhere between 0.5-120 seconds per site. It's still not finished by any stretch, but I'm up to about 322,000 screenshots now. It's just great.

riku_iki 5y ago

> I have a setup similar to the OP and there's a really good reason for it: cost. For $2k in hardware I think in you case colocation data center would be alternative solution.

perrygeo 5y ago
I would but my local colo offers max 25 Mbps (!) networking unless you get the enterprise plan. I'm guessing there's better colo options near a big city?
- rosndo 5y ago
  You don’t have to colo locally.

Dracophoenix 5y ago

A machine like that is only 2k? Can you break down the cost into parts?I'm guessing the CPU is the Ryzen 3990X. Is the RAM ECC ? Are the drives in a RAID configuration? Is it kept in a server rack? What router, switch, and firewall are your using?

perrygeo 5y ago
It's a refurb supermicro 2U. DDR4 ECC Ram, 6 spinny disks in a zfs pool. It's older gen hardware and it's really loud. I put it on a makeshift rack and put it far away, heating the corner of my basement! I've got a consumer wifi router with DD-WRT, router has never been the bottleneck (that would be Comcast).
- Dracophoenix 5y ago
  Are you running Linux, FreeBSD or Solaris?

dangus 5y ago

I wouldn’t necessarily argue that the cloud is cheaper as a rule, but I would say that this comparison misrepresents how you’re supposed to use cloud computing and what its strengths are. That 64 core server is probably sitting nearly idle most of the time. You don’t just spin up big systems on AWS and leave them idle. The whole point of cloud computing is the on-demand scalability. You can basically spend nothing until a request comes in, and most hobby projects are low traffic. For example, I found a project that runs plex media server encoding jobs as Kubernetes pods: https://github.com/munnerz/kube-plex https://github.com/munnerz/kube-plex When you’re not encoding anything, you’re not running much compute. I think I’ve heard of game servers that wait for players to attempt to connect before starting the instance. If all you’re doing is playing a multiplayer game with friends that instance is going to be off 20+ hours a day. Still, I haven’t done any napkin math on what applications represent cost savings.

perrygeo 5y ago
> The whole point of cloud computing is the on-demand scalability. True, my system is fairly static. But it scales when I need it. As mentioned, it's for my personal use. "on-demand scalability" is a bullshit buzzword in this context. My goal was to buy a machine capable of several specific tasks. > That 64 core server is probably sitting nearly idle most of the time > ... plex media server > ... encoding > ... game servers > ... that instance is going to be off 20+ hours a day. Not those tasks :-) I have no trouble maxxing out my disk IO and memory, slightly more challenging to keep my CPU busy but it's keeping an average load of 4-8. If you can't keep a machine busy, just turn it off. Simple as that. I would not have bought such a beast if I didn't have a plan keep it busy!