14 ms·

EPYC Server Configuration

102 points by ihsw2 9y ago

dvfjsdhgfv 9y ago
I'm a great fan of Hetzner services. It's not that their servers don't break - it's the way they react to hardware failures. Average response time is in minutes - they quickly replace the faulty equipment and everything gets back to normal. I'm using their services intensively and it always amaze me how quickly they can fix the problem.
- sparrish 9y ago
  I've had exactly the opposite experience with Hetzner support. Slow to respond. Difficult to convince that hardware failure has occurred and never less than 3 hours, often more than 5 to get it replaced.
  - derefr 9y ago
    I'm guessing the difference comes down to failures they detect first, vs. failures you detect first. If they have a monitoring system that tells them something broke, they'll immediately go and fix it. If you tell them something broke, they'll have to go check. If you tell them something broke and they have a monitoring system that says it's fine, then you'll likely have problems.
    - TimWolla 9y ago
      If you are talking about dedicated servers: Those are unmanaged, Hetzner does not monitor those. Regarding grandparent: What kind of hardware failure? I had to change different types of hardware in the past. For HDD changes you send a full SMART result + possibly the logs showing that the RAID kicked the faulty disk and they replace the faulty disk in 30 minutes (and one time even apologized that it took 2 hours for a low-priority-support server). In all cases the actual downtime was < 5 minutes, because of RAID. For possibly broken fans I simply stated that according to my Munin logs show consistent temperatures near the upper limit and asked whether they could take a look. They added an additional fan within 30 minutes, again with actual downtime < 5 minutes. Most of my replacements were during the night in Germany, that might have helped, because there are possibly less requests in parallel. And I always had hard logs proving my HDD failures.
- katastic 9y ago
  I know this is silly, but I would have laughed like crazy if your comment about using their reliable services ended with your comment stopping halfway (like your station/internet died).
- therealidiot 9y ago
  I've had similar experiences with them. Had a CPU cooler fail somehow (resulting in one very toasty, unstable server) and they responded within minutes, the whole thing was probably resolved within half an hour.
- frozenice 9y ago
  I'm very satisfied, too. Servers are ready in minutes, not hours, after ordering. We currently run ~30 servers there (dedi) and over the last ~7 years roughly had 1 HDD fail per year. Each time the replacement was done on our chosen time and quickly in about 5 - 20 minutes (the installabled images come with pre-configured software RAID 1). We recently tried a AX60 (Ryzen 7 1700X Octa-Core) and had some sporadic crashes, support changed some BIOS settings, no problems since. Service and prices are unbeatable, imho.
ihsw2 9y ago
This is probably a better link than the product purchase page: https://www.hetzner.com/news/neuer-dedicated-root-server-ax160-high-performance-erleben/ https://www.hetzner.com/news/neuer-dedicated-root-server-ax1...
asah 9y ago
I've had a wonderful experience with hetzner. The web UI isn't as slick as AWS/GCP/Azure and bandwidth is slow, but the actual machines with great and they're a fraction of the cost. For my workload, I got 20x (2000%) better price-performance than Google, which itself beat Amazon. Killer for R&D.
- mschuster91 9y ago
  > but the actual machines with great and they're a fraction of the cost. The advantage of cloud stuff is if you need an extremely high amount of power for an extremely short amount of time. For example, I needed a hundred CPU cores + TB of RAM for a 15-minute time span, paid less than 100$ - renting this or, god forbid, buying this, would have racked up 10k+ bills for sure, for buying probably over 200k. Once you have base-line load that can't be served with micro/mini instances, there is no financial alternative to renting, and in some cases buying+colo may come in even cheaper (but requires HW maintenance personnel). Snapchat, for example, pays 2 billion dollars for 5Y to Google, equivalent to 400M$ a year (per http://www.zdnet.com/article/snapchat-spending-2-billion-over-5-years-for-google-cloud/ http://www.zdnet.com/article/snapchat-spending-2-billion-ove...) ... I wonder what their CFO (or any board member who approved this) has smoked. No way this huge amount of spend is justified - not even the taxing differences between capex (buying) and opex (cloud/renting) can justify this in any way. In addition, all cloud providers charge heavily for data egress, which really makes me wonder how on earth a service like Netflix can actually be profitable.
  - namibj 9y ago
    Netflix operates its own hardware in ISP networks and at internet exchanges to actually deliver the video. The only thing running in the cloud is all the database and UI backend, but no big video files.
  - ihsw2 9y ago
    Netflix serves data from its OpenConnect rack-mounted servers, of which are deployed to every major ISP/IXP. Previously they were using mainstream CDNs but there were growing pains and cost concerns. To sum it up, everything outside of "the play button" is on AWS. This includes billing, account management, et al.
Radim 9y ago
We've had our dev servers (heavy R&D work in machine learning) with Hetzner for 4 years now, never a glitch. Super nice, responsive support too. I've had their Head of Product, Arno Pirner, engage in a long email exchange when we inquired about GPU servers last year (they finally launched a GPU line recently). Great service.
- 3cham 9y ago
  I just thought: could it be Radim Rehurek. It is.
globile 9y ago
This is great news for DDOSers! </end_irony> 80% of threats and DDOS attacks we receive come from Hetzner IPs! We stopped wasting time reporting this to Hetzner and simply block whole IP ranges in Cloudflare. Sad but true.
- nh2 9y ago
  IP range blocking is extremely bad practice. You'll lose customers that way. It's like rejecting email addresses that don't end with .com. There are lots of services these days that can "smartly" handle DoS attacks (e.g. only drop traffic from an IP in the range when it starts flooding you).
  - Nokinside 9y ago
    Sorry for my ignorance, but how likely it is that customer traffic originates from Hetzner servers? Maybe someone runs VPN's trough them?
    - globile 9y ago
      This is one possible explanation. For example, You'd be surprised how many people run intrusion attacks from Kaperskys VPN service.... You see Kaperskys IPS on your logs and wonder...
  - globile 9y ago
    Range blocking is the last resort, and we've only done it after alerting Hetzner and getting no response. In any case, we only block for a few hours.... and yes, you definitely risk losing clients. I know Hetzner is not only DE based and many user cases for IPs originating are possible, but luckily DE is not our market.
- malikNF 9y ago
  Wouldn't cloudflare block these attacks automatically for you? I used cloudflare on a server that used to get dossed often and the attacks are now almost non existent when we got cloudflare in-front of us. Range blocking is a really silly thing to do don't you think? Hope you guys find a better way that that. Something tells me your application might be leaking your site's IP?
TazeTSchnitzel 9y ago
> 4 x 32 GB RAM Because of their four dual-channel memory controllers, EPYC CPUs support eight-channel RAM. So, for optimum performance, it would need twice that number of modules, right? I don't know the performance differential.
- namibj 9y ago
  No, they have 4 memory channels per Socket, and can use up to 2 DIMMs per channel. So provided only one DIMM is user per channel, this can fully utilize the meory bandwidth of a single socket. (It also depends on the memory module having enough ranks to provide enough internal parallelism, but the effect of that only shows in small memory accesses).
  - my123 9y ago
    It's actually 8 memory channels per socket (4 times dual-channel)
  - TazeTSchnitzel 9y ago
    Are you thinking of Threadripper? EPYC has twice as many dies.
thetopher 9y ago
My experience with Hetzner was mixed. Great pricing, pretty quick to recover from problems, but the problems were frequent. They're not great about maintaining good availability. I would definitely consider using them again for any project where availability isn't a serious concern, but never anything customer-facing.
KaiserPro 9y ago
as a smallish datapoint, an i3.4xlarge costs $ 1059.94 ondemand which is about 8 times as much, with nowhere near the throughput. I assume with reserved one could knock it down to ~$650
- tyingq 9y ago
  It also comes with an unmetered 1 Gbit/sec port. Assuming it's perfect, and you push it for a full month, AWS egress charges for that 328 terabytes: $20684.
  - asb 9y ago
    It's 50TB/month, not unmetered. Only 1.17EUR/TB over that though.
    - tyingq 9y ago
      Ahh, okay, so $4403 of AWS egress charges to match the 50TB included.
chx 9y ago
Note 512GB RAM costing 433.00 EUR a month (500 USD). Reminder: https://twitter.com/garybernhardt/status/600783770925420546 https://twitter.com/garybernhardt/status/600783770925420546 > Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000. Previously, the cheapest such server was 700 USD at OVH and it only included half this much of NVMe SSD. Also, I gave a talk years ago at NYCCamp trying to hammer in that most websites do not need all the scalability tricks as they fit into a single server. Well, this server can happily house a 7TB database on mirrored SSDs and that's still below 500 EUR (with "only" 128GB of RAM). You need to think very hard whether you need to shard, cluster etc.
- dvfjsdhgfv 9y ago
  Your consulting advice will give you cold looks from those whose living depends on selling people complex server/"cloud" configurations under fancy names.
  - DarronWyke 9y ago
    The Cloud-to-Butt plugin makes that particularly poignant.
  - chx 9y ago
    I am just channeling Gary Bernhardt who is the best. He also tweeted at one point that we will oil snakes into extinction.
- nikanj 9y ago
  You'll never get funding with that pitch, though. Hyper-modular microservice blockchain bus is what investors want to hear. They read on the internet that the only way to survive FB levels of users is webscaling, and they won't make a bet unless they think you'll be FB sized in three years.
  - wmf 9y ago
    But what if you call it NoScale™?
FBISurveillance 9y ago
My biggest issue / bottleneck with Hetzner has been their network. 1Gbps is nothing these days if you run something larger. A mid-sized Elasticsearch cluster will eat this 1Gbps for breakfast during shard reallocation. Other than than it has neat-perfect value per dollar.
- nh2 9y ago
  You can order 10Gbps network cards and ask Hetzner to put your servers in a rack and a 10Gbps switch between them. Pricing: https://wiki.hetzner.de/index.php/Root_Server_Hardware/en#Miscellaneous https://wiki.hetzner.de/index.php/Root_Server_Hardware/en#Mi... Excerpt: > 10 Gbit Intel network card: € 13.50 > 12-Port 10 Gbit switch: € 43.00
Nux 9y ago
Epic indeed. I've been using their services for a few years now. Very good service all in all and very good value for the money. Thumbs up, Hetzner.
noir_lord 9y ago
I had massive issues with Hetzner the last time I tried a dedicated box with them (though that was 3ish years ago now). In about a month I had a handful of days where I wasn't having issues, half the time I'd open a ticket, report a problem then a bit later get "yes, we are looking into that now". Left a bad taste, these days I just use linode for most things, if I needed heavy duty processing power there are some decently priced options near to me (bytemark is about 60 miles away from me).
mtgx 9y ago
Huge fan of Hetzner being so aggressive with its AMD Ryzen/EPYC chip adoption, but I can't ignore the fact that they had a huge data breach recently, and all customer info, including passwords, was in plaintext! I'm surprised nobody here mentioned it. Didn't Hetzner notify its customers about it? https://techcentral.co.za/hetzner-hacked-customer-details-compromised/77920/ https://techcentral.co.za/hetzner-hacked-customer-details-co... https://techcentral.co.za/hetzner-deeply-distressed-data-breach/77963/ https://techcentral.co.za/hetzner-deeply-distressed-data-bre...
- TimWolla 9y ago
  Hetzner Online GmbH is a different entity than Hetzner (Pty) Ltd.
nextos 9y ago
Is it possible to buy EPYC workstations already? Components are quite scarce in the UK both for EPYC and Xeon Skylake.

102 points by ihsw2 9y ago

I'm a great fan of Hetzner services. It's not that their servers don't break - it's the way they react to hardware failures. Average response time is in minutes - they quickly replace the faulty equipment and everything gets back to normal. I'm using their services intensively and it always amaze me how quickly they can fix the problem.

sparrish 9y ago
I've had exactly the opposite experience with Hetzner support. Slow to respond. Difficult to convince that hardware failure has occurred and never less than 3 hours, often more than 5 to get it replaced.
- derefr 9y ago
  I'm guessing the difference comes down to failures they detect first, vs. failures you detect first. If they have a monitoring system that tells them something broke, they'll immediately go and fix it. If you tell them something broke, they'll have to go check. If you tell them something broke and they have a monitoring system that says it's fine, then you'll likely have problems.
  - TimWolla 9y ago
    If you are talking about dedicated servers: Those are unmanaged, Hetzner does not monitor those. Regarding grandparent: What kind of hardware failure? I had to change different types of hardware in the past. For HDD changes you send a full SMART result + possibly the logs showing that the RAID kicked the faulty disk and they replace the faulty disk in 30 minutes (and one time even apologized that it took 2 hours for a low-priority-support server). In all cases the actual downtime was < 5 minutes, because of RAID. For possibly broken fans I simply stated that according to my Munin logs show consistent temperatures near the upper limit and asked whether they could take a look. They added an additional fan within 30 minutes, again with actual downtime < 5 minutes. Most of my replacements were during the night in Germany, that might have helped, because there are possibly less requests in parallel. And I always had hard logs proving my HDD failures.
katastic 9y ago
I know this is silly, but I would have laughed like crazy if your comment about using their reliable services ended with your comment stopping halfway (like your station/internet died).
therealidiot 9y ago
I've had similar experiences with them. Had a CPU cooler fail somehow (resulting in one very toasty, unstable server) and they responded within minutes, the whole thing was probably resolved within half an hour.
frozenice 9y ago
I'm very satisfied, too. Servers are ready in minutes, not hours, after ordering. We currently run ~30 servers there (dedi) and over the last ~7 years roughly had 1 HDD fail per year. Each time the replacement was done on our chosen time and quickly in about 5 - 20 minutes (the installabled images come with pre-configured software RAID 1). We recently tried a AX60 (Ryzen 7 1700X Octa-Core) and had some sporadic crashes, support changed some BIOS settings, no problems since. Service and prices are unbeatable, imho.

ihsw2 9y ago

This is probably a better link than the product purchase page: https://www.hetzner.com/news/neuer-dedicated-root-server-ax160-high-performance-erleben/ https://www.hetzner.com/news/neuer-dedicated-root-server-ax1...

asah 9y ago

I've had a wonderful experience with hetzner. The web UI isn't as slick as AWS/GCP/Azure and bandwidth is slow, but the actual machines with great and they're a fraction of the cost. For my workload, I got 20x (2000%) better price-performance than Google, which itself beat Amazon. Killer for R&D.

mschuster91 9y ago
> but the actual machines with great and they're a fraction of the cost. The advantage of cloud stuff is if you need an extremely high amount of power for an extremely short amount of time. For example, I needed a hundred CPU cores + TB of RAM for a 15-minute time span, paid less than 100$ - renting this or, god forbid, buying this, would have racked up 10k+ bills for sure, for buying probably over 200k. Once you have base-line load that can't be served with micro/mini instances, there is no financial alternative to renting, and in some cases buying+colo may come in even cheaper (but requires HW maintenance personnel). Snapchat, for example, pays 2 billion dollars for 5Y to Google, equivalent to 400M$ a year (per http://www.zdnet.com/article/snapchat-spending-2-billion-over-5-years-for-google-cloud/ http://www.zdnet.com/article/snapchat-spending-2-billion-ove...) ... I wonder what their CFO (or any board member who approved this) has smoked. No way this huge amount of spend is justified - not even the taxing differences between capex (buying) and opex (cloud/renting) can justify this in any way. In addition, all cloud providers charge heavily for data egress, which really makes me wonder how on earth a service like Netflix can actually be profitable.
- namibj 9y ago
  Netflix operates its own hardware in ISP networks and at internet exchanges to actually deliver the video. The only thing running in the cloud is all the database and UI backend, but no big video files.
- ihsw2 9y ago
  Netflix serves data from its OpenConnect rack-mounted servers, of which are deployed to every major ISP/IXP. Previously they were using mainstream CDNs but there were growing pains and cost concerns. To sum it up, everything outside of "the play button" is on AWS. This includes billing, account management, et al.

Radim 9y ago

We've had our dev servers (heavy R&D work in machine learning) with Hetzner for 4 years now, never a glitch. Super nice, responsive support too. I've had their Head of Product, Arno Pirner, engage in a long email exchange when we inquired about GPU servers last year (they finally launched a GPU line recently). Great service.

3cham 9y ago
I just thought: could it be Radim Rehurek. It is.

globile 9y ago

This is great news for DDOSers! </end_irony> 80% of threats and DDOS attacks we receive come from Hetzner IPs! We stopped wasting time reporting this to Hetzner and simply block whole IP ranges in Cloudflare. Sad but true.

nh2 9y ago
IP range blocking is extremely bad practice. You'll lose customers that way. It's like rejecting email addresses that don't end with .com. There are lots of services these days that can "smartly" handle DoS attacks (e.g. only drop traffic from an IP in the range when it starts flooding you).
- Nokinside 9y ago
  Sorry for my ignorance, but how likely it is that customer traffic originates from Hetzner servers? Maybe someone runs VPN's trough them?
  - globile 9y ago
    This is one possible explanation. For example, You'd be surprised how many people run intrusion attacks from Kaperskys VPN service.... You see Kaperskys IPS on your logs and wonder...
- globile 9y ago
  Range blocking is the last resort, and we've only done it after alerting Hetzner and getting no response. In any case, we only block for a few hours.... and yes, you definitely risk losing clients. I know Hetzner is not only DE based and many user cases for IPs originating are possible, but luckily DE is not our market.
malikNF 9y ago
Wouldn't cloudflare block these attacks automatically for you? I used cloudflare on a server that used to get dossed often and the attacks are now almost non existent when we got cloudflare in-front of us. Range blocking is a really silly thing to do don't you think? Hope you guys find a better way that that. Something tells me your application might be leaking your site's IP?

TazeTSchnitzel 9y ago

> 4 x 32 GB RAM Because of their four dual-channel memory controllers, EPYC CPUs support eight-channel RAM. So, for optimum performance, it would need twice that number of modules, right? I don't know the performance differential.

namibj 9y ago
No, they have 4 memory channels per Socket, and can use up to 2 DIMMs per channel. So provided only one DIMM is user per channel, this can fully utilize the meory bandwidth of a single socket. (It also depends on the memory module having enough ranks to provide enough internal parallelism, but the effect of that only shows in small memory accesses).
- my123 9y ago
  It's actually 8 memory channels per socket (4 times dual-channel)
- TazeTSchnitzel 9y ago
  Are you thinking of Threadripper? EPYC has twice as many dies.

thetopher 9y ago

My experience with Hetzner was mixed. Great pricing, pretty quick to recover from problems, but the problems were frequent. They're not great about maintaining good availability. I would definitely consider using them again for any project where availability isn't a serious concern, but never anything customer-facing.

KaiserPro 9y ago

as a smallish datapoint, an i3.4xlarge costs $ 1059.94 ondemand which is about 8 times as much, with nowhere near the throughput. I assume with reserved one could knock it down to ~$650

tyingq 9y ago
It also comes with an unmetered 1 Gbit/sec port. Assuming it's perfect, and you push it for a full month, AWS egress charges for that 328 terabytes: $20684.
- asb 9y ago
  It's 50TB/month, not unmetered. Only 1.17EUR/TB over that though.
  - tyingq 9y ago
    Ahh, okay, so $4403 of AWS egress charges to match the 50TB included.

chx 9y ago

Note 512GB RAM costing 433.00 EUR a month (500 USD). Reminder: https://twitter.com/garybernhardt/status/600783770925420546 https://twitter.com/garybernhardt/status/600783770925420546 > Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000. Previously, the cheapest such server was 700 USD at OVH and it only included half this much of NVMe SSD. Also, I gave a talk years ago at NYCCamp trying to hammer in that most websites do not need all the scalability tricks as they fit into a single server. Well, this server can happily house a 7TB database on mirrored SSDs and that's still below 500 EUR (with "only" 128GB of RAM). You need to think very hard whether you need to shard, cluster etc.

dvfjsdhgfv 9y ago
Your consulting advice will give you cold looks from those whose living depends on selling people complex server/"cloud" configurations under fancy names.
- DarronWyke 9y ago
  The Cloud-to-Butt plugin makes that particularly poignant.
- chx 9y ago
  I am just channeling Gary Bernhardt who is the best. He also tweeted at one point that we will oil snakes into extinction.
nikanj 9y ago
You'll never get funding with that pitch, though. Hyper-modular microservice blockchain bus is what investors want to hear. They read on the internet that the only way to survive FB levels of users is webscaling, and they won't make a bet unless they think you'll be FB sized in three years.
- wmf 9y ago
  But what if you call it NoScale™?

FBISurveillance 9y ago

My biggest issue / bottleneck with Hetzner has been their network. 1Gbps is nothing these days if you run something larger. A mid-sized Elasticsearch cluster will eat this 1Gbps for breakfast during shard reallocation. Other than than it has neat-perfect value per dollar.

nh2 9y ago
You can order 10Gbps network cards and ask Hetzner to put your servers in a rack and a 10Gbps switch between them. Pricing: https://wiki.hetzner.de/index.php/Root_Server_Hardware/en#Miscellaneous https://wiki.hetzner.de/index.php/Root_Server_Hardware/en#Mi... Excerpt: > 10 Gbit Intel network card: € 13.50 > 12-Port 10 Gbit switch: € 43.00

Nux 9y ago

Epic indeed. I've been using their services for a few years now. Very good service all in all and very good value for the money. Thumbs up, Hetzner.

noir_lord 9y ago

I had massive issues with Hetzner the last time I tried a dedicated box with them (though that was 3ish years ago now). In about a month I had a handful of days where I wasn't having issues, half the time I'd open a ticket, report a problem then a bit later get "yes, we are looking into that now". Left a bad taste, these days I just use linode for most things, if I needed heavy duty processing power there are some decently priced options near to me (bytemark is about 60 miles away from me).

mtgx 9y ago

Huge fan of Hetzner being so aggressive with its AMD Ryzen/EPYC chip adoption, but I can't ignore the fact that they had a huge data breach recently, and all customer info, including passwords, was in plaintext! I'm surprised nobody here mentioned it. Didn't Hetzner notify its customers about it? https://techcentral.co.za/hetzner-hacked-customer-details-compromised/77920/ https://techcentral.co.za/hetzner-hacked-customer-details-co... https://techcentral.co.za/hetzner-deeply-distressed-data-breach/77963/ https://techcentral.co.za/hetzner-deeply-distressed-data-bre...

TimWolla 9y ago
Hetzner Online GmbH is a different entity than Hetzner (Pty) Ltd.

nextos 9y ago

Is it possible to buy EPYC workstations already? Components are quite scarce in the UK both for EPYC and Xeon Skylake.