4 ms·

AMD Inception Mitigations Cause 20% Performance Drop in Database Servers

101 points by nitinreddy88 3y ago

icegreentea2 3y ago
Other workloads with noticeable performance hits (at least relevant to my day to day work): * Code compilation - Linux kernel compilation takes ~15% longer with any of the mitigations on. LLVM and Node compilation are less substantial for kernel only, or safe RET microcode fix, but still ~15% longer for the IBPD mitigation. Though the Linux kernel gap seems much smaller (~7%) when tested on desktop ryzen (zen 4) instead of EPYC (zen 3) * Numpy benchmark - at least ~10% loss with mitigations (on EPYC) * nginx - at least ~15% with loss with mitigations (on EPYC) Most other impacts limited to less than 5% (if any).
ncann 3y ago
Too bad https://make-linux-fast-again.com/ https://make-linux-fast-again.com/ isn't loading anymore. Anyone knows an up to date guide for disabling all mitigations? I found https://gist.github.com/jfeilbach/f06bb8408626383a083f68276fa4df8a https://gist.github.com/jfeilbach/f06bb8408626383a083f68276f... but it was last updated in March (?). Also is there an equivalent for Windows?
- jraph 3y ago
  > Anyone knows an up to date guide for disabling all mitigations? IIUC, mitigations=off should disable them all. No need to specify them one by one and keep track of all of them. Of course, I'm not recommending this, and you should only do it if you know what you are doing.
- viraptor 3y ago
  The official docs contain it: Linux: https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html https://www.kernel.org/doc/html/latest/admin-guide/kernel-pa... (see the mitigations option)
vardump 3y ago
Any word on impact on zen4?
- icegreentea2 3y ago
  Not sure if you noticed, but page 6-8 from the article are actually benchmarking results from a zen 4 (Desktop Ryzen 7950).
  - vardump 3y ago
    Completely missed it, thanks!
- mycall 3y ago
  For M2 for that matter.
stephen_g 3y ago
Unfortunate, but I guess it's a bit like how safety regulations make building slower but we accept it for the far, far fewer workers dying or being maimed in accidents... Is this Inception issue the kind of thing that can be solved more efficiently in silicon with future generations, or is the mitigation something we have to accept and just go for faster processors to make up for it in the future?
- ars 3y ago
  I'm not familiar with this specific mitigation, but aren't they often only need in multi user environments? Meaning if you have a dedicated machine for your database, you probably don't want to turn these on. i.e. it's good to know that this is something you probably don't want.
  - deleted 3y ago
    [deleted]
  - hulitu 3y ago
    Your web browser is a multiuser environment with no access control.
    - oliwarner 3y ago
      Yes, but the scenario was >> if you have a dedicated machine for your database That's not a machine with a browser. The game then becomes: can somebody "inject" this sort of attack through a database connection. Good practice should insulate you but who knows.
  - danpalmer 3y ago
    Even in that scenario it’s normal to run the database as a non-root user. This is important so that if there’s a database escape vulnerability, some sort of RCE, then the attacker doesn’t have full control over the machine and would still need another privilege escalation path to do the most damage. You could skip this and run the DB as root, but that’s a big security trade off that most users shouldn’t be taking.
rob74 3y ago
Actual title: "Benchmarking The Performance Impact To AMD Inception Mitigations"
bjoli 3y ago
I discovered this myself yesterday. I suddenly saw a drop in performance after an update and ran a phoronix benchmark I have been running to see results from my overclock. I got a 5s (~8% ) increase in timed Linux kernel compilation. I turned mitigations off and was back to normal. I was going to try to figure it out today, but this is probably it.
pezezin 3y ago
Honest question, what are the real world impacts of these kind of vulnerabilities? If I am running a few on-premises private servers where I control all the software, should I be worried?
- NullPrefix 3y ago
  >I control all the software A bold statement to make. Even if it would be true, do you control all the inputs for that software or can other people feed it with data?
- x-complexity 3y ago
  If you can lock down ALL the software used & disable remote access, you'll likely never encounter this vulnerability. It requires local unprivileged access in order to be used, so if you can restrict that to a few allowed processes, you'll be 90% fine.
- viraptor 3y ago
  Do you execute any untrusted code submitted by the users? It not, then it's very likely not an issue for you.* This mostly allows escaping from a sandbox or getting elevated privileges on a host. But in those cases you're already hacked and the attacker has access to your data, so getting root on some server likely isn't a bigger deal than the initial hack. * there's a chance that this enables a remote exploit through some specially crafted data for some popular library... the chances are very slim though
exabrial 3y ago
Apologies if I ask a stupid question: is it necessary to run the mitigations on a database server?
- sebazzz 3y ago
  And even better: can we disable the mitigations while still keeping up to date with patches and BIOS updates?
- yencabulator 3y ago
  If that database server is a VM (cloud host or on-prem), the host will likely decide to have mitigations on. If the database server gives shell to DBAs or such who don't get root, perhaps yes. If the owners wish to restrict the attack surface of most of the non-database software running on a current day fat linux distro, perhaps yes. No mitigations means buffer overflow in any network-exposed code (DHCP client, DNS resolver, SSH server, Prometheus metrics exporter, ...) could lead to a ransomware attack on the database etc.
washadjeffmad 3y ago
Related story (5 days ago) "Intel's Downfall Mitigations Drop Performance Up to 39%, Tests Show": https://news.ycombinator.com/item?id=37085043 https://news.ycombinator.com/item?id=37085043
bob1029 3y ago
By the time the security bullies are done "fixing" my CPU, I will be able to open my start menu faster using an abacus.
paul_funyun 3y ago
Mitigations=off is aging like fine wine.

101 points by nitinreddy88 3y ago

Other workloads with noticeable performance hits (at least relevant to my day to day work): * Code compilation - Linux kernel compilation takes ~15% longer with any of the mitigations on. LLVM and Node compilation are less substantial for kernel only, or safe RET microcode fix, but still ~15% longer for the IBPD mitigation. Though the Linux kernel gap seems much smaller (~7%) when tested on desktop ryzen (zen 4) instead of EPYC (zen 3) * Numpy benchmark - at least ~10% loss with mitigations (on EPYC) * nginx - at least ~15% with loss with mitigations (on EPYC) Most other impacts limited to less than 5% (if any).

ncann 3y ago

Too bad https://make-linux-fast-again.com/ https://make-linux-fast-again.com/ isn't loading anymore. Anyone knows an up to date guide for disabling all mitigations? I found https://gist.github.com/jfeilbach/f06bb8408626383a083f68276fa4df8a https://gist.github.com/jfeilbach/f06bb8408626383a083f68276f... but it was last updated in March (?). Also is there an equivalent for Windows?

jraph 3y ago
> Anyone knows an up to date guide for disabling all mitigations? IIUC, mitigations=off should disable them all. No need to specify them one by one and keep track of all of them. Of course, I'm not recommending this, and you should only do it if you know what you are doing.
viraptor 3y ago
The official docs contain it: Linux: https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html https://www.kernel.org/doc/html/latest/admin-guide/kernel-pa... (see the mitigations option)

vardump 3y ago

Any word on impact on zen4?

icegreentea2 3y ago
Not sure if you noticed, but page 6-8 from the article are actually benchmarking results from a zen 4 (Desktop Ryzen 7950).
- vardump 3y ago
  Completely missed it, thanks!
mycall 3y ago
For M2 for that matter.

stephen_g 3y ago

Unfortunate, but I guess it's a bit like how safety regulations make building slower but we accept it for the far, far fewer workers dying or being maimed in accidents... Is this Inception issue the kind of thing that can be solved more efficiently in silicon with future generations, or is the mitigation something we have to accept and just go for faster processors to make up for it in the future?

ars 3y ago
I'm not familiar with this specific mitigation, but aren't they often only need in multi user environments? Meaning if you have a dedicated machine for your database, you probably don't want to turn these on. i.e. it's good to know that this is something you probably don't want.
- deleted 3y ago
  [deleted]
- hulitu 3y ago
  Your web browser is a multiuser environment with no access control.
  - oliwarner 3y ago
    Yes, but the scenario was >> if you have a dedicated machine for your database That's not a machine with a browser. The game then becomes: can somebody "inject" this sort of attack through a database connection. Good practice should insulate you but who knows.
- danpalmer 3y ago
  Even in that scenario it’s normal to run the database as a non-root user. This is important so that if there’s a database escape vulnerability, some sort of RCE, then the attacker doesn’t have full control over the machine and would still need another privilege escalation path to do the most damage. You could skip this and run the DB as root, but that’s a big security trade off that most users shouldn’t be taking.

rob74 3y ago

Actual title: "Benchmarking The Performance Impact To AMD Inception Mitigations"

bjoli 3y ago

I discovered this myself yesterday. I suddenly saw a drop in performance after an update and ran a phoronix benchmark I have been running to see results from my overclock. I got a 5s (~8% ) increase in timed Linux kernel compilation. I turned mitigations off and was back to normal. I was going to try to figure it out today, but this is probably it.

pezezin 3y ago

Honest question, what are the real world impacts of these kind of vulnerabilities? If I am running a few on-premises private servers where I control all the software, should I be worried?

NullPrefix 3y ago
>I control all the software A bold statement to make. Even if it would be true, do you control all the inputs for that software or can other people feed it with data?
x-complexity 3y ago
If you can lock down ALL the software used & disable remote access, you'll likely never encounter this vulnerability. It requires local unprivileged access in order to be used, so if you can restrict that to a few allowed processes, you'll be 90% fine.
viraptor 3y ago
Do you execute any untrusted code submitted by the users? It not, then it's very likely not an issue for you.* This mostly allows escaping from a sandbox or getting elevated privileges on a host. But in those cases you're already hacked and the attacker has access to your data, so getting root on some server likely isn't a bigger deal than the initial hack. * there's a chance that this enables a remote exploit through some specially crafted data for some popular library... the chances are very slim though

exabrial 3y ago

Apologies if I ask a stupid question: is it necessary to run the mitigations on a database server?

sebazzz 3y ago
And even better: can we disable the mitigations while still keeping up to date with patches and BIOS updates?
yencabulator 3y ago
If that database server is a VM (cloud host or on-prem), the host will likely decide to have mitigations on. If the database server gives shell to DBAs or such who don't get root, perhaps yes. If the owners wish to restrict the attack surface of most of the non-database software running on a current day fat linux distro, perhaps yes. No mitigations means buffer overflow in any network-exposed code (DHCP client, DNS resolver, SSH server, Prometheus metrics exporter, ...) could lead to a ransomware attack on the database etc.

washadjeffmad 3y ago

Related story (5 days ago) "Intel's Downfall Mitigations Drop Performance Up to 39%, Tests Show": https://news.ycombinator.com/item?id=37085043 https://news.ycombinator.com/item?id=37085043

bob1029 3y ago

By the time the security bullies are done "fixing" my CPU, I will be able to open my start menu faster using an abacus.

paul_funyun 3y ago

Mitigations=off is aging like fine wine.