4 ms·

> Disabling [spectre] mitigations gives us a performance boost of around 28% Every couple months these last several years there always seems to be some bug whe

by jart 5y ago

> Disabling [spectre] mitigations gives us a performance boost of around 28% Every couple months these last several years there always seems to be some bug where the fix only costs us 3% performance. Since those tiny performance hits add up over time, security is sort of like inflation in the compute economy. What I want to know is how high can we make that 28% go? The author could likely build a custom kernel that turns off stuff like pie, aslr, retpoline, etc. which would likely yield another 10%. Can anyone think of anything else?

ronsor 5y ago
Most of these mitigations are worse than useless in an environment not executing untrusted code. Simply put, if you have a dedicated server and you aren't running user code, you don't need them.
- vlz 5y ago
  But of course other exploits (e.g. in your webapp) might lead to "running user code" where you didn't expect it and then the mitigations could prevent privilege escalation, couldn't they?
  - vladvasiliu 5y ago
    But if you have a dedicated server for your web app, if there's some kind of exploit in it allowing for random code to be run, said code already has access to everything it needs, right? The interesting data will probably be whatever secrets the app handles, say database credentials, so the attacker is off to the races. They probably don't care about having root in particular.
    - ex_amazon_sde 5y ago
      > if there's some kind of exploit in it allowing for random code to be run, said code already has access to everything it needs On the same host there could be SSL certificates, credentials in a local MTA, credentials used to run backups and so on. Or the application itself could be made of multiple components where the vulnerable one is sandboxed.
      - vladvasiliu 5y ago
        All those points are true - though I'd argue this is stretching the "one app per VM" thing -, but I guess this is just the usual case of understanding your situation and realizing there's no one size fits all. My take on this question is rather that there shouldn't be any dogma around this, such as disabling mitigations should not be considered absolutely, 100% harmful and never, ever, ever disabled. In the context of the OP, where the application is running on AWS, backups, email, etc are all likely to be handled either externally (say EBS snapshots) in which case there's no issue, or via "trusting the machine", so getting credentials via the instance role which every process on the VM can do, so no need for privilege escalation. So I guess if you trust EC2 or Task roles or similar (not familiar with EKS) to access sensitive data and only run a "single" application, there's likely little to no reason to use the mitigations. But, yeah, if you're running an application with multiple components, each in their own processes and don't use instance roles for sensitive access, maybe leave them on. Also, maybe, this means you're not running a single app per vm?
        ex_amazon_sde 5y ago
        Why "app"? These are services. > there shouldn't be any dogma around this Like everything in security, it's about tradeoffs. > Also, maybe, this means you're not running a single app per vm? This is an argument for unikernels. Instead, on 99.9% of your services you want to run multiple independent processes, especially in a datacenter environment: your service, web server, sshd, logging forwarder, monitoring daemon, dhcp client, NTP client, backup service. Often some additional "bigcorp" services like HIDS, credential provider, asset management, power management, deployment tools.
        vladvasiliu 5y ago
        > Why "app"? These are services. Yes, but I was using my initial post's parent's terminology. But I agree, in my mind, the subject was one single "service", as in process (or a process hierarchy, like say with gunicorn for python deployments). > This is an argument for unikernels. It is. And I'm also very interested in the developments around Firecraker and similar technologies. If we'd be able to have the kind of isolation AWS promises between ec2 instances on a single physical machine, while at the same time being able to launch a process in an isolated container as easy as with docker right now, I'd consider that really great. And all the other "infrastructure" services you talk about could just live their lives in their dedicated containers. Not sure how all this would compare, performance-wise, with just enabling the mitigations.
seoaeu 5y ago
The puzzling thing was that spectre V2 mitigations were cited as the main culprit. They were responsible by themselves for a 15-20% slowdown, which is about an order of magnitude worse than in my experience. I wonder if the system had IBRS enabled instead of using retpolines at the mitigation strategy?
astrange 5y ago
PIE and ASLR are free on x86-64, unless someone has a bad ABI I don't know of. Spectre mitigations are also free or not needed on new enough hardware. Many security changes also help you find memory corruption bugs, which is good for developer productivity.
imhoguy 5y ago
I am not full deep in SecOps these days and would gladly hear opinion of some expert: Can disabling these mitigations bring any risks assuming the server is sending static content to the Internet over port 80/443 and it is practically stateless with read-only file system?
- syoc 5y ago
  I am not an expert but you shall have my take either way. The most important question here is "Am I executing arbitrary untrusted code?". HTTP servers will parse the incoming requests so they are executed to some extent. But I would not worry about it unless there is some backend application doing more involved processing with the data. repl.it should not disable mitigations.
jiggawatts 5y ago
Does anyone know of a quick & easy PowerShell script I can run on Windows servers to disable Spectre mitigations? The last time I looked I found a lot of waffle but no simple way I can just turn that stuff off...

by jart 5y ago

ronsor 5y ago

Most of these mitigations are worse than useless in an environment not executing untrusted code. Simply put, if you have a dedicated server and you aren't running user code, you don't need them.

vlz 5y ago
But of course other exploits (e.g. in your webapp) might lead to "running user code" where you didn't expect it and then the mitigations could prevent privilege escalation, couldn't they?
- vladvasiliu 5y ago
  But if you have a dedicated server for your web app, if there's some kind of exploit in it allowing for random code to be run, said code already has access to everything it needs, right? The interesting data will probably be whatever secrets the app handles, say database credentials, so the attacker is off to the races. They probably don't care about having root in particular.
  - ex_amazon_sde 5y ago
    > if there's some kind of exploit in it allowing for random code to be run, said code already has access to everything it needs On the same host there could be SSL certificates, credentials in a local MTA, credentials used to run backups and so on. Or the application itself could be made of multiple components where the vulnerable one is sandboxed.
    - vladvasiliu 5y ago
      All those points are true - though I'd argue this is stretching the "one app per VM" thing -, but I guess this is just the usual case of understanding your situation and realizing there's no one size fits all. My take on this question is rather that there shouldn't be any dogma around this, such as disabling mitigations should not be considered absolutely, 100% harmful and never, ever, ever disabled. In the context of the OP, where the application is running on AWS, backups, email, etc are all likely to be handled either externally (say EBS snapshots) in which case there's no issue, or via "trusting the machine", so getting credentials via the instance role which every process on the VM can do, so no need for privilege escalation. So I guess if you trust EC2 or Task roles or similar (not familiar with EKS) to access sensitive data and only run a "single" application, there's likely little to no reason to use the mitigations. But, yeah, if you're running an application with multiple components, each in their own processes and don't use instance roles for sensitive access, maybe leave them on. Also, maybe, this means you're not running a single app per vm?
      - ex_amazon_sde 5y ago
        Why "app"? These are services. > there shouldn't be any dogma around this Like everything in security, it's about tradeoffs. > Also, maybe, this means you're not running a single app per vm? This is an argument for unikernels. Instead, on 99.9% of your services you want to run multiple independent processes, especially in a datacenter environment: your service, web server, sshd, logging forwarder, monitoring daemon, dhcp client, NTP client, backup service. Often some additional "bigcorp" services like HIDS, credential provider, asset management, power management, deployment tools.
        vladvasiliu 5y ago
        > Why "app"? These are services. Yes, but I was using my initial post's parent's terminology. But I agree, in my mind, the subject was one single "service", as in process (or a process hierarchy, like say with gunicorn for python deployments). > This is an argument for unikernels. It is. And I'm also very interested in the developments around Firecraker and similar technologies. If we'd be able to have the kind of isolation AWS promises between ec2 instances on a single physical machine, while at the same time being able to launch a process in an isolated container as easy as with docker right now, I'd consider that really great. And all the other "infrastructure" services you talk about could just live their lives in their dedicated containers. Not sure how all this would compare, performance-wise, with just enabling the mitigations.

seoaeu 5y ago

The puzzling thing was that spectre V2 mitigations were cited as the main culprit. They were responsible by themselves for a 15-20% slowdown, which is about an order of magnitude worse than in my experience. I wonder if the system had IBRS enabled instead of using retpolines at the mitigation strategy?

astrange 5y ago

PIE and ASLR are free on x86-64, unless someone has a bad ABI I don't know of. Spectre mitigations are also free or not needed on new enough hardware. Many security changes also help you find memory corruption bugs, which is good for developer productivity.

imhoguy 5y ago

I am not full deep in SecOps these days and would gladly hear opinion of some expert: Can disabling these mitigations bring any risks assuming the server is sending static content to the Internet over port 80/443 and it is practically stateless with read-only file system?

syoc 5y ago
I am not an expert but you shall have my take either way. The most important question here is "Am I executing arbitrary untrusted code?". HTTP servers will parse the incoming requests so they are executed to some extent. But I would not worry about it unless there is some backend application doing more involved processing with the data. repl.it should not disable mitigations.

jiggawatts 5y ago

Does anyone know of a quick & easy PowerShell script I can run on Windows servers to disable Spectre mitigations? The last time I looked I found a lot of waffle but no simple way I can just turn that stuff off...