18 ms·
Meltdown fix impact on Redis performances in virtualized environments
- freen 9y agoHere's hoping your infrastructure scaling plan has more than 30% CPU headroom...
- forgot-my-pw 9y agoIs this good news for AWS? Some customers will be forced to upgrade to a larger instance.
- a2tech 9y agoRedddit has been running like garbage the last few days-I bet they're getting slammed with their instances needing mass reboots+big performance decreases
- innagadadavida 9y agoAre they running in a virtualized environment? They could just disable it as all the code is their own and is trusted?
- meritt 9y agoUnless customers absolutely raise hell, this is going to be very good for AWS. Suddenly all of these autoscaling setups are going to consume 30% more resources in order to achieve the same throughput. My expectation is we may see AWS apply some significant discounting to AWS services as a result, though that doesn't help all of us who utilize mostly prepaid reserved instances.
- epmaybe 9y agoAre the majority of aws revenues from casual users that probably wouldn't care? If so, I think you're going to be proven right. If, however, major customers make up the majority of revenue, then the answer seems less clear to me.
- meritt 9y agoMajor customers are absolutely the primary revenue source. AWS revenue follows the pareto principle.
- jacques_chester 9y ago> My expectation is we may see AWS apply some significant discounting to AWS services as a result, though that doesn't help all of us who utilize mostly prepaid reserved instances. It'll happen once one of the 3 main clouds breaks ranks and discounts their rates to match.
- deleted 9y ago[deleted]
- foobarbazetc 9y agoThere’s a case to be made that AWS is suddenly providing you 30% less than what you paid for. Imagine having reserved instances and they get 30% slower.
- PuffinBlue 9y agoNot sure if you saw this a few days back but this exact issue is already impacting some folks and they're obviously not happy. https://forums.aws.amazon.com/thread.jspa?threadID=269858 https://forums.aws.amazon.com/thread.jspa?threadID=269858
- guelo 9y agoMaybe, except that it makes running in non-virtualized environments more attractive.
- empath75 9y agoOr they’ll go bankrupt.
- lettergram 9y agoBreakdown from post: ~8% pipelined SET performance reduction after patch ~15% pipelined GET performance reduction after patch ~29% non-pipelined SET performance reduction after patch ~25% non-pipelines GET performance reduction after patch
- BonesJustice 9y agoNote also that the host OS (the one running the hypervisor) was already patched in all cases. These tests only reflect the difference between a patched and unpatched guest OS. The total performance reduction is likely greater.
- eric_h 9y agoAnd this all seems in line with what the expected slowdown would be. Non-pipelined calls will have more syscalls and therefore a bigger slowdown. Glad antirez took the time to do the benchmarks.
- jnordwick 9y agoAnd this is only going to be for installations that actually care about this level of security, so mostly just shared infrastructure/serverless/cloud. Internal systems aren't going to need the mitigation. I think people are blowing this way out of proportion.
- saurik 9y agoI personally think "the price of cloud computing just went up 5-10%" is "a big deal" and not some niche scenario.
- jacksmith21006 9y agoThing is I/O fees are contractual so Amazon has to suck up the increased cost or try to pass it somehow to Intel. But computer instances the extra cost will be on the customer and it is hard to see Amazon taking the PR hit and not somehow trying to put it on Intel.
- Xylakant 9y agoLike “all installations on AWS, GCE, DO and so on are affected? Even semi-private clouds might need mitigations since people rely on VMs as a security boundary. So yes, this is a big deal.
- cookiecaper 9y agoThese test results do not show the real effect of PTI, they only show the effect of enabling pti within a virtualized kernel. That's still useful because a lot of people are going to be enabling it (even though the value of this seems dubious), but real tests showing the approximate performance loss would need to happen on host hardware.
- fpoling 9y agoMeltdown on Linux allows to read all physical memory, not only the kernel data. The defence against it is only dubious if one does not rely at all on process isolation as extra layer of defence.
- cookiecaper 9y agoIs there a real reason to set `pti=on` in virtualized kernels, other than an abundance of caution? I'm not familiar with the internals of VT-x or intel-kvm.ko, but as far as I know, this is just doubling the hit for no real reason (unless you're doing nKVM or similar?).
- zaroth 9y agoI have this question as well. I can understand docker containers being vulnerable to other containers reading kernel memory across the boundary, but with VMs you would have to read into an entirely different kernel, right? I can understand how the attack works for JavaScript to read from the parent process / I.e. steal session cookies, but how could this attack work across VM boundaries?
- cookiecaper 9y agoThe Project Zero article [0] demonstrates breaking KVM this way by using indirect jumps to deduce the state of the CPU's internal branch history buffer: >The branch history buffer state is leaked in steps of 2 bits by measuring misprediction rates of an indirect call with two targets. That leaks out the physical address of the KVM driver, against which more code is speculative executed, with side channel being used to make these results deducible within the guest's memory area. Technically speaking, the guest can't fully unsandbox (i.e., they can't run commands on the physical host OS) with Spectre/Meltdown alone, but they can read the memory content of the entire physical host in the case of Meltdown, or all of userspace in the case of a Meltdown-less Spectre. This is being mitigated by retpoline + microcode update + new kernel parameters IBRS and IBPB, which will make it so that users can't mislead the CPU with speculative indirection (by killing the functionality in risky situations, another layer of slowdowns we will all have to confront soon; impact is not seen until major distros recompile everything with retpoline). PTI itself wouldn't mitigate that leakage, it just mitigates its exploitation to read back the host memory. However, this is a hardware thing that affects the host OS. If the host already has these features enabled on the real kernel, what value do they provide within the guest itself? Is timing _within the guest kernel_ accurate enough for sandboxed exploitation, making this a universal sandbox-constrained privilege escalation? I don't know. It'd be nice to avoid the redundant overhead, of course. [0] https://googleprojectzero.blogspot.com/2018/01/reading-privileged-memory-with-side.html https://googleprojectzero.blogspot.com/2018/01/reading-privi...
- misterbowfinger 9y agoI don't quite understand what "pipelining" means. Can anyone TL;DR?
- bdibs 9y agoIt's basically just consolidating multiple requests into one larger request.
- misterbowfinger 9y agoSo applications that use Redis mainly for caching won't be hit as hard then?
- ovao 9y agoApplications that don’t use Redis as a persistent store, and enable pipelining, will be hit less hard, yes. Applications that connect to Redis remotely will also be hit less hard — loopback makes this somewhat of a worst-case scenario.
- misterbowfinger 9y agoAhhhh that's this line was about: > Test performed with AOF enabled AOF is a type of persistence: https://redis.io/topics/persistence https://redis.io/topics/persistence Most of the time, I've used Redis with (I think) RDB persistence, just in case. Usually we've used it as an in-memory cache. Ok, that makes a lot of sense, thanks!
- teej 9y agoFairly certain this is talking about Redis pipelining (https://redis.io/topics/pipelining https://redis.io/topics/pipelining). It let's you send multiple commands to Redis without waiting for a response between each one.
- deleted 9y ago[deleted]
- 9y ago
- justinsaccount 9y agohttps://www.phoronix.com/scan.php?page=article&item=linux-kpti-pcid&num=3 https://www.phoronix.com/scan.php?page=article&item=linux-kp... shows that the kernel version has more of an impact.
- antirez 9y agoThat's super informative, as I wrote my methodology was flawed and indeed... Tests to redo. Btw what's the reason for this terrible regression?
- justinsaccount 9y agoThe regression being the drop in performance you saw? I don't know.. I'd be interested in seeing what the performance looks like on multiple "identical" C4.X8LARGE instances, regardless of KPTI. If you mean their results.. they actually show an increase in perf in later kernel versions. Based on https://github.com/phoronix-test-suite/test-profiles/tree/master/pts/redis-1.0.0 https://github.com/phoronix-test-suite/test-profiles/tree/ma... it looks like they just use the default configuration and run the benchmark with -n 1000000 -P 32 -q -c 50 --csv
- scrollaway 9y agoYep, seeing a much more severe hit on all our elasticache instances than anywhere else. CPU graph from two underused ones: https://twitter.com/Adys/status/949432228727218177 https://twitter.com/Adys/status/949432228727218177
- necubi 9y agoWe also saw a huge hit on our memcached elasticache nodes. Eyeballing from our CPU graph, seems ~30%.
- berbec 9y agoI've noticed a big slowdown in PV EC2. One machine used to be fine running apache and mysql. I had to add a 2nd cpu and split the mysql off onto a different instance to keep load to manageable levels.
- static_noise 9y agoThe two graphs have different y-axis-scales.
- scrollaway 9y agoObviously? They're two different instances.
- Washuu 9y agoHydra/Gamepedia got hit by the patch on its Elasticache instances around the same time. Redis is used heavily on its caching infrastructure and the web nodes starting thrashing due to the sudden performance loss.(Until the resources were increased.)
- jacksmith21006 9y agoWould be curious to see this also done on the Google cloud.
- jedmeyers 9y agoGoogle Cloud does not have Redis as a service. You can deploy your own Redis to GCE VM and try.
- jacksmith21006 9y agoYes deploy your own and do benchmarks before and after they implemented the patch.
- jedmeyers 9y agoCheck Container-optimized OS release notes to see which version has the fix. If I am not mistaken stable-63 has the latest set of fixes. https://cloud.google.com/container-optimized-os/docs/release-notes https://cloud.google.com/container-optimized-os/docs/release...
- korzun 9y agoThose benchmarks are garbage since the EC2 envoirment is shared.
- whalesalad 9y agoFairly substantial slowdown BUT still a phenomenal set of performance numbers. 800k GET’s per second is way above my pay grade.
- jacksmith21006 9y agoRather shocked that Amazon and the other cloud providers are not screaming harder at Intel. Maybe they are behind the scenes. I believe the increase cost of I/O is going to hit Amazon and can not be passed to the customer as what you pay is contractual. Then with computer instances the customer is paying more for ultimately the same amount of compute and that is a PR issue for Amazon and the other cloud providers. Seems like Amazon would not be OK with that and will want to pass it to Intel somehow.