14 ms·

CPU Usage Differences After Applying Meltdown Patch at Epic Games

• simooooo 9y ago
  Wow that's a huge jump. Scared for my web servers now
  • lsd5you 9y ago
    Not entirely sure why we need to update/protect most servers, since generally they won't be running untrusted code, right?
    • ZenoArrow 9y ago
      The reason for doing so is minimising the risk if an attacker breaks into a server. This is also why systems like ATMs should be patched. Untrusted code execution on servers and ATMs may not be the norm, but it's far from impossible.
      • lsd5you 9y ago
        Ok, sure, but we have to consider whether that is worth it, and if they can run unpriveleged code, there is a good chance there is a software privelege escalation available anyway. Of course on shared servers this is going to be a nightmare.
      • randyrand 9y ago
        couldn't they just use a myriad of other priveledge escalation bugs? there have been ~4600 privledge escalation bugs found since 1999. 250 a year. Almost one every day. https://www.cvedetails.com/vulnerabilities-by-types.php https://www.cvedetails.com/vulnerabilities-by-types.php At this point we still won't have security and now we won't have performance either. It's putting the cart before the horse.
        • ZenoArrow 9y ago
          > "couldn't they just use a myriad of other priveledge escalation bugs?" The idea is to patch all known privilege escalation bugs. It's rare for an attacker to rely on a single attack vector. They may get to the point where they only have limited access to running code, and want to break out of that sandbox. Spectre/Meltdown may be the route they take to do so.
    • s_kilk 9y ago
      If they're operating on data supplied by the attacker, they're potentially a single hop away from executing untrusted code.
      • dagss 9y ago
        ...at which point that untrusted code gains full access to the only userspace process that matters on that node. If you gain access to run code on a server process why escalate further, you already have access to everything that matters?... Assuming bare metal. In shared hosting / cloud / VMs it is different.
        • chrisper 9y ago
          With Meltdown / Spectre you can break out of sandboxes as far as I understand.
          • yorwba 9y ago
            That only matters if not everything is in the same sandbox. Many non-cloud servers probably either only run a single service, or they run all processes as the same user, which means that a sandbox escape doesn't really matter. You can't lose any security layers you never had in the first place.
        • lsd5you 9y ago
          This is a great point. Probably someone will argue properly configured apache will not have access to data ... etc. I think the practical reality is in many setups it already does, and what these hardware bugs mean is that the lazy imperfect - effectively single layer - security that probably exists on the majority of servers is now essentially equivalent to the security of systems where the security minded have been incredibly diligent (and at considerable cost) ensuring multi layer security ... etc. So in some sense it is an attack on their value system and their worth/usefulness.
          • jameshart 9y ago
    • Matt3o12_ 9y ago
      In case there is a security hole in their Webserver (Apache, Nginx, HAProxy, etc) or their application (Wordpress, etc), that an attack cannot escalate privileges even further. Image someone manages to execute arbitrary PHP code, now they can gain gain root access, read private keys, etc. Things PHP should not have access to. For spectre, the consequences of not applying those patches are not as bad but it wouldn’t surprise me if you could still do a lot of harm. PHP is somewhat sandboxes these days (by using chroot and what not) and meltdown/spectre could be used to escape the sandbox.
      • angelsl 9y ago
        Err, what? You can't get root access or escape a chroot from being able to read arbitrary memory.
        • Matt3o12_ 9y ago
          Yes you can? If password login is enabled, you could wait until someone SSHs in and read their decrypted password. You could read all kinds of secrets which will be very useful for gaining privileged information. Sure you can’t just become root but it is one way to get there. For my server, the attacker could probably read mosh‘s secret key which is used to authenticate mosh clients. Mosh is a “mobile shell” which allows is resistent toward network changes (including IP changes), is low bandwidth and extremely quick on Eadge connection. An attacker can probably come up with many solutions where reading arbitary memory can be used to gain root access. Granted the low bandwidth aspect of this attack might make it difficult but far from impossible.
        • baq 9y ago
          You don't need anything special to escape a chroot, it's never been intended as a security feature.
        • ZenoArrow 9y ago
          If you can dump the memory holding security certificates, I'd suggest you probably can. There will be processes outside a users control that run with elevated privileges, attacks that allow you to monitor the setup of these processes could prove risky to the overall security of a system.
    • InclinedPlane 9y ago
      Belt and suspenders. In principle you don't need to protect against spectre and meltdown if you can ensure that no untrusted code runs on a particular machine. How do you ensure that though? There are many many routes for that to happen, and it only has to happen once for it to completely invalidate your security model if you have no other levels of protection. On a server you can run untrusted code by exploiting other vulnerabilities in server software such as buffer overruns. It's unrealistic to expect that all servers will have no such vulnerabilities in any of their software at any time. That's why operating systems implement many layers of protections to prevent those exploits from being elevated into even worse breaches that could leak sensitive data, provide local root access, and so on. One of the more significant protections that exists today is address space layout randomization, which makes it much more difficult for a buffer overflow exploit to result in immediate privilege escalation. Meltdown/spectre make these sorts of issues much, much worse. They mean that absolute any vulnerability in any service could be exploited to read the entire contents of memory and deliver that to an attacker, including encryption keys, including passwords, and so forth. That's bad. You can essentially 100% guarantee that there will be some vulnerability in some service which enables a minor exploit, and you want to avoid having that minor issue become catastrophic through meltdown/spectre based attacks.
    • rocqua 9y ago
      The replies are missing something from the original article. They are running on the cloud, and Meltdown / spectre means that exploits can escape a VM. This means you don't just need to trust your VM, but also any VMs you are sharing the hardware with.
      • Asdfbla 9y ago
        I suppose if a Cloud provider could ensure that all your VM instances run on the same host and no other VM is allowed there then the issue would be a bit mitigated. Although this restriction on how the cloud provider is allowed to schedule your VMs probably would somewhat defeat the point of cloud hosting in the first place. Out of curiosity, how many VM/container instances usually run on a physical host at any given time (for your typical cloud computing provider)?
        • asheldon 9y ago
          AWS provides the ability to guarantee you don't share physical hardware with other customers with "EC2 Dedicated Instances" (https://aws.amazon.com/ec2/purchasing-options/dedicated-instances/ https://aws.amazon.com/ec2/purchasing-options/dedicated-inst...). I'm not familiar enough with other cloud provider offerings to say if they do or do not have similar features.
        • Fargren 9y ago
          With AWS, and I assume most other cloud computing providers, you can pay extra for you instances to run in a host without someone else's VMs. You probably should be doing this for any servers were you handle sensitive data, but it is a place were many will be cutting corners.
          • mrep 9y ago
            You don't really even have to pay extra. Just use the biggest instance since and you are guaranteed to be isolated because there is no room for anyone else. Granted, that only works for workloads that are spread across enough small instances.
      • 9y ago
• dannyw 9y ago
  Surprise, surprise, Intel’s Meltdown patch has significant and serious performance impacts for specific kinds of workloads. All because Intel decided to “optimise” by checking for permissions after speculative retirement, instead of before like AMD.
  • imjasonmiller 9y ago
    If, as you say, AMD is not affected by Meltdown unlike Intel, will this significantly change the server market? Excuse the pun, but would this make e.g. AMD EPYC a lot more attractive for such data centers?
    • zeusk 9y ago
      AMD has confirmed it isn't affected by Meltdown. https://lkml.org/lkml/2017/12/27/2 https://lkml.org/lkml/2017/12/27/2
    • arien 9y ago
      Aren't they still vulnerable to Spectre?
      • naasking 9y ago
        Pretty much every fast CPU is vulnerable to some subset of Spectre.
      • Choco31415 9y ago
        Correct, and so are Samsung and Qualcomm CPU’s, according to the original paper: https://spectreattack.com/spectre.pdf https://spectreattack.com/spectre.pdf
      • Tuna-Fish 9y ago
        Yes to the bounds check violation version, so far no to the BTB poisoning version. The bounds check version only works inside a single process, so is only relevant when you run untrusted code within the same process as some private data (such as JS in a web browser). AMD claims that they believe that the way their branch predictor works effectively makes the BTB poisoning unusable, but there is no actual proof, and their statement regarding it was much more wishy-washy than they were with meltdown. (Which they specifically state they are completely immune to.)
• ZenoArrow 9y ago
  Considering the performance impact, I wonder how console manufacturers are going to handle this (assuming that the processors they used are vulnerable to Spectre/Meltdown).
  • rcarmo 9y ago
    Console manufacturers don't run unsigned code, so I expect they'll just sit still until the next hardware refresh.
    • ZenoArrow 9y ago
      Modern consoles now come with web browsers, and the researchers proved that the attacks could be performed via web browsers, did they not?
      • deleted 9y ago
        [deleted]
      • lrem 9y ago
        Next patch: we enhanced your security by disabling execution of JavaScript from untrusted domains. In unrelated news, we now block ads!
        • ZenoArrow 9y ago
          If they are going to block JavaScript, they need to do it for all domains, not just untrusted domains. For example, if I can MITM my own website activity (which I can, by having a device that sits between a router and a console), then I can change the JavaScript coming from trusted domains. In other words, if I visit a site like Google or Facebook on an affected device, I can change the JavaScript that is run, and make it still appear like it came from a trusted domain.
          • yorwba 9y ago
            You can be prevented from a successful MITM by enforcing HTTPS using HSTS and pinning the keys of trusted sites. That's not a viable solution for accessing the internet at large, but for a console that is only incidentally used for web browsing, it's absolutely an option.
• lathiat 9y ago
  I’d encourage anyway seeing super huge hits to make sure they are not using paravirt (particularly on Amazon). As that needs mitigation the the virt level the impact seems very large.
  • deleted 9y ago
    [deleted]
• contrarian_ 9y ago
  Pretty much what I predicted here: https://news.ycombinator.com/item?id=16054674 https://news.ycombinator.com/item?id=16054674 > Sounds like servers handling lots of small UDP packets would be hit pretty hard.
  • tgb 9y ago
    Since they describe this as log-in issues, should we expect that to be a server using lots of UDP? Or do you think that the log-in service is hemmed in by the load on the game servers?
    • dogma1138 9y ago
      No, login in Fortnite is over 443 and it uses normal HTTPS. It also uses HTTPs to load a lot of other data such as server data, friends list, chat etc. Unreal Engine comes with a version of chromium built in which is used for many in game things like social tabs, news, and in game purchases these all work over HTTP/S. Game data is sent over 5222 TCP.
      • jbg_ 9y ago
        Out of interest, do you know if the game data uses XMPP or if it just uses the same port?
        • one_thawt 9y ago
          It does not. It's all based on the UE4 network stack.
        • dogma1138 9y ago
          5222 is pretty much used for all Epyc games I don’t think that I has anything to do with XMMP tho.
      • tgb 9y ago
        Right, that's what I was guessing and am not sure why I was downvoted when everyone is confirming that contrarian's post doesn't really make sense since UDP isn't very relevant here.
• aceoflolo 9y ago
  >We wanted to provide a bit more context for the most recent login issues and service instability. All of our cloud services are affected by updates required to mitigate the Meltdown vulnerability. We heavily rely on cloud services to run our back-end and we may experience further service issues due to ongoing updates. So they are saying "you can't log in because we're too cheap to rent more servers now that this patch has increased CPU usage"
  • transpostmeta 9y ago
    Resolving scalability issues is not as simple as "rent more servers".
    • aceoflolo 9y ago
      For a CPU-bound problem it mostly is as simple as that, yes
      • bhouston 9y ago
        Only if the algorithm can be split across multiple machines easily. Sometimes people assume that you can use local shared memory or something between threads in order to synchronize state. You figure out how many individuals can be on a server at once and then ensure that you can handle that load on a specific machine. I've seen this type of stuff for game state before because they need to keep everyone in a specific game domain (level or city depending on the type of game) synchronized and be nearly real-time. It can be hard to pull this across different machines without introducing significantly latency, redis or DBs is slow for a FPS shootter. Not saying this is the case but I can see it could be something like that.
        • tetha 9y ago
          Yup. When I was in a team working on distributed game servers, this forced us to shard games instead of distributing individual games. Terminology wise, if a game session was fully distributed across multiple instances, each server could accept traffic for this game session. Think elasticsearch - each node can answer searches for any index in the cluster. If you shard your games, you just put all games with an even ID on box 1 and all games with an odd ID on box 2. If box 1 dies, all games on that box disappear. And then you usually end up with a lobby server as an initial connection and redirect to the actual game server in the background. This is a very simple architecture. It's easy to develop for this architecture, because you don't need to worry about complex clustering issues - exactly 1 client talks to exactly 1 server and it doesn't matter if there's 50 other servers answering other clients. This is also very nice to scale. Most server side code for games tends to have very predictable resource consumption, because it's running a pretty predictable simulation loop on a pretty predictable and bounded data set. Especially from this perspective, I can see why the Epic guys are bugged. It's not pretty to put a factor of 2 into those calculations.
• mmaunder 9y ago
  Post. More. Benchmarks. I'll do the same once I have them. More data on this across platforms and apps is incredibly helpful for all.
  • chrisseaton 9y ago
    Real data like this post are more useful than more benchmarks.
    • mmaunder 9y ago
      Guess that's what I meant, but I'll take either!
      • deleted 9y ago
        [deleted]
    • jacquesm 9y ago
      They are benchmarks, just not microbenchmarks. Actual application performance is the gold standard for benchmarks. https://en.wikipedia.org/wiki/Benchmark_(computing)#Types_of_benchmark https://en.wikipedia.org/wiki/Benchmark_(computing)#Types_of...
      • wolfgke 9y ago
        > They are benchmarks, just not microbenchmarks. Actual application performance is the gold standard for benchmarks. This depends on your intended purpose of the benchmark. If you want to evaluate performance of a whole system, sure. But one often does benchmarks to find parts of the system that are critical to performance so that you can do further optimizations or do change in the software architecture to avoid these critical performance hotsports. For this case other benchmarks that allow an easy fine-grained interpretation of the data are surely better.
• Abishek_Muthian 9y ago
  If it's helpful, Our Node.js, MongoDB, Python servers all with significant network traffic didn't have any measurable impact after KPTI patches on Amazon Linux on T2.medium(burst), M4.large, T2.large(burst) respectively. Our impact is lesser than the figures suggested by redhat's advisory - https://access.redhat.com/articles/3307751 https://access.redhat.com/articles/3307751
  • urbanxs 9y ago
    That’s most likely because neither node, mongo nor python actually use the optimisation features of cpus. Says a lot about their performance quality.
  • bonzini 9y ago
    Are you using PV or HVM instances?
    • Abishek_Muthian 9y ago
      All of them on HVM.
      • falcolas 9y ago
        IIRC, XEN said that 32bit HVM VMs are not affected by Meltdown, and so probably don't get impacted by AWS' patches. They still require Linux kernel updates to protect the kernel space, so changes might still be seen there.
        • Abishek_Muthian 9y ago
          Yes, but XEN said 64bit PV aren't affected either because they already run in KPTI like environment. So I assume 64bit HVM like ours aren't impacted for our work load.
        • bonzini 9y ago
          They are affected but there is no fix yet. 64-bit PV is unaffected and won't suffer a performance penalty (more precisely, it was already suffering it!!!), hence my original question, but a 64-bit PV guest can use Meltdown to attack the hypervisor. The fix is to update Xen, though I am not sure if fixes are already publicly available.
• jacksmith21006 9y ago
  Really like to seem some benchmarks from different cloud providers before and after patches are applied to see if there is a material difference. The one most interested in is Google versus Amazon.
• erdbeerkuchen 9y ago
  The image didn't show up for me, here is a direct link for anyone with the same problem: https://lh6.googleusercontent.com/MwzsHRXQLVbmJ3pusNuGwn0ZQVjo9h8nRJHJhIo4d3XFqbvUYCj8EPq5jV7zeVEEcHAkraNBesbbNDW_UAlIjvw-hZBd80rKt7ZYl35nBIcfCCVyRvW5V7M7KVejv9tvVBHfgSKr https://lh6.googleusercontent.com/MwzsHRXQLVbmJ3pusNuGwn0ZQV...
  • snowpanda 9y ago
    Sorry to ask, but does anyone have a third mirror? Neither are working for me and I really am interested in seeing this image.
    • contrarian_ 9y ago
      https://i.imgur.com/pA7w4Tf.png https://i.imgur.com/pA7w4Tf.png
    • zzzcpan 9y ago
      http://anonymouse.org/cgi-bin/anon-www.cgi/https://lh6.googleusercontent.com/MwzsHRXQLVbmJ3pusNuGwn0ZQVjo9h8nRJHJhIo4d3XFqbvUYCj8EPq5jV7zeVEEcHAkraNBesbbNDW_UAlIjvw-hZBd80rKt7ZYl35nBIcfCCVyRvW5V7M7KVejv9tvVBHfgSKr http://anonymouse.org/cgi-bin/anon-www.cgi/https://lh6.googl...
  • fermienrico 9y ago
    Can someone explain what 1, 2 and 3 series in the plot means? What are we looking at? Why did one of them all of a sudden spiked?
    • johnbellone 9y ago
      Perhaps it’s multiple instances?
    • djsumdog 9y ago
      Yea I hated that too. They put up a graph but with no key! Arg! The article said they updated one host, so I'm assuming the spike is from that one VM that got patched.
• viraptor 9y ago
  I know they don't have to share the details, but the "patched" part is not really clear. Did they update to a new image / more recent kernel / anything else? Much like the redis post linked in HN before, we don't know if the impact is because of the "pti turned off/on" change, or are there more moving parts involved.
  • arien 9y ago
    If I recall correctly from older posts (I played Fortnite for a while), they're using AWS.
    • viraptor 9y ago
      That's still not answering many questions. I hope they publish a full analysis at some point.
      • lawrenceyan 9y ago
        Since they're using default provisioned EC2 instances, it's likely that the developers don't necessarily even fully understand their performance degradation. They just expect the service that they pay for to work properly.
        • viraptor 9y ago
          It's true, but it's not what I meant. They wrote "after a host was patched". This is ambiguous. So they mean the host as in instance, or host as in AWS host machine? Did they just reboot it get on the new/updated VM host, or did they rebuild to include the PTI fixes as well. Did they upgrade anything, or did everything else stay on the same version.
      • teej 9y ago
        From what I can tell, Amazon isn’t giving people deep technical information. They’ll just send you an email telling you which instances are being forced to restart and when.
  • 9y ago
• jedisct1 9y ago
  The Meltdown patch also introduces a serious performance hit for DNS servers and resolvers. Before giving figures, I want to run the same tests on a more recent CPU, but my current benchmarks are not great to say the least.
  • pixl97 9y ago
    Is this a Program -> CPU interaction that is slowing things down, or is this a CPU -> network interaction. I'm wondering if there are classes of network drivers that are having a much larger effect in performance. Network cards these days can do many things to improve performance like TCP/UDP offloading and because of that their drivers are very complex and I'm going to assume that there well be Meltdown fallout because of this.
• Darthy 9y ago
  The Meltdown attack requires an attacker to have a piece of code executed on your server. Epic's servers are used for login, where people send you data, and for game logic, where people also just send you data like "player x moved his avatar here, player y shoots etc". If all the server does is execute the code which Epic wrote themselves and already trust, why would it need to apply the Meltdown patch?
  • em3rgent0rdr 9y ago
    Your argument also applied to other circumstances where the computer only runs trusted code. For example a fully FLOSS software stack on home computer with JavaScript disabled in the browser.
  • akira2501 9y ago
    > why would it need to apply the Meltdown patch? They're using a "monolithic" cloud provider and don't have a choice in their current deployment?
  • drawnwren 9y ago
    This is a horrible approach to security. If you only secure against attacks you expect, you're gonna have a bad time.
    • deleted 9y ago
      [deleted]
    • toomuchtodo 9y ago
      Security is about risk mitigation. You cannot derisk entirely, so you make tradeoffs. Without knowing all of the parameters, it’s disingenuous to say it’s a horrible approach to security. The most secure computer is powered off, enclosed in concrete 6 feet below the surface of the earth. It is not very useful though.
      • em3rgent0rdr 9y ago
        Reminds me of Battlestar Galactica. The humans were so (wisely) fearful of the Cylons that their ship computers were not networked in anyway. For their risk/reward trade-off curve, the benefit from having computer networks were not worth the risk of the Cylons being able to compromise the entire ship. All communication was either verbal or done via fax printed to paper and so had to go through human intermediaries.
• ChildOfChaos 9y ago
  I have turned windows updates off so I don’t have to deal with this crap. Screw you intel + Microsoft.
  • bartread 9y ago
    Whilst I sympathise with your frustration, at least with Intel, I can't help feeling like you might be storing up bigger problems for yourself with this course of action.
    • ChildOfChaos 9y ago
      I get what you are saying but right now there are no known exploits and with so much patching happening will there ever? These things are always over blown in the media and the reality is very little damage happens to the average user. It’s servers perhaps that are most at risk. Also I don’t do much on my Windows, mostly gaming, I run an iMac and boot into it only for certain tasks, so it’s extremely unlikely I will ever have an issue. Any performance hit, even if small is just not worth updating for to me.
      • kazagistar 9y ago
        What do you mean "no known exploits"? The authors of the paper have an exploit that reads arbitrary system memory from a browser. And even after the meltdown patches, spectre "fixes" we have seen are only partial mitigation, and still potentially allow reading of passwords and third party cookies. But I guess if you want to wait til it's too late...
      • nemothekid 9y ago
        What do you mean by “no known exploits”? There are several PoCs out, one of which is in JavaScript for meltdown.
      • pixie_ 9y ago
        Yea sounds like you need all the CPU you can get. Good decision.
• jacksmith21006 9y ago
  Really like to see similar done on the Google Cloud infrastructure and see if there is any difference.
• j1vms 9y ago
  Maybe off-topic: Is formal verification viable anywhere in CPU logic design? Also, could any existing "CPU static analyzers" have caught the issue that caused Meltdown? Edit: It looks like the answer to the first is a definite yes.
  • andrewaylett 9y ago
    Formal verification of what? You can only verify properties you've thought of, and no-one conceived of this particular 'feature' causing issues like this until now. So I don't think formal verification would have helped: if anyone was in a position to realise the issue was worth verifying, they'd have been able to raise it without formal verification too.
    • zzzcpan 9y ago
      Presumably of isolation features. I'm sure everyone knew what side channel attacks were and knew that formal verification could've helped find bugs in isolation. They just chose not to do it, because it's only important for high-assurance, not for your regular insecure linux/bsd/windows. And I'm sure they are not going to bother with verification even after everything.
    • serf 9y ago
      >You can only verify properties you've thought of, and no-one conceived of this particular 'feature' causing issues like this until now. I've read for years about supposed insecurities with branch prediction -- it just wasn't shown practically. To say that it wasn't conceived of is a little off.
  • Sephr 9y ago
    Not until we have a fully accurate theory of physics and perfect models for the CPUs.
  • sehugg 9y ago
    Spitballing, but you'd have to prove that either no side channels exist, or that the information leaked by side channel(s) is indistinguishable from random noise, which I believe is an area of ongoing research.
• herf 9y ago
  Graph says pretty clearly the patch is 60-70% slower! or 2.5-3x hardware costs. I wonder which CPU family this is?
  • jug 9y ago
    This surprised me a lot. I thought 30-50% would be worst case and then with additive effects from both Spectre and Meltdown fixes. Not sure how it can be this bad. I imagine it could get even worse if you are running in a virtualized environment on top where the server is affected in turn, but I figure thst wouldn’t show in a CPU graph like this..
    • Tuna-Fish 9y ago
      The kernel address space isolation makes syscalls much more expensive than they were. A toy program that just repeatedly calls the cheapest syscall in the kernel would lose much more than half it's speed, but that's not what's reported because no-one actually needs to run that workload. Epic seems to have been particularly unlucky in how KPTI impacts them.
• mtgx 9y ago
  Looks like it may be time for Epic Games to consider switching to AMD Ryzen/EPYC.
  • dijit 9y ago
    I have been looking at AMD/Epyc CPUs recently. (Because making NUMA aware C++ code is hard, and AMD Epyc is a single socket on a server with 4 very closely knit NUMA zones so non-NUMA code will run better on that vs Intel) But unfortunately there's no comoddity server from HP/Dell available yet. But I hear one is on the way on the Dell side.
    • snuxoll 9y ago
      HP has paper launched the DL385 G10 with EPYC, I'm not sure if it's fully available through sales channels or not. Supermicro also has had EPYC systems available for a while - though that won't do you any good at all if you need a big name OEM for "nobody got fired for buying Cisco" reasons.
• forgotpw2018 9y ago
  Huge real world performance impact, they didn't say how much but it looks like close to 100%. I smell a class action lawsuit coming.
  • pitaj 9y ago
    Why? This wasn't gross incompetence in processor design, this kind of attack is completely new. I don't see how a class action could apply here, by IANAL
    • Filligree 9y ago
      Doesn't stop people from filing one. They already have, in fact. I agree that it shouldn't get anywhere, but I'm not as sanguine about whether or not it actually will.
    • jjeaff 9y ago
      It doesn't have to be gross incompetence. If you pay for something and it doesn't deliver as promised, you may be entitled to a partial or full refund.
    • chrisper 9y ago
      If you know that your product is flawed, do you keep selling it or will you pull it from the shelves? To this date I can still buy broken Intel CPUs.... They knew about the flaw in June. Yet they still kept selling Coffee Lake CPUs. If my 8700k wouldn't still be significantly faster than Amd Ryzen (or I wouldn't have to also return the MB), I would have switched to an Amd in a heart beat.
      • djsumdog 9y ago
        To be fair to Intel, mitigating this type of issue isn't at all trivial. If it were, it could have been fixed in a microcode update. You can't rush a chip design for something as complicated as x86_64. There are long multi-year development cycles and tons of regression tests. With this they need to add even more tests before they can start on attacking the issues with the design.
• lazyjones 9y ago
  Those who have looked at the patches in detail: does it treat older generation CPUs differently than 7th gen? The Paper authors who wrote the KAISER patch expected much worse performance on older CPUs due to implementation issues... Unfortunately, Epic Games don't provide any details about their CPUs AFAICT.
  • simfoo 9y ago
    Really interested in this as well. Details matter here
  • johnbellone 9y ago
    They did mention that they were using a cloud provider. But that’s about it.
  • cthalupa 9y ago
    I believe PCID is supposed to help mitigate the performance impact from the KPTI changes. Both the hardware and kernel have to support PCID
• merb 9y ago
  if people would still use good old owned hardware without any virtualizsation, spectre and meltdown would not be as scary as it really is on server hardware. which means that only clients would be affected. but since everything runs on the cloud, we basically need to update the whole world.
  • ebbv 9y ago
    First off, having dedicated hardware doesn't make these a non-issue, it just makes them less of an issue. If you have dedicated hardware they are concerns. Applications still need to be isolated from each other, and it's a path for someone who gets in as a user to potentially leverage it to root access. Second, saying everyone should have dedicated hardware is just nonsense. It's like saying everyone should run their own datacenter. For most people neither of those makes financial sense.
  • kazagistar 9y ago
    It's still pretty scary to have computers where all of memory is readabe from every process, even if you own them.
    • merb 9y ago
      well 50:50 chances are high that if a process goes bogus, that you have more problems than just "memory is readable from every process". (of course that does not apply to clients where code can run in jit (javascript), software that communicates with the internet, etc and runs on a remote machine). most servers probably should only run trusted code (of course that is mostly never the case because no company evaluates every process they running) but chances are high that if some uses linux and gnu stuff that most shady stuff gets catched. (I mean if not, some people could do a lot of bad stuff, consider a misbehaving systemd, nginx/apache, databases, which could basically do a lot of harm.)
• stefantalpalaru 9y ago
  It's time to move to userspace network drivers like https://github.com/snabbco/snabb/ https://github.com/snabbco/snabb/
• deleted 9y ago
  [deleted]
• brendangregg 9y ago
  What's the syscall/sec per CPU rate for this workload? My guess: over 1M.
• jitbit 9y ago
  Thought I'd share our graph too: https://www.jitbit.com/alexblog/270-cpu-usage-stats-after-patching-for-meltdown/ https://www.jitbit.com/alexblog/270-cpu-usage-stats-after-pa... Seeing almost 40% increase in CPU load
  • simik 9y ago
    Do you mean 40 percentage points? In other words, tripling?
    • lucb1e 9y ago
      Looking at the graph: probably. It was already increasing (from 16-19%, one outlier to 27%, first 8 datapoints) to around 25-30% (the last 6 before the big jump). Then a big jump to 62-85% (N=20) with most around 75% (N≃13). The last bit is around 60-72% (N=12) and the very last datapoint is below 40% again but that is half outside the graph (cherry-picking datapoints?). So to summarize, from 25% to 65%, or 2.6× the original. Note that this is all from eyeballing that graph and estimating what percentage a datapoint is at, since the scale has an interval of 20 percent points and no minor grid or anything.