4 ms·

If it's helpful, Our Node.js, MongoDB, Python servers all with significant network traffic didn't have any measurable impact after KPTI patches on Amazon Linux

If it's helpful, Our Node.js, MongoDB, Python servers all with significant network traffic didn't have any measurable impact after KPTI patches on Amazon Linux on T2.medium(burst), M4.large, T2.large(burst) respectively. Our impact is lesser than the figures suggested by redhat's advisory - https://access.redhat.com/articles/3307751 https://access.redhat.com/articles/3307751

urbanxs 9y ago
That’s most likely because neither node, mongo nor python actually use the optimisation features of cpus. Says a lot about their performance quality.
bonzini 9y ago
Are you using PV or HVM instances?
- Abishek_Muthian 9y ago
  All of them on HVM.
  - falcolas 9y ago
    IIRC, XEN said that 32bit HVM VMs are not affected by Meltdown, and so probably don't get impacted by AWS' patches. They still require Linux kernel updates to protect the kernel space, so changes might still be seen there.
    - Abishek_Muthian 9y ago
      Yes, but XEN said 64bit PV aren't affected either because they already run in KPTI like environment. So I assume 64bit HVM like ours aren't impacted for our work load.
    - bonzini 9y ago
      They are affected but there is no fix yet. 64-bit PV is unaffected and won't suffer a performance penalty (more precisely, it was already suffering it!!!), hence my original question, but a 64-bit PV guest can use Meltdown to attack the hypervisor. The fix is to update Xen, though I am not sure if fixes are already publicly available.
bartread 9y ago
Would you be able to post some ballpark figures, please?
- Abishek_Muthian 9y ago
  Actually, there is not any measurable difference. Our architecture is completely state-less and when compared with equivalent load there's no difference at all. I guess, default network throughput bottle-neck itself is higher than the random memory cache bottle-neck in my case. There was no impact on the latency either. Fearing automatic update by AWS resulting in performance issue, we rushed to update all our serves and gladly there weren't any performance impact. I don't think not every one has been fortunate, there's still lot of variables reg performance impact of KPTI. This user here using PHP server mentioning 50% performance - https://twitter.com/timgostony/status/948682862844248065 https://twitter.com/timgostony/status/948682862844248065 and of-course OP issue is being covered here.
  - bartread 9y ago
    Thanks - this is what's worrying me: the result of patching seems to be all over the map, and it's somewhat hard to predict. My biggest worry is SQL Server, so our plan is to fix a performance issue we isolated recently before we patch - I'd like that system running optimally before we take the hit.
    - Abishek_Muthian 9y ago
      Judging from our MongoDB instance, I'm optimistic that your SQL server wouldn't face much impact either. Even Redhat tests put the risk at modest. But for something like Redis, the case could be different. If you are on a cloud provider, do check that automatic update is not scheduled; if so it's better to update manually to follow up with mitigation if necessary.
      - dijit 9y ago
        This is almost entirely subjective and workload dependent. We know that the biggest performance factor in both of these patches (spectre, meltdown) is the kernel boundary. So saying something along the lines of "Redis is more affected than Mongo" is.. odd. MongoDB (I HOPE) will save files more frequently than redis, since redis is primarily in in-memory DB with some lazy IO access (depending on configuration). So if you have a lot of small network calls to redis (which, is a kernel boundary event.) then it will be more impacted than a mongodb which takes only a few very large requests. My gut is that the opposite would be true for the same workloads on both, MongoDB needs its files to return data, redis doesn't. Even the vfs-cache is in kernel "space", so if your entire data sat in vfs cache it wouldn't help. But that's my gut, and your exact usage may not mirror the 1:1 of this scenario.
yborg 9y ago
"Significant network traffic" tells me almost nothing, that's a totally subjective description and you don't describe your workload at all. A back-end MMO game infrastructure server I have some understanding of in terms of the kind of load it is handling.
- burnte 9y ago
  It tells us his workload is more network bound than computationally intensive.
  - therealdrag0 9y ago
    Exactly, in my company's SaaS monitoring, I excluded db/external calls, and I saw a 1-2ms increase (on-top of 15ms), so yeah that's a 10% increase, but with external calls make the overall transaction 50+ms, so 1-2ms is trivial. Similarly, the CPU on our Azure cloud box holding a half dozen dockerized Java apps I can't really see went up at all. Hovering around 10% CPU usage before and after patch.
  - em3rgent0rdr 9y ago
    More specifically, the meltdown patch primarily affect memory bound applications, not necessarily computations, although those usually overlap for CPU bound applications.
EgoIncarnate 9y ago
Depending on when you started your instances, you may have already been patched. I've read reports of AWS performance drops as early as Dec 15th.
floatingatoll 9y ago
It's not necessarily helpful, since "Python" doesn't indicate 'single-threaded' or 'multi-threaded', and whether or not your threading is constrained on outside resources (MongoDB) or purely internal worker calculations, and etc. But it's interesting!
keltor 9y ago
We're using KVM on OpenStack. We use a number of the services and PostgreSQL is our main database platform with our main backend platform being C++ based. I have performance trending going back 1 year and honestly other than Postgres, the performance #s are like rounding errors. PostgreSQL is getting about a 7% hit on our databases that don't fit in memory. We also run SAP and again we're not seeing anything. The SAP Database runs on POWER Servers, so obviously unaffected at least by Meltdown. In our case, there was a major kernel version change, so we're just at the tail end of QC before I can release things to production to begin the rolling restarts tomorrow morning.

urbanxs 9y ago

That’s most likely because neither node, mongo nor python actually use the optimisation features of cpus. Says a lot about their performance quality.

bonzini 9y ago

Are you using PV or HVM instances?

Abishek_Muthian 9y ago
All of them on HVM.
- falcolas 9y ago
  IIRC, XEN said that 32bit HVM VMs are not affected by Meltdown, and so probably don't get impacted by AWS' patches. They still require Linux kernel updates to protect the kernel space, so changes might still be seen there.
  - Abishek_Muthian 9y ago
    Yes, but XEN said 64bit PV aren't affected either because they already run in KPTI like environment. So I assume 64bit HVM like ours aren't impacted for our work load.
  - bonzini 9y ago
    They are affected but there is no fix yet. 64-bit PV is unaffected and won't suffer a performance penalty (more precisely, it was already suffering it!!!), hence my original question, but a 64-bit PV guest can use Meltdown to attack the hypervisor. The fix is to update Xen, though I am not sure if fixes are already publicly available.

bartread 9y ago

Would you be able to post some ballpark figures, please?

Abishek_Muthian 9y ago
Actually, there is not any measurable difference. Our architecture is completely state-less and when compared with equivalent load there's no difference at all. I guess, default network throughput bottle-neck itself is higher than the random memory cache bottle-neck in my case. There was no impact on the latency either. Fearing automatic update by AWS resulting in performance issue, we rushed to update all our serves and gladly there weren't any performance impact. I don't think not every one has been fortunate, there's still lot of variables reg performance impact of KPTI. This user here using PHP server mentioning 50% performance - https://twitter.com/timgostony/status/948682862844248065 https://twitter.com/timgostony/status/948682862844248065 and of-course OP issue is being covered here.
- bartread 9y ago
  Thanks - this is what's worrying me: the result of patching seems to be all over the map, and it's somewhat hard to predict. My biggest worry is SQL Server, so our plan is to fix a performance issue we isolated recently before we patch - I'd like that system running optimally before we take the hit.
  - Abishek_Muthian 9y ago
    Judging from our MongoDB instance, I'm optimistic that your SQL server wouldn't face much impact either. Even Redhat tests put the risk at modest. But for something like Redis, the case could be different. If you are on a cloud provider, do check that automatic update is not scheduled; if so it's better to update manually to follow up with mitigation if necessary.
    - dijit 9y ago
      This is almost entirely subjective and workload dependent. We know that the biggest performance factor in both of these patches (spectre, meltdown) is the kernel boundary. So saying something along the lines of "Redis is more affected than Mongo" is.. odd. MongoDB (I HOPE) will save files more frequently than redis, since redis is primarily in in-memory DB with some lazy IO access (depending on configuration). So if you have a lot of small network calls to redis (which, is a kernel boundary event.) then it will be more impacted than a mongodb which takes only a few very large requests. My gut is that the opposite would be true for the same workloads on both, MongoDB needs its files to return data, redis doesn't. Even the vfs-cache is in kernel "space", so if your entire data sat in vfs cache it wouldn't help. But that's my gut, and your exact usage may not mirror the 1:1 of this scenario.

yborg 9y ago

"Significant network traffic" tells me almost nothing, that's a totally subjective description and you don't describe your workload at all. A back-end MMO game infrastructure server I have some understanding of in terms of the kind of load it is handling.

burnte 9y ago
It tells us his workload is more network bound than computationally intensive.
- therealdrag0 9y ago
  Exactly, in my company's SaaS monitoring, I excluded db/external calls, and I saw a 1-2ms increase (on-top of 15ms), so yeah that's a 10% increase, but with external calls make the overall transaction 50+ms, so 1-2ms is trivial. Similarly, the CPU on our Azure cloud box holding a half dozen dockerized Java apps I can't really see went up at all. Hovering around 10% CPU usage before and after patch.
- em3rgent0rdr 9y ago
  More specifically, the meltdown patch primarily affect memory bound applications, not necessarily computations, although those usually overlap for CPU bound applications.

EgoIncarnate 9y ago

Depending on when you started your instances, you may have already been patched. I've read reports of AWS performance drops as early as Dec 15th.

floatingatoll 9y ago

It's not necessarily helpful, since "Python" doesn't indicate 'single-threaded' or 'multi-threaded', and whether or not your threading is constrained on outside resources (MongoDB) or purely internal worker calculations, and etc. But it's interesting!

keltor 9y ago

We're using KVM on OpenStack. We use a number of the services and PostgreSQL is our main database platform with our main backend platform being C++ based. I have performance trending going back 1 year and honestly other than Postgres, the performance #s are like rounding errors. PostgreSQL is getting about a 7% hit on our databases that don't fit in memory. We also run SAP and again we're not seeing anything. The SAP Database runs on POWER Servers, so obviously unaffected at least by Meltdown. In our case, there was a major kernel version change, so we're just at the tail end of QC before I can release things to production to begin the rolling restarts tomorrow morning.