I think you're right it's probably nvidias fault rather than xorg - but my problems were solved by Wayland so it got me out of trouble. I had a similar problem

I think you're right it's probably nvidias fault rather than xorg - but my problems were solved by Wayland so it got me out of trouble. I had a similar problem to what the sibling comment mentioned - scaling was causing me performance issues. I disabled the laptop monitor and just ran multiple screens of the same resolution, and need about 150 % scaling to have text visible on these 4 X 32 inch 4 K monitors, but it was causing a huge amount of lag in xorg. It was solved by switching to Wayland which reduced the lag completely, and also let me run fractional scaling (I think I use 135 %) which wasn't available in xorg. I spent a lot of time with combinations of drivers and inputs and docks and scaling and couldn't get xorg to work. The laptop is a dell precision 3580 with a 4 GB NVIDIA gpu so a pretty standard piece of hardware