5 ms·

I feel like using spinlocks in user space at all without kernel support like rseq is just asking for weird performance degradations.

jcalvinowens 7mo ago
> I feel like using spinlocks in user space at all without kernel support like rseq is just asking for weird performance degradations. Yeah, exactly. "Doctor, help, somebody replaced my wooden hammer with a metal one, and now I can't hit myself in the face with it as many times." If you use spinlocks in userspace, you're gonna have a bad time.
- mgaunard 7mo ago
  Most people looking for performance will reach for the spinlock. The expectation is that the kernel should somehow detect applications that are spinning, and avoid preempting them early.
  - IshKebab 7mo ago
    Well that seems like an unreasonable expectation no? Also isn't the point of spinlocks that they get released before the kernel does anything? Otherwise you could just use a futex... Which maybe you should do anyway... https://matklad.github.io/2020/01/04/mutexes-are-faster-than-spinlocks.html https://matklad.github.io/2020/01/04/mutexes-are-faster-than...
    - mgaunard 7mo ago
      The scheduling is based on how much the LWP made use of its previous time slices. A spinning program clearly is using every cycle it's given without yielding, and so you can clearly tell preemption should be minimized.
  - silon42 7mo ago
    If you are spinning so long that it requires preemption, you're doing something wrong, no?
    - jcalvinowens 7mo ago
      It doesn't matter, it's a long tail thing: on average user spinlocks can work, and even appear to be beneficial on benchmarks (for many reasons, Andy alludes to some above). But if you have enough users, some of them will experience the apocalyptic long tail, no matter what you do: that's why user spinlocks are unacceptable. RSEQ is the first real answer for this, but it's still not a guarantee: it is not possible to disable SCHED_OTHER preemption in userspace. If I make something 1% faster on average, but now a random 0.000001% of its users see a ten-second stall every day, I lose. It is tempting to think about it as a latency/throughput tradeoff. But it isn't that simple, the unbounded thrashing can be more like a crash in terms of impact to the system.
      - mgaunard 7mo ago
        Well, you can always pin to a core and move other threads out of that core. That's what you'd do if manually scheduling. Ideally the dynamic scheduler would do that on its own.
        jcalvinowens 7mo ago
        Sure. But if you squint even that isn't good enough, you'll still take interrupts on that core in the critical section sometimes when somebody else wants the lock. The other problem with spin-wait is that it overshoots, especially with an increasing backoff. Part of the overhead of sleeping is paid back by being woken up immediately. When it's made to work, the backoff is often "overfit" in that very slight random differences in kernel scheduler behavior can cause huge apparent regressions.
      - silon42 6mo ago
        Yeah, the thrashing thing I'm very familiar on OOM scenario... the far most common Linux "crash" that I experience (at least monthly, sometimes daily, depending on what I'm doing)... I've waited overnight a few times but OOM killer still didn't activate.
- deleted 7mo ago
  [deleted]
jeltz 7mo ago
PostgreSQL is old and had to support kernels which did not support spinlocks. But, yes, maybe PostgreSQL should stop doing so now that kernels do.
anarazel 7mo ago
I really dislike the use of spinlocks in postgres (and have been replacing a lot of uses over time), but it's not always easy to replace them from a performance angle. On x86 a spinlock release doesn't need a memory barrier (unless you do insane things) / lock prefix, but a futex based lock does (because you otherwise may not realize you need to futex wake). Turns out that that increase in memory barriers causes regressions that are nontrivial to avoid. Another difficulty is that most of the remaining spinlocks are just a single bit in a 8 larger byte atomic. Futexes still don't support anything but 4 bytes (we could probably get away with using it on a part of the 8 byte atomic with some reordering) and unfortunately postgres still supports platforms with no 8 byte atomics (which I think is supremely silly), and the support for a fallback implementation makes it harder to use futexes. The spinlock triggering the contention in the report was just stupid and we only recently got around to removing it, because it isn't used during normal operation. Edit: forgot to add that the spinlock contention is not measurable on much more extreme workloads when using huge pages. A 100GB buffer pool with 4KB pages doesn't make much sense.
- anarazel 7mo ago
  Addendum big enough to warrant a separate post: The fact the contention is a spinlock, rather than a futex is unrelated to the "regression". A quick hack shows the contended performance to be nearly indistinguishable with a futex based lock. Which makes sense, non-PI futexes don't transfer the scheduler slice the lock owner, because they don't know who the lock owner is. Postgres' spinlock use randomized exponential backoff, so they don't prevent the lock owner from getting scheduled. Thus the contention is worse with PREEMPT_LAZY, even with non-PI futexes (which is what typical lock implementations are based on), because the lock holder gets scheduled out more often. Probably worth repeating: This contention is due to an absurd configuration that should never be used in practice.
  - menaerus 7mo ago
    Contention doesn't exist in older kernel versions even with huge-pages disabled, no?

jcalvinowens 7mo ago

> I feel like using spinlocks in user space at all without kernel support like rseq is just asking for weird performance degradations. Yeah, exactly. "Doctor, help, somebody replaced my wooden hammer with a metal one, and now I can't hit myself in the face with it as many times." If you use spinlocks in userspace, you're gonna have a bad time.

mgaunard 7mo ago
Most people looking for performance will reach for the spinlock. The expectation is that the kernel should somehow detect applications that are spinning, and avoid preempting them early.
- IshKebab 7mo ago
  Well that seems like an unreasonable expectation no? Also isn't the point of spinlocks that they get released before the kernel does anything? Otherwise you could just use a futex... Which maybe you should do anyway... https://matklad.github.io/2020/01/04/mutexes-are-faster-than-spinlocks.html https://matklad.github.io/2020/01/04/mutexes-are-faster-than...
  - mgaunard 7mo ago
    The scheduling is based on how much the LWP made use of its previous time slices. A spinning program clearly is using every cycle it's given without yielding, and so you can clearly tell preemption should be minimized.
- silon42 7mo ago
  If you are spinning so long that it requires preemption, you're doing something wrong, no?
  - jcalvinowens 7mo ago
    It doesn't matter, it's a long tail thing: on average user spinlocks can work, and even appear to be beneficial on benchmarks (for many reasons, Andy alludes to some above). But if you have enough users, some of them will experience the apocalyptic long tail, no matter what you do: that's why user spinlocks are unacceptable. RSEQ is the first real answer for this, but it's still not a guarantee: it is not possible to disable SCHED_OTHER preemption in userspace. If I make something 1% faster on average, but now a random 0.000001% of its users see a ten-second stall every day, I lose. It is tempting to think about it as a latency/throughput tradeoff. But it isn't that simple, the unbounded thrashing can be more like a crash in terms of impact to the system.
    - mgaunard 7mo ago
      Well, you can always pin to a core and move other threads out of that core. That's what you'd do if manually scheduling. Ideally the dynamic scheduler would do that on its own.
      - jcalvinowens 7mo ago
        Sure. But if you squint even that isn't good enough, you'll still take interrupts on that core in the critical section sometimes when somebody else wants the lock. The other problem with spin-wait is that it overshoots, especially with an increasing backoff. Part of the overhead of sleeping is paid back by being woken up immediately. When it's made to work, the backoff is often "overfit" in that very slight random differences in kernel scheduler behavior can cause huge apparent regressions.
    - silon42 6mo ago
      Yeah, the thrashing thing I'm very familiar on OOM scenario... the far most common Linux "crash" that I experience (at least monthly, sometimes daily, depending on what I'm doing)... I've waited overnight a few times but OOM killer still didn't activate.
deleted 7mo ago
[deleted]

jeltz 7mo ago

PostgreSQL is old and had to support kernels which did not support spinlocks. But, yes, maybe PostgreSQL should stop doing so now that kernels do.

anarazel 7mo ago

I really dislike the use of spinlocks in postgres (and have been replacing a lot of uses over time), but it's not always easy to replace them from a performance angle. On x86 a spinlock release doesn't need a memory barrier (unless you do insane things) / lock prefix, but a futex based lock does (because you otherwise may not realize you need to futex wake). Turns out that that increase in memory barriers causes regressions that are nontrivial to avoid. Another difficulty is that most of the remaining spinlocks are just a single bit in a 8 larger byte atomic. Futexes still don't support anything but 4 bytes (we could probably get away with using it on a part of the 8 byte atomic with some reordering) and unfortunately postgres still supports platforms with no 8 byte atomics (which I think is supremely silly), and the support for a fallback implementation makes it harder to use futexes. The spinlock triggering the contention in the report was just stupid and we only recently got around to removing it, because it isn't used during normal operation. Edit: forgot to add that the spinlock contention is not measurable on much more extreme workloads when using huge pages. A 100GB buffer pool with 4KB pages doesn't make much sense.

anarazel 7mo ago
Addendum big enough to warrant a separate post: The fact the contention is a spinlock, rather than a futex is unrelated to the "regression". A quick hack shows the contended performance to be nearly indistinguishable with a futex based lock. Which makes sense, non-PI futexes don't transfer the scheduler slice the lock owner, because they don't know who the lock owner is. Postgres' spinlock use randomized exponential backoff, so they don't prevent the lock owner from getting scheduled. Thus the contention is worse with PREEMPT_LAZY, even with non-PI futexes (which is what typical lock implementations are based on), because the lock holder gets scheduled out more often. Probably worth repeating: This contention is due to an absurd configuration that should never be used in practice.
- menaerus 7mo ago
  Contention doesn't exist in older kernel versions even with huge-pages disabled, no?