6 ms·

The Intel docs mention that PAUSE isn't the right choice for waits of "thousands of cycles or more" - even on the old architecture, 10ms is waaaaaay longer than

The Intel docs mention that PAUSE isn't the right choice for waits of "thousands of cycles or more" - even on the old architecture, 10ms is waaaaaay longer than recommended. I also have to quibble with this conclusion: This is not a .NET issue. It affects all Spinlock implementations which use the pause instruction. The instruction isn't the issue, the exponential backoff that .NET includes is. At the extreme (last iteration, on a 64-core CPU) it will spin for 1.2 million pause instructions...

phkahler 8y ago
Right. The whole time I was thinking the hardware change probably exacerbated an existing software issue. Then I saw this: "The issue has been fixed with .NET Core 2.1 and .NET Framework 4.8 Preview contains also the fixes for it." And by the end of the article I was thinking their code might just run faster as a single thread on a ~4GHz processor. Another thought was why do they have so much resource contention in the first place? That seems fishy. Another thought: Perhaps they should write it in Rust with all of its "fearless concurrency" claims. While I'm skeptical of those claims, The exercise would probably highlight some of the real issues. Any way I look at it, I keep thinking it's a software design problem.
- smaddox 8y ago
  Rust doesn't magically fix resources contention. It just provides safe concurrency primitives (and the surrounding language semantics) that make it difficult -to-impossible to misuse in such a way as to cause a deadlock or race condition.
  - phkahler 8y ago
    Right, so if the language forces you to eliminate deadlocks and race conditions you may find less need for spinlocks which are the problem here. Speculation on my part. I'm not actually advocating Rust but I am thinking about a project to give it a try.
    - ehsanu1 8y ago
      if the language forces you to eliminate deadlocks and race conditions It does not, it only prevents a specific problem known as a data race, which is an issue related to memory/ownership. There is no deadlock protection. And race conditions are a much broader set of issues than just data races. I like rust, just want to keep your expectations in check.
  - lunchbreak 8y ago
    If I remember correctly, rust doesn't help with deadlocks - you still have to be careful of them. Though it does have primitives to help significantly with race conditions
    - bonzini 8y ago
      The primitives help with taking the same lock around every access to a location. They don't help in finding locations that should use the same lock; you can easily write Rust code that uses too many separate locks and as such can still have race conditions.
  - coldseattle 8y ago
    It's amusing to see the Rust astroturfers jump in to any and all threads on hackernews spreading the gospel.
    - detaro 8y ago
      > While I'm skeptical of those claims, That looks like astroturfing to you? Please don't throw accusations like this around willy-nilly.
      - deleted 8y ago
        [deleted]
      - phkahler 8y ago
        >> That looks like astroturfing to you? No, it looks like people exaggerating to me. I don't doubt there is some merit to the safety claims made about Rust, but many people have a tendency toward unwarranted idealization. I admitted that I have not learned Rust yet which should add doubt to my own skepticism. The project I'm thinking of trying with Rust looks like it will be predominantly Unsafe Rust - using a lot of structures that are multiply referenced by necessity (or not?). I'm doing a lot of reading and thinking prior to starting because I really want the benefit from Rust as well as changing my thinking if there are good changes to be made there. Using an array of items and having them reference each other by index should satisfy the Rust compiler, but that doesn't actually solve the potential problems of trying to traverse (update) a complex data structure with parallel threads. The challenge of doing this suggests to me that the claims are overblown. A modestly complex data structure seems to require disabling all that safety but I am still learning and contemplating different approaches. Questioning basic assumptions seems prudent. I want to believe, but at this point I'm not convinced.
        deleted 8y ago
        [deleted]
        detaro 8y ago
        You might have missed that I was responding to a (flagged) response to you talking about "astroturfers" and not to you directly. Or I'm misreading your response?
  - bad_user 8y ago
    > difficult -to-impossible to misuse in such a way as to cause a deadlock or race condition Rust doesn't solve deadlocks or race conditions. Rust only makes it hard to accidentally share a memory location, but that's only a very small part of the problem, race conditions being a fundamental part of computer science, being baked into any language that supports expressing and joining asynchronous processes. I would be generous in agreeing with a statement about Rust making it a little more difficult to end up with a deadlock or unwanted race condition, except that's not true when compared with many other high level languages. Also nothing is impossible until you prove it via formal methods (e.g. TLA+).
    - lmm 8y ago
      > Rust only makes it hard to accidentally share a memory location, Not just a memory location, any value. You can use the tools Rust gives you on any piece of state you care to manage - a database connection, a remote server, the office coffee machine - represent it as a value and Rust will let you check that it's owned where you want it to be owned, borrowed where you want it to be borrowed and so on. > Also nothing is impossible until you prove it via formal methods (e.g. TLA+). What's the distinction you're drawing between something like TLA+ and something like the Rust borrow checker?
    - pcwalton 8y ago
      > I would be generous in agreeing with a statement about Rust making it a little more difficult to end up with a deadlock or unwanted race condition, except that's not true when compared with many other high level languages. It is easy to specifically describe what Rust prevents: unsynchronized memory access from multiple threads. What other high-level languages allow read-write shared-memory multithreading and guarantee that? > Also nothing is impossible until you prove it via formal methods (e.g. TLA+). The fact that unsynchronized memory access is impossible in safe Rust (modulo compiler bugs, CPU bugs, etc.) is something that nobody seriously doubts. It's obvious. I'm glad people are working on proving it, but the proof is an academic exercise.
      - Svenskunganka 8y ago
        > It is easy to specifically describe what Rust prevents: unsynchronized memory access from multiple threads. What other high-level languages allow read-write shared-memory multithreading and guarantee that? Pony maybe? https://www.ponylang.org/discover/#what-is-pony https://www.ponylang.org/discover/#what-is-pony At least that's my impression of its reference capabilities.
setquk 8y ago
It's definitely .net. We've observed deadlocks in spin scenarios which came down to this implementation rather than our callers.
- jclulow 8y ago
  You're suggesting, then, that the spinlock implementation is not merely slower on some CPUs but actually _incorrect_? It seems more likely that your software had some combination of a deadlock and a race condition exposed by different lock timing -- inserting delays in execution is one way to expand race windows to reproduce these kinds of defects.
  - setquk 8y ago
    Yes I am suggesting the implementation is incorrect. We had a standalone test case for this which could reproduce the condition after a random wait of 1-45 minutes. I will dig it out and raise it against .Net Core if it's still valid there. The problem was GC initiated i.e. the entry into GC caused the deadlock to start with which the process never recovered from as it was spinlocking all threads and no resources were available to complete GC.
  - deleted 8y ago
    [deleted]
eloff 8y ago
The whole point of spinning is to avoid the expense of a context switch by putting the thread to sleep, letting the OS schedule another. If you're going to spin for 10ms, that's longer than the timeslice most threads run for. That is going to be well above the overhead of any context switch. It makes no sense.
- dragontamer 8y ago
  > That is going to be well above the overhead of any context switch. Is it though? The "pause" instruction gives all resources to the hyperthread sibling. So under high-utilization (ie: your core has a hyperthread-sibling ready and willing), a "pause" instruction is basically a super-low latency task-switch. I wouldn't be surprised that a "spinlock with pause" would be better, even in 5ms to 10ms chunks of time (Windows has 15ms quantas, and may task switch as late as 30ms to 75ms depending on the situation). Because its better to do the hardware-assisted "pause" task switch rather than actually cause the OS-scheduler to get involved. In fact, that's what Microsoft AND Intel have concluded!! To get better real-world metrics, both Microsoft AND Intel felt like the "pause" instruction should wait longer! Skylake-X increases the latency, while .NET created this exponential-backoff. I bet it was good in isolation, but when combined, the overall effect was too much pausing.
  - eloff 8y ago
    Giving resources to the sibling hyperthread via pause is not the kind of context switch I'm talking about. I'm talking about the expensive context switch when the OS scheduler must swap out a running thread for another. That carries a steep obvious penalty switching between user space and the kernel and back, but actually can involve much heavier penalties where the cache and TLB are concerned - depending what thread runs next. Pausing for as much as 10ms could be better under some circumstances, but that would seem to me to be heavier than a context-switch. Whether the system actually achieves better throughput or not in that case would be workload dependent. Worst case the hyperthread is also running the spin lock loop and no useful work at all is accomplished on the core.
    - 8y ago

phkahler 8y ago

Right. The whole time I was thinking the hardware change probably exacerbated an existing software issue. Then I saw this: "The issue has been fixed with .NET Core 2.1 and .NET Framework 4.8 Preview contains also the fixes for it." And by the end of the article I was thinking their code might just run faster as a single thread on a ~4GHz processor. Another thought was why do they have so much resource contention in the first place? That seems fishy. Another thought: Perhaps they should write it in Rust with all of its "fearless concurrency" claims. While I'm skeptical of those claims, The exercise would probably highlight some of the real issues. Any way I look at it, I keep thinking it's a software design problem.

smaddox 8y ago
Rust doesn't magically fix resources contention. It just provides safe concurrency primitives (and the surrounding language semantics) that make it difficult -to-impossible to misuse in such a way as to cause a deadlock or race condition.
- phkahler 8y ago
  Right, so if the language forces you to eliminate deadlocks and race conditions you may find less need for spinlocks which are the problem here. Speculation on my part. I'm not actually advocating Rust but I am thinking about a project to give it a try.
  - ehsanu1 8y ago
    if the language forces you to eliminate deadlocks and race conditions It does not, it only prevents a specific problem known as a data race, which is an issue related to memory/ownership. There is no deadlock protection. And race conditions are a much broader set of issues than just data races. I like rust, just want to keep your expectations in check.
- lunchbreak 8y ago
  If I remember correctly, rust doesn't help with deadlocks - you still have to be careful of them. Though it does have primitives to help significantly with race conditions
  - bonzini 8y ago
    The primitives help with taking the same lock around every access to a location. They don't help in finding locations that should use the same lock; you can easily write Rust code that uses too many separate locks and as such can still have race conditions.
- coldseattle 8y ago
  It's amusing to see the Rust astroturfers jump in to any and all threads on hackernews spreading the gospel.
  - detaro 8y ago
    > While I'm skeptical of those claims, That looks like astroturfing to you? Please don't throw accusations like this around willy-nilly.
    - deleted 8y ago
      [deleted]
    - phkahler 8y ago
      >> That looks like astroturfing to you? No, it looks like people exaggerating to me. I don't doubt there is some merit to the safety claims made about Rust, but many people have a tendency toward unwarranted idealization. I admitted that I have not learned Rust yet which should add doubt to my own skepticism. The project I'm thinking of trying with Rust looks like it will be predominantly Unsafe Rust - using a lot of structures that are multiply referenced by necessity (or not?). I'm doing a lot of reading and thinking prior to starting because I really want the benefit from Rust as well as changing my thinking if there are good changes to be made there. Using an array of items and having them reference each other by index should satisfy the Rust compiler, but that doesn't actually solve the potential problems of trying to traverse (update) a complex data structure with parallel threads. The challenge of doing this suggests to me that the claims are overblown. A modestly complex data structure seems to require disabling all that safety but I am still learning and contemplating different approaches. Questioning basic assumptions seems prudent. I want to believe, but at this point I'm not convinced.
      - deleted 8y ago
        [deleted]
      - detaro 8y ago
        You might have missed that I was responding to a (flagged) response to you talking about "astroturfers" and not to you directly. Or I'm misreading your response?
- bad_user 8y ago
  > difficult -to-impossible to misuse in such a way as to cause a deadlock or race condition Rust doesn't solve deadlocks or race conditions. Rust only makes it hard to accidentally share a memory location, but that's only a very small part of the problem, race conditions being a fundamental part of computer science, being baked into any language that supports expressing and joining asynchronous processes. I would be generous in agreeing with a statement about Rust making it a little more difficult to end up with a deadlock or unwanted race condition, except that's not true when compared with many other high level languages. Also nothing is impossible until you prove it via formal methods (e.g. TLA+).
  - lmm 8y ago
    > Rust only makes it hard to accidentally share a memory location, Not just a memory location, any value. You can use the tools Rust gives you on any piece of state you care to manage - a database connection, a remote server, the office coffee machine - represent it as a value and Rust will let you check that it's owned where you want it to be owned, borrowed where you want it to be borrowed and so on. > Also nothing is impossible until you prove it via formal methods (e.g. TLA+). What's the distinction you're drawing between something like TLA+ and something like the Rust borrow checker?
  - pcwalton 8y ago
    > I would be generous in agreeing with a statement about Rust making it a little more difficult to end up with a deadlock or unwanted race condition, except that's not true when compared with many other high level languages. It is easy to specifically describe what Rust prevents: unsynchronized memory access from multiple threads. What other high-level languages allow read-write shared-memory multithreading and guarantee that? > Also nothing is impossible until you prove it via formal methods (e.g. TLA+). The fact that unsynchronized memory access is impossible in safe Rust (modulo compiler bugs, CPU bugs, etc.) is something that nobody seriously doubts. It's obvious. I'm glad people are working on proving it, but the proof is an academic exercise.
    - Svenskunganka 8y ago
      > It is easy to specifically describe what Rust prevents: unsynchronized memory access from multiple threads. What other high-level languages allow read-write shared-memory multithreading and guarantee that? Pony maybe? https://www.ponylang.org/discover/#what-is-pony https://www.ponylang.org/discover/#what-is-pony At least that's my impression of its reference capabilities.

setquk 8y ago

It's definitely .net. We've observed deadlocks in spin scenarios which came down to this implementation rather than our callers.

jclulow 8y ago
You're suggesting, then, that the spinlock implementation is not merely slower on some CPUs but actually _incorrect_? It seems more likely that your software had some combination of a deadlock and a race condition exposed by different lock timing -- inserting delays in execution is one way to expand race windows to reproduce these kinds of defects.
- setquk 8y ago
  Yes I am suggesting the implementation is incorrect. We had a standalone test case for this which could reproduce the condition after a random wait of 1-45 minutes. I will dig it out and raise it against .Net Core if it's still valid there. The problem was GC initiated i.e. the entry into GC caused the deadlock to start with which the process never recovered from as it was spinlocking all threads and no resources were available to complete GC.
- deleted 8y ago
  [deleted]

eloff 8y ago

The whole point of spinning is to avoid the expense of a context switch by putting the thread to sleep, letting the OS schedule another. If you're going to spin for 10ms, that's longer than the timeslice most threads run for. That is going to be well above the overhead of any context switch. It makes no sense.

dragontamer 8y ago
> That is going to be well above the overhead of any context switch. Is it though? The "pause" instruction gives all resources to the hyperthread sibling. So under high-utilization (ie: your core has a hyperthread-sibling ready and willing), a "pause" instruction is basically a super-low latency task-switch. I wouldn't be surprised that a "spinlock with pause" would be better, even in 5ms to 10ms chunks of time (Windows has 15ms quantas, and may task switch as late as 30ms to 75ms depending on the situation). Because its better to do the hardware-assisted "pause" task switch rather than actually cause the OS-scheduler to get involved. In fact, that's what Microsoft AND Intel have concluded!! To get better real-world metrics, both Microsoft AND Intel felt like the "pause" instruction should wait longer! Skylake-X increases the latency, while .NET created this exponential-backoff. I bet it was good in isolation, but when combined, the overall effect was too much pausing.
- eloff 8y ago
  Giving resources to the sibling hyperthread via pause is not the kind of context switch I'm talking about. I'm talking about the expensive context switch when the OS scheduler must swap out a running thread for another. That carries a steep obvious penalty switching between user space and the kernel and back, but actually can involve much heavier penalties where the cache and TLB are concerned - depending what thread runs next. Pausing for as much as 10ms could be better under some circumstances, but that would seem to me to be heavier than a context-switch. Whether the system actually achieves better throughput or not in that case would be workload dependent. Worst case the hyperthread is also running the spin lock loop and no useful work at all is accomplished on the core.
  - 8y ago