3 ms·

Interesting project. I almost wish I had a concurrency bug to test it on. > Guest software running in the Antithesis platform still experiences concurrency sim

by tlb 3y ago

Interesting project. I almost wish I had a concurrency bug to test it on. > Guest software running in the Antithesis platform still experiences concurrency similar to a multi-core / multi-machine system, thanks to the process scheduling imposed by the guest OS This might not exercise the full set of race conditions. When two threads are running simultaneously on separate cores (or hyper-threaded on the same core) they can interleave instructions at a much finer granularity than any OS time slicing would cause, even within instructions. For example, could it find a race condition where two threads are executing INC [addr] on the same memory address, where context switching between instructions doesn't trigger it?

wmf 3y ago
If history is any guide, everybody has concurrency bugs.
metalcrow 3y ago
> For example, could it find a race condition where two threads are executing INC [addr] on the same memory address, where context switching between instructions doesn't trigger it? I'm not actually familiar with the details of hardware MMUs, but would they not enforce sequential access of the address? Or do MMUs allow parallel reads and writes?
- tlb 3y ago
  x86 processors have a LOCK instruction prefix, which makes some instructions atomic including increment. Increment is nontrivial to make atomic, because there are two memory accesses: read X and write back X+1. It's a bit slow, because it has to inform every other core in the system, "Hey, I just read this memory address and I'm about to write something back to it, so don't don't use it until I'm done." C++ has functions to do this like std::atomic_fetch_add, but they're clunky to use so people often forget.
  - metalcrow 3y ago
    Ahh, it's 2 separate memory accesses just encapsulated in one instruction, that explains it. Yeah, unless this hypervisor allows context switching at the level of microcode ops, that seems to be undetectable currently.
  - monocasa 3y ago
    The way that's currently implemented is actually just one memory operation these days. L2 (or really wherever coherency is mostly managed) has a tiny ALU so in addition to read or write, atomicop is an operation you can send to L2. It'll gain Modified or Exclusive access to the cache line(s) that op is addressed to and just do the operation right there. That way the normal cache protocol is all you need for atomicity, and really the line is only contended for a single cycle from the L2 controller's (and the rest of the coherency peers) perspective. That's why they retconned the lock prefix to not be an actual assertion of the #LOCK signal any more. That's also why TileLink and AMBA include atomic ops like addition and bitwise ops in their coherency protocols rather than just 'claim region'. That's also why you see newer archs like RISC-V and Arm64 that have both lr/sc style ops, in addition to direct atomic memory ops like amoadd.w, it better matches the primitives of the underlying memory system.
    - BeeOnRope 3y ago
      That's not how it works on x86 as far as I know. The atomic ops are simply performed by the ALU, against the L1 cache when the instruction is about to retire. Atomicity is guaranteed by not allowing the line to be stolen by another core between the operation, and memory order is ensured by draining the store buffer before executing the op and other speculation (eg loads can speculatively pass even atomic operations).
wwilson 3y ago
You are correct. There's a set of concurrency bugs that require actual SMP setups to trigger (like stuff with atomic operations, memory ordering, etc.). If you're trying to build a very low-level lock-free data structure where this is your primary threat model, Antithesis is not the right tool for you... for now... until we build a CPU simulator...

by tlb 3y ago

wmf 3y ago

If history is any guide, everybody has concurrency bugs.

metalcrow 3y ago

> For example, could it find a race condition where two threads are executing INC [addr] on the same memory address, where context switching between instructions doesn't trigger it? I'm not actually familiar with the details of hardware MMUs, but would they not enforce sequential access of the address? Or do MMUs allow parallel reads and writes?

tlb 3y ago
x86 processors have a LOCK instruction prefix, which makes some instructions atomic including increment. Increment is nontrivial to make atomic, because there are two memory accesses: read X and write back X+1. It's a bit slow, because it has to inform every other core in the system, "Hey, I just read this memory address and I'm about to write something back to it, so don't don't use it until I'm done." C++ has functions to do this like std::atomic_fetch_add, but they're clunky to use so people often forget.
- metalcrow 3y ago
  Ahh, it's 2 separate memory accesses just encapsulated in one instruction, that explains it. Yeah, unless this hypervisor allows context switching at the level of microcode ops, that seems to be undetectable currently.
- monocasa 3y ago
  The way that's currently implemented is actually just one memory operation these days. L2 (or really wherever coherency is mostly managed) has a tiny ALU so in addition to read or write, atomicop is an operation you can send to L2. It'll gain Modified or Exclusive access to the cache line(s) that op is addressed to and just do the operation right there. That way the normal cache protocol is all you need for atomicity, and really the line is only contended for a single cycle from the L2 controller's (and the rest of the coherency peers) perspective. That's why they retconned the lock prefix to not be an actual assertion of the #LOCK signal any more. That's also why TileLink and AMBA include atomic ops like addition and bitwise ops in their coherency protocols rather than just 'claim region'. That's also why you see newer archs like RISC-V and Arm64 that have both lr/sc style ops, in addition to direct atomic memory ops like amoadd.w, it better matches the primitives of the underlying memory system.
  - BeeOnRope 3y ago
    That's not how it works on x86 as far as I know. The atomic ops are simply performed by the ALU, against the L1 cache when the instruction is about to retire. Atomicity is guaranteed by not allowing the line to be stolen by another core between the operation, and memory order is ensured by draining the store buffer before executing the op and other speculation (eg loads can speculatively pass even atomic operations).

wwilson 3y ago

You are correct. There's a set of concurrency bugs that require actual SMP setups to trigger (like stuff with atomic operations, memory ordering, etc.). If you're trying to build a very low-level lock-free data structure where this is your primary threat model, Antithesis is not the right tool for you... for now... until we build a CPU simulator...