4 ms·
> The most effective practical implementation I am aware of is the LMAX Disruptor, as this implementation can essentially funnel multiple threads into a single
by memco 5y ago
> The most effective practical implementation I am aware of is the LMAX Disruptor, as this implementation can essentially funnel multiple threads into a single writer w/ aggregate rates of tens-to-hundreds of millions of operations per second.
Do you have more info on this? I work on magnetic tapes where any writes must be sequential but I want to process data to it and from it in parallel where possible.
- bob1029 5y agoNo specific references aside from my own experimentation around these ideas. The core essence is as follows: - Use an append-only log structure as the exclusive means of interfacing with the storage medium. If you are concerned about future cleanup (i.e. on block-wise rewritable media), segment this log out into file chunks of reasonable size. Cleanup of old segments would involve scanning each and rewriting alive data to the front of the log. - Consider using a key-value abstraction as the basis for all of this, as it allows for trivially constructing dynamically-programmed tree structures with ideal locality-of-reference semantics (i.e. the splay tree and friends). - All writes to the append-only log conclude with a consistent snapshot of system state. This is where the magic happens. The ring buffer results in batching of transaction request (e.g. SetValue, GetValue) such that a single final byte array can be constructed all at once that ultimately goes out to disk. You may have 1000 transactions bundled up into 10KB of contiguous bytes. This allows for you to say things like "transactions per block i/o". Callers into the system do a busy wait against a boolean status flag on the transaction object (or more ideally, structs in a fixed-sized array). This certainly consumes more power, but you wanted to go fast, right?