4 ms·

This is trying to replicate Java/Log4J2/Disruptor in Lua?

jsmthrowaway 11y ago
No. You could build dtrace with software like this, whereas the entire stack you mentioned only cares indirectly about per-message cycle overhead. This is for when your tight loop, like your routing engine or game loop, wants to kick out numbers at timestamps as fast as it can and other software deals with making them presentable. It's a great technique. This is built for networking software/hardware, where every cycle matters, and is actually extremely useful to me in a couple domains. I'm impressed disabled messages only sink five cycles so I assume there was hand tuning done here. Edit: And I'm right, it's a hand tuned logical compare (cool): https://github.com/lukego/snabb/blob/timeline/src/core/timeline.dasl https://github.com/lukego/snabb/blob/timeline/src/core/timel... If you want reasons to learn assembly, there you go. Too few folks wielding it in typical Web operations, where it would be extraordinarily useful in any number of hot paths. Just requires care.
- vardump 11y ago
  local memory = shm.map(shmpath, ffi.typeof("char["..size.."]")) mmapp'ed, so buffer memory is always 4096 aligned. -- Log entry format (ring item) ffi.cdef[[ struct timeline_entry { uint64_t tsc; // CPU timestamp uint64_t ip; // Instruction pointer of logger uint32_t aux; // TSC_AUX: core (bits 0-7) + numa (12-15) uint32_t reserved; // (pad to 64 bytes) uint64_t msg_id; // category:24, message:24, priority:16 uint64_t msg0, msg1, msg2, msg3; // message arguments }; ]] And 64 bytes long. Just like a cache line, probably not a coincidence? No unnecessary slow memory traffic or cache pollution. Nice. Such a shame TSC_AUX is always 0 on Windows and most Linux kernels as well. Oh well. It's not that much code or work to write CPU core index and NUMA group to an MSR when bringing up a CPU core.
  - jsmthrowaway 11y ago
    Yeah, no kidding. The author did their homework and this is really good code to study. I'm with you, it's almost certainly not a coincidence. It's aging a bit, but Ulrich Drepper's seminal paper on memory comes to mind: https://www.akkadia.org/drepper/cpumemory.pdf https://www.akkadia.org/drepper/cpumemory.pdf Say what you want about Drepper's style and personality, but that paper told me (a) he's an incredibly knowledgable dude and (b) I'll always have more to learn, especially about cache.
    - vardump 11y ago
      Looks like he's not writing to "reserved" field. So CPU will definitely need to read the cache line at current log entry before write (RFO, read for ownership). I'm not sure whether current CPUs are smart enough, but in theory writing 64 bytes at once to cache aligned address could avoid RFO. If it's L1/L2/L3 miss, that could be 100-200 cycles saving. Might have also been better to do RDTSCP first. Otherwise it'll also avoid reordering instructions. And sabotage attempt to avoid RFO. Anyways, not sure, didn't profile. (And by profiling I mean using those countless CPU performance counters to figure out what's going on in the mysterious black box.)
frugalmail 11y ago
Not sure why a legitimate question gets downvoted here. In reality with the intelligent response by jsmthrowaway, and me spending way more time than I wanted to looking at their repo, it clearly is using the same strategy that I mentioned above. The difference is that it's at a lower level for a different platform (not lua).

No. You could build dtrace with software like this, whereas the entire stack you mentioned only cares indirectly about per-message cycle overhead. This is for when your tight loop, like your routing engine or game loop, wants to kick out numbers at timestamps as fast as it can and other software deals with making them presentable. It's a great technique. This is built for networking software/hardware, where every cycle matters, and is actually extremely useful to me in a couple domains. I'm impressed disabled messages only sink five cycles so I assume there was hand tuning done here. Edit: And I'm right, it's a hand tuned logical compare (cool): https://github.com/lukego/snabb/blob/timeline/src/core/timeline.dasl https://github.com/lukego/snabb/blob/timeline/src/core/timel... If you want reasons to learn assembly, there you go. Too few folks wielding it in typical Web operations, where it would be extraordinarily useful in any number of hot paths. Just requires care.

vardump 11y ago
local memory = shm.map(shmpath, ffi.typeof("char["..size.."]")) mmapp'ed, so buffer memory is always 4096 aligned. -- Log entry format (ring item) ffi.cdef[[ struct timeline_entry { uint64_t tsc; // CPU timestamp uint64_t ip; // Instruction pointer of logger uint32_t aux; // TSC_AUX: core (bits 0-7) + numa (12-15) uint32_t reserved; // (pad to 64 bytes) uint64_t msg_id; // category:24, message:24, priority:16 uint64_t msg0, msg1, msg2, msg3; // message arguments }; ]] And 64 bytes long. Just like a cache line, probably not a coincidence? No unnecessary slow memory traffic or cache pollution. Nice. Such a shame TSC_AUX is always 0 on Windows and most Linux kernels as well. Oh well. It's not that much code or work to write CPU core index and NUMA group to an MSR when bringing up a CPU core.
- jsmthrowaway 11y ago
  Yeah, no kidding. The author did their homework and this is really good code to study. I'm with you, it's almost certainly not a coincidence. It's aging a bit, but Ulrich Drepper's seminal paper on memory comes to mind: https://www.akkadia.org/drepper/cpumemory.pdf https://www.akkadia.org/drepper/cpumemory.pdf Say what you want about Drepper's style and personality, but that paper told me (a) he's an incredibly knowledgable dude and (b) I'll always have more to learn, especially about cache.
  - vardump 11y ago
    Looks like he's not writing to "reserved" field. So CPU will definitely need to read the cache line at current log entry before write (RFO, read for ownership). I'm not sure whether current CPUs are smart enough, but in theory writing 64 bytes at once to cache aligned address could avoid RFO. If it's L1/L2/L3 miss, that could be 100-200 cycles saving. Might have also been better to do RDTSCP first. Otherwise it'll also avoid reordering instructions. And sabotage attempt to avoid RFO. Anyways, not sure, didn't profile. (And by profiling I mean using those countless CPU performance counters to figure out what's going on in the mysterious black box.)

frugalmail 11y ago

Not sure why a legitimate question gets downvoted here. In reality with the intelligent response by jsmthrowaway, and me spending way more time than I wanted to looking at their repo, it clearly is using the same strategy that I mentioned above. The difference is that it's at a lower level for a different platform (not lua).