3 ms·
local memory = shm.map(shmpath, ffi.typeof("char["..size.."]")) mmapp'ed, so buffer memory is always 4096 aligned. -- Log entry format (ring item) f
by vardump 11y ago
local memory = shm.map(shmpath, ffi.typeof("char["..size.."]"))
mmapp'ed, so buffer memory is always 4096 aligned.
-- Log entry format (ring item)
ffi.cdef[[
struct timeline_entry {
uint64_t tsc; // CPU timestamp
uint64_t ip; // Instruction pointer of logger
uint32_t aux; // TSC_AUX: core (bits 0-7) + numa (12-15)
uint32_t reserved; // (pad to 64 bytes)
uint64_t msg_id; // category:24, message:24, priority:16
uint64_t msg0, msg1, msg2, msg3; // message arguments
};
]]
And 64 bytes long. Just like a cache line, probably not a coincidence? No unnecessary slow memory traffic or cache pollution. Nice.
Such a shame TSC_AUX is always 0 on Windows and most Linux kernels as well. Oh well. It's not that much code or work to write CPU core index and NUMA group to an MSR when bringing up a CPU core.
- jsmthrowaway 11y agoYeah, no kidding. The author did their homework and this is really good code to study. I'm with you, it's almost certainly not a coincidence. It's aging a bit, but Ulrich Drepper's seminal paper on memory comes to mind: https://www.akkadia.org/drepper/cpumemory.pdf https://www.akkadia.org/drepper/cpumemory.pdf Say what you want about Drepper's style and personality, but that paper told me (a) he's an incredibly knowledgable dude and (b) I'll always have more to learn, especially about cache.
- vardump 11y agoLooks like he's not writing to "reserved" field. So CPU will definitely need to read the cache line at current log entry before write (RFO, read for ownership). I'm not sure whether current CPUs are smart enough, but in theory writing 64 bytes at once to cache aligned address could avoid RFO. If it's L1/L2/L3 miss, that could be 100-200 cycles saving. Might have also been better to do RDTSCP first. Otherwise it'll also avoid reordering instructions. And sabotage attempt to avoid RFO. Anyways, not sure, didn't profile. (And by profiling I mean using those countless CPU performance counters to figure out what's going on in the mysterious black box.)