3 ms·

> With multicore processors becoming the norm, a combination of isocpus, some low latency kernel options, and some virtualization are all that's needed imo. H

> With multicore processors becoming the norm, a combination of isocpus, some low latency kernel options, and some virtualization are all that's needed imo. Have you worked in vehicle embedded systems or are you just pulling requirements (or lack thereof) out of your hat? I'm not an expert, but I would guess that they do require deterministic behaviour, which, AFAIK, Linux doesn't offer. What happens if for some weird reason the scheduler starves the critical process for 40 ms in some uncommon situation that only happens 0.00001% of the time? How can you prove that such a thing won't ever happen?

ncmncm 6y ago
His point is that there are ways to keep the OS from ever fooling with certain cores, after startup. At the extreme, you put an exo-, para-, iso-, whatsit-kernel on them, but just starting a process that does all its kernel activity at startup, then never does another system call after, is more common. Usually that involves mapping device registers into the process address space via a custom ioctl, and then just polling in a tight loop.
- lmilcin 6y ago
  That's not really enough. As an example ANY operation involving filesystem or memory access has no defined limit on how long it can take. Your filesystem can decide to do some housekeeping while you do write and as simple thing as dereferencing a pointer can cause cause context switch to Linux kernel to do the TLB thing. You might want to allocate some memory and then Linux might decide to move a bunch of memory around before it gives the pointer back to you. What I mean is, to be able to guarantee execution of a piece of code is not a simple task for the faint hearted. You need to forget IO. You need to forget switching context to Linux (basically forget any syscalls). You need to forget allocating any memory after program startup. You need to use really huge pages. Forget being green, because changing CPU frequency or sleeping can cause unexpected latency in the middle of some operation. Also doing this inside VM is neigh impossible.
  - ncmncm 6y ago
    If you are not doing any system calls, it means you are perforce not doing file system operations. Memory is normally pre-allocated and mlocked into place at startup. TLB misses can happen, but those are filled in hardware, just like regular cache misses, not by kernel activity (and often from cache). The "TLB shootdowns" you read about happen when memory is unmapped, so you don't. The kernel can block you if you write to memory mapping a file backed by a physical disk, so you don't. And, yes, the core is locked in C0 state, drawing maximum power full time. It is certainly possible to do I/O, by operating directly on the device registers that I noted were mapped into process memory. And, in fact, this is a routine method of conducting such operations. You can read about it by searching "kernel bypass". There is usually no need or desire to do it in a VM, but kernel-bypass libraries are quite capable of mapping their registers into a VM's memory, so somebody must be doing it. Commonly, input from network interfaces is dumped into one or more big-ass ring buffers mapped onto /dev/hugepages; and other, regular processes, running on other cores, map those read-only and pick over it for interesting bits, and maybe do file system operations of their own.

ncmncm 6y ago

His point is that there are ways to keep the OS from ever fooling with certain cores, after startup. At the extreme, you put an exo-, para-, iso-, whatsit-kernel on them, but just starting a process that does all its kernel activity at startup, then never does another system call after, is more common. Usually that involves mapping device registers into the process address space via a custom ioctl, and then just polling in a tight loop.

lmilcin 6y ago
That's not really enough. As an example ANY operation involving filesystem or memory access has no defined limit on how long it can take. Your filesystem can decide to do some housekeeping while you do write and as simple thing as dereferencing a pointer can cause cause context switch to Linux kernel to do the TLB thing. You might want to allocate some memory and then Linux might decide to move a bunch of memory around before it gives the pointer back to you. What I mean is, to be able to guarantee execution of a piece of code is not a simple task for the faint hearted. You need to forget IO. You need to forget switching context to Linux (basically forget any syscalls). You need to forget allocating any memory after program startup. You need to use really huge pages. Forget being green, because changing CPU frequency or sleeping can cause unexpected latency in the middle of some operation. Also doing this inside VM is neigh impossible.
- ncmncm 6y ago
  If you are not doing any system calls, it means you are perforce not doing file system operations. Memory is normally pre-allocated and mlocked into place at startup. TLB misses can happen, but those are filled in hardware, just like regular cache misses, not by kernel activity (and often from cache). The "TLB shootdowns" you read about happen when memory is unmapped, so you don't. The kernel can block you if you write to memory mapping a file backed by a physical disk, so you don't. And, yes, the core is locked in C0 state, drawing maximum power full time. It is certainly possible to do I/O, by operating directly on the device registers that I noted were mapped into process memory. And, in fact, this is a routine method of conducting such operations. You can read about it by searching "kernel bypass". There is usually no need or desire to do it in a VM, but kernel-bypass libraries are quite capable of mapping their registers into a VM's memory, so somebody must be doing it. Commonly, input from network interfaces is dumped into one or more big-ass ring buffers mapped onto /dev/hugepages; and other, regular processes, running on other cores, map those read-only and pick over it for interesting bits, and maybe do file system operations of their own.