9 ms·

> Worse, not even the ARM ISA is ready for this. An astute reader might realize that computing the cache line on every invocation is not enough for user space c

by wolfgke 10y ago

> Worse, not even the ARM ISA is ready for this. An astute reader might realize that computing the cache line on every invocation is not enough for user space code: It can happen that a process gets scheduled on a different CPU while executing the __clear_cache function with a certain cache line size, where it might not be valid anymore. I rather see the problem in the fact that there seems to be no possibility to say to the Linux scheduler: Only schedule this process/thread between cores that have the same cache line size. Or add an attribute when some thread is created on the cache line size of the cores it is allowed to run. Or an attribute when some thread is created for "allow arbitrary cache line size but don't let it run on cores with a different size". This way it would suffice to check for the cache size on program or thread start.

monocasa 10y ago
In that case, you could read /proc/cpuinfo and set your affinity mask.
- wolfgke 10y ago
  Suggest this to the authors of the original article. :-)
tveita 10y ago
That's too specific a feature to expose to developers - most people wouldn't even know about the feature, and the ones that did might needlessly enable it just to be conservative. The intention of the big.LITTLE architecture is to let processes be migrated seamlessly between the small and big core and let the unused core be turned off to save power. The kernel and the hardware should work together to make the core switching transparent and expose a safe way to invalidate the cache independent of the current processor.
- wolfgke 10y ago
  > The intention of the big.LITTLE architecture is to let processes be migrated seamlessly between the small and big core and let the unused core be turned off to save power. On the other hand flushing specific cache lines is a rather special feature. If the code uses such obscure low-level features (much more than the "typical" application) such as the invariant that the size of a cache-line stays constant over the execution (which most applications really don't care about) one can at least expect from the developer to pass this information to the scheduler so that the scheduler can take that this invariant will indeed be satisfied.
  - comex 10y ago
    Anything that uses a JIT necessarily uses this "obscure low-level feature", including any program implemented in one of many programming languages. Forcing such programs onto the heavy processor makes no sense; there just needs to be a way to properly clear the cache.
    - wolfgke 10y ago
      On https://news.ycombinator.com/item?id=12483698 https://news.ycombinator.com/item?id=12483698 I wrote a better idea how this might be implemented without causing this problem. Nevertheless even if we use my first, worse interface this should not be a problem: Spawn a thread that does the lowlevel stuff, sync it and after that run the JITted code.
lambda 10y ago
But then you would never be able to take advantage of the power/performance tradeoff that big.LITTLE gives you; of being able to migrate you to a big core when you need it, but save power by migrating you to the little core when you're not doing anything CPU intensive, and shutting off the power-hungry big core.
- wolfgke 10y ago
  > But then you would never be able to take advantage of the power/performance tradeoff that big.LITTLE gives you Rather: Only applications that don't depend on the invariant that the size of a cache line will stay constant over the execution time will take advantage of this. I can imagine ways how one could increase the advantage, but I don't want to go too much into technical details here.
  - lambda 10y ago
    Why would programs want to "depend on the invariant that the size of a cache line will stay constant"? That's a minor detail that most applications don't care about. In this case, it probably mattered because when JITing code, you need to explicitly flush instruction cache after changing code. But that's not anything the application cares about; that's something the runtime cares about, and the runtime is going to want to support taking advantage of the speed/power tradeoff without making application developers have to go to the trouble of thinking about it. Also, from a user perspective, you don't want applications to be able to demand only the big, powerful cores. That will eat through your battery much faster, with likely very little benefit.
    - wolfgke 10y ago
      > Also, from a user perspective, you don't want applications to be able to demand only the big, powerful cores. This should rather be a setting in the rights system that a user can refuse the right to an application to be able to demand big, powerful cores. I can nevertheless imagine quite well scenarios where enabling a (specific) application to do so can be quite useful.
wolfgke 10y ago
Another idea how one could solve this problem by a clever kernel interface. This idea is probably a better than the approach in my post above, since this enables to migrate threads from big to little cores and vice versa except for critical regions. Add an interface to the kernel/scheduler void lock_cacheline_size(...) void unlock_cacheline_size(...) Calling lock_cacheline_size tells that from now on the thread must stay on a core with the same cache line size. Consider it as locking a mutex on the cache line size. After this you can run critical code that depends on the fact that the cache line size stays constant for the whole time. After you are finished with it, you call unlock_cacheline_size (as a kind of unlocking the mutex on the cache line size) - from now on the scheduler is free again to migrate the code again to cores with different cache line sizes.
- cesarb 10y ago
  A simpler and more robust approach would be to add a system call to flush a memory range. The kernel can then do all the locking, without the risk that migration to the other core gets indefinitely blocked by user space code.
Senji 10y ago
>An astute reader might realize that computing the cache line on every invocation is not enough for user space code: It can happen that a process gets scheduled on a different CPU while executing the __clear_cache function with a certain cache line size, where it might not be valid anymore. Doesn't ARM have an "Enter Critical Region" instruction?

by wolfgke 10y ago

monocasa 10y ago

In that case, you could read /proc/cpuinfo and set your affinity mask.

wolfgke 10y ago
Suggest this to the authors of the original article. :-)

tveita 10y ago

That's too specific a feature to expose to developers - most people wouldn't even know about the feature, and the ones that did might needlessly enable it just to be conservative. The intention of the big.LITTLE architecture is to let processes be migrated seamlessly between the small and big core and let the unused core be turned off to save power. The kernel and the hardware should work together to make the core switching transparent and expose a safe way to invalidate the cache independent of the current processor.

wolfgke 10y ago
> The intention of the big.LITTLE architecture is to let processes be migrated seamlessly between the small and big core and let the unused core be turned off to save power. On the other hand flushing specific cache lines is a rather special feature. If the code uses such obscure low-level features (much more than the "typical" application) such as the invariant that the size of a cache-line stays constant over the execution (which most applications really don't care about) one can at least expect from the developer to pass this information to the scheduler so that the scheduler can take that this invariant will indeed be satisfied.
- comex 10y ago
  Anything that uses a JIT necessarily uses this "obscure low-level feature", including any program implemented in one of many programming languages. Forcing such programs onto the heavy processor makes no sense; there just needs to be a way to properly clear the cache.
  - wolfgke 10y ago
    On https://news.ycombinator.com/item?id=12483698 https://news.ycombinator.com/item?id=12483698 I wrote a better idea how this might be implemented without causing this problem. Nevertheless even if we use my first, worse interface this should not be a problem: Spawn a thread that does the lowlevel stuff, sync it and after that run the JITted code.

lambda 10y ago

But then you would never be able to take advantage of the power/performance tradeoff that big.LITTLE gives you; of being able to migrate you to a big core when you need it, but save power by migrating you to the little core when you're not doing anything CPU intensive, and shutting off the power-hungry big core.

wolfgke 10y ago
> But then you would never be able to take advantage of the power/performance tradeoff that big.LITTLE gives you Rather: Only applications that don't depend on the invariant that the size of a cache line will stay constant over the execution time will take advantage of this. I can imagine ways how one could increase the advantage, but I don't want to go too much into technical details here.
- lambda 10y ago
  Why would programs want to "depend on the invariant that the size of a cache line will stay constant"? That's a minor detail that most applications don't care about. In this case, it probably mattered because when JITing code, you need to explicitly flush instruction cache after changing code. But that's not anything the application cares about; that's something the runtime cares about, and the runtime is going to want to support taking advantage of the speed/power tradeoff without making application developers have to go to the trouble of thinking about it. Also, from a user perspective, you don't want applications to be able to demand only the big, powerful cores. That will eat through your battery much faster, with likely very little benefit.
  - wolfgke 10y ago
    > Also, from a user perspective, you don't want applications to be able to demand only the big, powerful cores. This should rather be a setting in the rights system that a user can refuse the right to an application to be able to demand big, powerful cores. I can nevertheless imagine quite well scenarios where enabling a (specific) application to do so can be quite useful.

wolfgke 10y ago

Another idea how one could solve this problem by a clever kernel interface. This idea is probably a better than the approach in my post above, since this enables to migrate threads from big to little cores and vice versa except for critical regions. Add an interface to the kernel/scheduler void lock_cacheline_size(...) void unlock_cacheline_size(...) Calling lock_cacheline_size tells that from now on the thread must stay on a core with the same cache line size. Consider it as locking a mutex on the cache line size. After this you can run critical code that depends on the fact that the cache line size stays constant for the whole time. After you are finished with it, you call unlock_cacheline_size (as a kind of unlocking the mutex on the cache line size) - from now on the scheduler is free again to migrate the code again to cores with different cache line sizes.

cesarb 10y ago
A simpler and more robust approach would be to add a system call to flush a memory range. The kernel can then do all the locking, without the risk that migration to the other core gets indefinitely blocked by user space code.

Senji 10y ago

>An astute reader might realize that computing the cache line on every invocation is not enough for user space code: It can happen that a process gets scheduled on a different CPU while executing the __clear_cache function with a certain cache line size, where it might not be valid anymore. Doesn't ARM have an "Enter Critical Region" instruction?