18 ms·
Jeez - how is this supposed to work with OS scheduling? The charm of big.little on arm is that the instruction sets between the big and the little cores are ide
by CountSessine 6y ago
Jeez - how is this supposed to work with OS scheduling? The charm of big.little on arm is that the instruction sets between the big and the little cores are identical. Now the OS has to pin processes based on support for different instructions? What a ridiculous nuisance. Intel really couldn’t discipline themselves just this once and actually implement the same instructions, even if in microcode, for both types of cores?
Are the little cores’ instructions at least a complete subset of the big cores’? Are we going to have some ridiculous situations where the little cores are completely pegged but the OS can’t migrate their processes off to the big core?
Or do kernel programmers need to start chasing Intel and trap/software implement every single future AVX8192AESNISSSE instruction Intel jams into future instruction sets to provide Xeon market differentiation?
- kllrnohj 6y ago> Now the OS has to pin processes based on support for different instructions? The only complication here would be if they have differing extensions like AVX512, but that's easily solved by the OS by just advertising the common baseline. Nothing about this looks difficult to support?
- CountSessine 6y agoby the OS by just advertising the common baseline. What does that mean? Advertise to whom? The process/process loader? Does it mean that I can’t compile with -mavx2 anymore? What if I do? The extensions are the whole problem.
- kllrnohj 6y ago> What does that mean? Advertise to whom? Runtime detection is the process querying what extensions are available, and then selectively using those. You adjust what the query returns to only return the common set. Runtime detection has been a pretty standard thing for well over a decade now - it's how we all manage to run the same compiled binaries over the years despite variability in SSE & AVX support. You don't download different versions of Chrome/Photoshop/Gimp/Premiere/Blender/Whatever compiled for different CPU micro-architectures, do you? You might if you run Gentoo I suppose, but that'd be about it. > Does it mean that I can’t compile with -mavx2 anymore? You already can't if you're shipping binaries to users unless you only support Skylake & newer? There's a lot of CPUs currently in use that don't support AVX2. So... you either already have this problem and you're familiar with it, or you're not doing this and it's moot.
- loeg 6y agoThe typical x86 extension query (cpuid) is an unpriviliged user-mode instruction — unless the application takes care to ask the OS, there's no real way for the OS to select the common denominator. Edit: Also: AVX2 is a lot older than Skylake. You're probably thinking of AVX512.
- kllrnohj 6y agoThe OS could still intercept CPUID - this is, after all, what VMs do. But it looks like Intel is doing this anyway, as Sunny Cove in this application has had its AVX-512 removed anyway: "One thing we can confirm in advance – the Sunny Cove does not appear to be AVX-512 enabled."
- saagarjha 6y agoWell then you need to set up a VMM for the OS…
- loeg 6y agoRunning every thread in a hypervisor just to trap cpuid sounds like extreme overkill. And probably some applications want the features only available on the big core, and masking those out doesn't solve that end of the problem.
- close04 6y ago> You adjust what the query returns to only return the common set. But this shoots the big core in the foot. You run at the lowest common denominator and the big core doesn't have the advantage of higher clocks. And this might just lead to a lot of software that only runs on the big core. Imagine you wanted to fly around between multiple points but if you want the option to ever switch to a bus then your plane will circle around each airport until it's as slow as the bus. Or you can opt for "plane only".
- kllrnohj 6y agoIt only shoots the big core in the foot for things that would make meaningful use of the extensions present on the big cores but not on the little ones. Which in a 7w application is what, exactly?
- KenoFischer 6y agoThe OS can just turn off any instruction extensions by setting the XCR0 register to whatever it wants. Unsupported instructions will then cause general protection faults (SIGSEGV on Linux). Whether that's what the OS wants to do is a totally separate question. It would be kinda funny if this thing lead to the ability of a process to request a certain XCR0 that the kernel would then serve, because I've been asking for that feature for unrelated reasons for quite some time and didn't get the warmest response to it on the mailing lists.
- agumonkey 6y agowait for OpenMultiCore that will standardize heterogeneous cpu sets
- masklinn 6y ago> The only complication here would be if they have differing extensions like AVX512 Which they do, Atom tops out at SSE4.2.
- chx 6y agoThey can easily disable the differing extensions in the BIG core.
- NullPrefix 6y agoWon't be BIG anymore if it's gimped, will it?
- kllrnohj 6y agoIt will. The performance different between an Intel Atom and a Core i3 has basically nothing to do with differing instruction set extensions. AVX2 support is not why a Core i3 runs circles around an Atom, particularly since the overwhelming majority of instructions executed are not AVX anyway.
- rbanffy 6y agoI'm pretty sure my computer does a lot of AVX while it matches virtual backgrounds to my video feed, or while it tries to figure out voice commands. All that can be done with SSE and probably with x87 instructions, but, still, software will try to pick the best option.
- fluffy87 6y agoBinaries need to target a minimum subset, but can use run time feature detection to query differences. If a binary does that, the OS can just not migrate that binary across cores.
- loeg 6y agoHow would the OS know? The only mechanism I can come up with is to detect illegal instruction traps on small cores, then flag the thread as big-core-only and re-start the execution at the bad instruction. That's not ideal but maybe workable. (If it traps on the big core, too, it's just a bad instruction and SIGILL is raised to userspace like usual.)
- deleted 6y ago[deleted]
- cesarb 6y agoThis is not ARM; on Intel, the CPUID instruction can also be used by normal unprivileged programs. Not all programs look only at /proc/cpuinfo or the ELF auxiliary vector.
- Klinky 6y agoThere were some issues with big.LITTLE, like this[1] issue with cache line size differences between Big and Little cores. 1. https://www.mono-project.com/news/2016/09/12/arm64-icache/ https://www.mono-project.com/news/2016/09/12/arm64-icache/
- CountSessine 6y agoYeah, I remember this. But I think it was just Samsung that made this blunder, no?
- Klinky 6y agoYes, you look to be right that the Samsung-designed Mongoose core in the Exynos SoC had 128-bytes cache line size, while ARM designed their generic Big cores to also use 64-bytes like their Little cores. Still though, seems like ARM did not provide guidance that these specs must match between Big & Little cores, and Samsung also didn't catch this either.
- murderfs 6y agoNVIDIA had a much funnier (infuriating) one, where Tegra X1 had caches between the big and little cores that weren't always coherent. Their solution was to just turn off the little cores while continuing to spew marketing bullshit about it being an 8-core chipset.
- qubex 6y agoI assume OS schedulers are going to have to include instruction-set support when seeking to maintain core affinity.
- ww520 6y agoDifferent instruction sets are not too bad. Copy from my comment elsewhere. > The OS can install an illegal-opcode exception handler. When a process is first run on the small CPU, the unsupported opcode will raise the exception. The exception handler can simply set the processor affinity of the process to the main CPU and put it to sleep. The OS will handle it like it normally would - putting the process in the run queue of the affined processor.
- saagarjha 6y agoThat sounds really slow.
- ahupp 6y agoIt only has to happen once.
- wtallis 6y agoOnce per process. I doubt operating systems will start modifying executables and libraries on-disk to tag them as using relevant instruction set extensions. And generally speaking, when an application first starts issuing SIMD instructions, that's probably not the a great time to be interrupting it even if it only needs to happen once.
- loeg 6y agoYou could imagine it as an ELF note of some kind. It's not a direction I've love to go in, and catching illegal exception traps seems fast enough that the more complicated design doesn't seem worthwhile.
- gpderetta 6y agoonce per process is perfectly fine. the issue is that once you have an active process using a feature only available on the larger cores, you can't shut off the larger cores to save power without paying a large latency to wake up that process.
- dmitrygr 6y agoIf this is going to be anything like how arm did it, intel most likely upgraded the atom CPUs to support all the instructions of the big core, probably just very slowly. For example, you will find that the lowly Cortex-A7 supports virtualization extensions. Why? Because it was paired with big cores that supported them. The only way to make big.LITTLE work and allow task migration between them is to have them support the same instruction set exactly. So, I suspect that that is why AVX512 was removed from the big core here, and I suspect the atom cores aren't exactly garden variety either. They probably support most other extensions of modern x86 CPUs, at a slow speed. For example, AVX2 can be microcoded by running multiple pieces of the data through the ALU, one at a time.
- ip26 6y agoI have to imagine the differing instruction sets means different extensions, not entirely different ISA. I'm not a programmer, but doesn't compiled code often offer multiple codepaths, checking flags at runtime? So if your code runs on the big core, it finds avx256 is supported, and uses the avx256 codepath. If it runs on the little core, it finds avx256 is not supported and it takes a legacy floating point codepath instead.
- mooman219 6y agoTons of asterisks here, but: You'll have to explicitly write the code to do that at runtime if APIs are provided to do so. When you build for a target, say x86, you're usually targeting the lowest common denominator that you're willing to support and the compiler might not output less supported instructions even if the CPU it's compiling on supports them.
- gpderetta 6y agoYes, most programs target the generic x86-64 ISA which means sse2 at most. Still some have accellerated paths that are enabled looking at feature bits. So a simple solution is for the os to mask out some of those bits for processes that intends to potentially migrate to the less featured cores, while potentially giving the option to let the user "pin" a process to the fast cores enabling the all features.
- Dylan16807 6y agoThe problem is that you want to be able to migrate processes between cores. It's not the end of the world if a number-crunching process gets stuck on the big core. But if a process detects an instruction that it's only using for convenience purposes, and that causes it to get stuck on the big core or stuck on the little cores, that's not good. In a different scenario, if it was set up so that a process starting on a little core never detected an important instruction, and was stuck in slow mode forever, that would also not be good. It depends a lot on which instructions differ and how the feature flags work, of course. They might have set it up in a way that's fine.
- StefanHamminga 6y agoI suppose they'll leverage 'function multi-version'[1], as Clear Linux already does[2] to enable extra instructions on the big cores. [1]: `target_clones` -> https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attribute... [2]: https://github.com/clearlinux/make-fmv-patch https://github.com/clearlinux/make-fmv-patch
- davrosthedalek 6y agoThat seems hard to do with preemptive scheduling. AFAIK, the decision is on function level, and done at startup. Even if you can decide on call-time, it's only function level, and the preemption can happen mid-function.
- loeg 6y agoThat only works if your code is never migrated between cores.
- Someone 6y ago“even if in microcode” I don’t think users would be happy if it were a lottery whether their code runs slow, using microcode instructions, or fast, on the better CPU, especially given that, I guess, the difference in instruction set is for vector instructions (because adding those is what’s makes a CPU big nowadays) So, performance oriented code will pin its performance-critical threads to the fast CPU. Now, the question is: what code won’t try to claim the faster CPU, given that benchmarks typically run programs while there is no contention from other programs?
- wmf 6y agoSo, performance oriented code will pin its performance-critical threads to the fast CPU. Now, the question is: what code won’t try to claim the faster CPU Do apps have CCX pinning logic for Ryzen? No? Why would they add it for an Intel processor that probably won't even be sold in volume.
- notriddle 6y agoI'm not sure how this is different from the existing big.LITTLE "lottery", between being scheduled on a CPU with a deep speculative pipeline and a high clock rate or a CPU with less. If the microcoded vector ops cause your code to run slow, then you will be consuming lots of CPU time, and should be migrated to the big core anyway, right?
- Symmetry 6y agoThe big core has SMT and AVX-512 turned off, I assume that they're just fusing off everything that the little cores don't have.
- gpderetta 6y agoI believe the os can turn off features in the big cores, disabling the corresponding bits in cpuid so that the two set of cores will advertise the same capabilities.
- gridlockd 6y agoI think the obvious solution is that AVX (or any other extension not in Tremont) will not be supported at all on these CPUs. These are 7W chips, after all.
- Legogris 6y agoBrings me back to coding for the ARM7TDMI on the Gameboy Advance - it supported both 16-bit THUMB[0] and the standard 32-bit instruction sets, and you would switch modes at runtime. Fun times. This does seem more annoying, though. [0]: https://en.wikipedia.org/wiki/ARM_architecture#Thumb https://en.wikipedia.org/wiki/ARM_architecture#Thumb
- waltpad 6y agoMaybe it wouldn't be possible to have several processes working in parallel, but instead a single process with 5 cores available for threads? Consider the PS3 arch for instance: it had 7 cores, 1 for management and the rest for computing. Could it be the way Intel envision its future lines of products?