4 ms·

If someone was to decide to develop a game for a minimum performance requirement of a Pentium III at 500mhz and some GPU, for some reason (which would be a comp

by phire 5y ago

If someone was to decide to develop a game for a minimum performance requirement of a Pentium III at 500mhz and some GPU, for some reason (which would be a completely arbitrary choice, but that's the hobby, so roll with it) then the only way they can possibly check that out meets that minimum requirement is to test on a machine with that configuration. Either a real machine, or a accurate emulator. It doesn't matter if there is other PC hardware configurations out there with different preformance. A minimum requirements just means "I tested on this machine, and it meets the minimums." Ideally you should underspec your minimum requirement test machine so that your target audience can be reasonably expected meet it. You can't substitute in virtualization. That has zero chance in hell of providing a realistic estimate of performance, even if you paired it with an accurate gpu emulator. Modern CPUs simply have very different performance characters, instructions that might have huge stalls on the P3 might be extremely cheap under virtualization. Caches are also widely different sizes. If you use a proper, but inaccurate emulator, you get different issues. Even if it was tuned to provide a decent estimate of cpu performance over average code (and they are typically tuned to overestimate cpu preformance, because people playing games would rather framedrops from real hardware are not emulated), it's just an average that doesn't take into account things like cache misses and branch misspredicts. If you were to write code with a lot of cache misses or branch misspredicts, your inaccurate emulator would massively overestimate it's preformance compared to a real cpu. The various issues just add up and it becomes impossible to profile and optimise the game you are developing unless you have an accurate emulator. Other solutions will all point to different parts of the code being hot. Also, remember this is within the era when you might be still developing a game with a software renderer, and if not you still have to do vertex transform and lighting on the cpu. Personally I'm not that interested in accurate emulation of PC, the issues get a lot worse when it comes to developing games for 5th, 6th and maybe even 7th gen consoles. That's were my true interest in accurate OoO emulation lies. But I can see why someone might want accurate PC emulation too.

AshamedCaptain 5y ago
> If someone was to decide to develop a game for a minimum performance requirement of a Pentium III at 500mhz and some GPU, for some reason (which would be a completely arbitrary choice, but that's the hobby, so roll with it) then the only way they can possibly check that out meets that minimum requirement is to test on a machine with that configuration. This just doesn't happen in modern PC development, save for heavyweights who can afford multiple identical hardware configurations (e.g. HPC clusters). I know I'm repeating myself, but the variety of configurations just makes this highly implausible. Sure, you can be some demoscene type of guy who decides to target specifically this configuration, but then you're literally targeting one processor out of hundreds, and per your own words, the fact that it works on the 500Mhz doesn't mean it will work with the same performance on the next generation or even on the 550Mhz variant. I guess this is obviously fine, but really stretching it. You'll quickly end up having something that only works on your machine, with the same starting disk image, etc. Even PCem doesn't fully simulate the x86 cache because there is no benefit to it, and that includes cores from eras which were much more sensitive to timing. Branch mispredictions? Forget about it. Most P3 software is going to run concurrently to some other software, anyway. I'm not saying that you don't need a cycle-accurate simulator to get real timings. I'm saying that with such a large divergence in configurations and environments, virtualization (or any other inaccurate emulator) is likely to provide a performance level that is quite accurately somewhere in the interval. Most specially since you will have actually calibrated it to that interval beforehand :) Now on consoles I can see the benefit. Consoles are lots of identical hardware, operating systems that tend to get out of the way, and the people who develop for them only test (for obvious reasons) on the console hardware itself or at most a developer edition which has the same hardware (for obvious reasons again). You can have a silent bug that depends on timing of a mispredicted branch or the relative speed between the bus accesses of two cores and _never_ notice it since your testing environment is exactly 1 device (such a bug would immediate flare on a PC on like the 2nd reboot). Whatever it is that you develop for any one such console, it is highly likely it will work on all the million other sold consoles. Consoles are practically designed to have reproduce-able environments. On the other hand you practically can emulate the entire x86 software catalog with emulators which _still_ have large differences in behavior at the actual instruction level compared to the hardware, so the instruction timing doesn't really seem important, and creating now some software that does depend on it seems .. complicated. As an anecdote, not long ago I was working on a x86 emulator, and to my horror I realized that the push/pop instructions were actually miscomputing the operand size on a rather common but not primary situation (long mode but with a 32-bit segment). The emulator was pushing the stack by double the amount it should, and pushing/popping the high dword of registers it shouldn't have clobbered. This was actually happening in some of the most critical operating system code out there (bootloaders, WoW, etc.) ... and yet the bug had been in the emulator for years and no one had been the wiser, booting 64-bit OSes just fine :)
- phire 5y ago
  I think the variety in configurations is completely irrelevant. If you aren't using at least one accurate configuration for your testing, there is a huge risk that you miss your performance target by a huge margin. Your 60fps game could end up running at 20fps on your target minimum hardware. Small performance inaccuracies can massively add up if you have a non-emulated cache-miss or branch miss-predict delay in your inner-most loop. I think you are massively overestimating how accuracy of timings that you can get though virtualisation or semi-accurate emulation. Yes. They are probably accurate enough for running any historic software from the era, as most code for the PC is well-behaved to not do the wrong thing when running too fast. It's just for the use-case of developing new software, as soon as you start optimising or profiling, you need accurate timings. And yes, we might be talking about weird demo-scene style projects along the lines of "I want to get the absolute best possible graphics out of the computer I had 25 years ago, no frames dropped, no wasted cpu cycles". I'm talking about the kind of project were someone is writing inner loops with intrinsics or in assembly. You might argue that such a project is a massive edge case that it's not worth catering too. And if you are writing an emulator, that's a 100% legitimate position to hold, emulators shouldn't have to cater for every possible usecase. My point is only "If you don't have a 100% accurate emulator, and there is some niche use case it can't emulate, then the hardware isn't fully preserved" and that it would be nice if an accurate emulator existed. > Now on consoles I can see the benefit. Consoles are lots of identical hardware, operating systems that tend to get out of the way, and the people who develop for them only test (for obvious reasons) on the console hardware itself or at most a developer edition which has the same hardware (for obvious reasons again). Yes, I've chased after bugs in console emulation (Dolphin Emulator) that were impossible to fix correctly without significantly more accurate emulation. Like the game which memset a staging buffer before data had finished DMAing out. The game was only saved on real hardware because after memsetting, it invalidated the cachelines and in typical situations, none of the memset cachelines had been evicted. Impossible to correctly fix without emulating the existence of an L2 cache. We eventually resorted to patching the game to fix the bug. Or games where video decoding stutters, because it has a hot inner loop that push the out-of-order CPU and has very few cache misses. It executes faster over the whole frame on real hardware than Dolphin's CPU timing model, which assumes a certain number of cache misses. The game must have tuned it's video codec to use as much CPU time as possible. We have speed running tricks that don't work in Dolphin, because the depend on lagging the game. And games that freak out when the GPU executes too fast, but when you adjust the timings for those other games freak out because the GPU is executing too slow. It's impossible to calculate accurate GPU timings without running much of vertex transform and a basic depth rasterizer. These are projects I'd love to work on at some point, accurate CPU and GPU timings for Dolphin, even if they don't run at full speed and bus contention is still ignored. I think might be possible to get within the correct order of magnitude (so 10-50% of realtime), which is workable for some usecases like TASes and testing bugs.

by phire 5y ago

AshamedCaptain 5y ago

> If someone was to decide to develop a game for a minimum performance requirement of a Pentium III at 500mhz and some GPU, for some reason (which would be a completely arbitrary choice, but that's the hobby, so roll with it) then the only way they can possibly check that out meets that minimum requirement is to test on a machine with that configuration. This just doesn't happen in modern PC development, save for heavyweights who can afford multiple identical hardware configurations (e.g. HPC clusters). I know I'm repeating myself, but the variety of configurations just makes this highly implausible. Sure, you can be some demoscene type of guy who decides to target specifically this configuration, but then you're literally targeting one processor out of hundreds, and per your own words, the fact that it works on the 500Mhz doesn't mean it will work with the same performance on the next generation or even on the 550Mhz variant. I guess this is obviously fine, but really stretching it. You'll quickly end up having something that only works on your machine, with the same starting disk image, etc. Even PCem doesn't fully simulate the x86 cache because there is no benefit to it, and that includes cores from eras which were much more sensitive to timing. Branch mispredictions? Forget about it. Most P3 software is going to run concurrently to some other software, anyway. I'm not saying that you don't need a cycle-accurate simulator to get real timings. I'm saying that with such a large divergence in configurations and environments, virtualization (or any other inaccurate emulator) is likely to provide a performance level that is quite accurately somewhere in the interval. Most specially since you will have actually calibrated it to that interval beforehand :) Now on consoles I can see the benefit. Consoles are lots of identical hardware, operating systems that tend to get out of the way, and the people who develop for them only test (for obvious reasons) on the console hardware itself or at most a developer edition which has the same hardware (for obvious reasons again). You can have a silent bug that depends on timing of a mispredicted branch or the relative speed between the bus accesses of two cores and _never_ notice it since your testing environment is exactly 1 device (such a bug would immediate flare on a PC on like the 2nd reboot). Whatever it is that you develop for any one such console, it is highly likely it will work on all the million other sold consoles. Consoles are practically designed to have reproduce-able environments. On the other hand you practically can emulate the entire x86 software catalog with emulators which _still_ have large differences in behavior at the actual instruction level compared to the hardware, so the instruction timing doesn't really seem important, and creating now some software that does depend on it seems .. complicated. As an anecdote, not long ago I was working on a x86 emulator, and to my horror I realized that the push/pop instructions were actually miscomputing the operand size on a rather common but not primary situation (long mode but with a 32-bit segment). The emulator was pushing the stack by double the amount it should, and pushing/popping the high dword of registers it shouldn't have clobbered. This was actually happening in some of the most critical operating system code out there (bootloaders, WoW, etc.) ... and yet the bug had been in the emulator for years and no one had been the wiser, booting 64-bit OSes just fine :)

phire 5y ago
I think the variety in configurations is completely irrelevant. If you aren't using at least one accurate configuration for your testing, there is a huge risk that you miss your performance target by a huge margin. Your 60fps game could end up running at 20fps on your target minimum hardware. Small performance inaccuracies can massively add up if you have a non-emulated cache-miss or branch miss-predict delay in your inner-most loop. I think you are massively overestimating how accuracy of timings that you can get though virtualisation or semi-accurate emulation. Yes. They are probably accurate enough for running any historic software from the era, as most code for the PC is well-behaved to not do the wrong thing when running too fast. It's just for the use-case of developing new software, as soon as you start optimising or profiling, you need accurate timings. And yes, we might be talking about weird demo-scene style projects along the lines of "I want to get the absolute best possible graphics out of the computer I had 25 years ago, no frames dropped, no wasted cpu cycles". I'm talking about the kind of project were someone is writing inner loops with intrinsics or in assembly. You might argue that such a project is a massive edge case that it's not worth catering too. And if you are writing an emulator, that's a 100% legitimate position to hold, emulators shouldn't have to cater for every possible usecase. My point is only "If you don't have a 100% accurate emulator, and there is some niche use case it can't emulate, then the hardware isn't fully preserved" and that it would be nice if an accurate emulator existed. > Now on consoles I can see the benefit. Consoles are lots of identical hardware, operating systems that tend to get out of the way, and the people who develop for them only test (for obvious reasons) on the console hardware itself or at most a developer edition which has the same hardware (for obvious reasons again). Yes, I've chased after bugs in console emulation (Dolphin Emulator) that were impossible to fix correctly without significantly more accurate emulation. Like the game which memset a staging buffer before data had finished DMAing out. The game was only saved on real hardware because after memsetting, it invalidated the cachelines and in typical situations, none of the memset cachelines had been evicted. Impossible to correctly fix without emulating the existence of an L2 cache. We eventually resorted to patching the game to fix the bug. Or games where video decoding stutters, because it has a hot inner loop that push the out-of-order CPU and has very few cache misses. It executes faster over the whole frame on real hardware than Dolphin's CPU timing model, which assumes a certain number of cache misses. The game must have tuned it's video codec to use as much CPU time as possible. We have speed running tricks that don't work in Dolphin, because the depend on lagging the game. And games that freak out when the GPU executes too fast, but when you adjust the timings for those other games freak out because the GPU is executing too slow. It's impossible to calculate accurate GPU timings without running much of vertex transform and a basic depth rasterizer. These are projects I'd love to work on at some point, accurate CPU and GPU timings for Dolphin, even if they don't run at full speed and bus contention is still ignored. I think might be possible to get within the correct order of magnitude (so 10-50% of realtime), which is workable for some usecases like TASes and testing bugs.