6 ms·

Fans of a 2013 Japanese soccer game tracked down a bizarre floating point bug

246 points by politelemon 5y ago

gpvos 5y ago
"This makes sense, so of course nothing else does it this way." :)
- OldHand2018 5y ago
  Is the "broken" approach something from the Intel-designed x86 or the AMD-designed AMD64? And is aarch64 just following what the PC world did? Seems like it. Too bad PowerPC fell by the wayside.
  - adrian_b 5y ago
    Nothing was really broken, the problem was just that the 3 CPU architectures, POWER, Intel/AMD and 64-bit ARM include in their instruction sets slightly different FMA instructions, so porting a program between them requires care. The fused multiply-add instructions (FMA) have been used for the first time in 1990, in the first IBM POWER CPUs. Some years later FMA instructions have been included in many other CPUs from various vendors, the main exceptions being the Intel/AMD CPUs (where the first CPU with FMA was AMD Bulldozer in 2011, 21 years later than POWER) and the ARM CPUs, which also added FMA quite late, but before the introduction of the 64-bit variant. The problem is that both Intel and ARM have chosen to define FMA instructions that differ between themselves and also from POWER, in the position of the negation signs. With real numbers of unlimited precision, that would not have mattered, because the negation is commutative with the multiplication and distributive over addition. However with floating-point numbers these mathematical properties do not hold exactly, which is what the developers of the Dolphin emulator have discovered now. This is a general rule for any floating-point computation, that changing a formula using associativity/distributivity/commutativity is not guaranteed to produce the same results. Sometimes the differences are too small to matter, but in this case they made a game unplayable in the emulator, so they had to emulate the exact formula used in the original game, without trying to reduce the execution time by reducing the number of emulated instructions (emulating the original formula requires 2 instructions, while they initially optimized the code to use a single instruction but that computed a formula with the negation sign moved in another place).
    - esrauch 5y ago
      They mention precision bit then actually gave the example that if all inputs were exactly 0 that it gave differing results of +0 vs -0. Presumably the game checked the sign of the result and hit an error recovery codepath if it was negative. That's not really something that unlimited precision would avoid, but also not something that "real" arithmetic suffers from since real arithmetic doesn't have -0 and +0 as distinguishable concepts.
      - adrian_b 5y ago
        You are right that for that place in the game this probably exposed a bug in the original game. Normally testing the sign of a floating-point number that is the result of a multiplication or division should always be preceded by a test for zero, unless you intentionally want to take different paths for zeros with different signs.
  - MontagFTB 5y ago
    Based on the article it sounds like the main difference comes down to calculations resulting in +0 or -0. As long as the architecture is self-consistent, it wouldn’t be a problem. Once that floating point value is sent to another architecture, things could go bad. I suppose sending FP values over the wire and used for dsync detection doesn’t happen often?
Narishma 5y ago
I think changing game to video game would make the title less confusing.
- vmception 5y ago
  I think floating point would give the assumption that it has nothing to do with meatspace, but it would allow room for a fascinating surprise if it did! I vote for keeping it as is.
  - hnlmorg 5y ago
    If we're following HN guidelines correctly, the title should be 'Dolphin Progress Report: August 2021' -- what has been submitted has been editorialised.
  - minitoar 5y ago
    I was immediately trying to figure out if it somehow had to do with a stop watch being used in meatspace.
  - chromatin 5y ago
    Counterpoint: I assumed real life scoreboard.
  - Agentlien 5y ago
    I overlooked the URL. Seeing that or the original title I would have understood what this was. But now, my first guess was actually that this was about some much debated football match and that the floating point bug was present in some software used by the referees when reviewing replays.
    - tokai 5y ago
      Football referees didn't review replays in 2013.
      - samhw 5y ago
deleted 5y ago
[deleted]
userbinator 5y ago
The order of negation in those instructions may have been due to a very low-level hardware optimisation: it could save some time and circuit area to have the negation happen "inside" instead of "after", because the former is part of existing circuitry to do FP multiplication (ermember that FP is sign-magnitude) whereas the latter would require an additional stage on the output.
adwn 5y ago
It seems the author gets a little bit confused by the FMA instructions: > AArch64 nmadd equation is -(A * C) - B [...] Having an add instruction subtract is a curious decision from ARM [...] AArch64's nmsub equation is (A * C) - B, which isn't negated at all. Yet AArch64's msub is negated, for whatever reason, so we used that instead. We've learned not to question it. Re-arrange those equations a bit, and you get: msub: (b - ac) nmadd: -(b + ac) nmsub: -(b - ac) And suddenly, it all makes sense :-)
- deleted 5y ago
  [deleted]
- masklinn 5y ago
  The negated variant make sense compared to the original, it's really the definition of msub which is strange, I would expect msub to subtract the last operand from the multiplication, not the other way around. The AMD64 mnemonics actually encode this order e.g. VFMADD213PDy is 2 * 1 + 3 (ba + c) while VFMADD132PDy is 1 * 3 + 2 (ac+b). Though apparently the Dolphin folks disagree either way and interpret nmadd as (nm)add, rather than n(madd).
  - pm215 5y ago
    The Arm ARM says FNMADD is "Floating-point Negated fused Multiply-Add", so the position of the hyphens points strongly to "n(madd)" rather than "(nm)add".
    - a1369209993 5y ago
      > ARM says FNMADD is "Floating-point Negated fused Multiply-Add", so the position of the hyphens points Never mind the hyphens, what do they think a "Negated fused Multiply" is?
- pm215 5y ago
  In the spec, the pseudocode defines them as FMADD: addend + ( op1 * op2) FMSUB: addend + (-op1 * op2) FNMADD: -addend + (-op1 * op2) FNMSUB: -addend + ( op1 * op2) which is to say that in all three cases the result is the output of a fused multiply-add; the difference is just whether (a) the addend and (b) op1 are negated before feeding them in. IEEE 754-2008 only defines that you have to provide a fusedMultiplyAdd() operation, so it's up to the CPU architecture whether folding in the free negations is something they want to provide, and if so, where exactly the negation is done.
MontagFTB 5y ago
{paragraphs of subtle floating point math differences between architectures} “We've learned not to question it.” Truly a veteran developer crew! Congrats on the amazing fix.
rvnx 5y ago
These developers are really amazing :) I hope Nintendo will reuse their work one day on future consoles and offer them to work together.
- meibo 5y ago
  Nintendo has its own (buggy) Wii/GC emulator that has to be optimized for each game.
robarker 5y ago
That's a great initiative by the fans right here
Aissen 5y ago
Funny that they are finally starting to bypass a few of the protections of the closed-source version of the Homebrew channel. This highlights how hard it can be to accurately emulate some hardware (mis)features.
- svenpeter 5y ago
  To be fair, we deliberately used very obscure hardware "features" which we knew were not implemented by any emulators and probably not used by any games to build these protections :-) I'd have to dig up the old code but I'm fairly sure some of them rely on an operating system (Nintendo IOS, unrelated to both Cisco's IOS and Apple's iOS) running on the co-processor (nicknamed "Starlet"). Dolphin doesn't emulate that part at all because IOS exposes a high-level interface that can just be emulated instead. Works amazingly well for games, but will probably trip our protections.
  - 93po 5y ago
    Is it weird that you sound proud about this? Is there something noble about doing work that makes lives harder for hobbyists trying to preserve video games for the future, and has nearly zero impact on the actual sales of the game at release?
    - svenpeter 5y ago
      > Is it weird that you sound proud about this? Is there something noble about doing work that makes lives harder for hobbyists trying to preserve video games for the future, and has nearly zero impact on the actual sales of the game at release? What makes you think we did it for any of those reasons? And how does us abusing hardware bugs make video game preservation any harder? This was not for a commercial game, this was for an entry point to load your own software on a locked down video game console. And back then people were selling our (free!) software and we added those protections to make sure we could show a "you have been scammed if you paid for this" screen that couldn't be removed. Unfortunately that also meant breaking our loader from running in emulators, but that didn't matter at all: Those could just directly launch actual .elf files anyway and didn't need the detour through the homebrew channel. The code (minus those protections) for the Homebrew Channel is also available as open source these days.
Snetry 5y ago
Its always fun to read Dolphin Progress Reports
rawoke083600 5y ago
Honest question: Hows the experience with dolphin and the remotes ?? I assume you have to buy a wii-remote to have a decent experience ? The few times i tried to play games with keyboard or gamepad(logitech) it kinda was frustrating ! *Otherwise - good job on a great project !
- emodendroket 5y ago
  Depends on what you want to play (a regular controller is a fine substitute for the Wii Classic Control Pad) but the Wiimote works well over Bluetooth.
pronoiac 5y ago
The original title: Dolphin Progress Report: August 2021 I've emailed the mods to suggest a change.
wiradikusuma 5y ago
I'm curious how well can my PC handle emulated console games (esp. newer gens). Where can I get some ROMs? For research purpose, of course.
- LeoPanthera 5y ago
  This "magic" URL showing the most-favorited items on the Internet Archive is very interesting. https://archive.org/advancedsearch.php?q=*&fl%5B%5D=identifier&sort%5B%5D=num_favorites+desc&sort%5B%5D=&sort%5B%5D=&rows=50&page=1&callback=callback&save=yes&output=tables https://archive.org/advancedsearch.php?q=*&fl%5B%5D=identifi...

246 points by politelemon 5y ago

gpvos 5y ago

"This makes sense, so of course nothing else does it this way." :)

OldHand2018 5y ago
Is the "broken" approach something from the Intel-designed x86 or the AMD-designed AMD64? And is aarch64 just following what the PC world did? Seems like it. Too bad PowerPC fell by the wayside.
- adrian_b 5y ago
  Nothing was really broken, the problem was just that the 3 CPU architectures, POWER, Intel/AMD and 64-bit ARM include in their instruction sets slightly different FMA instructions, so porting a program between them requires care. The fused multiply-add instructions (FMA) have been used for the first time in 1990, in the first IBM POWER CPUs. Some years later FMA instructions have been included in many other CPUs from various vendors, the main exceptions being the Intel/AMD CPUs (where the first CPU with FMA was AMD Bulldozer in 2011, 21 years later than POWER) and the ARM CPUs, which also added FMA quite late, but before the introduction of the 64-bit variant. The problem is that both Intel and ARM have chosen to define FMA instructions that differ between themselves and also from POWER, in the position of the negation signs. With real numbers of unlimited precision, that would not have mattered, because the negation is commutative with the multiplication and distributive over addition. However with floating-point numbers these mathematical properties do not hold exactly, which is what the developers of the Dolphin emulator have discovered now. This is a general rule for any floating-point computation, that changing a formula using associativity/distributivity/commutativity is not guaranteed to produce the same results. Sometimes the differences are too small to matter, but in this case they made a game unplayable in the emulator, so they had to emulate the exact formula used in the original game, without trying to reduce the execution time by reducing the number of emulated instructions (emulating the original formula requires 2 instructions, while they initially optimized the code to use a single instruction but that computed a formula with the negation sign moved in another place).
  - esrauch 5y ago
    They mention precision bit then actually gave the example that if all inputs were exactly 0 that it gave differing results of +0 vs -0. Presumably the game checked the sign of the result and hit an error recovery codepath if it was negative. That's not really something that unlimited precision would avoid, but also not something that "real" arithmetic suffers from since real arithmetic doesn't have -0 and +0 as distinguishable concepts.
    - adrian_b 5y ago
      You are right that for that place in the game this probably exposed a bug in the original game. Normally testing the sign of a floating-point number that is the result of a multiplication or division should always be preceded by a test for zero, unless you intentionally want to take different paths for zeros with different signs.
- MontagFTB 5y ago
  Based on the article it sounds like the main difference comes down to calculations resulting in +0 or -0. As long as the architecture is self-consistent, it wouldn’t be a problem. Once that floating point value is sent to another architecture, things could go bad. I suppose sending FP values over the wire and used for dsync detection doesn’t happen often?

Narishma 5y ago

I think changing game to video game would make the title less confusing.

vmception 5y ago
I think floating point would give the assumption that it has nothing to do with meatspace, but it would allow room for a fascinating surprise if it did! I vote for keeping it as is.
- hnlmorg 5y ago
  If we're following HN guidelines correctly, the title should be 'Dolphin Progress Report: August 2021' -- what has been submitted has been editorialised.
- minitoar 5y ago
  I was immediately trying to figure out if it somehow had to do with a stop watch being used in meatspace.
- chromatin 5y ago
  Counterpoint: I assumed real life scoreboard.
- Agentlien 5y ago
  I overlooked the URL. Seeing that or the original title I would have understood what this was. But now, my first guess was actually that this was about some much debated football match and that the floating point bug was present in some software used by the referees when reviewing replays.
  - tokai 5y ago
    Football referees didn't review replays in 2013.
    - samhw 5y ago

deleted 5y ago

[deleted]

userbinator 5y ago

The order of negation in those instructions may have been due to a very low-level hardware optimisation: it could save some time and circuit area to have the negation happen "inside" instead of "after", because the former is part of existing circuitry to do FP multiplication (ermember that FP is sign-magnitude) whereas the latter would require an additional stage on the output.

adwn 5y ago

It seems the author gets a little bit confused by the FMA instructions: > AArch64 nmadd equation is -(A * C) - B [...] Having an add instruction subtract is a curious decision from ARM [...] AArch64's nmsub equation is (A * C) - B, which isn't negated at all. Yet AArch64's msub is negated, for whatever reason, so we used that instead. We've learned not to question it. Re-arrange those equations a bit, and you get: msub: (b - ac) nmadd: -(b + ac) nmsub: -(b - ac) And suddenly, it all makes sense :-)

deleted 5y ago
[deleted]
masklinn 5y ago
The negated variant make sense compared to the original, it's really the definition of msub which is strange, I would expect msub to subtract the last operand from the multiplication, not the other way around. The AMD64 mnemonics actually encode this order e.g. VFMADD213PDy is 2 * 1 + 3 (ba + c) while VFMADD132PDy is 1 * 3 + 2 (ac+b). Though apparently the Dolphin folks disagree either way and interpret nmadd as (nm)add, rather than n(madd).
- pm215 5y ago
  The Arm ARM says FNMADD is "Floating-point Negated fused Multiply-Add", so the position of the hyphens points strongly to "n(madd)" rather than "(nm)add".
  - a1369209993 5y ago
    > ARM says FNMADD is "Floating-point Negated fused Multiply-Add", so the position of the hyphens points Never mind the hyphens, what do they think a "Negated fused Multiply" is?
pm215 5y ago
In the spec, the pseudocode defines them as FMADD: addend + ( op1 * op2) FMSUB: addend + (-op1 * op2) FNMADD: -addend + (-op1 * op2) FNMSUB: -addend + ( op1 * op2) which is to say that in all three cases the result is the output of a fused multiply-add; the difference is just whether (a) the addend and (b) op1 are negated before feeding them in. IEEE 754-2008 only defines that you have to provide a fusedMultiplyAdd() operation, so it's up to the CPU architecture whether folding in the free negations is something they want to provide, and if so, where exactly the negation is done.

MontagFTB 5y ago

{paragraphs of subtle floating point math differences between architectures} “We've learned not to question it.” Truly a veteran developer crew! Congrats on the amazing fix.

rvnx 5y ago

These developers are really amazing :) I hope Nintendo will reuse their work one day on future consoles and offer them to work together.

meibo 5y ago
Nintendo has its own (buggy) Wii/GC emulator that has to be optimized for each game.

robarker 5y ago

That's a great initiative by the fans right here

Aissen 5y ago

Funny that they are finally starting to bypass a few of the protections of the closed-source version of the Homebrew channel. This highlights how hard it can be to accurately emulate some hardware (mis)features.

svenpeter 5y ago
To be fair, we deliberately used very obscure hardware "features" which we knew were not implemented by any emulators and probably not used by any games to build these protections :-) I'd have to dig up the old code but I'm fairly sure some of them rely on an operating system (Nintendo IOS, unrelated to both Cisco's IOS and Apple's iOS) running on the co-processor (nicknamed "Starlet"). Dolphin doesn't emulate that part at all because IOS exposes a high-level interface that can just be emulated instead. Works amazingly well for games, but will probably trip our protections.
- 93po 5y ago
  Is it weird that you sound proud about this? Is there something noble about doing work that makes lives harder for hobbyists trying to preserve video games for the future, and has nearly zero impact on the actual sales of the game at release?
  - svenpeter 5y ago
    > Is it weird that you sound proud about this? Is there something noble about doing work that makes lives harder for hobbyists trying to preserve video games for the future, and has nearly zero impact on the actual sales of the game at release? What makes you think we did it for any of those reasons? And how does us abusing hardware bugs make video game preservation any harder? This was not for a commercial game, this was for an entry point to load your own software on a locked down video game console. And back then people were selling our (free!) software and we added those protections to make sure we could show a "you have been scammed if you paid for this" screen that couldn't be removed. Unfortunately that also meant breaking our loader from running in emulators, but that didn't matter at all: Those could just directly launch actual .elf files anyway and didn't need the detour through the homebrew channel. The code (minus those protections) for the Homebrew Channel is also available as open source these days.

Snetry 5y ago

Its always fun to read Dolphin Progress Reports

rawoke083600 5y ago

Honest question: Hows the experience with dolphin and the remotes ?? I assume you have to buy a wii-remote to have a decent experience ? The few times i tried to play games with keyboard or gamepad(logitech) it kinda was frustrating ! *Otherwise - good job on a great project !

emodendroket 5y ago
Depends on what you want to play (a regular controller is a fine substitute for the Wii Classic Control Pad) but the Wiimote works well over Bluetooth.

pronoiac 5y ago

The original title: Dolphin Progress Report: August 2021 I've emailed the mods to suggest a change.

wiradikusuma 5y ago

I'm curious how well can my PC handle emulated console games (esp. newer gens). Where can I get some ROMs? For research purpose, of course.

LeoPanthera 5y ago
This "magic" URL showing the most-favorited items on the Internet Archive is very interesting. https://archive.org/advancedsearch.php?q=*&fl%5B%5D=identifier&sort%5B%5D=num_favorites+desc&sort%5B%5D=&sort%5B%5D=&rows=50&page=1&callback=callback&save=yes&output=tables https://archive.org/advancedsearch.php?q=*&fl%5B%5D=identifi...