18 ms·

How a 20 year old bug in GTA San Andreas surfaced in Windows 11 24H2

1371 points by yett 1y ago

adzm 1y ago
I'm more curious in what changed with the critical section locking/unlocking implementation in this version of Windows!
- mjevans 1y ago
  It looks like the utilized stack, or a stack protection area, increased.
  - asveikau 1y ago
    When I worked at Microsoft and I had downtime I would sometimes read the code for app compatibility shims out of pure curiosity. Win9x video games that made bad assumptions about the stack were a theme I saw. One of the differences between win9x and NT based windows is that kernel32 (later kernelbase) is a now user mode wrapper atop ntdll, whereas in the olden days kernel32 would trap directly into the kernel. This means that kernel32 uses more user mode stack space in NT. A badly behaving app that stored data to the left of the stack pointer and called into kernel32 might see its data structures clobbered in NT and not in 9x. So there were compatibility hacks that temporarily moved the stack pointer for certain apps.
    - hoten 1y ago
      What was the testing like for such bugs? Is it somehow automated, or is there a lengthy doc describing the manual testing steps, or are there no tests at all?
      - asveikau 1y ago
        I don't know. I wasn't on the team doing this. I was just looking at the source tree.
      - twoodfin 1y ago
        I interned with the AppCompat team shortly before the release of Windows XP, which was huge for them as it was the first Windows for consumers on the NT kernel. IIRC, they had a significant lab and tons of infrastructure for exercising and identifying compatibility issues in thousands of popular and less popular software packages. It all got distilled into a huge database of app fingerprints and corresponding compatibility shims to be applied at runtime.
    - tom_ 1y ago
      I wonder how many people think of the call stack as running left to right, most recent return first, rather than top to bottom, likewise? If you stare at enough hex dumps, it makes perfect sense.
db48x 1y ago
[flagged]
- deleted 1y ago
  [deleted]
- mschuster91 1y ago
  To u/db48x whose post got flagged and doesn't reappear despite me vouching for it as I think they have a point (at least for modern games): GTA San Andreas was released in 2004. Back then, YAML was in its infancy (2001) and JSON was only standardized informally in 2006, and XML wasn't something widely used outside of the Java world. On top of that, the hardware requirements (256MB of system RAM, and the PlayStation 2 only had 32MB) made it enough of a challenge to get the game running at all. Throwing in a heavyweight parsing library for either of these three languages was out of the question.
  - ceejayoz 1y ago
    Vouching seems to be time-lagged and require more than one.
  - Magma7404 1y ago
    The comment reappeared, and while you're right about using proper libraries to handle data, it doesn't excuse the "undefined behavior (uninitialized local variables)" that I still see all the time despite all the warning and error flags that can be fed to the compiler. Most of the time, the programmers who do this do not follow the simple rule that Stroustrup said which is to define or initialize a variable where you declare it (i.e. declare it before using it), and which would solve a lot of bugs in C++.
    - mschuster91 1y ago
      > it doesn't excuse the "undefined behavior (uninitialized local variables)" that I still see all the time despite all the warning and error flags that you can feed to the compiler. Yeah but we're talking about a 2004 game that was pretty rushed after 2002's Vice City (and I wouldn't be surprised if the bug in the ingestion code didn't exist there as well, just wasn't triggered due to the lack of planes except that darn RC Chopper and RC plane from that bombing run mission). Back then, the tooling to spot UB and code smell didn't even exist or, if at all, it was very rudimentary, or the warnings that did come up were just ignored because everything seemed to work.
maz1b 1y ago
I always enjoy reading deeply technical writeups like these. I only wonder how much more rare they may or may not get in the AI era.
- senda 1y ago
  i think the shift will be from craftmens to trademens in regards to general software engineers, but these are type of writes up stem of a artisan style all to its own.
  - eduardofcgo 1y ago
    We have been seeing this shift for a while, where "software engineers" graduate from 3 month bootcamps. Except now most likely they will not be earning 500k making crud apps.
    - sitzkrieg 1y ago
      and thats a good thing
    - throwaway2037 1y ago
      I call bullshit. What 3mo bootcamp grads were earning 500k writing CRUD apps? Zero.
  - nonethewiser 1y ago
    Compare python devs of today to fortran devs of the 60s. Something like that distance. Maybe more. But the trend isnt new.
  - throwaway2037 1y ago
    What about the incredible front end Devs that only know JS/CSS/HTML? They can still be true craftspeople in their art, be it cross-browser/platform issues or performance tweaking.
- Cthulhu_ 1y ago
  I don't think they will get more rare; there will always be a top % of engineers that do deep dives. I hope anyway. But AI won't replace them, nor did the past 50+ years of software development innovation. There's millions (tens of millions?) of higher programming language developers that don't know the difference between stack or heap besides maybe some theory they half remember from school but they don't care because they don't have to think about it for their day job.
bombcar 1y ago
This is the kind of thing I'd expect from Raymond Chen - which is extremely high praise! I'm glad they tracked it down even further to figure out exactly why.
- martinsnow 1y ago
  Raymond is a wizard. Read his blogs for many years and love his style and knowledge.
  - RcouF1uZ4gsC 1y ago
    Raymond knows everything. From microcode bugs on Alpha AXP to template meta programming to UI.
    - transcriptase 1y ago
      I wonder how many times a Deloitte, PwC, KPMG, Bain, EY, McKinsey, or BCG consultant naively tried putting him on a shortlist for being “impacted” over the years because he was in the Top X of a spreadsheet sorted on Y.
      - billforsternz 1y ago
        "Look this guy's job seems to be mainly writing blog posts. We could replace that with AI and get it to regularly pitch the new Visual Enshitify 2.0 product launch as a bonus. Win win win!"
    - gosub100 1y ago
      [flagged]
  - Discordian93 1y ago
    He's a total legend, yet apparently he's never met Bill Gates in person from what he said in an interview in the Dave's Garage YouTube channel a few years ago. You'd think that someone who's been that prominent for so long in the company would have been invited to a company dinner where he was present or something.
ge96 1y ago
I like the one where you shoot at the moon and it gets closer
- philippsh 1y ago
  [dead]
- Shekelphile 1y ago
  That isn't a bug. https://x.com/ObbeVermeij/status/1757572432863384046 https://x.com/ObbeVermeij/status/1757572432863384046
  - db48x 1y ago
    But it is a very likable story :)
carlos-menezes 1y ago
Much love to Silent, who’s been improving my favorite game for over... a decade now?
deleted 1y ago
[deleted]
xxpor 1y ago
For some reason this domain is blocked by work dns filtering?
- kstrauser 1y ago
  Let your IT department know that their denylist is broken.
  - xxpor 1y ago
    Yes, there doesn't seem to be anything even remotely suspicious here.
mjevans 1y ago
For anyone with access issues https://web.archive.org/web/20250423144746/https://cookieplmonster.github.io/2025/04/23/gta-san-andreas-win11-24h2-bug/ https://web.archive.org/web/20250423144746/https://cookieplm...
deepcurryshit 1y ago
[dead]
gitroom 1y ago
pretty wild how bugs can stick around that long - id never think something from 20 years ago would pop up just cause windows changed
nayuki 1y ago
> all these findings prove that the bug is NOT an issue with Windows 11 24H2, as things like the way the stack is used by internal WinAPI functions are not contractual and they may change at any time, with no prior notice. The real issue here is the game relying on undefined behavior (uninitialized local variables), and to be honest, I’m shocked that the game didn’t hit this bug on so many OS versions, although as I pointed out earlier, it was extremely close This sentence is the real takeaway point of the article. Undefined behavior is extremely insidious and can lull you into the belief that you were right, when you already made a mistake 1000 steps ago but it only got triggered now. I emphasized this point in my article from years ago (but after the game was released): > When a C or C++ program triggers undefined behavior, anything is allowed to happen in the program execution. And by anything, I really mean anything: The program can crash with an error message, it can silently corrupt data, it can morph into a colorful video game, or it can even give the right result. > If you’re lucky, the program triggering UB will show an appropriate error message and/or crash, making you immediately aware that something went wrong. If you’re unlucky, the program will quietly mangle data, and by the time you notice the problem (via effects such as crashes or incorrect output) the root cause has been buried in the past execution history. And if you’re very unlucky, the program will do exactly what you hoped it should do, until you change some unrelated code / compiler versions / compiler vendors / operating systems / hardware platforms – and then a new bug becomes visible, and you have no clue why seemingly correct code now fails to work properly. -- https://www.nayuki.io/page/undefined-behavior-in-c-and-cplusplus-programs https://www.nayuki.io/page/undefined-behavior-in-c-and-cplus... As I wrote in my article, this point really got hammered into me when a coworker showed me a patch that he made - which added a couple of innocuous, totally correct print statements to an existing C++ program - and that triggered a crash. But without his print statements, there was no crash. It turned out that there was a preexisting out-of-bounds array write, and the layout of the stack/heap somehow masked that problem before, and his unlucky prints unmasked the problem. Okay so then, how can we do better as developers today? 0) Read, understand, and memorize what actions in C or C++ are undefined behavior. Avoid them in your code at all costs. Also obey the preconditions of any API you use, whether in the standard library, operating system, etc. 1) Compile your application in Debug mode and compare its behavior to Release mode. If they differ by anything other than speed, then you have a serious problem on your hands. 2) Compile and run with sanitizers like -fsanitize=undefined,address to catch undefined behavior at runtime. 3) Use managed languages like Java, C#, Python, etc. where you basically don't have to worry about UB in normal day-to-day code. Or use very well-designed low-level languages like Rust that are safe by default and minimize your exposure to UB when you really need to do advanced things. Whereas C and C++ have been a bonanza of UB like we have never seen before in any other language.
rkunde 1y ago
I wonder if they fixed the vehicle definition file as well, or just the parser. The latter would be an incomplete fix.
- tonmoy 1y ago
  Given that those parameters are for wheels on a plane that doesn’t have wheels, I would say fixing the parser is the better fix
  - doublerabbit 1y ago
    It is, but who enjoys fixing parser issues...
- ddm999 1y ago
  SilentPatch (for GTAs, at least) specifically is a code-only mod, such that the single .asi file can be removed to uninstall it & all it's changes. A real update should fix both (note: I don't believe the later releases did, they also just added defaults to the parser) but for SilentPatch: a mod is not a real update, and being as simple as possible to remove & reducing conflicts with other mods is more important here than a fix that digs as deep as possible.
jandrese 1y ago
> Not ignore the compilation warnings – this code most likely threw a warning in the original code that was either ignored or disabled! What compiler error would you expect here? Maybe not checking the return value from scanf to make sure it matches the number of parameters? Otherwise this seems like a data file error that the compiler would have no clue about.
- burch45 1y ago
  Undefined behavior to access the uninitialized memory. A sanitizer would have flagged that.
  - jandrese 1y ago
    The compiler has no way of knowing that the memory would be undefined, not unless it somehow can verify the data file. The most I think it can do is flag the program for not checking the return value of scanf, but even that is unlikely to be true since the program probably was checking for end of file which is also in the return value. It was failing to check the number of matched parameters. This is the kind of error that is easy to miss given the semantics of scanf.
    - andrewmcwatters 1y ago
      Uninitialized variables are a really common case.
      - gmueckl 1y ago
        The pointer to the uninitialized variable is passed to scanf, which writes a value there unless it encounters an error. The compiler cannot understand this contract from the scanf declaration alone.
    - nayuki 1y ago
      > The compiler has no way of knowing that the memory would be undefined Yes it would. -fsanitize=address does a bunch of instrumentation - it allocates shadow memory to keep track of what main memory is defined, and it checks every read and write address against the shadow memory. It is a combination of compile-time instrumentation and run-time checking. And yes, it is expensive, so it should be used for debugging and not the final release. https://clang.llvm.org/docs/AddressSanitizer.html https://clang.llvm.org/docs/AddressSanitizer.html , https://learn.microsoft.com/en-us/cpp/sanitizers/asan?view=msvc-170 https://learn.microsoft.com/en-us/cpp/sanitizers/asan?view=m...
josephcsible 1y ago
tl;dr of the explanation: the Skimmer vehicle is missing a wheel scale definition, so its wheel scale gets read from uninitialized memory. On previous versions of Windows, this happened to be the wheel scale of the previously-loaded vehicle, so things happened to work fine. Starting on Windows 11 24H2, LeaveCriticalSection (which gets called between loading vehicles) uses more stack space than before, so it now overwrites that memory with a gigantic value, resulting in the Skimmer spawning so high up that it may as well not exist at all.
pmarreck 1y ago
My takeaway, speaking as someone who leans towards functional programming and immutability, is "this is yet another example of a mutability problem that could never happen in a functional context" (so, for example, this bug would have never been created by Rust unless it was deeply misused)
- grishka 1y ago
  This is more of a problem of the C/C++ standard that it allows uninitialized variables but doesn't give them defined values, considering it "undefined behavior" to read from an uninitialized variable. Java, for example, doesn't have this particular problem because it does specify default values for variables.
  - mabster 1y ago
    But it's this and many other features of C/C++ that make it faster than Java. C/C++ developers really don't want to "pay" for something for safety. Though, I really like the _mm_undefined_ps() intrinsics for SSE that make it clear that you're purposefully not initialising a variable. Something like that for ints and floats would be pretty sweet.
    - Dylan16807 1y ago
      Statically proving the variables get initialized wouldn't change the performance except by making sure you check the return value of sscanf, or turning refusal to check into a couple register wipes. Either way, that's a negligible increase to a hefty function call. It wouldn't require default initializing variables in all circumstances. When I think of the "no runtime cost" mentality of C/C++ I don't think that normally extends to ignoring errors in I/O functions.
    - twic 1y ago
      And yet, there is a good chance that C++ will start doing exactly this [1]. Because [2]: > The performance impact is negligible (less that 0.5% regression) to slightly positive (that is, some code gets faster by up to 1%). The code size impact is negligible (smaller than 0.5%). Compile-time regressions are negligible. Were overheads to matter for particular coding patterns, compilers would be able to obviate most of them. > The only significant performance/code regressions are when code has very large automatic storage duration objects. We provide an attribute to opt-out of zero-initialization of objects of automatic storage duration. We then expect that programmer can audit their code for this attribute, and ensure that the unsafe subset of C++ is used in a safe manner. > This change was not possible 30 years ago because optimizations simply were not as good as they are today, and the costs were too high. The costs are now negligible. [1] https://github.com/cplusplus/papers/issues/1401 https://github.com/cplusplus/papers/issues/1401 [2] https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2723r1.html https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p27...
csours 1y ago
Once this category of error is raised to your attention, you start to notice it more and more. A little piece of technology made sense in the original context, but then it got moved to a different context without realizing that move broke the contract. Specifically in this case a flying boat became an airplane. --- I recently worked a bug that feels very similar: A linux cups printer would not print to the selected tray, instead it always requested manual feed. Ok. Try a bunch of command line options, same issue. Ok. Make the selection directly in the PPD (postscript printer definition) file. Same issue. Ok! Decompile the PXL file. Wrong tray is set in pxl file... why? Check Debug2 log level for cups - Wrong MediaPosition is being sent to ghostscript (which compiles the printer options into the print job) by a cups filter... why? Cups filter is translating the MediaPosition from the PPD file... because the philosophy of cups is to do what the user intended. The intention inferred from MediaPosition in the PPD file (postscript printer definition) is that the MediaPosition corresponds to the PWG (Printer Working Group) MediaPosition, NOT the vendor MediaPosition (or local equivalent - in this case MediaSource). AHA!! My PPD file had been copied from a previous generation of server, from a time when that cups filter did NOT translate the MediaPosition, so the VENDOR MediaSource numbers were used. Historically, this makes sense. The vendor tray number is set in the vendor ppd file because cups didn't know how to translate that. Fast forward to a new execution context, and cups filters have gotten better at translating user intention, now it's translating a number that doesn't need to be translated, and silently selecting the wrong tray. TLDR; There is no such thing as a printer command, only printer suggestions.
- twic 1y ago
  Infamously, this is also why Ariane 501 blew up. (a component being reused in a new context where a contract is broken, not bad CUPS drivers)
smcameron 1y ago
Surprised to see the return value of sscanf being ignored, that seems like a pretty rookie mistake, and this bug would never have made it out of the original programmer's system if they had bothered to check it.
- canucker2016 1y ago
  Yes, it would have made it out of the original programmer's system for that initial commit. FTA: I have a likely explanation for why Rockstar made this specific mistake in the data to begin with – in Vice City, Skimmer was defined as a boat, and therefore did not have those values defined by design! When in San Andreas they changed Skimmer’s vehicle type to a plane, someone forgot to add those now-required extra parameters. Since this game seldom verifies the completeness of its data, this mistake simply slipped under the radar. So the original code (or at least a working code + data version) in GTA Vice City had no visible problems, at least with the Skimmer object, since the vehicles.ide file had the correct number of values for the Skimmer boat object. Someone changed the Skimmer object from a boat to a plane for GTA San Andreas, BUT they DID NOT update the object to have the REQUIRED wheel values for a plane object. Now the GTA code is expecting more values than it gets. The vehicles.ide wasn't validated for correctness after the Skimmer object change to plane. Maybe there are more gotchas in that file... At least users can fix the problem with a text editor instead of waiting and hoping that RockStar would fix the problem and release an update.
amenghra 1y ago
IMHO, if something isn’t part of the contract, it should be randomized. Eg if iteration order of maps isn’t guaranteed in your language, then your language should go out of its way to randomize it. Otherwise, you end up with brittle code: code that works fine until it doesn’t.
- roseway4 1y ago
  iirc, Go intentionally randomizes map ordering for just this reason.
  - withinboredom 1y ago
    Yep, and then you get crash reports you can’t reproduce.
    - cristaloleg 1y ago
      Same can be said about pointer addresses (random for each run). But ASLR exists for a specific reason.
- willcipriano 1y ago
  Then you are wasting runtime clock cycles randomizing lists.
  - nayuki 1y ago
    Any sane language would design a list iterator to follow the order of the list. No, the difference is when you're iterating over orderless hash-based sets or maps/dictionaries. Many languages choose to leave the iteration order undefined. I think Python did that up to a point, but afterward they defined dictionaries (but not sets) to be iterated over in the order that keys were added. Also, some languages intentionally randomize the order per program run, to avoid things like users intentionally stuffing hash tables with colliding keys.
    - 1y ago
qingcharles 1y ago
20 years and still nobody has realized what the back of this sign is about. IYKYK. https://static.wikia.nocookie.net/gta-myths/images/f/f1/Egg_8.jpg/revision/latest?cb=20130526152144 https://static.wikia.nocookie.net/gta-myths/images/f/f1/Egg_...
- spopejoy 1y ago
  fixed link (the above did not work for me) https://static.wikia.nocookie.net/gta-myths/images/f/f1/Egg_8.jpg https://static.wikia.nocookie.net/gta-myths/images/f/f1/Egg_...
- rideontime 1y ago
  And if I don't know?
nubinetwork 1y ago
Apparently the planes are buggy in all sorts of ways, although I don't remember them being this bad... https://youtu.be/hrJ0eVY5ACw https://youtu.be/hrJ0eVY5ACw
olvy0 1y ago
Just like $dayjob.
gigatexal 1y ago
Use a debugger folks. A 10x dev cited this story to me about the ills of not using one.
- ajross 1y ago
  Tools like valgrind/asan/msan would have flagged this instantly too. Just a unit test of that vehicle loader would have seen it. Really this is more a story about poor development practice than it is an interesting bug.
  - Thaxll 1y ago
    As if tools in early 2000's were any good...
    - ajross 1y ago
      Valgrind was released in 2002 to immediate celebration. It was available and surely known to the team. All they needed to do was write a unit test that loaded and instantiated those vehicle files and run it with "valgrind" in front of the command line.
      - gmueckl 1y ago
        I don't know whether Valgrind ever gained support for any of the GTA target platforms. It wasn't available on Windows for a very, very long time.
        zerd 1y ago
        Still isn't available for windows from what I gather.
      - db48x 1y ago
        It was available and widely celebrated, but there was no guarantee that these Windows and console developers had heard of it yet or that they could have used it if they had.
rossant 1y ago
Am I the only one to be annoyed by this...? while (this->m_fBladeAngle > 6.2831855) { this->m_fBladeAngle = this->m_fBladeAngle - 6.2831855; } Like, "let's just write a while loop that could turn into an infinite loop coz I'm too lazy to do a division"
- hoten 1y ago
  for real. The author clearly never heard of fmod
  - zerd 1y ago
    fmod takes in the order of 30+ cycles, probably more in year 2003 CPUs, vs 1 for cmp, 1 for sub, 1 for jmp.
    - hoten 1y ago
      Sure the lower bound is nicer here. But when the tradeoff includes an unlimited upper bound it's not a very attractive option. I guess the most robust code handling both performance and unexpected input would be one iteration of this (leveraging the assumption that angles are either always within the bounds, or had one frame of going out of bounds by a small amount); followed by a fmod if that assumption is just totally off.
- nemothekid 1y ago
  I want to assume that the GTA developers did this hack because it was faster than floating point division on the Playstation 2 or something. But knowing they were able to they were able to blow up loading GTA5 by 5 minutes by just parsing json with sscanf, I don't have much hope.
  - badsectoracula 1y ago
    IIRC the whole parsing performance issue was because the original code was written for the SP campaign of GTA5 that only had a handful of objects to parse data for. That was barely a blip in terms of performance impact and AFAIK was written years before GTAOnline was made (where it became an issue - and even then only became an issue much after GTAOnline was first made). Writing some simple code that works with the data you expect to have without bothering with optimizations is fine, if anything it is one of the actual cases of "premature optimization": even with profiling no real time is spent on that code, your data wont make it spend any time and you should avoid wild guesses since chances are you'll be wrong (even if in this case it could be a correct guess, it'd be like a broken clock guessing the time is always 13:37). The actual issue with that code was that, after they reused it for GTAOnline and started becoming a performance issue after some time as they added more objects, nobody thought to try and see what is wrong.
anal_reactor 1y ago
I love it how many bugs go from "why doesn't this work?!" to "how on Earth did this work previously?!"
userbinator 1y ago
On Windows 11 24H2, more stack space was modified by a new implementation of Critical Sections. IMHO this shows the downfall of Microsoft. Why did they do that? Critical sections have been there for many decades and should be basically bug-free by now. My best guess is someone thought they'd "improve" things and rewrote it, then made some microbenchmark that maybe showed the dubious improvement. The other comment here mentions Raymond Chen, who wrote this article about why backwards-compatibility is very important (and arguably what got Microsoft into the position it's in today): https://devblogs.microsoft.com/oldnewthing/20031224-00/?p=41363 https://devblogs.microsoft.com/oldnewthing/20031224-00/?p=41... and also this memorable case: https://news.ycombinator.com/item?id=2281932 https://news.ycombinator.com/item?id=2281932
- kittoes 1y ago
  Really? Someone depending on UB in their software represents the downfall of Microsoft?! What a hot take...
  - devnullbrain 1y ago
    User has working software. User updates operating system. User has broken software. That's a problem for the party trying to sell operating system updates.
    - voidspark 1y ago
      The software was fundamentally broken before the OS update. It was working by pure random chance with undefined behaviour. It’s a C++ issue, not an OS issue. The same code compiled for another OS would have different random results.
      - Uvix 1y ago
        While this is technically correct that doesn’t get the customer to put the blame where it belongs.
      - smallstepforman 1y ago
cadamsdotcom 1y ago
It has always been too easy to read & write beyond the stack. This should fail, plain and simple. Mitigations exist - ASLR, NX pages, stack-smashing protection etc. but nothing comprehensively stops reads of stale data beyond the stack. Thought experiment for a moment. What if the hardware ensures the unused part of a stack region cannot be read or written. There are many ways to skin this cat, here’s one based around tracking each stack’s start address A, size S, and current depth D 1. Add an instruction to inform the CPU there is a stack at address A of size S. Its depth D is initially 0. 2. Add a jump instruction which reserves N bytes on the stack at address A, growing depth D to (D+N). Maybe this can be its own “reserve” instruction so as not to need a new jump instruction. 3. Give existing return instructions stack awareness. If returning to an address inside a stack, un-reserve the bytes reserved by the most recent jump, making the new depth (D-N). 4. Fail reads or writes to the stack region beyond its current depth. In other words fail all reads and writes between A+S-D and A+S. 5. The arithmetic is reversed on architectures whose stacks grow downwards. Downsides I can see: It cements one calling convention. The CPU memory manager will need a lot of state per stack, of which there are many per process: address A, size S, current depth D, plus a reservation stack - ie. sizes of each frame’s stack memory. That’s a lot of bookkeeping! It’s far from zero cost. The limits of how much bookkeeping the CPU can do impose limits on how deep a stack can go and how many stacks are supported - so when there are too many stacks or one goes too deep, either the CPU needs to signal failure or engage a fallback mode and revert to behaving as CPUs do today. And of course fallback puts things back to the start. It’d therefore only mitigate situations in which an attacker cannot control the depth of the stack / a bug always happens inside the max depth the CPU can bookkeep for. That said, stacks are ubiquitous! Hardware stack awareness opens up all kinds of new mitigations. Why isn’t this a common idea? Has it been tried?
- LegionMammal978 1y ago
  This bug wasn't caused by a read beyond the current bounds of the stack, but a stale value from a prior call to the same function at the exact same location on the stack. Buffer-overflow protections like you describe wouldn't help here.
  - 1y ago
kristofferR 1y ago
I hope someone can figure out the Red Dead Redemption 2 bug where random animals and characters disappear silently if you have too many texture mods installed. I spent hours looking for a badger.
DuckOnFire 1y ago
How does a bug from 20 years ago even still work today?
dusted 1y ago
> all these findings prove that the bug is NOT an issue with Windows 11 24H2, as things like the way the stack is used by internal WinAPI functions are not contractual and they may change at any time, with no prior notice. This reminds me of an excellent article I read a while back, the gist of it was that, given sufficient success, there's no such thing as a private API.
- kidfiji 1y ago
  I know there’s an XKCD comic about this
  - ChrisRR 1y ago
    Just bring back spacebar heating
  - VMG 1y ago
    https://xkcd.com/1172/ https://xkcd.com/1172/
- someperson 1y ago
  Could you please find this article and link it here. I'm curious about the arguments.
  - dusted 1y ago
    I really can't remember if it was this one, I'll have to check if I saved the link somewhere at work.. but maybe it was Hyrum's law https://www.hyrumslaw.com/ https://www.hyrumslaw.com/
robohoe 1y ago
Maybe it’s the real reason why CJ couldn’t follow the dang train.
vortico 1y ago
>Scientists claim to have discovered a ‘new color’ no one has seen before. LOL!
claiir 1y ago
Okay, but why did `LeaveCriticalSection` change? Compiler changes, new features, refactoring, etc? That’s the most interesting part (and absent)!
smallstepforman 1y ago
The core problem is some compilers initialising memory to zero in Debug mode, masking behaviour of unitialised data, since in most cases zero is a legit value. In Release mode, this zeroing doesn’t happen. Devs need to be aware that the following C++ initisliser exists which zeros data structures for you: MyStruct s = { };
rs186 1y ago
Knowing C/C++, I more or less guessed what's happening (uninitialized variable) early in the blog post. It blows my mind that the languages allow you to leave variables uninitialized which has caused countless bugs (including production bugs that I have seen first hand), and you often need to rely on additional compiler flags or static analysis tools/valgrind etc to catch them. Even though newer languages often use a different solution (default zero value or must initialize a variable before use), people still go back to C/C++ all the time.
techlatest_net 1y ago
[dead]

1371 points by yett 1y ago

adzm 1y ago

I'm more curious in what changed with the critical section locking/unlocking implementation in this version of Windows!

mjevans 1y ago
It looks like the utilized stack, or a stack protection area, increased.
- asveikau 1y ago
  When I worked at Microsoft and I had downtime I would sometimes read the code for app compatibility shims out of pure curiosity. Win9x video games that made bad assumptions about the stack were a theme I saw. One of the differences between win9x and NT based windows is that kernel32 (later kernelbase) is a now user mode wrapper atop ntdll, whereas in the olden days kernel32 would trap directly into the kernel. This means that kernel32 uses more user mode stack space in NT. A badly behaving app that stored data to the left of the stack pointer and called into kernel32 might see its data structures clobbered in NT and not in 9x. So there were compatibility hacks that temporarily moved the stack pointer for certain apps.
  - hoten 1y ago
    What was the testing like for such bugs? Is it somehow automated, or is there a lengthy doc describing the manual testing steps, or are there no tests at all?
    - asveikau 1y ago
      I don't know. I wasn't on the team doing this. I was just looking at the source tree.
    - twoodfin 1y ago
      I interned with the AppCompat team shortly before the release of Windows XP, which was huge for them as it was the first Windows for consumers on the NT kernel. IIRC, they had a significant lab and tons of infrastructure for exercising and identifying compatibility issues in thousands of popular and less popular software packages. It all got distilled into a huge database of app fingerprints and corresponding compatibility shims to be applied at runtime.
  - tom_ 1y ago
    I wonder how many people think of the call stack as running left to right, most recent return first, rather than top to bottom, likewise? If you stare at enough hex dumps, it makes perfect sense.

db48x 1y ago

[flagged]

deleted 1y ago
[deleted]
mschuster91 1y ago
To u/db48x whose post got flagged and doesn't reappear despite me vouching for it as I think they have a point (at least for modern games): GTA San Andreas was released in 2004. Back then, YAML was in its infancy (2001) and JSON was only standardized informally in 2006, and XML wasn't something widely used outside of the Java world. On top of that, the hardware requirements (256MB of system RAM, and the PlayStation 2 only had 32MB) made it enough of a challenge to get the game running at all. Throwing in a heavyweight parsing library for either of these three languages was out of the question.
- ceejayoz 1y ago
  Vouching seems to be time-lagged and require more than one.
- Magma7404 1y ago
  The comment reappeared, and while you're right about using proper libraries to handle data, it doesn't excuse the "undefined behavior (uninitialized local variables)" that I still see all the time despite all the warning and error flags that can be fed to the compiler. Most of the time, the programmers who do this do not follow the simple rule that Stroustrup said which is to define or initialize a variable where you declare it (i.e. declare it before using it), and which would solve a lot of bugs in C++.
  - mschuster91 1y ago
    > it doesn't excuse the "undefined behavior (uninitialized local variables)" that I still see all the time despite all the warning and error flags that you can feed to the compiler. Yeah but we're talking about a 2004 game that was pretty rushed after 2002's Vice City (and I wouldn't be surprised if the bug in the ingestion code didn't exist there as well, just wasn't triggered due to the lack of planes except that darn RC Chopper and RC plane from that bombing run mission). Back then, the tooling to spot UB and code smell didn't even exist or, if at all, it was very rudimentary, or the warnings that did come up were just ignored because everything seemed to work.

maz1b 1y ago

I always enjoy reading deeply technical writeups like these. I only wonder how much more rare they may or may not get in the AI era.

senda 1y ago
i think the shift will be from craftmens to trademens in regards to general software engineers, but these are type of writes up stem of a artisan style all to its own.
- eduardofcgo 1y ago
  We have been seeing this shift for a while, where "software engineers" graduate from 3 month bootcamps. Except now most likely they will not be earning 500k making crud apps.
  - sitzkrieg 1y ago
    and thats a good thing
  - throwaway2037 1y ago
    I call bullshit. What 3mo bootcamp grads were earning 500k writing CRUD apps? Zero.
- nonethewiser 1y ago
  Compare python devs of today to fortran devs of the 60s. Something like that distance. Maybe more. But the trend isnt new.
- throwaway2037 1y ago
  What about the incredible front end Devs that only know JS/CSS/HTML? They can still be true craftspeople in their art, be it cross-browser/platform issues or performance tweaking.
Cthulhu_ 1y ago
I don't think they will get more rare; there will always be a top % of engineers that do deep dives. I hope anyway. But AI won't replace them, nor did the past 50+ years of software development innovation. There's millions (tens of millions?) of higher programming language developers that don't know the difference between stack or heap besides maybe some theory they half remember from school but they don't care because they don't have to think about it for their day job.

bombcar 1y ago

This is the kind of thing I'd expect from Raymond Chen - which is extremely high praise! I'm glad they tracked it down even further to figure out exactly why.

martinsnow 1y ago
Raymond is a wizard. Read his blogs for many years and love his style and knowledge.
- RcouF1uZ4gsC 1y ago
  Raymond knows everything. From microcode bugs on Alpha AXP to template meta programming to UI.
  - transcriptase 1y ago
    I wonder how many times a Deloitte, PwC, KPMG, Bain, EY, McKinsey, or BCG consultant naively tried putting him on a shortlist for being “impacted” over the years because he was in the Top X of a spreadsheet sorted on Y.
    - billforsternz 1y ago
      "Look this guy's job seems to be mainly writing blog posts. We could replace that with AI and get it to regularly pitch the new Visual Enshitify 2.0 product launch as a bonus. Win win win!"
  - gosub100 1y ago
    [flagged]
- Discordian93 1y ago
  He's a total legend, yet apparently he's never met Bill Gates in person from what he said in an interview in the Dave's Garage YouTube channel a few years ago. You'd think that someone who's been that prominent for so long in the company would have been invited to a company dinner where he was present or something.

ge96 1y ago

I like the one where you shoot at the moon and it gets closer

philippsh 1y ago
[dead]
Shekelphile 1y ago
That isn't a bug. https://x.com/ObbeVermeij/status/1757572432863384046 https://x.com/ObbeVermeij/status/1757572432863384046
- db48x 1y ago
  But it is a very likable story :)

carlos-menezes 1y ago

Much love to Silent, who’s been improving my favorite game for over... a decade now?

deleted 1y ago

[deleted]

xxpor 1y ago

For some reason this domain is blocked by work dns filtering?

kstrauser 1y ago
Let your IT department know that their denylist is broken.
- xxpor 1y ago
  Yes, there doesn't seem to be anything even remotely suspicious here.

mjevans 1y ago

For anyone with access issues https://web.archive.org/web/20250423144746/https://cookieplmonster.github.io/2025/04/23/gta-san-andreas-win11-24h2-bug/ https://web.archive.org/web/20250423144746/https://cookieplm...

deepcurryshit 1y ago

[dead]

gitroom 1y ago

pretty wild how bugs can stick around that long - id never think something from 20 years ago would pop up just cause windows changed

nayuki 1y ago

> all these findings prove that the bug is NOT an issue with Windows 11 24H2, as things like the way the stack is used by internal WinAPI functions are not contractual and they may change at any time, with no prior notice. The real issue here is the game relying on undefined behavior (uninitialized local variables), and to be honest, I’m shocked that the game didn’t hit this bug on so many OS versions, although as I pointed out earlier, it was extremely close This sentence is the real takeaway point of the article. Undefined behavior is extremely insidious and can lull you into the belief that you were right, when you already made a mistake 1000 steps ago but it only got triggered now. I emphasized this point in my article from years ago (but after the game was released): > When a C or C++ program triggers undefined behavior, anything is allowed to happen in the program execution. And by anything, I really mean anything: The program can crash with an error message, it can silently corrupt data, it can morph into a colorful video game, or it can even give the right result. > If you’re lucky, the program triggering UB will show an appropriate error message and/or crash, making you immediately aware that something went wrong. If you’re unlucky, the program will quietly mangle data, and by the time you notice the problem (via effects such as crashes or incorrect output) the root cause has been buried in the past execution history. And if you’re very unlucky, the program will do exactly what you hoped it should do, until you change some unrelated code / compiler versions / compiler vendors / operating systems / hardware platforms – and then a new bug becomes visible, and you have no clue why seemingly correct code now fails to work properly. -- https://www.nayuki.io/page/undefined-behavior-in-c-and-cplusplus-programs https://www.nayuki.io/page/undefined-behavior-in-c-and-cplus... As I wrote in my article, this point really got hammered into me when a coworker showed me a patch that he made - which added a couple of innocuous, totally correct print statements to an existing C++ program - and that triggered a crash. But without his print statements, there was no crash. It turned out that there was a preexisting out-of-bounds array write, and the layout of the stack/heap somehow masked that problem before, and his unlucky prints unmasked the problem. Okay so then, how can we do better as developers today? 0) Read, understand, and memorize what actions in C or C++ are undefined behavior. Avoid them in your code at all costs. Also obey the preconditions of any API you use, whether in the standard library, operating system, etc. 1) Compile your application in Debug mode and compare its behavior to Release mode. If they differ by anything other than speed, then you have a serious problem on your hands. 2) Compile and run with sanitizers like -fsanitize=undefined,address to catch undefined behavior at runtime. 3) Use managed languages like Java, C#, Python, etc. where you basically don't have to worry about UB in normal day-to-day code. Or use very well-designed low-level languages like Rust that are safe by default and minimize your exposure to UB when you really need to do advanced things. Whereas C and C++ have been a bonanza of UB like we have never seen before in any other language.

rkunde 1y ago

I wonder if they fixed the vehicle definition file as well, or just the parser. The latter would be an incomplete fix.

tonmoy 1y ago
Given that those parameters are for wheels on a plane that doesn’t have wheels, I would say fixing the parser is the better fix
- doublerabbit 1y ago
  It is, but who enjoys fixing parser issues...
ddm999 1y ago
SilentPatch (for GTAs, at least) specifically is a code-only mod, such that the single .asi file can be removed to uninstall it & all it's changes. A real update should fix both (note: I don't believe the later releases did, they also just added defaults to the parser) but for SilentPatch: a mod is not a real update, and being as simple as possible to remove & reducing conflicts with other mods is more important here than a fix that digs as deep as possible.

jandrese 1y ago

> Not ignore the compilation warnings – this code most likely threw a warning in the original code that was either ignored or disabled! What compiler error would you expect here? Maybe not checking the return value from scanf to make sure it matches the number of parameters? Otherwise this seems like a data file error that the compiler would have no clue about.

burch45 1y ago
Undefined behavior to access the uninitialized memory. A sanitizer would have flagged that.
- jandrese 1y ago
  The compiler has no way of knowing that the memory would be undefined, not unless it somehow can verify the data file. The most I think it can do is flag the program for not checking the return value of scanf, but even that is unlikely to be true since the program probably was checking for end of file which is also in the return value. It was failing to check the number of matched parameters. This is the kind of error that is easy to miss given the semantics of scanf.
  - andrewmcwatters 1y ago
    Uninitialized variables are a really common case.
    - gmueckl 1y ago
      The pointer to the uninitialized variable is passed to scanf, which writes a value there unless it encounters an error. The compiler cannot understand this contract from the scanf declaration alone.
  - nayuki 1y ago
    > The compiler has no way of knowing that the memory would be undefined Yes it would. -fsanitize=address does a bunch of instrumentation - it allocates shadow memory to keep track of what main memory is defined, and it checks every read and write address against the shadow memory. It is a combination of compile-time instrumentation and run-time checking. And yes, it is expensive, so it should be used for debugging and not the final release. https://clang.llvm.org/docs/AddressSanitizer.html https://clang.llvm.org/docs/AddressSanitizer.html , https://learn.microsoft.com/en-us/cpp/sanitizers/asan?view=msvc-170 https://learn.microsoft.com/en-us/cpp/sanitizers/asan?view=m...

josephcsible 1y ago

tl;dr of the explanation: the Skimmer vehicle is missing a wheel scale definition, so its wheel scale gets read from uninitialized memory. On previous versions of Windows, this happened to be the wheel scale of the previously-loaded vehicle, so things happened to work fine. Starting on Windows 11 24H2, LeaveCriticalSection (which gets called between loading vehicles) uses more stack space than before, so it now overwrites that memory with a gigantic value, resulting in the Skimmer spawning so high up that it may as well not exist at all.

pmarreck 1y ago

My takeaway, speaking as someone who leans towards functional programming and immutability, is "this is yet another example of a mutability problem that could never happen in a functional context" (so, for example, this bug would have never been created by Rust unless it was deeply misused)

grishka 1y ago
This is more of a problem of the C/C++ standard that it allows uninitialized variables but doesn't give them defined values, considering it "undefined behavior" to read from an uninitialized variable. Java, for example, doesn't have this particular problem because it does specify default values for variables.
- mabster 1y ago
  But it's this and many other features of C/C++ that make it faster than Java. C/C++ developers really don't want to "pay" for something for safety. Though, I really like the _mm_undefined_ps() intrinsics for SSE that make it clear that you're purposefully not initialising a variable. Something like that for ints and floats would be pretty sweet.
  - Dylan16807 1y ago
    Statically proving the variables get initialized wouldn't change the performance except by making sure you check the return value of sscanf, or turning refusal to check into a couple register wipes. Either way, that's a negligible increase to a hefty function call. It wouldn't require default initializing variables in all circumstances. When I think of the "no runtime cost" mentality of C/C++ I don't think that normally extends to ignoring errors in I/O functions.
  - twic 1y ago
    And yet, there is a good chance that C++ will start doing exactly this [1]. Because [2]: > The performance impact is negligible (less that 0.5% regression) to slightly positive (that is, some code gets faster by up to 1%). The code size impact is negligible (smaller than 0.5%). Compile-time regressions are negligible. Were overheads to matter for particular coding patterns, compilers would be able to obviate most of them. > The only significant performance/code regressions are when code has very large automatic storage duration objects. We provide an attribute to opt-out of zero-initialization of objects of automatic storage duration. We then expect that programmer can audit their code for this attribute, and ensure that the unsafe subset of C++ is used in a safe manner. > This change was not possible 30 years ago because optimizations simply were not as good as they are today, and the costs were too high. The costs are now negligible. [1] https://github.com/cplusplus/papers/issues/1401 https://github.com/cplusplus/papers/issues/1401 [2] https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2723r1.html https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p27...

csours 1y ago

Once this category of error is raised to your attention, you start to notice it more and more. A little piece of technology made sense in the original context, but then it got moved to a different context without realizing that move broke the contract. Specifically in this case a flying boat became an airplane. --- I recently worked a bug that feels very similar: A linux cups printer would not print to the selected tray, instead it always requested manual feed. Ok. Try a bunch of command line options, same issue. Ok. Make the selection directly in the PPD (postscript printer definition) file. Same issue. Ok! Decompile the PXL file. Wrong tray is set in pxl file... why? Check Debug2 log level for cups - Wrong MediaPosition is being sent to ghostscript (which compiles the printer options into the print job) by a cups filter... why? Cups filter is translating the MediaPosition from the PPD file... because the philosophy of cups is to do what the user intended. The intention inferred from MediaPosition in the PPD file (postscript printer definition) is that the MediaPosition corresponds to the PWG (Printer Working Group) MediaPosition, NOT the vendor MediaPosition (or local equivalent - in this case MediaSource). AHA!! My PPD file had been copied from a previous generation of server, from a time when that cups filter did NOT translate the MediaPosition, so the VENDOR MediaSource numbers were used. Historically, this makes sense. The vendor tray number is set in the vendor ppd file because cups didn't know how to translate that. Fast forward to a new execution context, and cups filters have gotten better at translating user intention, now it's translating a number that doesn't need to be translated, and silently selecting the wrong tray. TLDR; There is no such thing as a printer command, only printer suggestions.

twic 1y ago
Infamously, this is also why Ariane 501 blew up. (a component being reused in a new context where a contract is broken, not bad CUPS drivers)

smcameron 1y ago

Surprised to see the return value of sscanf being ignored, that seems like a pretty rookie mistake, and this bug would never have made it out of the original programmer's system if they had bothered to check it.

canucker2016 1y ago
Yes, it would have made it out of the original programmer's system for that initial commit. FTA: I have a likely explanation for why Rockstar made this specific mistake in the data to begin with – in Vice City, Skimmer was defined as a boat, and therefore did not have those values defined by design! When in San Andreas they changed Skimmer’s vehicle type to a plane, someone forgot to add those now-required extra parameters. Since this game seldom verifies the completeness of its data, this mistake simply slipped under the radar. So the original code (or at least a working code + data version) in GTA Vice City had no visible problems, at least with the Skimmer object, since the vehicles.ide file had the correct number of values for the Skimmer boat object. Someone changed the Skimmer object from a boat to a plane for GTA San Andreas, BUT they DID NOT update the object to have the REQUIRED wheel values for a plane object. Now the GTA code is expecting more values than it gets. The vehicles.ide wasn't validated for correctness after the Skimmer object change to plane. Maybe there are more gotchas in that file... At least users can fix the problem with a text editor instead of waiting and hoping that RockStar would fix the problem and release an update.

amenghra 1y ago

IMHO, if something isn’t part of the contract, it should be randomized. Eg if iteration order of maps isn’t guaranteed in your language, then your language should go out of its way to randomize it. Otherwise, you end up with brittle code: code that works fine until it doesn’t.

roseway4 1y ago
iirc, Go intentionally randomizes map ordering for just this reason.
- withinboredom 1y ago
  Yep, and then you get crash reports you can’t reproduce.
  - cristaloleg 1y ago
    Same can be said about pointer addresses (random for each run). But ASLR exists for a specific reason.
willcipriano 1y ago
Then you are wasting runtime clock cycles randomizing lists.
- nayuki 1y ago
  Any sane language would design a list iterator to follow the order of the list. No, the difference is when you're iterating over orderless hash-based sets or maps/dictionaries. Many languages choose to leave the iteration order undefined. I think Python did that up to a point, but afterward they defined dictionaries (but not sets) to be iterated over in the order that keys were added. Also, some languages intentionally randomize the order per program run, to avoid things like users intentionally stuffing hash tables with colliding keys.
  - 1y ago

qingcharles 1y ago

20 years and still nobody has realized what the back of this sign is about. IYKYK. https://static.wikia.nocookie.net/gta-myths/images/f/f1/Egg_8.jpg/revision/latest?cb=20130526152144 https://static.wikia.nocookie.net/gta-myths/images/f/f1/Egg_...

spopejoy 1y ago
fixed link (the above did not work for me) https://static.wikia.nocookie.net/gta-myths/images/f/f1/Egg_8.jpg https://static.wikia.nocookie.net/gta-myths/images/f/f1/Egg_...
rideontime 1y ago
And if I don't know?

nubinetwork 1y ago

Apparently the planes are buggy in all sorts of ways, although I don't remember them being this bad... https://youtu.be/hrJ0eVY5ACw https://youtu.be/hrJ0eVY5ACw

olvy0 1y ago

Just like $dayjob.

gigatexal 1y ago

Use a debugger folks. A 10x dev cited this story to me about the ills of not using one.

ajross 1y ago
Tools like valgrind/asan/msan would have flagged this instantly too. Just a unit test of that vehicle loader would have seen it. Really this is more a story about poor development practice than it is an interesting bug.
- Thaxll 1y ago
  As if tools in early 2000's were any good...
  - ajross 1y ago
    Valgrind was released in 2002 to immediate celebration. It was available and surely known to the team. All they needed to do was write a unit test that loaded and instantiated those vehicle files and run it with "valgrind" in front of the command line.
    - gmueckl 1y ago
      I don't know whether Valgrind ever gained support for any of the GTA target platforms. It wasn't available on Windows for a very, very long time.
      - zerd 1y ago
        Still isn't available for windows from what I gather.
    - db48x 1y ago
      It was available and widely celebrated, but there was no guarantee that these Windows and console developers had heard of it yet or that they could have used it if they had.

rossant 1y ago

Am I the only one to be annoyed by this...? while (this->m_fBladeAngle > 6.2831855) { this->m_fBladeAngle = this->m_fBladeAngle - 6.2831855; } Like, "let's just write a while loop that could turn into an infinite loop coz I'm too lazy to do a division"

hoten 1y ago
for real. The author clearly never heard of fmod
- zerd 1y ago
  fmod takes in the order of 30+ cycles, probably more in year 2003 CPUs, vs 1 for cmp, 1 for sub, 1 for jmp.
  - hoten 1y ago
    Sure the lower bound is nicer here. But when the tradeoff includes an unlimited upper bound it's not a very attractive option. I guess the most robust code handling both performance and unexpected input would be one iteration of this (leveraging the assumption that angles are either always within the bounds, or had one frame of going out of bounds by a small amount); followed by a fmod if that assumption is just totally off.
nemothekid 1y ago
I want to assume that the GTA developers did this hack because it was faster than floating point division on the Playstation 2 or something. But knowing they were able to they were able to blow up loading GTA5 by 5 minutes by just parsing json with sscanf, I don't have much hope.
- badsectoracula 1y ago
  IIRC the whole parsing performance issue was because the original code was written for the SP campaign of GTA5 that only had a handful of objects to parse data for. That was barely a blip in terms of performance impact and AFAIK was written years before GTAOnline was made (where it became an issue - and even then only became an issue much after GTAOnline was first made). Writing some simple code that works with the data you expect to have without bothering with optimizations is fine, if anything it is one of the actual cases of "premature optimization": even with profiling no real time is spent on that code, your data wont make it spend any time and you should avoid wild guesses since chances are you'll be wrong (even if in this case it could be a correct guess, it'd be like a broken clock guessing the time is always 13:37). The actual issue with that code was that, after they reused it for GTAOnline and started becoming a performance issue after some time as they added more objects, nobody thought to try and see what is wrong.

anal_reactor 1y ago

I love it how many bugs go from "why doesn't this work?!" to "how on Earth did this work previously?!"

userbinator 1y ago

On Windows 11 24H2, more stack space was modified by a new implementation of Critical Sections. IMHO this shows the downfall of Microsoft. Why did they do that? Critical sections have been there for many decades and should be basically bug-free by now. My best guess is someone thought they'd "improve" things and rewrote it, then made some microbenchmark that maybe showed the dubious improvement. The other comment here mentions Raymond Chen, who wrote this article about why backwards-compatibility is very important (and arguably what got Microsoft into the position it's in today): https://devblogs.microsoft.com/oldnewthing/20031224-00/?p=41363 https://devblogs.microsoft.com/oldnewthing/20031224-00/?p=41... and also this memorable case: https://news.ycombinator.com/item?id=2281932 https://news.ycombinator.com/item?id=2281932

kittoes 1y ago
Really? Someone depending on UB in their software represents the downfall of Microsoft?! What a hot take...
- devnullbrain 1y ago
  User has working software. User updates operating system. User has broken software. That's a problem for the party trying to sell operating system updates.
  - voidspark 1y ago
    The software was fundamentally broken before the OS update. It was working by pure random chance with undefined behaviour. It’s a C++ issue, not an OS issue. The same code compiled for another OS would have different random results.
    - Uvix 1y ago
      While this is technically correct that doesn’t get the customer to put the blame where it belongs.
    - smallstepforman 1y ago

cadamsdotcom 1y ago

It has always been too easy to read & write beyond the stack. This should fail, plain and simple. Mitigations exist - ASLR, NX pages, stack-smashing protection etc. but nothing comprehensively stops reads of stale data beyond the stack. Thought experiment for a moment. What if the hardware ensures the unused part of a stack region cannot be read or written. There are many ways to skin this cat, here’s one based around tracking each stack’s start address A, size S, and current depth D 1. Add an instruction to inform the CPU there is a stack at address A of size S. Its depth D is initially 0. 2. Add a jump instruction which reserves N bytes on the stack at address A, growing depth D to (D+N). Maybe this can be its own “reserve” instruction so as not to need a new jump instruction. 3. Give existing return instructions stack awareness. If returning to an address inside a stack, un-reserve the bytes reserved by the most recent jump, making the new depth (D-N). 4. Fail reads or writes to the stack region beyond its current depth. In other words fail all reads and writes between A+S-D and A+S. 5. The arithmetic is reversed on architectures whose stacks grow downwards. Downsides I can see: It cements one calling convention. The CPU memory manager will need a lot of state per stack, of which there are many per process: address A, size S, current depth D, plus a reservation stack - ie. sizes of each frame’s stack memory. That’s a lot of bookkeeping! It’s far from zero cost. The limits of how much bookkeeping the CPU can do impose limits on how deep a stack can go and how many stacks are supported - so when there are too many stacks or one goes too deep, either the CPU needs to signal failure or engage a fallback mode and revert to behaving as CPUs do today. And of course fallback puts things back to the start. It’d therefore only mitigate situations in which an attacker cannot control the depth of the stack / a bug always happens inside the max depth the CPU can bookkeep for. That said, stacks are ubiquitous! Hardware stack awareness opens up all kinds of new mitigations. Why isn’t this a common idea? Has it been tried?

LegionMammal978 1y ago
This bug wasn't caused by a read beyond the current bounds of the stack, but a stale value from a prior call to the same function at the exact same location on the stack. Buffer-overflow protections like you describe wouldn't help here.
- 1y ago

kristofferR 1y ago

I hope someone can figure out the Red Dead Redemption 2 bug where random animals and characters disappear silently if you have too many texture mods installed. I spent hours looking for a badger.

DuckOnFire 1y ago

How does a bug from 20 years ago even still work today?

dusted 1y ago

> all these findings prove that the bug is NOT an issue with Windows 11 24H2, as things like the way the stack is used by internal WinAPI functions are not contractual and they may change at any time, with no prior notice. This reminds me of an excellent article I read a while back, the gist of it was that, given sufficient success, there's no such thing as a private API.

kidfiji 1y ago
I know there’s an XKCD comic about this
- ChrisRR 1y ago
  Just bring back spacebar heating
- VMG 1y ago
  https://xkcd.com/1172/ https://xkcd.com/1172/
someperson 1y ago
Could you please find this article and link it here. I'm curious about the arguments.
- dusted 1y ago
  I really can't remember if it was this one, I'll have to check if I saved the link somewhere at work.. but maybe it was Hyrum's law https://www.hyrumslaw.com/ https://www.hyrumslaw.com/

robohoe 1y ago

Maybe it’s the real reason why CJ couldn’t follow the dang train.

vortico 1y ago

>Scientists claim to have discovered a ‘new color’ no one has seen before. LOL!

claiir 1y ago

Okay, but why did `LeaveCriticalSection` change? Compiler changes, new features, refactoring, etc? That’s the most interesting part (and absent)!

smallstepforman 1y ago

The core problem is some compilers initialising memory to zero in Debug mode, masking behaviour of unitialised data, since in most cases zero is a legit value. In Release mode, this zeroing doesn’t happen. Devs need to be aware that the following C++ initisliser exists which zeros data structures for you: MyStruct s = { };

rs186 1y ago

Knowing C/C++, I more or less guessed what's happening (uninitialized variable) early in the blog post. It blows my mind that the languages allow you to leave variables uninitialized which has caused countless bugs (including production bugs that I have seen first hand), and you often need to rely on additional compiler flags or static analysis tools/valgrind etc to catch them. Even though newer languages often use a different solution (default zero value or must initialize a variable before use), people still go back to C/C++ all the time.

techlatest_net 1y ago

[dead]