33 ms·

Morrowind rebooted the original Xbox

240 points by pmarin 5y ago

chews 5y ago
that was a great post, typical fun exploitation to rid yourself of the problem of memory management. There was a way to force garbage collection in Flash as well that was used by the clever pros at the time.
varsketiz 5y ago
Wow, that's a cool hack. I wonder if they did more stuff like that to go around Xbox limitations. Reminds me of Crash Bandicoot hacks for ps1 :))
- ToFundorNot 5y ago
  For those who haven't read/watched it: https://arstechnica.com/gaming/2020/02/war-stories-how-crash-bandicoot-hacked-the-original-playstation/ https://arstechnica.com/gaming/2020/02/war-stories-how-crash... It is indeed a story of creativity in design.
  - DicIfTEx 5y ago
    Complete with a cautionary tale about the dangers of being too creative: https://www.gamasutra.com/blogs/DaveBaggett/20131031/203788/My_Hardest_Bug_Ever.php https://www.gamasutra.com/blogs/DaveBaggett/20131031/203788/...
    - failuser 5y ago
      Bumping the timer spreed is not the creative part, creative part was zeroing on the issue. For software guys used to reliable hardware and no prior knowledge how to debug hardware bugs this is hell.
    - bogwog 5y ago
      Wow. That story is going to give me nightmares.
    - Firehawke 5y ago
      It's notable that the controller and memory card slot are on the same bus-- in fact, the system isn't talking to the memory card directly! It has to send requests and data through the controller, back to the system, and then to the card port. This is because the original specification would have had a memory card slot in the controller but it was too expensive. They also had considered using floppies for storage, but they were too unreliable (and my experiences with 1.44MB 3.5s definitely agrees that consumer-grade cheap disks sucked hard)
hinkley 5y ago
Presumably then if the game was glitching on you and you did a save/load then the problem would magically fix itself. (have you tried turning it off and back on again?)
deleted 5y ago
[deleted]
flatiron 5y ago
His video on Monday was great about this. I like that they checked first if they were running low on memory and then just rebooted the darn thing if it was.
geuis 5y ago
Video from Modern Vintage Gamer walking through the process of how this worked along with a demonstration https://www.youtube.com/watch?v=x0TKwPnHc-M https://www.youtube.com/watch?v=x0TKwPnHc-M
- simlevesque 5y ago
  This guy is the best video game tech youtuber out here. He knows his stuff deeply.
  - Nextgrid 5y ago
    He was part of the original Xbox homebrew scene.
kashanjunaid 5y ago
great hack! thanks for share
Impossible 5y ago
This was common in console games and console SDKs had special APIs to support this. It's a decent solution to memory leak and fragmentation issues
thomasfromcdnjs 5y ago
Great to see Morrowind on HN, just sharing my personal story; I was in highschool living in a trailer park with my mother. My ma ended up getting a pay out from the government for a work related injury. She bought me a PC and we set it up in the caravan and I was able to run Morrowind. It was a great escape from the conditions at the time and I kept learning programming when I wasn't hunting Daedra's.
- gnulinux 5y ago
  Morrowind looks fun but every time I try playing, it feels like doing Sunday chores. 90% of my game experience thus far has been walking. What am I doing wrong?
  - taurath 5y ago
    It’s one of those games that it feels best to not really know what type of game it is at first going in. It felt like the ultimate “lots and lots under the surface” game when it came out, but now so many features have been coopted what seemed really clever before now feels like a waste of time since you are more intimately familiar with the mechanics. It’s a “what’s over there?” game.
  - codezero 5y ago
    For me the novelty of Morrowind doesn't quite last, you could just play Skyrim imho. Morrowind had a world that was so expansive at the time it was awe inspiring. I think that was the main thing that made it so wild. It lacked a lot of quality of life features that make it hard to play today, and I remember routinely taking a break then forgetting which five month long plot I had been playing before going on vacation for a week. Just not the kind of game you can pick up in a weekend.
    - hombre_fatal 5y ago
      Most quality of life features just make games a bit boring though, like WoW-style waypoint markers where you don’t even read the quests or talk to people anymore because the game tells you where to go. Skyrim is especially bad imo because it all takes place in the same biome with no sense of “wtf is going on here?” My advice to people here: Get OpenMW and start playing Morrowind with a notepad open that you can alt-tab into for notes. I never beat the game until last year. If the pace is too slow, fine, just open up the console and find the command to double or triple your walking speed. It’s a game best enjoyed slow otherwise, as you unfold the mysteries of the world. I think it definitely holds up to 2021.
corysama 5y ago
Yep. The function would reboot into a new executable without disrupting what was on the screen. I don’t recall which one, but this was used by an FPS to switch between single-player vs. multi-player exes that were developed by two completely different studios.
- maccard 5y ago
  Call of duty did it first I think but it's not uncommon to do these days...
- adamparsons 5y ago
  PC versions of CoD do this, and switching between mode on a 360 was always kinda slow so I could definitely believe its doing something similar
yroc92 5y ago
It blew my mind to hear that the Xbox ran on 64mb of RAM.
- gfrangakis 5y ago
  That blew my mind as well. I remember it looking so good. Xbox - 64mb Xbox 360 - 512mb Xbox One - 8 GB (later 12 GB) Xbox Series X - 16 GB This last one also surprises me, was expecting at least 32GB
  - deleted 5y ago
    [deleted]
  - ahartmetz 5y ago
    At least in the PS5, the ultra-fast SSD and some technology around it are supposed to drastically reduce the need to keep things in RAM in case they are needed quickly later. There's a talk about that and other PS5 innovations by Mark Czerny. It's a very high quality talk.
  - mikepurvis 5y ago
    I think the theory with the current gen is that devs get a lot more out a mere doubling because of all the improvements to SSD streaming. Like, it's 16gb, but it's 16gb that only has to cover the next few seconds of gameplay, not the next 30 seconds.
  - Dylan16807 5y ago
    > Xbox Series X - 16 GB > This last one also surprises me, was expecting at least 32GB Well the GPU is comparable to 8GB models, so I can see why 16GB would work fine. You don't need that much space for game state. Though looking into the Series X more, they sure set up the ram weirdly. There are 4 1GB memory chips and 6 2GB chips. This gives it 10GB of very fast memory, and 6GB of somewhat fast memory. The operating system reserves 2.5GB of the somewhat fast memory, and games get 13.5GB The 'natural' setup would be 20GB, and they didn't even go that high, so they must be plenty confident that 32GB is well beyond what they need.
kabdib 5y ago
I was surprised to see that (at least on the 360) there was no way for a title to terminate itself; the game process ran until it was killed by something else (e.g., the user launching another title, or shutting the console down). Another surprise was the utter lack of a date-and-time clock chip. Without a network (and I'm guessing, without actually connecting to XBox Live), the 360 had no idea what time it was. That saved a bunch of money on hardware, and removed headaches involving coin cells (such as shipping policies).
- rodgerd 5y ago
  It's still the case on a Series X. There are separate pools for applications and games, as well, and the Series X is able to keep state for multiple running games at once, which is nice. The only downside is that it seems to cause a certain amount of pain with applications sometimes not working when there's a contest over sending Dolby to a receiver. On the time thing: turns out that PlayStation 3 and 4 will stop letting you run games if you replace the battery and it can't re-sync time with the PSN.
  - kevingadd 5y ago
    To be more precise, the "if your battery dies you can't play games" thing only applies to downloaded PS3 games. It does apply to all PS4 games (both physical and digital) though.
- grishka 5y ago
  What happened if you deliberately did something stupid like dereferencing a null pointer? What if you returned from the loop on the main thread?
  - kabdib 5y ago
    I don't remember for sure. I think that if the title exited (by returning, or hitting an unhandled exception), the console tore everything down and relaunched the title . . . but it also might have just let threads die until there weren't any left to run, and then just sat there (running just system and UI threads). It wasn't something I expected, and there were good reasons for the behavior other than "let's hassle developers with more annoying cert requirements".
munk-a 5y ago
A MUD I worked (uh volunteered, as a total newbie) on had a soft-reboot implementation for deploys that blew my mind when I first saw it in action. Open telnet connections were all sent keep alives before the MUD launched a new instance of itself from the executable (which may have been a different version from the currently running copy) and then killed itself. To users on the other end a message appeared saying "Relaunching the server" and was followed up shortly by "You may now resume play". Since this engine was designed to constantly fall over due to instabilities and OOMs it stored the current game state in a persistent state (originally on-disk files - most of it was migrated to MySQL) so the only things that really needed hacking were re-acquiring the right sockets and remembering which user is which. This included, FYI, under development new game elements which were mostly written and launched using a mix of things called RPROGs, OPROGs, CPROGs and Descs (among other things) all of which were written in a custom domain language input into the MUD itself - devs like myself worked only on underlying mechanics and had no need to wander into the specifics of all those triggers. This probably opened up a number of security holes but, given that the game only ever checked the first six characters of a password (and checked them one by one against a plaintext copy - that was patched) there were lower fruit available to criticize. If you want to check it out an older copy is available open-sourced[1] - it's a DIKU variant MUD. 1. https://github.com/webbj74/RPI-Engine https://github.com/webbj74/RPI-Engine
- endgame 5y ago
  Copyover MUD servers used to do all sorts of cool stuff. I remember seeing one that would reboot by: 1. Open a pipe 2. fork() 3. parent exec()'s the new binary with an argument saying "reload yourself from pipe at FD #n" 4. child writes the entire game state including open connections down the pipe, and closes It's done this way so the PID of the process doesn't change.
- akkartik 5y ago
  Great example of the benefits of crash-only software. https://www.usenix.org/legacy/events/hotos03/tech/full_papers/candea/candea.pdf https://www.usenix.org/legacy/events/hotos03/tech/full_paper...
pvorb 5y ago
Well, that's what I call thinking outside the (X)box to work around a limitation.
pseudosavant 5y ago
It is funny they say the Xbox was known for only having 64mb of RAM. The PS2 only had 32mb. The GameCube only had 27mb.
- pezezin 5y ago
  Your numbers are a bit off. The PS2 had 36 MB of RAM (32 main, 4 graphics), and the Gamecube had 43 MB (24 main, 16 audio, 3 graphics).
  - creata 5y ago
    Why did the GameCube dedicate so much of its RAM to audio?
    - bombcar 5y ago
      It wasn’t dedicated to audio - it was DRAM (slower than the main RAM) that could be used for various things - but audio was one of the most common.
    - kevingadd 5y ago
      The RAM could be used for things other than audio, but audio on the GC was driven by a dedicated special DSP chip, so it's not weird for that thing to have its own bank of memory. AFAIK the Wii used the same audio chip and I think the Wii U might have as well?
- mypalmike 5y ago
  Indeed. XBox was the easy target of those 3.
CyberRabbi 5y ago
Interesting article but I just have to say that XLaunchNewImage is basically execve or kexec. Calling it “rebooting” makes it seem like the game developers put in some dirty/crazy hacks when in reality all they did was call a public API that was explicitly designed to accommodate this use case (e.g. not resetting video state).
celesti 5y ago
Soft Reboot, is that like Sudo Purge?
trashface 5y ago
The original Xbox was a huge pain in terms of memory. I'm amazed they got Morrowind to run on it at all. I worked on an Xbox game, and me and another programmer (much more skilled) spent a couple weeks trying to get memory usage down so that it would fit - and to eliminate almost all leaks. There was a TCR that required the game to be able to run for an extended period, don't remember how long. But to prepare for it, I'd set it up to run in a loop overnight on the dev kit, and if it was locked up in the morning when I came in, I knew I still had work to do.
geerlingguy 5y ago
It took me a few minutes, after reading both the article and the first few comments here, before I realized Morrowind was a game (in the Elder Scrolls series, it seems), and not the code name for some early Xbox firmware (what I presumed at first).
shmerl 5y ago
Good to mention OpenMW: https://openmw.org https://openmw.org
jbluepolarbear 5y ago
Morrowind for Xbox is the reason I’m a game developer today. I loved the game so much, but the Xbox version was really, really slow and wasn’t able to use mods. I convinced my parents to split the difference with me on a pc barely capable to play the PC version. My brother and I made countless mods and quests.
1MachineElf 5y ago
Today you can run Morrowind quite well on the Xbox One (not to be confused with the 1st gen Xbox.) I highly recommend running it on PC with OpenMW, especially with the Tamriel Rebuilt mod installed.
bitwize 5y ago
There was a Sega Genesis game (I think it was X-Men) where it prompted you to "reset the master computer" or summat. There was no button on the controller that would reset the computer. Instead you had to mash RESET on the console itself! Somehow, it saved a bit of state that would detect a soft reboot at that point and resume the game from before you hit reset. Too bad later Genesis models lacked a RESET button...
sarak12070 5y ago
How to stay fit and healthy at home http://healthwithbeauty.xyz/2021/04/15/how-to-stay-fit-and-healthy-at-home/ http://healthwithbeauty.xyz/2021/04/15/how-to-stay-fit-and-h... secretary bird cool facts I am sure you don't know about these birds https://www.interestingnews.club/2021/04/secretary-bird-cool-facts.html https://www.interestingnews.club/2021/04/secretary-bird-cool...
flohofwoe 5y ago
As far as I remember, this wasn't a "hack", but "booting into a new map" was a recommended technique in the SDK documentation to reduce loading times, simplify memory management and reduce the memory needed for code (e.g. you'd compile different "executables" for different parts of the game, like main menu vs singleplayer-mode vs multiplayer-mode).
- genocidicbunny 5y ago
  Exactly. This isn't some amazing thing that only Morrowind did -- it was a standard practice that was made available via a well documented API. In fact, your example of main menu vs single player vs multiplayer is still the example provided as the use case for similar APIs on more recent consoles.
  - ascagnel_ 5y ago
    Not just consoles -- games based on id Tech 3 (Quake 3's engine) would often ship with separate executable files on PC for single- and multi-player.
- grenoire 5y ago
  The video linked in the article is a lot more technical and refers to the, indeed documented, the XLaunchNewImage function. I'd give it a watch!
Cthulhu_ 5y ago
I think that's a pretty clever hack, and a strong case to make startup and reboot times as fast as possible, especially for a single purpose device (or operating mode, e.g. boot straight to the game in this case).
rusk 5y ago
Interesting parallel to the “nightly reboot”, the Java community’s dirty little secret ... memory management is hard.
- DaiPlusPlus 5y ago
  Why Java though? Java uses object handles, not raw pointers, so a stop-the-world GC in a JVM can relocate and defragment memory whenever it wants.
  - rusk 5y ago
    Java because GC lures people into a false sense of security. You can still leak memory by failing to release references here and there and in a heavily used system this can rack up quite considerably over a 24 hour period and it's often more pragmatic just to reboot the application than get somebody to fix it. I suppose most other languages/environments could suffer this but with Java it's drummed into you from the very start that "you can't get memory leaks" so people just don't worry about those things.
- MaxBarraclough 5y ago
  I'm reminded of the way GitLab built a whole architecture to cope with their serious memory leaks, rather than fixing the leaks. [0] This is despite that GitLab is written in Ruby, a garbage-collected language. [0] https://docs.gitlab.com/ee/administration/operations/unicorn.html https://docs.gitlab.com/ee/administration/operations/unicorn...
MayeulC 5y ago
It's indeed a bit like a kexec. On system initialization, kernel routines are copied into RAM by the bootloader[1]. Executables run in ring 0, and have a jump table to call kernel routines in their own address space (IIRC) [2]. As such, loading any executable counts as a soft reboot. Also, there is a functioning open source emulator, Xqemu [3] (and its sibling/fork focused on speed and compatibility more than accuracy, xemu [4]) I recommend reading "17 mistakes Microsoft made in the Xbox security system", which is very informative [5]. [1]: https://xboxdevwiki.net/Boot_Process https://xboxdevwiki.net/Boot_Process [2]: https://xboxdevwiki.net/Kernel https://xboxdevwiki.net/Kernel [3]: https://xqemu.com/ https://xqemu.com/ [4]: https://xemu.app/ https://xemu.app/ [5]: https://xboxdevwiki.net/17_Mistakes_Microsoft_Made_in_the_Xbox_Security_System https://xboxdevwiki.net/17_Mistakes_Microsoft_Made_in_the_Xb...
thomond 5y ago
Skip the blogspam: https://www.youtube.com/watch?v=x0TKwPnHc-M https://www.youtube.com/watch?v=x0TKwPnHc-M
laurensr 5y ago
Nice write-up! The Nintendo Wii did more or less the same thing. It would (re-load) various IOSes [1] when opening a game or applet. [1]: https://wiibrew.org/wiki/IOS https://wiibrew.org/wiki/IOS
MaxBarraclough 5y ago
> Presumably, the reason for this behaviour was that it was simpler to boot the game fresh when loading a save, rather than trying to unload all the game assets in memory from the current game. So, 'arena'/'region-based' memory management, of a sort. Figures. Wonder if any other games used the same trick.
lunaticlabs 5y ago
I’ve been working in games since before then, and it’s a common trick. It’s often portrayed as if it would occur because of poor memory management (I.e. memory not being freed), but even with perfect memory management it can happen in constrained memory environments (like consoles). What happens is you would are allocating large memory chunks at level loading time, and potentially a number of smaller allocations during the a user playing that need to be persistent (for example, player save data). You would inevitably end up with a small block of memory allocated in the middle of a large free chunk, and would unable to allocate a large chunk that spanned the small allocated chunk. This was manageable up to point, you would try and pre allocate as much of the persistent memory as possible so that it would end up at similar addresses and contiguous, but it was difficult to guarantee. Theoretically, of course, you can develop without allocating persistent memory in game, but it becomes difficult to maintain, and limits you in other areas. If you used any 3rd party libraries that allocated memory, or certain template functions or standard library functions that allocated memory, you had to work around that. If your game wasn’t linear and players can move around dynamically, this becomes exponentially difficult. On consoles, we also had to pass “soak” tests, where the game has to run in a demo mode for 24-48 hours or longer, continuously without crashing or errors. Inevitably the only solution was to force all your memory to get remapped, but the only way to do that was to reallocate everything in a relatively predictable order, and the only way to do that was to save your state, force a reboot, and reload. Hence the “hack”, except that this was so common, facilities existed that helped you do it. Normally, there was a chunk of memory that persisted between soft reboots, so you could store your state here. You could also pass flags to the rebooted executable, to tell it where or how to load.

240 points by pmarin 5y ago

that was a great post, typical fun exploitation to rid yourself of the problem of memory management. There was a way to force garbage collection in Flash as well that was used by the clever pros at the time.

varsketiz 5y ago

Wow, that's a cool hack. I wonder if they did more stuff like that to go around Xbox limitations. Reminds me of Crash Bandicoot hacks for ps1 :))

ToFundorNot 5y ago
For those who haven't read/watched it: https://arstechnica.com/gaming/2020/02/war-stories-how-crash-bandicoot-hacked-the-original-playstation/ https://arstechnica.com/gaming/2020/02/war-stories-how-crash... It is indeed a story of creativity in design.
- DicIfTEx 5y ago
  Complete with a cautionary tale about the dangers of being too creative: https://www.gamasutra.com/blogs/DaveBaggett/20131031/203788/My_Hardest_Bug_Ever.php https://www.gamasutra.com/blogs/DaveBaggett/20131031/203788/...
  - failuser 5y ago
    Bumping the timer spreed is not the creative part, creative part was zeroing on the issue. For software guys used to reliable hardware and no prior knowledge how to debug hardware bugs this is hell.
  - bogwog 5y ago
    Wow. That story is going to give me nightmares.
  - Firehawke 5y ago
    It's notable that the controller and memory card slot are on the same bus-- in fact, the system isn't talking to the memory card directly! It has to send requests and data through the controller, back to the system, and then to the card port. This is because the original specification would have had a memory card slot in the controller but it was too expensive. They also had considered using floppies for storage, but they were too unreliable (and my experiences with 1.44MB 3.5s definitely agrees that consumer-grade cheap disks sucked hard)

hinkley 5y ago

Presumably then if the game was glitching on you and you did a save/load then the problem would magically fix itself. (have you tried turning it off and back on again?)

deleted 5y ago

[deleted]

flatiron 5y ago

His video on Monday was great about this. I like that they checked first if they were running low on memory and then just rebooted the darn thing if it was.

geuis 5y ago

Video from Modern Vintage Gamer walking through the process of how this worked along with a demonstration https://www.youtube.com/watch?v=x0TKwPnHc-M https://www.youtube.com/watch?v=x0TKwPnHc-M

simlevesque 5y ago
This guy is the best video game tech youtuber out here. He knows his stuff deeply.
- Nextgrid 5y ago
  He was part of the original Xbox homebrew scene.

kashanjunaid 5y ago

great hack! thanks for share

Impossible 5y ago

This was common in console games and console SDKs had special APIs to support this. It's a decent solution to memory leak and fragmentation issues

thomasfromcdnjs 5y ago

Great to see Morrowind on HN, just sharing my personal story; I was in highschool living in a trailer park with my mother. My ma ended up getting a pay out from the government for a work related injury. She bought me a PC and we set it up in the caravan and I was able to run Morrowind. It was a great escape from the conditions at the time and I kept learning programming when I wasn't hunting Daedra's.

gnulinux 5y ago
Morrowind looks fun but every time I try playing, it feels like doing Sunday chores. 90% of my game experience thus far has been walking. What am I doing wrong?
- taurath 5y ago
  It’s one of those games that it feels best to not really know what type of game it is at first going in. It felt like the ultimate “lots and lots under the surface” game when it came out, but now so many features have been coopted what seemed really clever before now feels like a waste of time since you are more intimately familiar with the mechanics. It’s a “what’s over there?” game.
- codezero 5y ago
  For me the novelty of Morrowind doesn't quite last, you could just play Skyrim imho. Morrowind had a world that was so expansive at the time it was awe inspiring. I think that was the main thing that made it so wild. It lacked a lot of quality of life features that make it hard to play today, and I remember routinely taking a break then forgetting which five month long plot I had been playing before going on vacation for a week. Just not the kind of game you can pick up in a weekend.
  - hombre_fatal 5y ago
    Most quality of life features just make games a bit boring though, like WoW-style waypoint markers where you don’t even read the quests or talk to people anymore because the game tells you where to go. Skyrim is especially bad imo because it all takes place in the same biome with no sense of “wtf is going on here?” My advice to people here: Get OpenMW and start playing Morrowind with a notepad open that you can alt-tab into for notes. I never beat the game until last year. If the pace is too slow, fine, just open up the console and find the command to double or triple your walking speed. It’s a game best enjoyed slow otherwise, as you unfold the mysteries of the world. I think it definitely holds up to 2021.

corysama 5y ago

Yep. The function would reboot into a new executable without disrupting what was on the screen. I don’t recall which one, but this was used by an FPS to switch between single-player vs. multi-player exes that were developed by two completely different studios.

maccard 5y ago
Call of duty did it first I think but it's not uncommon to do these days...
adamparsons 5y ago
PC versions of CoD do this, and switching between mode on a 360 was always kinda slow so I could definitely believe its doing something similar

yroc92 5y ago

It blew my mind to hear that the Xbox ran on 64mb of RAM.

gfrangakis 5y ago
That blew my mind as well. I remember it looking so good. Xbox - 64mb Xbox 360 - 512mb Xbox One - 8 GB (later 12 GB) Xbox Series X - 16 GB This last one also surprises me, was expecting at least 32GB
- deleted 5y ago
  [deleted]
- ahartmetz 5y ago
  At least in the PS5, the ultra-fast SSD and some technology around it are supposed to drastically reduce the need to keep things in RAM in case they are needed quickly later. There's a talk about that and other PS5 innovations by Mark Czerny. It's a very high quality talk.
- mikepurvis 5y ago
  I think the theory with the current gen is that devs get a lot more out a mere doubling because of all the improvements to SSD streaming. Like, it's 16gb, but it's 16gb that only has to cover the next few seconds of gameplay, not the next 30 seconds.
- Dylan16807 5y ago
  > Xbox Series X - 16 GB > This last one also surprises me, was expecting at least 32GB Well the GPU is comparable to 8GB models, so I can see why 16GB would work fine. You don't need that much space for game state. Though looking into the Series X more, they sure set up the ram weirdly. There are 4 1GB memory chips and 6 2GB chips. This gives it 10GB of very fast memory, and 6GB of somewhat fast memory. The operating system reserves 2.5GB of the somewhat fast memory, and games get 13.5GB The 'natural' setup would be 20GB, and they didn't even go that high, so they must be plenty confident that 32GB is well beyond what they need.

kabdib 5y ago

I was surprised to see that (at least on the 360) there was no way for a title to terminate itself; the game process ran until it was killed by something else (e.g., the user launching another title, or shutting the console down). Another surprise was the utter lack of a date-and-time clock chip. Without a network (and I'm guessing, without actually connecting to XBox Live), the 360 had no idea what time it was. That saved a bunch of money on hardware, and removed headaches involving coin cells (such as shipping policies).

rodgerd 5y ago
It's still the case on a Series X. There are separate pools for applications and games, as well, and the Series X is able to keep state for multiple running games at once, which is nice. The only downside is that it seems to cause a certain amount of pain with applications sometimes not working when there's a contest over sending Dolby to a receiver. On the time thing: turns out that PlayStation 3 and 4 will stop letting you run games if you replace the battery and it can't re-sync time with the PSN.
- kevingadd 5y ago
  To be more precise, the "if your battery dies you can't play games" thing only applies to downloaded PS3 games. It does apply to all PS4 games (both physical and digital) though.
grishka 5y ago
What happened if you deliberately did something stupid like dereferencing a null pointer? What if you returned from the loop on the main thread?
- kabdib 5y ago
  I don't remember for sure. I think that if the title exited (by returning, or hitting an unhandled exception), the console tore everything down and relaunched the title . . . but it also might have just let threads die until there weren't any left to run, and then just sat there (running just system and UI threads). It wasn't something I expected, and there were good reasons for the behavior other than "let's hassle developers with more annoying cert requirements".

munk-a 5y ago

A MUD I worked (uh volunteered, as a total newbie) on had a soft-reboot implementation for deploys that blew my mind when I first saw it in action. Open telnet connections were all sent keep alives before the MUD launched a new instance of itself from the executable (which may have been a different version from the currently running copy) and then killed itself. To users on the other end a message appeared saying "Relaunching the server" and was followed up shortly by "You may now resume play". Since this engine was designed to constantly fall over due to instabilities and OOMs it stored the current game state in a persistent state (originally on-disk files - most of it was migrated to MySQL) so the only things that really needed hacking were re-acquiring the right sockets and remembering which user is which. This included, FYI, under development new game elements which were mostly written and launched using a mix of things called RPROGs, OPROGs, CPROGs and Descs (among other things) all of which were written in a custom domain language input into the MUD itself - devs like myself worked only on underlying mechanics and had no need to wander into the specifics of all those triggers. This probably opened up a number of security holes but, given that the game only ever checked the first six characters of a password (and checked them one by one against a plaintext copy - that was patched) there were lower fruit available to criticize. If you want to check it out an older copy is available open-sourced[1] - it's a DIKU variant MUD. 1. https://github.com/webbj74/RPI-Engine https://github.com/webbj74/RPI-Engine

endgame 5y ago
Copyover MUD servers used to do all sorts of cool stuff. I remember seeing one that would reboot by: 1. Open a pipe 2. fork() 3. parent exec()'s the new binary with an argument saying "reload yourself from pipe at FD #n" 4. child writes the entire game state including open connections down the pipe, and closes It's done this way so the PID of the process doesn't change.
akkartik 5y ago
Great example of the benefits of crash-only software. https://www.usenix.org/legacy/events/hotos03/tech/full_papers/candea/candea.pdf https://www.usenix.org/legacy/events/hotos03/tech/full_paper...

pvorb 5y ago

Well, that's what I call thinking outside the (X)box to work around a limitation.

pseudosavant 5y ago

It is funny they say the Xbox was known for only having 64mb of RAM. The PS2 only had 32mb. The GameCube only had 27mb.

pezezin 5y ago
Your numbers are a bit off. The PS2 had 36 MB of RAM (32 main, 4 graphics), and the Gamecube had 43 MB (24 main, 16 audio, 3 graphics).
- creata 5y ago
  Why did the GameCube dedicate so much of its RAM to audio?
  - bombcar 5y ago
    It wasn’t dedicated to audio - it was DRAM (slower than the main RAM) that could be used for various things - but audio was one of the most common.
  - kevingadd 5y ago
    The RAM could be used for things other than audio, but audio on the GC was driven by a dedicated special DSP chip, so it's not weird for that thing to have its own bank of memory. AFAIK the Wii used the same audio chip and I think the Wii U might have as well?
mypalmike 5y ago
Indeed. XBox was the easy target of those 3.

CyberRabbi 5y ago

Interesting article but I just have to say that XLaunchNewImage is basically execve or kexec. Calling it “rebooting” makes it seem like the game developers put in some dirty/crazy hacks when in reality all they did was call a public API that was explicitly designed to accommodate this use case (e.g. not resetting video state).

celesti 5y ago

Soft Reboot, is that like Sudo Purge?

trashface 5y ago

The original Xbox was a huge pain in terms of memory. I'm amazed they got Morrowind to run on it at all. I worked on an Xbox game, and me and another programmer (much more skilled) spent a couple weeks trying to get memory usage down so that it would fit - and to eliminate almost all leaks. There was a TCR that required the game to be able to run for an extended period, don't remember how long. But to prepare for it, I'd set it up to run in a loop overnight on the dev kit, and if it was locked up in the morning when I came in, I knew I still had work to do.

geerlingguy 5y ago

It took me a few minutes, after reading both the article and the first few comments here, before I realized Morrowind was a game (in the Elder Scrolls series, it seems), and not the code name for some early Xbox firmware (what I presumed at first).

shmerl 5y ago

Good to mention OpenMW: https://openmw.org https://openmw.org

jbluepolarbear 5y ago

Morrowind for Xbox is the reason I’m a game developer today. I loved the game so much, but the Xbox version was really, really slow and wasn’t able to use mods. I convinced my parents to split the difference with me on a pc barely capable to play the PC version. My brother and I made countless mods and quests.

1MachineElf 5y ago

Today you can run Morrowind quite well on the Xbox One (not to be confused with the 1st gen Xbox.) I highly recommend running it on PC with OpenMW, especially with the Tamriel Rebuilt mod installed.

bitwize 5y ago

There was a Sega Genesis game (I think it was X-Men) where it prompted you to "reset the master computer" or summat. There was no button on the controller that would reset the computer. Instead you had to mash RESET on the console itself! Somehow, it saved a bit of state that would detect a soft reboot at that point and resume the game from before you hit reset. Too bad later Genesis models lacked a RESET button...

sarak12070 5y ago

How to stay fit and healthy at home http://healthwithbeauty.xyz/2021/04/15/how-to-stay-fit-and-healthy-at-home/ http://healthwithbeauty.xyz/2021/04/15/how-to-stay-fit-and-h... secretary bird cool facts I am sure you don't know about these birds https://www.interestingnews.club/2021/04/secretary-bird-cool-facts.html https://www.interestingnews.club/2021/04/secretary-bird-cool...

flohofwoe 5y ago

As far as I remember, this wasn't a "hack", but "booting into a new map" was a recommended technique in the SDK documentation to reduce loading times, simplify memory management and reduce the memory needed for code (e.g. you'd compile different "executables" for different parts of the game, like main menu vs singleplayer-mode vs multiplayer-mode).

genocidicbunny 5y ago
Exactly. This isn't some amazing thing that only Morrowind did -- it was a standard practice that was made available via a well documented API. In fact, your example of main menu vs single player vs multiplayer is still the example provided as the use case for similar APIs on more recent consoles.
- ascagnel_ 5y ago
  Not just consoles -- games based on id Tech 3 (Quake 3's engine) would often ship with separate executable files on PC for single- and multi-player.
grenoire 5y ago
The video linked in the article is a lot more technical and refers to the, indeed documented, the XLaunchNewImage function. I'd give it a watch!

Cthulhu_ 5y ago

I think that's a pretty clever hack, and a strong case to make startup and reboot times as fast as possible, especially for a single purpose device (or operating mode, e.g. boot straight to the game in this case).

rusk 5y ago

Interesting parallel to the “nightly reboot”, the Java community’s dirty little secret ... memory management is hard.

DaiPlusPlus 5y ago
Why Java though? Java uses object handles, not raw pointers, so a stop-the-world GC in a JVM can relocate and defragment memory whenever it wants.
- rusk 5y ago
  Java because GC lures people into a false sense of security. You can still leak memory by failing to release references here and there and in a heavily used system this can rack up quite considerably over a 24 hour period and it's often more pragmatic just to reboot the application than get somebody to fix it. I suppose most other languages/environments could suffer this but with Java it's drummed into you from the very start that "you can't get memory leaks" so people just don't worry about those things.
MaxBarraclough 5y ago
I'm reminded of the way GitLab built a whole architecture to cope with their serious memory leaks, rather than fixing the leaks. [0] This is despite that GitLab is written in Ruby, a garbage-collected language. [0] https://docs.gitlab.com/ee/administration/operations/unicorn.html https://docs.gitlab.com/ee/administration/operations/unicorn...

MayeulC 5y ago

It's indeed a bit like a kexec. On system initialization, kernel routines are copied into RAM by the bootloader[1]. Executables run in ring 0, and have a jump table to call kernel routines in their own address space (IIRC) [2]. As such, loading any executable counts as a soft reboot. Also, there is a functioning open source emulator, Xqemu [3] (and its sibling/fork focused on speed and compatibility more than accuracy, xemu [4]) I recommend reading "17 mistakes Microsoft made in the Xbox security system", which is very informative [5]. [1]: https://xboxdevwiki.net/Boot_Process https://xboxdevwiki.net/Boot_Process [2]: https://xboxdevwiki.net/Kernel https://xboxdevwiki.net/Kernel [3]: https://xqemu.com/ https://xqemu.com/ [4]: https://xemu.app/ https://xemu.app/ [5]: https://xboxdevwiki.net/17_Mistakes_Microsoft_Made_in_the_Xbox_Security_System https://xboxdevwiki.net/17_Mistakes_Microsoft_Made_in_the_Xb...

thomond 5y ago

Skip the blogspam: https://www.youtube.com/watch?v=x0TKwPnHc-M https://www.youtube.com/watch?v=x0TKwPnHc-M

laurensr 5y ago

Nice write-up! The Nintendo Wii did more or less the same thing. It would (re-load) various IOSes [1] when opening a game or applet. [1]: https://wiibrew.org/wiki/IOS https://wiibrew.org/wiki/IOS

MaxBarraclough 5y ago

> Presumably, the reason for this behaviour was that it was simpler to boot the game fresh when loading a save, rather than trying to unload all the game assets in memory from the current game. So, 'arena'/'region-based' memory management, of a sort. Figures. Wonder if any other games used the same trick.

lunaticlabs 5y ago

I’ve been working in games since before then, and it’s a common trick. It’s often portrayed as if it would occur because of poor memory management (I.e. memory not being freed), but even with perfect memory management it can happen in constrained memory environments (like consoles). What happens is you would are allocating large memory chunks at level loading time, and potentially a number of smaller allocations during the a user playing that need to be persistent (for example, player save data). You would inevitably end up with a small block of memory allocated in the middle of a large free chunk, and would unable to allocate a large chunk that spanned the small allocated chunk. This was manageable up to point, you would try and pre allocate as much of the persistent memory as possible so that it would end up at similar addresses and contiguous, but it was difficult to guarantee. Theoretically, of course, you can develop without allocating persistent memory in game, but it becomes difficult to maintain, and limits you in other areas. If you used any 3rd party libraries that allocated memory, or certain template functions or standard library functions that allocated memory, you had to work around that. If your game wasn’t linear and players can move around dynamically, this becomes exponentially difficult. On consoles, we also had to pass “soak” tests, where the game has to run in a demo mode for 24-48 hours or longer, continuously without crashing or errors. Inevitably the only solution was to force all your memory to get remapped, but the only way to do that was to reallocate everything in a relatively predictable order, and the only way to do that was to save your state, force a reboot, and reload. Hence the “hack”, except that this was so common, facilities existed that helped you do it. Normally, there was a chunk of memory that persisted between soft reboots, so you could store your state here. You could also pass flags to the rebooted executable, to tell it where or how to load.