5 ms·

Can you speak to some of the harder or more interesting challenges you faced during that time?

dinartem 2y ago
One challenge was that while I started working on the Xbox 360 about three years before it would ship, we knew that the custom CPU would not be available until early 2005 (first chips arrived in early February). And there was only supposed to be one hardware spin before final release. So I had no real hardware to test any of the software I was writing, and no other chips (like the Apple G5 we used as alpha kits) had the custom security hardware or boot sequence like the custom chip would have. But I still needed to provide the first stage boot loader which is stored in ROM inside the CPU weeks before first manufacture. I ended up writing a simulator of the CPU (instruction level), to make progress on writing the boot code. Obviously my boot code and hypervisor would run perfectly on my simulator since I wrote both! But IBM had also had a hardware accelerated cycle-accurate simulator that I got to use. I was required to boot the entire Xbox 360 kernel in their simulator before I could release the boot ROM. What takes a few seconds on hardware to boot took over 3 hours in simulation. The POST codes would be displayed every so often to let me know that progress was still being made. The first CPU arrived on a Friday, by Saturday the electrical engineers flew to Austin to help get the chip on the motherboard and make sure FSB and other busses were all working. I arrived on Monday evening with a laptop containing the source code to the kernel, on Tuesday I compiled and flashed various versions, working through the typical bring-up issues. By Wednesday afternoon the kernel was running Quake, including sound output and controller input. Three years of preparation to make my contribution to hardware bring-up as short as possible, since I would bottleneck everyone else in the development team until the CPU booted the kernel.
- fragmede 2y ago
  damn, that's bad ass. did that simulator run on a Windows system or was it something more esoteric?
  - dinartem 2y ago
    I called the simulator Sbox and it was just a simple console app. I didn't implement the GPU, so no graphics just the hypervisor and kernel and some simple non-graphics apps. I made it so that you could build the Xbox 360 kernel on your windows machine, then just run sbox.exe and it would automatically find the just built kernel image targeting the PPC64 and boot it. Then if you typed control-C it would drop into the kernel debugger as a sub process, and you could poke around at the machine state as if it were the real Xbox hardware, showing all the PPC instructions and registers. It was a lot of fun writing it, and quite useful.
    - fragmede 2y ago
      That sounds like a lot of fun! What was your career like before that, that led you to be in the place to do such fun things?
    - Zeetah 2y ago
      You should also talk about the lwarx/stecx bug. IIRC - in the first version of the chip there was a bug in one or both of these instructions. Your code booted on SBox but didn't on the hardware. You compared the two and then figured out it was these instructions. You filed a bug report and then dug into them and used SBox to figure out what must have been going wrong. The chip supplier came back with a workaround and within five minutes you simulated it on SBox and said it wouldn't work, why, and then said how it should be fixed. The supplier didn't believe you as yet. And you worked out a workaround so we could be unblocked. Two weeks later they agreed with your fix...
      - maximilianburke 2y ago
        I recall an issue when trying to use lwarx/stwcx on Xbox 360 directly that the compiler (or maybe even the kernel, on program load? it's been a while) raised an error and said to use the Interlocked intrinsics instead -- is that related?
        dinartem 2y ago
- bananaboy 2y ago
  Amazing! Thanks for sharing! What sort of things are you working on now?
- kridsdale1 2y ago
  This is the coolest HN post I’ve read in months. Cheers.
- brokenmachine 2y ago
  Kick ass. This kind of post is why HN is the best.
- n144q 2y ago
  > we knew that the custom CPU would not be available until early 2005 Sounds a little bit like the situation with Xbox Series? The SDKs were released late because Microsoft was waiting for certain features in AMD APU
- Zeetah 2y ago
  You have an awesome memory, Dinarte! Eric Mejdric from IBM called on Friday and said we have the chips, when are you guys getting here? I took a red eye that night and got to Austin on Saturday morning. We brought up the board, the IBM debugger, and then got stuck. I remember calling you on Sunday morning. You had just got a big screen TV for the Super bowl and had people over and in-between hosting them you dropped us new bits to make progress. I think Tracy came on Sunday or Monday and with you got the Kernel booted. Good times! This is Harjit by the way. Edit: added super bowl.
  - saturn8601 2y ago
    OMG Harjit! I saw you in the documentary! You and the entire team are total rockstars! I just cannot fathom ever being in a position to design something that provided so much joy and happy core memories to countless people around the world...you guys did it! Just the thought of how many people you touched with your work....just amazing! :) Had a question if you don't mind: Can you talk about the thought process behind the power supply design? Its very large even in the super slim models. Were you following a specific design driven by the hardware architecture or were there other reasons? I always wondered about that.
    - maroonblazer 2y ago
      >I saw you in the documentary! I presume you're referring to this one: https://www.xbox.com/en-US/power-on#watch https://www.xbox.com/en-US/power-on#watch
      - saturn8601 2y ago
        Yes! It is worth watching every second. What an amazing production.
  - dinartem 2y ago
    Actually Tracy never made it to Austin. He was going to fly in later in the week to continue bring-up, but since we were done by Wednesday, we just sent the chips to Redmond and he continued there. He was of course always available on the phone to answer my kernel questions I had.
- saturn8601 2y ago
  You sir are an inspiration! I am but a mediocre Angular developer and reading this has me in complete awe! The kind of drive you must have had to get this done well I dont know how people manage to do it but it is so cool to see! :)
- markus_zhang 2y ago
  This is really some blast from the past. Can you please shed more light on the simulator? Is it interpretation or JIT? But then I realize XBOX uses Pentium III, so maybe virtualization? Edit Sorry it was XBOX 360 so it's not Pentium. As someone who recently got interested in emulation and wrote two lc-3 emulators, would really love to learn from the masters.
- cbanek 2y ago
  Small world! I worked on Yellow Door / Golden Gate automation for releasing 360 titles and patches to prod, and the beta group / KDC service code.

One challenge was that while I started working on the Xbox 360 about three years before it would ship, we knew that the custom CPU would not be available until early 2005 (first chips arrived in early February). And there was only supposed to be one hardware spin before final release. So I had no real hardware to test any of the software I was writing, and no other chips (like the Apple G5 we used as alpha kits) had the custom security hardware or boot sequence like the custom chip would have. But I still needed to provide the first stage boot loader which is stored in ROM inside the CPU weeks before first manufacture. I ended up writing a simulator of the CPU (instruction level), to make progress on writing the boot code. Obviously my boot code and hypervisor would run perfectly on my simulator since I wrote both! But IBM had also had a hardware accelerated cycle-accurate simulator that I got to use. I was required to boot the entire Xbox 360 kernel in their simulator before I could release the boot ROM. What takes a few seconds on hardware to boot took over 3 hours in simulation. The POST codes would be displayed every so often to let me know that progress was still being made. The first CPU arrived on a Friday, by Saturday the electrical engineers flew to Austin to help get the chip on the motherboard and make sure FSB and other busses were all working. I arrived on Monday evening with a laptop containing the source code to the kernel, on Tuesday I compiled and flashed various versions, working through the typical bring-up issues. By Wednesday afternoon the kernel was running Quake, including sound output and controller input. Three years of preparation to make my contribution to hardware bring-up as short as possible, since I would bottleneck everyone else in the development team until the CPU booted the kernel.

fragmede 2y ago
damn, that's bad ass. did that simulator run on a Windows system or was it something more esoteric?
- dinartem 2y ago
  I called the simulator Sbox and it was just a simple console app. I didn't implement the GPU, so no graphics just the hypervisor and kernel and some simple non-graphics apps. I made it so that you could build the Xbox 360 kernel on your windows machine, then just run sbox.exe and it would automatically find the just built kernel image targeting the PPC64 and boot it. Then if you typed control-C it would drop into the kernel debugger as a sub process, and you could poke around at the machine state as if it were the real Xbox hardware, showing all the PPC instructions and registers. It was a lot of fun writing it, and quite useful.
  - fragmede 2y ago
    That sounds like a lot of fun! What was your career like before that, that led you to be in the place to do such fun things?
  - Zeetah 2y ago
    You should also talk about the lwarx/stecx bug. IIRC - in the first version of the chip there was a bug in one or both of these instructions. Your code booted on SBox but didn't on the hardware. You compared the two and then figured out it was these instructions. You filed a bug report and then dug into them and used SBox to figure out what must have been going wrong. The chip supplier came back with a workaround and within five minutes you simulated it on SBox and said it wouldn't work, why, and then said how it should be fixed. The supplier didn't believe you as yet. And you worked out a workaround so we could be unblocked. Two weeks later they agreed with your fix...
    - maximilianburke 2y ago
      I recall an issue when trying to use lwarx/stwcx on Xbox 360 directly that the compiler (or maybe even the kernel, on program load? it's been a while) raised an error and said to use the Interlocked intrinsics instead -- is that related?
      - dinartem 2y ago
bananaboy 2y ago
Amazing! Thanks for sharing! What sort of things are you working on now?
kridsdale1 2y ago
This is the coolest HN post I’ve read in months. Cheers.
brokenmachine 2y ago
Kick ass. This kind of post is why HN is the best.
n144q 2y ago
> we knew that the custom CPU would not be available until early 2005 Sounds a little bit like the situation with Xbox Series? The SDKs were released late because Microsoft was waiting for certain features in AMD APU
Zeetah 2y ago
You have an awesome memory, Dinarte! Eric Mejdric from IBM called on Friday and said we have the chips, when are you guys getting here? I took a red eye that night and got to Austin on Saturday morning. We brought up the board, the IBM debugger, and then got stuck. I remember calling you on Sunday morning. You had just got a big screen TV for the Super bowl and had people over and in-between hosting them you dropped us new bits to make progress. I think Tracy came on Sunday or Monday and with you got the Kernel booted. Good times! This is Harjit by the way. Edit: added super bowl.
- saturn8601 2y ago
  OMG Harjit! I saw you in the documentary! You and the entire team are total rockstars! I just cannot fathom ever being in a position to design something that provided so much joy and happy core memories to countless people around the world...you guys did it! Just the thought of how many people you touched with your work....just amazing! :) Had a question if you don't mind: Can you talk about the thought process behind the power supply design? Its very large even in the super slim models. Were you following a specific design driven by the hardware architecture or were there other reasons? I always wondered about that.
  - maroonblazer 2y ago
    >I saw you in the documentary! I presume you're referring to this one: https://www.xbox.com/en-US/power-on#watch https://www.xbox.com/en-US/power-on#watch
    - saturn8601 2y ago
      Yes! It is worth watching every second. What an amazing production.
- dinartem 2y ago
  Actually Tracy never made it to Austin. He was going to fly in later in the week to continue bring-up, but since we were done by Wednesday, we just sent the chips to Redmond and he continued there. He was of course always available on the phone to answer my kernel questions I had.
saturn8601 2y ago
You sir are an inspiration! I am but a mediocre Angular developer and reading this has me in complete awe! The kind of drive you must have had to get this done well I dont know how people manage to do it but it is so cool to see! :)
markus_zhang 2y ago
This is really some blast from the past. Can you please shed more light on the simulator? Is it interpretation or JIT? But then I realize XBOX uses Pentium III, so maybe virtualization? Edit Sorry it was XBOX 360 so it's not Pentium. As someone who recently got interested in emulation and wrote two lc-3 emulators, would really love to learn from the masters.
cbanek 2y ago
Small world! I worked on Yellow Door / Golden Gate automation for releasing 360 titles and patches to prod, and the beta group / KDC service code.