3 ms·

I got some exposure to this at SiCortex, where we had our own MIPS-based processors and so had to do many of these things in software. There was one ColdFire (e

I got some exposure to this at SiCortex, where we had our own MIPS-based processors and so had to do many of these things in software. There was one ColdFire (embedded 68K) processor running μClinux per board, plus 27 of our own. This "Module Service Processor" would boot first, fetch a boot image from the one-per-system "System Service Processor" (pretty generic PC), load that via JTAG into each node's cache, then finally set each one loose to do things like memory registration and interconnect setup. This all set the stage for the actual Linux boot, which itself involved two stages with a switch_root in between. My very first assignment was to work on some of that MSP-to-node stuff, then later I had to dive into memory registration at least twice, even though both were pretty far from my real specialty. Small company, y'know. This kind of low-level work is significantly more complicated than even most kernel developers realize - hence the need for articles like OP. Ditto for anything on large (more than single-board) systems. The intersection of the two was, frankly, a bit exhausting. Just keeping track of all the moving parts and their respective states induced a cognitive load that made debugging other already-hard problems that much more difficult. My hat's off for anyone who has kept on doing that stuff longer than I did, or who has to do it in an environment where vendors are keeping so many secrets.

lgg 3y ago
This stuff is certainly pretty rarified these days. I remember when the PPC970 came out people were shocked how difficult it was to bootstrap. IBM didn't really care as POWER4 (from which it was derived) was not a merchant chip and they had management processors (and very high margins) in all their machines to handle it. Apple was the launch partner and even back then had a lot of in house expertise doing this sort of work. Everyone else who tried to use it was in for some real pain and most of them gave up. The guys doing the eval boards with support from IBM literally posted this: https://web.archive.org/web/20060715134515/http://www.970eval.com/970FAQ.html#anchor#4 https://web.archive.org/web/20060715134515/http://www.970eva... TL;DR, the last line is "Once all of the above is completed, the processor will be able to successfully fetch instructions from a boot source. You are now effectively at the same point you would have been 5 months ago, had this been a standard 750 bringup... Board bringup from this point should be very straightforward and follow established methods."
- notacoward 3y ago
  That's an amazing document. Practically every sentence, though tersely stated, hints at hours (or worse) of experimentation and head-scratching. The "would have been 5 months ago" bit at the end is remarkably restrained. I'm certain I would have quit (or worse) by that point. Respect and condolences to whoever did this.
jacquesm 3y ago
I had to do some of this while bringing up a 486 to run my own kernel. Very frustrating, to the point that I had the reset switch of the machine wired to a sustain pedal just so that I didn't have to dive under the desk all the time.
bsoft16385 3y ago
That reminds me of a story about SiCortex. My university (University of Colorado Boulder) bought one of a very few SiCortex systems ever sold. As an undergraduate I competed at SC07 and SC08 in the Cluster Challenge competition. Our coach was a CU facility member, Doug, who also was responsible for the SiCortex box we bought. At SC08 he told us that another one of the teams was competing with a SiCortex box. We knew that they would win the LINPACK part of the challenge, but we didn't know that LINPACK was basically the only thing they managed to get working. We had also heard rumors that SiCortex was in trouble financially at that time. When we were walking the show floor at SC08, we came across the huge SiCortex booth, which had 10 or so machines of different sizes (I believe the smallest was a 64-core workstation and the largest was a ~5000 whole rack system). I remarked to Doug that SiCortex didn't look to be in such bad shape. Doug turned to me and said, "25% of the machines SiCortex has ever made are in that booth". The SiCortex idea was like VLIW. On paper the numbers look great. On highly optimized synthetic benchmarks it looks good. On real world code you find out how hard it is to get good performance.
- notacoward 3y ago
  Sounds about right. The processors were six-core 500MHz single issue, which was pretty damn slow even by the standards of the time (2006-2008), beefed up with some extra floating point and some relatively very fast interconnect stuff. The key is that there were a lot of processors - 972 in the big machine. So you really really had to rely on parallelism to get any kind of performance, and a lot of code doesn't parallelize well at all. Even in the HPC space, a surprising number of codes just aren't designed to work for more than about 64. Also the machine positively sucked for linear integer code - like, say, compilers or OS kernels. One of the first things customers would do, naturally, was compile. Bad first impression. Also, it was nearly impossible to get a Lustre MDS for a thousand-node cluster (which is what the biggest machine was) to run for any length of time without falling over, because Lustre was designed around the assumption that the MDS would be bigger and beefier than anything else and have "poor man's flow control" in the form of a relatively slow network. In our case it was exactly the same and completely unprotected because the interconnect was the fastest part of the system by quite a margin. That was my nightmare for those two years. I've heard that Lustre has since added some flow control ("network request scheduler") but I was never able to benefit from that. PVFS2 worked better, and Gluster (which I worked on for nearly a decade afterward) would probably have been better still because it's more fully distributed and less CPU-hungry. The reason I mention all this is that there's an important lesson: building a system with a very unusual set of performance characteristics is a terrible idea business-wise, because people won't be able to realize its potential. Not even in a fairly specialized market. They'll just think it's slow. Unless it's truly bespoke, literally a one-off or close to it, nobody will want it. P.S. I actually had to visit CU-Boulder to debug something on that machine, with the aforementioned Doug. It became one of my favorite "war stories" from a 30-year career, but this has gone on long enough so I'll skip it.

lgg 3y ago

This stuff is certainly pretty rarified these days. I remember when the PPC970 came out people were shocked how difficult it was to bootstrap. IBM didn't really care as POWER4 (from which it was derived) was not a merchant chip and they had management processors (and very high margins) in all their machines to handle it. Apple was the launch partner and even back then had a lot of in house expertise doing this sort of work. Everyone else who tried to use it was in for some real pain and most of them gave up. The guys doing the eval boards with support from IBM literally posted this: https://web.archive.org/web/20060715134515/http://www.970eval.com/970FAQ.html#anchor#4 https://web.archive.org/web/20060715134515/http://www.970eva... TL;DR, the last line is "Once all of the above is completed, the processor will be able to successfully fetch instructions from a boot source. You are now effectively at the same point you would have been 5 months ago, had this been a standard 750 bringup... Board bringup from this point should be very straightforward and follow established methods."

notacoward 3y ago
That's an amazing document. Practically every sentence, though tersely stated, hints at hours (or worse) of experimentation and head-scratching. The "would have been 5 months ago" bit at the end is remarkably restrained. I'm certain I would have quit (or worse) by that point. Respect and condolences to whoever did this.

jacquesm 3y ago

I had to do some of this while bringing up a 486 to run my own kernel. Very frustrating, to the point that I had the reset switch of the machine wired to a sustain pedal just so that I didn't have to dive under the desk all the time.

bsoft16385 3y ago

That reminds me of a story about SiCortex. My university (University of Colorado Boulder) bought one of a very few SiCortex systems ever sold. As an undergraduate I competed at SC07 and SC08 in the Cluster Challenge competition. Our coach was a CU facility member, Doug, who also was responsible for the SiCortex box we bought. At SC08 he told us that another one of the teams was competing with a SiCortex box. We knew that they would win the LINPACK part of the challenge, but we didn't know that LINPACK was basically the only thing they managed to get working. We had also heard rumors that SiCortex was in trouble financially at that time. When we were walking the show floor at SC08, we came across the huge SiCortex booth, which had 10 or so machines of different sizes (I believe the smallest was a 64-core workstation and the largest was a ~5000 whole rack system). I remarked to Doug that SiCortex didn't look to be in such bad shape. Doug turned to me and said, "25% of the machines SiCortex has ever made are in that booth". The SiCortex idea was like VLIW. On paper the numbers look great. On highly optimized synthetic benchmarks it looks good. On real world code you find out how hard it is to get good performance.

notacoward 3y ago
Sounds about right. The processors were six-core 500MHz single issue, which was pretty damn slow even by the standards of the time (2006-2008), beefed up with some extra floating point and some relatively very fast interconnect stuff. The key is that there were a lot of processors - 972 in the big machine. So you really really had to rely on parallelism to get any kind of performance, and a lot of code doesn't parallelize well at all. Even in the HPC space, a surprising number of codes just aren't designed to work for more than about 64. Also the machine positively sucked for linear integer code - like, say, compilers or OS kernels. One of the first things customers would do, naturally, was compile. Bad first impression. Also, it was nearly impossible to get a Lustre MDS for a thousand-node cluster (which is what the biggest machine was) to run for any length of time without falling over, because Lustre was designed around the assumption that the MDS would be bigger and beefier than anything else and have "poor man's flow control" in the form of a relatively slow network. In our case it was exactly the same and completely unprotected because the interconnect was the fastest part of the system by quite a margin. That was my nightmare for those two years. I've heard that Lustre has since added some flow control ("network request scheduler") but I was never able to benefit from that. PVFS2 worked better, and Gluster (which I worked on for nearly a decade afterward) would probably have been better still because it's more fully distributed and less CPU-hungry. The reason I mention all this is that there's an important lesson: building a system with a very unusual set of performance characteristics is a terrible idea business-wise, because people won't be able to realize its potential. Not even in a fairly specialized market. They'll just think it's slow. Unless it's truly bespoke, literally a one-off or close to it, nobody will want it. P.S. I actually had to visit CU-Boulder to debug something on that machine, with the aforementioned Doug. It became one of my favorite "war stories" from a 30-year career, but this has gone on long enough so I'll skip it.