10 ms·
Simulating the IBM 360/50 mainframe from its microcode
- kens 5y agoAuthor here if anyone has questions about IBM System/360 computers :-)
- boothby 5y agoI'm trying to understand how they read out that BCROS memory. From the picture, it appears to be a single layer of copper on a mylar sheet. How did they get the bits out?
- kens 5y agoThey read the data capacitively. A sheet with perpendicular drive lines was put on top of the copper sheet. Energizing a drive line capacitively coupled to the square capacitors on the BCROS sheet. Each bit had two capacitors (balanced), driving a differential amplifier (sense amplifier) to produce the output bit. Using two capacitors reduced noise since the differential amplifier would cancel out the noise.
- boothby 5y agoThanks, for the answer, and for maintaining the coolest website on the net. Keep up the good work!
- monocasa 5y agoI've heard from someone that used one that the 360/91 was microcoded, just very differently both compared to contemporaries and to today's machines. That the fetch, decode, and dispatch was in pure logic, but the execution units themselves (at least in the float unit, which was the only unit that was out of order) ran their own microcode programs. You wouldn't happen to know where more documents than what's on bitsavers exist for the 360/91 (or related like the 95 and 195) do you? Competing anecdotes are less than satisfying. My source has been worng before but he bats a pretty good average for his contrarian comparch statements.
- kens 5y agoHere's a very detailed paper on the 360/91 architecture: https://www.engineering.iastate.edu/~zzhang/cpre581/reading/ibm67-anderson-360.pdf https://www.engineering.iastate.edu/~zzhang/cpre581/reading/...
- monocasa 5y agoThat's a great paper! I really appreciate you finding that; it was a great read and does go further in depth than the funcChar docs. It unfortunately doesn't seem to disambiguate the specific question of if floating point execution units were implemented via microcode unless I'm missing it.
- pinewurst 5y agoI don't think the Mod 91 ever shipped. It was a combat machine designed to squash CDC (IBM was sued by them for their anti-competitive actions here) and was eventually transmogrified into the Mod 95/195 which did ship.
- monocasa 5y agoAccording to wiki, there were 15 model 91s, 4 kept by IBM and the rest went to customers including NASA. There were two 95s and both went to NASA (and really the 95 was a 91 with a different RAM technology but the same CPU). That lines up with my source who started his career doing simulation and analysis of rocket propellent in micro gravity for the tail end of the Apollo program and some of the initial work for the space shuttle.
- kens 5y agoThere were approximately 15 Model 91s produced. The Computer History Museum has the console from one, and the Living Computers Museum has another. The Model 95 was the even more rare version with thin-film memory instead of core memory. Only two of them were produced, for NASA. The Model 195 was a reimplementation of the Model 91 with "monolithic" integrated circuits instead of hybrid SLT modules. The System 370/195 was very similar to the System 360/195. Curiously, the System 360/195 has the black styling of System/370 control panels, rather than the beige control panel of System/360. https://en.wikipedia.org/wiki/IBM_System/360_Model_91#Models_built https://en.wikipedia.org/wiki/IBM_System/360_Model_91#Models...
- colejohnson66 5y agoThe IO "channels" are interesting. Are they analogous to the PIO in the RP2040 (Pi Pico)?
- kens 5y agoThe System/360 I/O channels are kind of like a DMA (direct memory access) controller, so I/O can happen in the background instead of making the CPU wait. The Raspberry Pi PIO is more of a programmable state machine for low-level I/O protocols. So they are kind of similar in that they offload I/O tasks from the main CPU. But they also act at different levels: a System/360 channel does something like reading a block of data from tape, while a PIO deals with individual up and down transitions on a line.
- bogomipz 5y agoIs this I/O channel architecture basically the same thing that makes mainframes so reliable even today? Might you or anyone else have any recommendations on mainframe I/O architecture evolution? I've tried to look it up in the past but without much success.
- chiph 5y agoThey are reliable through exhaustive testing of components, plus error-checking, plus redundancy (on some models). What channels get you is high I/O throughput. Especially in 80 character long records The machines (were and maybe still) are optimized to run COBOL, and what COBOL programs are commonly used for is operations like "Read a record, use columns 50 thru 59 to add to a total, repeat until EOF. Then print the total formatted as $**9,999.99" So the faster you can read records in, the better. Channels also did buffering when reading from devices like tape drives that did not like to start/stop their motion a lot. So the channel controller would tell the tape drive to read "a few" records and hold them for when the CPU asked for them. Mechanically, the tape reel would slow to a stop and a vacuum column would hold excess tape. The inertia of the reel did not allow for sudden starts/stops and if you tried you'd snap the tape. Curious Marc has a video: https://www.youtube.com/watch?v=7Lh4CMz_Z6M https://www.youtube.com/watch?v=7Lh4CMz_Z6M
- j_not_j 5y agoI wonder how the designers "knew" or hoped that the S/360 instruction set was achievable with that hardware/microcode design. Was it sufficient to cover all the instructions? The floating point portion seems like a likely risk. (Perhaps, more likely, the microcode drafts were co-written with hardware designs so they had less risk of a failure.)
- kens 5y agoThere's not really much risk. Because the S/360 was built from gates (SLT) rather than an integrated circuit, they could easily change the design if needed. (Even at a customer site!) It's not like a microprocessor chip that would require an expensive re-spin if something went wrong. So if they discovered they needed another micro-operation, they could just add it to the design. There are a few micro-operations that seem rather contrived, as if they realized it would really help performance to do this random combination of things.
- ziggus 5y agoBased on your (amazing) write-up, it seems like there was some sort of abstraction layer between a programmer writing in some kind of assembler and the actual CPU registers, is that correct?
- rjsw 5y agoThat is what microcode is for.
- colejohnson66 5y agoThat's what microcode is - an abstraction between machine code ("compiled" assembly) and control lines. The machine code is a CISC-like system that controls the microcode ROM/PLA which outputs either the internal RISC-like opcodes or the actual control lines.
- ziggus 5y agoThanks for the explanation. It's obviously been a long time since my machine code CS classes.
- r2sk5t 5y agoThanks for the blog post @picture. I worked on emulating the 360/370 instruction set using both an 8086 processor and a bit slice processor, when I was at Big Blue. The hardware and microcode development was insanely complex. This ended up being a shipped product called the XT and AT/370. https://en.wikipedia.org/wiki/PC-based_IBM_mainframe-compatible_systems https://en.wikipedia.org/wiki/PC-based_IBM_mainframe-compati...
- klelatti 5y agoThis is intriguing! What was the market for this machine? The PC and mainframe seem so different in their likely users.
- r2sk5t 5y agoMainframes are expensive and busy running production workloads. Offloading development and test to franken-PC's running 370 emulation had large cost saving potential. Plus, back in those days the IBM sales organization had a ton of power and could sell anything, the margins were always fantastic, and large numbers of sales reps earned more than the IBM CEO.
- klelatti 5y agoThanks! This makes a lot of sense. I worked for an IBM marketing unit in the 1980s so saw some of this up close!
- rbanffy 5y agoWhy suffer with MS-DOS when you could run VM/370 on your PC? IBM still sells emulators, now software-based, for companies that do mainframe software development. And they are freakishly expensive.
- Koshkin 5y agoFor the rest of us, there is Hercules! http://www.hercules-390.org/ http://www.hercules-390.org/
- tablespoon 5y ago> The IBM System/360 was a groundbreaking family of mainframe computers announced on April 7, 1964. System/360 was an extremely risky "bet-the-company" project for IBM, costing over $5 billion $5 billion in 2022 dollars or 1960s dollars?
- TedDoesntTalk 5y ago$5 billion in 1964 is about $45 trillion today. so, uh, yeah, 2022 dollars. https://www.in2013dollars.com/us/inflation/1964?amount=5000000000 https://www.in2013dollars.com/us/inflation/1964?amount=50000...
- amock 5y agoThe link your provided says $45 billion, not trillion.
- jecel 5y agoThe link says "$5,000,000,000 in 1964 is equivalent in purchasing power to about $44,968,064,516.13 today", which is $45 billion. Considering that the entire revenue for IBM in 1964 was $3.2 billion (1964 money), spending $5 billion on a project (over several years) was certainly a "bet the company" move.
- kingcharles 5y ago$45Tn is half the GDP of the whole planet (~$80Tn). I should know. Law enforcement were trying to indict me for $100Tn in currency fraud. I wanted to point out to them that this was higher than the GDP of Planet Earth, but I don't think they would have understood what GDP is.
- kanzenryu2 5y agoStory time?
- kingcharles 5y agoI was arrested. In my wallet was a $100Tn bill. Law enforcement send it to the State crime lab for analysis. Lab report says it is the most sophisticated piece of currency forgery they have ever seen. It checks 8 of their 9 boxes (it's been eight years since I saw the report) which I remember some were: correct paper, correct ink, metal strip, hologram, dots.. I don't remember the rest. I can't remember which one it failed. They finally decided not to follow-through on the prosecution for it. I guess someone eventually realized that: a) It was a real bank note, not a forgery b) It wasn't US currency as they had believed, but a note from Zimbabwe: https://www.banknoteworld.com/zimbabwe-currency/100-trillion-zimbabwe-dollars/ https://www.banknoteworld.com/zimbabwe-currency/100-trillion...
- samteeeee 5y agoI'm reading the excellent "The soul of a new machine" by Tracy Kidder right now. Your post has helped visualize the types of computers mentioned in the book.
- sho_hn 5y agoWonderful book. There's a snippet on conflicts between engineers and how engineers approach them - often not in the best way - that I've referenced so many times in my career.
- beachy 5y agoGreat book. I relentlessly and boringly quote the commandment from the CEO in that book that there was to be "no mode switch". I have found this to be a great rule of thumb in software too. Engineers often first instinct is to add "advanced mode", "legacy mode" or whatever. Often with no idea how much trouble having two modes of behaviour will cause downstream in training, understanding, compatibility etc.
- kens 5y agoThat's a great book. Keep in mind that the computer in "The soul of a new machine" (Data General Eclipse MV/8000) was a minicomputer, not a mainframe, and in 1980, not 1964. So there are a lot of differences as well as similarities. (I'm not trying to be pedantic, but just want to make sure you don't get the wrong impression of the Eclipse.)
- beachy 5y agoThese articles are fantastic. When I started at IBM in the early 80s there were plenty of 360s still around, but the new hotness was 370, so all the training courses, installations, management focus were for the new machines. The cool kids went to London, Tokyo and Poughkeepsie for training courses - the rest of us toughed it out fixing these venerable old beasts using oscilloscopes and mountains of manuals. There were no decent helicopter level documentation like Ken's available. Having access to articles like this back then would have been life changing.
- hprotagonist 5y ago> The cool kids went to London, Tokyo and Poughkeepsie for training courses I grew up around the city on that list that ain’t like the others. The idea that “cool kids” would be sent there, on _purpose_, and as some kind of _reward__, just made me laugh and laugh even though i know why it’s true and that it’s not a joke.
- TedDoesntTalk 5y agoPoughkeepsie itself may not be a great place, but the surrounding area is beautiful.
- ncmncm 5y agoCity government is still dysfunctional.
- hnthrowaway0315 5y ago> the rest of us toughed it out fixing these venerable old beasts using oscilloscopes and mountains of manuals. Actually this looks very, very cool to me. It resembles taming mythological beasts . I wish I had the chance to work on such projects.
- beachy 5y agoWith hindsight it was sure exciting but not really fun. There was huge pressure to get the machine back up again. An outage might mean that an entire bank branch was down, or that payroll could not be run. The CEO could be looking over your shoulder. That kind of pressure saps the fun quickly. Also the bugs could be very, very hard hard to find. In room-sized machines like 3033 for example, there are many thousands of signal leads (trileads) that were many yards long. A slightly bad connection could cause electrical ringing and highly intermittent errors that were excruciatingly difficult to reproduce, let alone to find and fix. Intermittent errors were common and brutal. You could think you'd fixed it, only to have another crash a few hours later. A common debugging technique was to swap cards around and see if the bug moved. Just doing that often introduced new bugs. In latter years the role of the on-site engineer was reduced, and often you would be a lacky for some remote engineer back in the plant where the machine was built.
- BXLE_1-1-BitIs1 5y agoOne of the stories I heard about the 50 is that if you threw it into the ocean it would sink intermittently. Anyway the 50 was the first computer I was paid to program on. Many were bought to run in 7070 emulator mode as the 50 was the smallest / cheapest machine that could run 7070. Happily I was on the 360 side. At another shop, we had 2 50s, one running DOS and the other OS with HASP. One day a student was having trouble with the Test and Set (superceded by Compare and Swap) instruction not setting the condition code properly. I wrote a short diagnostic program to exercise TS, store the resulting condition codes and dump. The instruction was not performing correctly. I showed the dump to the non IBM engineers who hemmed and hawed. A couple days later one phoned me up to say that TS wasn't setting the condition code correctly - exactly what I had been telling him. They fixed the problem. The next day I got a phone call that HASP wasn't starting on the other 50. Took a dump and found HASP was in a wait just after TS. The engineers had swapped microcode cards between the two machines. My manager was not pleased with the engineers.
- bogomipz 5y ago>"One of the stories I heard about the 50 is that if you threw it into the ocean it would sink intermittently." That's a good joke. Thanks for sharing. >"I wrote a short diagnostic program to exercise TS, store the resulting condition codes and dump." I had to look up HASP, which is an interesting bit of history. What were the IO devices that you spooled from?
- BXLE_1-1-BitIs1 5y agoHASP was rebranded JES2 when 370 came along. IO devices were principally card readers and printers, but the neat trick was the internal reader. Remember that JES2/3 were running batch jobs. TSO was shiny new.
- ncmncm 5y agoOK, I need the joke explained. Is this a pun on SYNC?
- 5y ago
- Koshkin 5y agoOne of the best intros to microprogramming: https://people.cs.clemson.edu/~mark/uprog.html https://people.cs.clemson.edu/~mark/uprog.html
- ch_123 5y ago> Microcode can be implemented in a variety of ways. Many computers use "vertical microcode", where a microcode instruction is similar to a machine instruction, just less complicated. The System/360 designs, on the other hand, used "horizontal microcode", with complex, wide instructions of up to 100 bits, depending on the model. I'm pretty sure that horizontal microcoding is more common, at least in modern CPU design. On x86, the micro-ops are generally wider than the machine code instructions.
- colejohnson66 5y agoThey're wider, but that's just because one "word" holds the whole instruction, instead of multiple bytes. In fact, reverse engineering efforts[0] (and the "RISC86" patent[1]) make clear that they're actually "vertical". Intel Goldmont (from [0]) has entries that are 176 bits each, but that's actually three (distinct) 48 bit uops and a 30 bit "sequence word". Horizontal microcode is much simpler for in-order processors, but my understanding of this stuff seems like they wouldn't work well with the superscalar processors of today. Gating the hundreds of control lines seems (to me) like more effort than gating a few dozen bits of a uop. [0]: https://github.com/chip-red-pill/uCodeDisasm https://github.com/chip-red-pill/uCodeDisasm [1]: https://patents.google.com/patent/US5926642A https://patents.google.com/patent/US5926642A
- rbanffy 5y agoIn college we were tasked with designing a CPU. Mine was a stack oriented (started register based and retained the registers but most ops were on the stack) that used a very large microcode word, one per clock cycle of the instruction being executed. In the end, I was saving bits from the control word and doing "ready" signals between the blocks so that the microcode didn't need to drive everything. In theory, it could do more than one thing in a clock cycle if the stars aligned just right and there would be no dependencies. No instruction used the feature in the end, because the deadline was too close. Wish I had the time to implement it. OTOH, I'm glad I never had to debug all the analog glitches and timing bugs that design certainly would show when it colided with reality
- monocasa 5y ago
- bogomipz 5y agoWow. What a fantastic post! I had question about the following passage: >"As you can see, the CPU performs many activities in parallel for one microinstruction, which increases the computer's performance." Would this microcode also be considered one of the first forms of ILP(instruction level parallelism) then?
- kens 5y agoIt's still executing a single instruction, so it's not ILP. Multiple functional units can do different parts of the instruction in parallel, but instructions are still sequential. The System/360 Model 91, a more advanced model, used instruction level parallelism for higher perormance.
- kragen 5y agoI forget, did Stretch have ILP?
- PaulHoule 5y agoI remember how the microcomputers of the 1980s (Apple ][, C64, TRS-80 Coco) were mainly dead ends. There were possible paths forward (6502 to the 24-bit 65C816, Z80 to the eZ80, ...) and second generation machines like the Apple 2gs, C128 and Coco 3. They were all dead ends, as were the machines based on the 68k series. There was a strange time in the early 1980s that every computer manufacturer feared obsolescence but machines like the Apple ][ kept selling because there wasn't a path for continuous improvement. Microcomputers at the time were all built around the video display system so you couldn't drop in a 10% faster CPU. Intel didn't have a clear plan for the 8086, in fact they expected the future to be the doomed i860 or iAPX 432. They stumbled into the 24-bit 80286 which was "brain damaged" in terms of it's memory protection model but just plain kicked ass in performance. I bought a 12 MHz 286 machine in 1987 which was by far the best computer I ever owned, powerful enough that I could emulate the Z80 and use it as a software development workstation for CP/M. That was when the PC crushed everything else. I could afford a 32-bit 486 and run Linux on it in 1993, and 10 years later I upgraded to a 64-bit version of the architecture. Like the 360/370/390/z-Architecture the Microsoft-centric PC maintained instruction set compatibility over the years. Yet, that's not the only path to a sustainable computing brand. Apple started the mac on the doomed 68k, switched to Power PC, switched again to Intel, and switched again to ARM. Microsoft tested the waters for such a migration multiple times but it's never amounted to much. Sometimes I dream of a world where Apple didn't abandon the ][, released the //gs a few years earlier than it did, and eventually came out with a 32-bit extension of the 6502 architecture.
- tasty_freeze 5y agoIIRC, the IBM/360 exception model is such that if an exception occurs, there instruction in question cannot have any side effects -- no partial execution. That leads to a clean exception model but can cause performance pain. Say the instruction is a string copy. The instruction itself might cross a page boundary, and the source might cross a page boundary (perhaps more than one), and the destination can also cross a page boundary. The microcode can't just begin processing the instruction, then hit a page fault, fix up the page mapping, and continue. Instead it must check that each possible read or write will not cause a page fault and only then proceed with the entire sequence.
- monocasa 5y agoInterestingly this is somewhere that x86 is cleaner. For instance on x86, the 'memcpy' instruction rep movsb is architecturally defined as simply not incrementing the program counter until it meets it's end condition. It can perform one move decrement remain count, increment addresses, check for exceptions and interrupts then repeat. Cortex M also exposes how far you are into a ldm/stm for recoverable exceptions as well.
- graycat 5y agoWhy are the official IBM 360/370 publications Principles of Operation not sufficient to write emulators?
- kens 5y agoThat's sufficient to write an emulator at the instruction set level. However, I want to light the console lights accurately. This requires implementation of the internal registers and the microcode, which varies from model to model and isn't documented in the Principles of Operation. For anyone who is curious, the 360 Principles of Operation are at bitsavers: http://bitsavers.org/pdf/ibm/360/princOps/ http://bitsavers.org/pdf/ibm/360/princOps/
- hungryforcodes 5y agoIt would be cool if we could get the 360 instruction set / architecture to run on something small like the ESP32. I'd definitely try that out. Similar things have been done with CP/M for example. https://hackaday.io/project/20552-cpm-8266 https://hackaday.io/project/20552-cpm-8266
- BXLE_1-1-BitIs1 5y agoAs for emulators, Platform Solutions developed a system that could run IBM mainframe Operating Systems and applications at a very competitive price. I recall there was a middle layer of microcode such that the computer could run whichever instruction set you wanted. IBM smothered the lawsuit by buying Platform Solutions and persuaded various antitrust authorities to look the other way. https://applecon.com/case-studies/ibm-v-platform-solutions-and-t3-technologies/ https://applecon.com/case-studies/ibm-v-platform-solutions-a... Up to that point many ISVs were using mainframe emulators such as the Flex system, but that license was killed off and ISVs were either pushed to remoting to a mainframe or going to z/PDT.