6 ms·

IBM Z systems are always fascinating to a computer engineer. I was fortunate at my first job to be able to access a z10 brand new, and was tasked with installin

IBM Z systems are always fascinating to a computer engineer. I was fortunate at my first job to be able to access a z10 brand new, and was tasked with installing, configuring, and trying out several POCs with z/OS, DB2 on z/OS, WebSphere Application server on z/OS, CICS transaction server, z/VM operating system, running 1000's of linux images on top of z/VM, making them communicate with each other and z/OS using Hipersockets. It was pure fun. The cool moments I distinctly remember were runing the z/OS installation binaries from tape drives, and swapping in and out - DASDs(Hard disks), and perhaps the coolest was being able to hot plug in/out CPU blades from the cabinet while the system is live. It's pity that thing is so expensive, and out of reach of the common folks. For those not in the know, and curious. Based on my knowledge from 2011, CPs are the general purpose CPUs that run z/OS - the main OS companies typically run on System Z. There were special CPUs(may be microcoded so), called IFL that were allowed to run z/VM and Linux. I don't know how things have evolved in the last 7 years :-)

phs2501 8y ago
Pretty sure it's more that IFLs are not allowed to run z/OS workloads. I think you can run Linux on CPs just fine, it's just more expensive.
mikehollinger 8y ago
So I joined IBM in 2005 working on firmware for a project called eCLipz, which eventually produced the POWER6 family of systems. We've just shipped POWER9 which is in the #1 and #2 supercomputers in the world, Summit and Sierra. [1] The "i/p/z" part of the project name is important, because it represents: i - the lineage of the AS/400, which was a 'minicomputer' design from the 80's designed for multiple users, which came up with the idea of "wizards" for administration tasks before there was even a name for a "wizard." The "i" means "integrated," since IBM i is meant to require very little administration. [2] [4] p - the lineage of the RS/6000, which begat AIX UNIX. The "p" indicated "performance." The box I worked on shipped at 5 GHz in 2008 (IIRC). z - the lineage of the System/360 mainframe which the above article discusses. The "z" indicated "zero downtime." The control structure and power and cooling infrastructure for the systems used common elements with that project. The hardware and firmware underneath the operating systems for i and p converged into Power systems, which eventually produced a variety of projects in the open source community (OpenCAPI, OpenPOWER itself, OpenBMC), and of course runs bi-endian to support things like little-endian Ubuntu or the classic Big-endian AIX or i/OS on the same processor core. Interesting trivia - In IBM systems (Mainframe or Power), the cold boot of the processor from an unpowered state is called an "Initial Program Load," or IPL. The IPL lingo dates back to System/360 (see its Wikipedia), and survives to this day in really interesting places in the wild. Take the OpenBMC project, which aims to create a 100% open source software stack for those sideband management processors found in servers (and increasingly in other devices too). This github bug from a few months ago which complains about a bug causing "IPL" problems. [3] ;-) [1] https://www.theverge.com/circuitbreaker/2018/11/12/18087470/ibm-summit-sierra-supercomputer-us-fastest https://www.theverge.com/circuitbreaker/2018/11/12/18087470/... [2] https://en.wikipedia.org/wiki/IBM_i https://en.wikipedia.org/wiki/IBM_i [3] https://github.com/openbmc/openbmc/issues/2831 https://github.com/openbmc/openbmc/issues/2831 [4] edit - fixed "70's" to "80's" per https://news.ycombinator.com/item?id=18494866 https://news.ycombinator.com/item?id=18494866 ;-)
- protomyth 8y ago
  I do love the lingo from these machines. I remember encountering the need to "vary off" an interface. The weirdest was the being told I needed to "wand on" the system. I guess the origin was from the AS/400 when they were pushing cards with a barcode as ids, so you would scan your card with a optical wand (pen) to login to your station. Years later I still hear "wand on".
- rbanffy 8y ago
  I have friends who still say their program "abended". Just a minor nitpick - the green screens we see today are also often descendants of the 3270 family as well as the 5250's that started life with the System/34 and found their way into the AS/400 and iSeries. I had to nitpick because the 3270 is where this beautiful (shameless plug) font originated: https://github.com/rbanffy/3270font https://github.com/rbanffy/3270font
  - mikehollinger 8y ago
    Ha! Fair enough. Lessee if I can edit...
  - rhizome 8y ago
    ABEND was also used in Novell Netware, I wouldn't be surprised if that's the most recent actual incarnation (rather than a reference, as in a game).
  - NikolaNovak 8y ago
    Are... are there people who don't? :O I haven't ever really worked on Mainframe as such, but I _am_ in the IBM/ERP world, and, well, a program Abends, that's just what it does when it goes South. Not even sure what else to call it - failed? Errorred out? ...Segfaulted its guts on the ground? :-> (edit: I suppose I should also fess up that we call virtual machines "LPARs" and physical machines "frames" :P )
    - rbanffy 8y ago
      It took me a long time to accept Docker's use of the word "container". To me, "virtual machine" has a distinctive Smalltalk 80 feel. And, of course, it's a bit underwhelming in comparison.
jhallenworld 8y ago
They used to provide a free z Linux guest OS available to anyone. I used it to verify that my open source projects worked on Z. It was RedHat 7.3 I think.
segmondy 8y ago
I can run a kubernetes cluster, with 1000 of linux images, have them communicate with each other. Hot swap in and out nodes. Such a thing is not out of reach of common folks if you change your mental model of what a computer is. "The network is the computer"
- andrewstuart2 8y ago
  You can also run Kubernetes on a mainframe now, with ZLinux and the hundreds of precompiled s390X (IBM CPU architecture) docker images: https://hub.docker.com/u/ibmcom/ https://hub.docker.com/u/ibmcom/. This is attractive if you happen to already be a large company and locked into mainframe contracts. Sure, you don't need a mainframe to do this, but it's also nice to not need to throw away your mainframe and buy a bunch of commodity hardware to do this. Options are good for everybody. :-)
- jstapels 8y ago
  But that comes at the expense of complicated software. Sure, once you've gone through the effort of procuring all that hardware, configuring it to run Kubernetes, reconfiguring your software to run in containers... you're good to go. Mainframes allow you to easily scale your software vertically (but at the expense of complicated hardware). That might seem silly in today's world, but a lot of the software running on Z was written decades ago and the risk of rewriting it is extremely high.
  - CarVac 8y ago
    How... does it work then? Was the programming model for mainframes made to be scalable from the beginning?
    - santoshalper 8y ago
      The vast majority of mainframe software, especially older software was batch processing oriented - often millions of discreet financial transactions - that can easily be scaled horizontally across multiple CPUs or processing engines.
    - 8y ago
pjmlp 8y ago
Also has stuff like the Language Environment, bytecode based distribution binaries, later followed up by IBM OS/400 as well (now IBM i). https://www.ibm.com/support/knowledgecenter/en/SSGMCP_4.2.0/com.ibm.cics.ts.applicationprogramming.doc/topics/dfhp3_langenv_oview.html https://www.ibm.com/support/knowledgecenter/en/SSGMCP_4.2.0/... Basically JME, DEX, MSIL, watchOS bitcode, are modern platforms catching up with mainframes.
technion 8y ago
At some point in history, I had the misfortune of working with Stratus hardware. Not quite where you describe the Z system, but it did operate with hot swappable components, and multiple OS's running in a "fault tolerant" manner. We hit show stopping firmware bugs that managed to take down every OS. The end result was that the three standalone Dells on top of it had better uptime. I always wondered how the mainframe world ever managed this. When you market is so small (relative to x86) every case is an edge case. Surely these extremely large and expensive devices still had bugs that transcended that redundancy.
neverartful 8y ago
The bulk of my mainframe exposure was with z/VM, but you're right, those systems are fascinating. I've never understood why IBM doesn't provide hobbyist licenses of z/OS and z/VM for individuals to learn on. Of course, the software would be run on Hercules [1] emulator and not a real mainframe. Even so, it would be a great avenue for people to learn about the mainframe software environment and maybe even help with the mainframe skills shortage. [1] https://en.wikipedia.org/wiki/Hercules_(emulator) https://en.wikipedia.org/wiki/Hercules_(emulator)
- PhantomGremlin 8y ago
  Things were different in the 1970s. And perhaps that explains why IBM is more secretive now. The Itel AS/5 was a clone of the IBM System 370 Model 158, a very popular mainframe for its time. How was Itel able to clone it? Simple. Just use the hardware schematics! http://www.silogic.com/NAS/NAS.html http://www.silogic.com/NAS/NAS.html ALD's were Automated Logic Diagrams. IBM sold these. They were complete schematics for the S-370. EXSYSCO basically duplicated the logic of the S-370. I had personal experience with the AS/5. It came supplied with IBM schematics and microcode listings. These were kept in giant binders in the machine room. When the machine booted up, a Z80 based processor that Itel designed was used to load IBM microcode into the clone mainframe. Oh, and the application software that ran on that 1970s machine will still run on IBMs latest mainframes. When IBM introduced the PC in 1981, hardware schematics and BIOS listings were also available for purchase. Times have changed!
neverartful 8y ago
Here's a 100,000 foot view of mainframes (based on when I was working with them and on my limited view). The mainframes that I worked with (not sure if this is true with all) had something called Capacity Upgrade on Demand. This allowed the data center to temporarily turn on extra processors/engines to handle unusually high processing demands. After the crunch is over, the extra processors can be turned off. I'm not sure how the money aspect of this works. What the x86 world thinks of as a bare-metal box/chassis is what looks like a full sized rack, except it's enclosed in a cabinet. This bare-metal box is carved up into LPARs (logical partitions) using software called PR/SM [1]. I've read a number of times that PR/SM is a stripped-down version of z/VM. I don't know if that's true or not. IBM has multiple operating systems for mainframes. The most well known is z/OS (formerly known as MVS). There are others like z/TPF and z/VSE (maybe even more?). There is also z/VM. JCL (Job Control Language) is used by z/OS (possibly z/VSE too, idk) for running "jobs". This is essentially like shell scripts. The job is submitted and then queued for execution. After execution, the results are made available in an output queue. From Linux on x86 point of view, it's like all program output is being redirected to a file. There are a number of z/OS add-ons that are typically licensed to run on z/OS. One is TSO (Time Sharing Option). I don't know if TSO is separately licensed or not. Basically, it makes a batch environment almost appear to be interactive. Another component commonly licensed is ISPF (I don't know what it stands for). ISPF is basically a menu-based system that is configurable and I think it comes with some helpful built-in utilities. z/VM's history goes back to the late 60's and early 70's. Supposedly, one of the primary reasons it was developed was to give z/OS system programmers a sandbox for developing and testing changes. Apparently, some mainframe shops also ran MVS and other operating systems as guests under VM in production (this was probably in 80's). This wasn't the only use of VM though. There was CMS that is a single-user OS that was typically used for end-user business users. Since VM with CMS guests were so common, one would often just see "VM/CMS" written. From within CMS, one could issue CMS commands as well as CP commands. CP is "Control Program" and refers to base z/VM (hypervisor). One can also run z/VM as a guest under z/VM. This is referred to as 2nd-level VM and is used for testing/evaluating new versions of z/VM before installing them into production. When Linux hit the scene, versions of Linux were built for mainframes and Linux was added to the list of guest OSs that z/VM supported. To be clear, this is real Linux -- not emulated. It's normal Linux, but compiled to mainframe opcodes. Once you get access to a Linux guest on z/VM, it's really hard to tell that it's different from x86. It's not weird or different. Any linux user would feel just as much as home with Linux under z/VM as with x86. Parent posted discussion of CPs and IFLs. Back when I was on the scene, there were also zIIPs and zAPPs. I think zIIP was for encryption or compression (maybe both). I believe zAPP was primarily for running Java workloads. It's been a long time, so my recollection may not be 100% accurate. Probably the biggest shock to a newcomer of z/VM and z/OS is that terminal sessions are block-oriented (3270) and not character-oriented. With 3270 session, you type whatever you need to on a screen and then when a special attention key is pressed, your screen information is passed to the mainframe to process whatever you typed. No vi editing with 3270. Text editing on 3270 is with line-oriented editors like XEdit. The scripting language for z/OS and z/VM is REXX. It's as high level as Python and is really a nice language once you get used to its quirks. Here ends my mainframe brain dump. Can't guarantee that it's all correct or even still relevant, but hopefully gives a little bit of the feel for the environment. [1] https://en.wikipedia.org/wiki/PR/SM https://en.wikipedia.org/wiki/PR/SM

phs2501 8y ago

Pretty sure it's more that IFLs are not allowed to run z/OS workloads. I think you can run Linux on CPs just fine, it's just more expensive.

mikehollinger 8y ago

So I joined IBM in 2005 working on firmware for a project called eCLipz, which eventually produced the POWER6 family of systems. We've just shipped POWER9 which is in the #1 and #2 supercomputers in the world, Summit and Sierra. [1] The "i/p/z" part of the project name is important, because it represents: i - the lineage of the AS/400, which was a 'minicomputer' design from the 80's designed for multiple users, which came up with the idea of "wizards" for administration tasks before there was even a name for a "wizard." The "i" means "integrated," since IBM i is meant to require very little administration. [2] [4] p - the lineage of the RS/6000, which begat AIX UNIX. The "p" indicated "performance." The box I worked on shipped at 5 GHz in 2008 (IIRC). z - the lineage of the System/360 mainframe which the above article discusses. The "z" indicated "zero downtime." The control structure and power and cooling infrastructure for the systems used common elements with that project. The hardware and firmware underneath the operating systems for i and p converged into Power systems, which eventually produced a variety of projects in the open source community (OpenCAPI, OpenPOWER itself, OpenBMC), and of course runs bi-endian to support things like little-endian Ubuntu or the classic Big-endian AIX or i/OS on the same processor core. Interesting trivia - In IBM systems (Mainframe or Power), the cold boot of the processor from an unpowered state is called an "Initial Program Load," or IPL. The IPL lingo dates back to System/360 (see its Wikipedia), and survives to this day in really interesting places in the wild. Take the OpenBMC project, which aims to create a 100% open source software stack for those sideband management processors found in servers (and increasingly in other devices too). This github bug from a few months ago which complains about a bug causing "IPL" problems. [3] ;-) [1] https://www.theverge.com/circuitbreaker/2018/11/12/18087470/ibm-summit-sierra-supercomputer-us-fastest https://www.theverge.com/circuitbreaker/2018/11/12/18087470/... [2] https://en.wikipedia.org/wiki/IBM_i https://en.wikipedia.org/wiki/IBM_i [3] https://github.com/openbmc/openbmc/issues/2831 https://github.com/openbmc/openbmc/issues/2831 [4] edit - fixed "70's" to "80's" per https://news.ycombinator.com/item?id=18494866 https://news.ycombinator.com/item?id=18494866 ;-)

protomyth 8y ago
I do love the lingo from these machines. I remember encountering the need to "vary off" an interface. The weirdest was the being told I needed to "wand on" the system. I guess the origin was from the AS/400 when they were pushing cards with a barcode as ids, so you would scan your card with a optical wand (pen) to login to your station. Years later I still hear "wand on".
rbanffy 8y ago
I have friends who still say their program "abended". Just a minor nitpick - the green screens we see today are also often descendants of the 3270 family as well as the 5250's that started life with the System/34 and found their way into the AS/400 and iSeries. I had to nitpick because the 3270 is where this beautiful (shameless plug) font originated: https://github.com/rbanffy/3270font https://github.com/rbanffy/3270font
- mikehollinger 8y ago
  Ha! Fair enough. Lessee if I can edit...
- rhizome 8y ago
  ABEND was also used in Novell Netware, I wouldn't be surprised if that's the most recent actual incarnation (rather than a reference, as in a game).
- NikolaNovak 8y ago
  Are... are there people who don't? :O I haven't ever really worked on Mainframe as such, but I _am_ in the IBM/ERP world, and, well, a program Abends, that's just what it does when it goes South. Not even sure what else to call it - failed? Errorred out? ...Segfaulted its guts on the ground? :-> (edit: I suppose I should also fess up that we call virtual machines "LPARs" and physical machines "frames" :P )
  - rbanffy 8y ago
    It took me a long time to accept Docker's use of the word "container". To me, "virtual machine" has a distinctive Smalltalk 80 feel. And, of course, it's a bit underwhelming in comparison.

jhallenworld 8y ago

They used to provide a free z Linux guest OS available to anyone. I used it to verify that my open source projects worked on Z. It was RedHat 7.3 I think.

segmondy 8y ago

I can run a kubernetes cluster, with 1000 of linux images, have them communicate with each other. Hot swap in and out nodes. Such a thing is not out of reach of common folks if you change your mental model of what a computer is. "The network is the computer"

andrewstuart2 8y ago
You can also run Kubernetes on a mainframe now, with ZLinux and the hundreds of precompiled s390X (IBM CPU architecture) docker images: https://hub.docker.com/u/ibmcom/ https://hub.docker.com/u/ibmcom/. This is attractive if you happen to already be a large company and locked into mainframe contracts. Sure, you don't need a mainframe to do this, but it's also nice to not need to throw away your mainframe and buy a bunch of commodity hardware to do this. Options are good for everybody. :-)
jstapels 8y ago
But that comes at the expense of complicated software. Sure, once you've gone through the effort of procuring all that hardware, configuring it to run Kubernetes, reconfiguring your software to run in containers... you're good to go. Mainframes allow you to easily scale your software vertically (but at the expense of complicated hardware). That might seem silly in today's world, but a lot of the software running on Z was written decades ago and the risk of rewriting it is extremely high.
- CarVac 8y ago
  How... does it work then? Was the programming model for mainframes made to be scalable from the beginning?
  - santoshalper 8y ago
    The vast majority of mainframe software, especially older software was batch processing oriented - often millions of discreet financial transactions - that can easily be scaled horizontally across multiple CPUs or processing engines.
  - 8y ago

pjmlp 8y ago

Also has stuff like the Language Environment, bytecode based distribution binaries, later followed up by IBM OS/400 as well (now IBM i). https://www.ibm.com/support/knowledgecenter/en/SSGMCP_4.2.0/com.ibm.cics.ts.applicationprogramming.doc/topics/dfhp3_langenv_oview.html https://www.ibm.com/support/knowledgecenter/en/SSGMCP_4.2.0/... Basically JME, DEX, MSIL, watchOS bitcode, are modern platforms catching up with mainframes.

technion 8y ago

At some point in history, I had the misfortune of working with Stratus hardware. Not quite where you describe the Z system, but it did operate with hot swappable components, and multiple OS's running in a "fault tolerant" manner. We hit show stopping firmware bugs that managed to take down every OS. The end result was that the three standalone Dells on top of it had better uptime. I always wondered how the mainframe world ever managed this. When you market is so small (relative to x86) every case is an edge case. Surely these extremely large and expensive devices still had bugs that transcended that redundancy.

neverartful 8y ago

The bulk of my mainframe exposure was with z/VM, but you're right, those systems are fascinating. I've never understood why IBM doesn't provide hobbyist licenses of z/OS and z/VM for individuals to learn on. Of course, the software would be run on Hercules [1] emulator and not a real mainframe. Even so, it would be a great avenue for people to learn about the mainframe software environment and maybe even help with the mainframe skills shortage. [1] https://en.wikipedia.org/wiki/Hercules_(emulator) https://en.wikipedia.org/wiki/Hercules_(emulator)

PhantomGremlin 8y ago
Things were different in the 1970s. And perhaps that explains why IBM is more secretive now. The Itel AS/5 was a clone of the IBM System 370 Model 158, a very popular mainframe for its time. How was Itel able to clone it? Simple. Just use the hardware schematics! http://www.silogic.com/NAS/NAS.html http://www.silogic.com/NAS/NAS.html ALD's were Automated Logic Diagrams. IBM sold these. They were complete schematics for the S-370. EXSYSCO basically duplicated the logic of the S-370. I had personal experience with the AS/5. It came supplied with IBM schematics and microcode listings. These were kept in giant binders in the machine room. When the machine booted up, a Z80 based processor that Itel designed was used to load IBM microcode into the clone mainframe. Oh, and the application software that ran on that 1970s machine will still run on IBMs latest mainframes. When IBM introduced the PC in 1981, hardware schematics and BIOS listings were also available for purchase. Times have changed!

neverartful 8y ago

Here's a 100,000 foot view of mainframes (based on when I was working with them and on my limited view). The mainframes that I worked with (not sure if this is true with all) had something called Capacity Upgrade on Demand. This allowed the data center to temporarily turn on extra processors/engines to handle unusually high processing demands. After the crunch is over, the extra processors can be turned off. I'm not sure how the money aspect of this works. What the x86 world thinks of as a bare-metal box/chassis is what looks like a full sized rack, except it's enclosed in a cabinet. This bare-metal box is carved up into LPARs (logical partitions) using software called PR/SM [1]. I've read a number of times that PR/SM is a stripped-down version of z/VM. I don't know if that's true or not. IBM has multiple operating systems for mainframes. The most well known is z/OS (formerly known as MVS). There are others like z/TPF and z/VSE (maybe even more?). There is also z/VM. JCL (Job Control Language) is used by z/OS (possibly z/VSE too, idk) for running "jobs". This is essentially like shell scripts. The job is submitted and then queued for execution. After execution, the results are made available in an output queue. From Linux on x86 point of view, it's like all program output is being redirected to a file. There are a number of z/OS add-ons that are typically licensed to run on z/OS. One is TSO (Time Sharing Option). I don't know if TSO is separately licensed or not. Basically, it makes a batch environment almost appear to be interactive. Another component commonly licensed is ISPF (I don't know what it stands for). ISPF is basically a menu-based system that is configurable and I think it comes with some helpful built-in utilities. z/VM's history goes back to the late 60's and early 70's. Supposedly, one of the primary reasons it was developed was to give z/OS system programmers a sandbox for developing and testing changes. Apparently, some mainframe shops also ran MVS and other operating systems as guests under VM in production (this was probably in 80's). This wasn't the only use of VM though. There was CMS that is a single-user OS that was typically used for end-user business users. Since VM with CMS guests were so common, one would often just see "VM/CMS" written. From within CMS, one could issue CMS commands as well as CP commands. CP is "Control Program" and refers to base z/VM (hypervisor). One can also run z/VM as a guest under z/VM. This is referred to as 2nd-level VM and is used for testing/evaluating new versions of z/VM before installing them into production. When Linux hit the scene, versions of Linux were built for mainframes and Linux was added to the list of guest OSs that z/VM supported. To be clear, this is real Linux -- not emulated. It's normal Linux, but compiled to mainframe opcodes. Once you get access to a Linux guest on z/VM, it's really hard to tell that it's different from x86. It's not weird or different. Any linux user would feel just as much as home with Linux under z/VM as with x86. Parent posted discussion of CPs and IFLs. Back when I was on the scene, there were also zIIPs and zAPPs. I think zIIP was for encryption or compression (maybe both). I believe zAPP was primarily for running Java workloads. It's been a long time, so my recollection may not be 100% accurate. Probably the biggest shock to a newcomer of z/VM and z/OS is that terminal sessions are block-oriented (3270) and not character-oriented. With 3270 session, you type whatever you need to on a screen and then when a special attention key is pressed, your screen information is passed to the mainframe to process whatever you typed. No vi editing with 3270. Text editing on 3270 is with line-oriented editors like XEdit. The scripting language for z/OS and z/VM is REXX. It's as high level as Python and is really a nice language once you get used to its quirks. Here ends my mainframe brain dump. Can't guarantee that it's all correct or even still relevant, but hopefully gives a little bit of the feel for the environment. [1] https://en.wikipedia.org/wiki/PR/SM https://en.wikipedia.org/wiki/PR/SM