8 ms·

ABCs of IBM z/OS System Programming Volume 1

75 points by emersonrsantos 11y ago

morenoh149 11y ago
Does anyone know what kinds of companies or applications make use of z/OS mainframes?
- PaulHoule 11y ago
  The State of New York uses them. Once a law was passed that the state needs to turn paperwork around in less than 48 hours they found the printing system was a bottleneck so they hired a greybeard friend of mine to build a new printer interface and device drivers for it. Cornell University also used to run HR and accounting on a 370 mainframe and people were happy, but then they switched to peoplesoft and now (i) they are not happy, and (ii) looking for a new accounting system.
- Spooky23 11y ago
  All US states, 90%+ of the big banks, retailers, airlines, brokerages, etc use mainframes. There are still places using non-IBM mainframes as well. Several state welfare systems use Unisys mainframes.
  - nickpsecurity 11y ago
    Unisys, Fujitsu, and Group Bull that I recall. Too bad Unisys bought Convergent's CTOS. Was the only thing close to Plan 9 that had big take-up in business. I'm not counting Inferno because we're talking mainframes and big business.
- ja27 11y ago
  Almost every Fortune 500 and most branches of the federal government. There are a few exceptions obviously. Microsoft, Apple, HP, etc. are unlikely to use IBM technologies internally. Facebook, Amazon, etc. are young enough to not have mainframes. You can go search for z/OS job openings and see everyone from Walmart to Lockheed Martin to Citibank.
  - deleted 11y ago
    [deleted]
  - nickpsecurity 11y ago
    Microsoft did get called out for using a mini-mainframe, the AS/400, back when they were pushing Windows Server. Reportedly they ran their entire business on one AS/400. Embarrassed, they got off the AS/400. Took about 20-23 Windows servers to match its performance and reliability. Told me what I needed to know. ;) But, yeah, they don't use it any more. Most mainframe users are legacy users. However, there's occasional conversions especially for consolidation. IBM's competitor, Group Bull, posted some case studies on that such as consolidating hundreds of desktops and servers onto 2 easily-managed mainframes with cost savings.
    - jussij 11y ago
      I do remember this event, when Microsoft was forced to admit they did not use any of their their own technology other than the desktop and it resulted in a big hit to the credibility of any the Microsoft server technologies. But the result actually meant Microsoft had to then eat their own dog food and in a strange way it forced Microsoft to greatly improve their server platforms to where they are today. So in a funny way it probably backfired on those early Microsoft detractors.
      - nickpsecurity 11y ago
        Never thought of it like that. Damn. Why did we have to run our mouths lol...
- kjs3 11y ago
  In my experience: Insurance, banking, credit card processing, airline reservations, payroll processing.
- mod 11y ago
  I worked for a company that does z/OS software and got to see their client list--it was pretty much a who's who of industry leaders in every industry, plus a bunch of colleges.
- mfisher87 11y ago
  Financial services are a good example. Lots of highly regulated legacy systems. For many big name banks and credit cards, it's pretty safe to assume that a Z/OS mainframe is doing some work with the data you've generated with every swipe.
  - zmonkeyz 11y ago
    Nationwide did a big linux on z consolidation a couple of years back.
- AnonNo15 11y ago
  Royal Bank of Canada for sure. Other banks from Big 5, probably too.
- surge 11y ago
  Every large financial institution, airline, or large company that does millions of transactions an hour that isn't competing with IBM or young enough to implement a different platform that can scale to that level. There is a z/OS security talk I saw that partially dispels the myths and prejudices against mainframes as legacy. In some cases z/OS is more modern than what you might find on your desktop and OSs like Linux are implementing features in its kernel that have been in zOS for years.
  - ghuntley 11y ago
    That would be Soldier of Fortran - https://www.youtube.com/watch?v=Xfl4spvM5DI https://www.youtube.com/watch?v=Xfl4spvM5DI
    - surge 11y ago
      Right I love his stuff. I was thinking of his talk at BSides LV 2014 which he gave the same talk but he also had talks this year also at BSides and DefCon. I also wanted to mention that where I've worked and many other places there is a crisis of brain drain at companies as the mainframe experts who are highly paid retire out and there are programs to recruit young people who are more attracted to other platforms. There is a lot of pay and job security in the field. I know of one situation where a guy past retirement is paid full salary to work 1/3 of full-time from his vacation home because the company could not afford to lose his knowledge and struggled to find someone competent enough to learn under him that was interested in mainframe. In other words, someone young with these skills would be highly paid and in higher demand at many companies.
      - DigitalJack 11y ago
        Is there any way to learn this sort of thing outside of being at a company with a z/architecture mainframe? I know about the Hercules project, but as far as I can tell, you can only run predecessors to Z/OS (due to licensing as I understand it), and zLinux.
        11y ago
- coleca 11y ago
  Many retail (brick and mortar chains) use them extensively. Way back in the day, that was what you did when you wanted to process "big data" in the 80's and 90's. Those systems became so ingrained into the business processes of the retailers that they are hard to remove. I've seen some retailers in the North East try and "get off the mainframe" by moving to a modern ERP from Oracle or SAP only to have the projects fail after spending $250m-$500m and to this day they are still running 20-40 year old backend software for things like financials, pricing, payroll and merchandising. I've seen production COBOL code running that was authored in the late 70's.
  - Spooky23 11y ago
    I was on the outskirts of a conversation where someone wanted to get away from the "inflexibility of the mainframe", so they were moving to SAP. I missed the rest of the conversation, because I was choking on my beer.
    - nickpsecurity 11y ago
      Lmao. I would've been to. Like with person you replied to, retail companies in my area still use mainframes, AS/400, and even IBM's DOS variant. Few people even understand fully how it works and they're not young people wanting a ton of change to happen. Improvements are incremental and stay within the terminal. However, they do use newer technologies on GUI's etc that integrate with their mainframe solutions. People often have to keep working with two, inconsistent systems at once, though.
joshbaptiste 11y ago
Airline GDS such a Sabre are on such systems.
- MyDogHasFleas 11y ago
  AFAIK, SABRE is/was an airlines reservation system, not a GDS. SABRE evolved into TPF which is not related to z/OS except inasmuch as they are both operating systems that run on IBM z-series mainframes. See http://www-03.ibm.com/systems/z/os/ http://www-03.ibm.com/systems/z/os/ for a list of IBM z-series operating systems.
  - ajdlinux 11y ago
    SABRE is a GDS - it's used by a whole bunch of US and non-US airlines.
  - bdg 11y ago
    I worked with SABRE -- it's a GDS, and a reason I really want to avoid the entire industry.
DigitalJack 11y ago
If your software stack has per core licensing, a z/architecture mainframe can make a lot of sense. They are extremely efficient at running virtual machines (with linux guests for example). Where an Intel cluster can handle 10-20 VMs per core, a z/arch mainframe can do 100s per core. They have dedicated processors for offloading IO operations, which has a big impact on reducing pipeline stalls, and also part of why they are so much better at handling a multitude of VMs. They have 4 levels of cache, with gigabytes of memory at level 4, data is kept close to the processors. The hardware costs a lot, but if your software is licensed per core, there is a good chance you could save millions compared to an Intel solution. It depends on your usage scenario though. If everything is open source and you aren't paying per core fees, then there is probably no advantage for you.
- Spooky23 11y ago
  cough Oracle cough Just be careful with this strategy. When you could start offloading per-socket IBM licensed software to more performant multi-core Intel or POWER CPUs, IBM switched from socket to "Processor Value Units". (Aka "give us more money")
  - savramescu 11y ago
    A single core is not equivalent to a hexa core so I think PVU's make more sense in today's world. What I would like from them is to be more transparent with the requirement to use ILMT.
- MichaelGG 11y ago
  Who licenses per core but then ignores virtualization? (For apps where you aren't throughput bound.)
wyclif 11y ago
What's the z/OS jobs scene like now?
- NelsonMinar 11y ago
  Good, in that many of the folks who know z/OS programming are retiring or dying.
- hvs 11y ago
  I'm not closely involved with it, but my understanding is that there is a big push to train software engineers in India on z/OS to make up for aging out of talent (i.e. retiring).
  - webwanderings 11y ago
    The people in India are cheap compared to the aging IBMers in US. The push is economical and forced. They could easily make the same push here, but it won't happen in the current environment.
    - nickpsecurity 11y ago
      That's not true: there is a push here. There's been previous articles, college partnerships, and so on. I'm not going to pretend they'll put more effort here than on cheap labor. However, there's been colleges teaching COBOL, mainframe, etc, even community colleges, with almost nobody trying to get into those jobs outside people already working one. This is one of few times where there might be a real labor shortage as one generation retires or dies off without another to take the reins.
      - StillBored 11y ago
        I talked to a professor at a community college teaching mainframe skills a couple years ago. I asked him point blank at one point, what the average starting salary for a graduate from his school hired to do mainframe work was. The number he threw out was less than 1/2 of the starting salary of the average comp-sci graduate from the state school down the road. Granted the people from the CC had a two year degree, and were being hired for pretty low level positions. But unless the pay ramp was really steep anyone doing it was crazy not to just enroll in the comp-sci department at the state school and put in the additional two years. I considered if becoming a mainframe expert was a good career move, and from what I can tell midlevel mainframe guys don't make as much as midlevel PHP experts. The top end is ok, but i've heard of people doing vmware designs/installs for more money. Maybe some mainframe guys are paid by the gold bar, but I sure didn't meet any. That is probably why there is a "shortage". People my age look at it and decide its not a good move because while companies will pay 8 figures for a machine they won't pay a fraction of it to the guys running said machine.
renownedmedia 11y ago
"... a 13-volume collection that provides an introduction ..."
zubairsaif007 11y ago
http://googler700.blogspot.com/ http://googler700.blogspot.com/
nickpsecurity 11y ago
Looks like an uphill, learning experience.
- StillBored 11y ago
  And frustrating for anyone with extensive experience with modern software stacks. The zseries is a strange combination of semi-modern hardware with ancient software/hardware practices. I spent a year of my life with a recent mainframe as part of my previous job. During that time I discovered a long list of things that are frankly unbelievable to anyone familiar with modern computing. I should write a book about all the crazy crap, but I don't really have too. Just reading the install/setup/operating guides with a mainframe in hand should be sufficient for anyone with a clue. Bonus points if you install any industry standard benchmarks to measure things like memory/disk io/networking latency/bandwidth. But, in the end I get it. People talk about all kinds of crap that isn't true about the mainframe, but that doesn't matter. What companies are getting when they buy a zeries from IBM is a software stack with a 99.99% future compatibility statement. That means the millions you invest in developing software today will still work two decades from now. IBM isn't going to pull the rug out from under you when the latest OS/language/etc shows up. That is why its possible to run JCL/COBOL/etc written decades ago without change on modern machines. For companies that use computers to assist their main line of business (rather than being the main line itself) this is an extremely valuable feature.
  - nickpsecurity 11y ago
    "What companies are getting when they buy a zeries from IBM is a software stack with a 99.99% future compatibility statement. " That is the greatest benefit. I think there's a potential market for a company using something modern (and maintainable) that offers same benefit. Especially if it can work side-by-side with mainframes and/or popular stacks.

morenoh149 11y ago

Does anyone know what kinds of companies or applications make use of z/OS mainframes?

PaulHoule 11y ago
The State of New York uses them. Once a law was passed that the state needs to turn paperwork around in less than 48 hours they found the printing system was a bottleneck so they hired a greybeard friend of mine to build a new printer interface and device drivers for it. Cornell University also used to run HR and accounting on a 370 mainframe and people were happy, but then they switched to peoplesoft and now (i) they are not happy, and (ii) looking for a new accounting system.
Spooky23 11y ago
All US states, 90%+ of the big banks, retailers, airlines, brokerages, etc use mainframes. There are still places using non-IBM mainframes as well. Several state welfare systems use Unisys mainframes.
- nickpsecurity 11y ago
  Unisys, Fujitsu, and Group Bull that I recall. Too bad Unisys bought Convergent's CTOS. Was the only thing close to Plan 9 that had big take-up in business. I'm not counting Inferno because we're talking mainframes and big business.
ja27 11y ago
Almost every Fortune 500 and most branches of the federal government. There are a few exceptions obviously. Microsoft, Apple, HP, etc. are unlikely to use IBM technologies internally. Facebook, Amazon, etc. are young enough to not have mainframes. You can go search for z/OS job openings and see everyone from Walmart to Lockheed Martin to Citibank.
- deleted 11y ago
  [deleted]
- nickpsecurity 11y ago
  Microsoft did get called out for using a mini-mainframe, the AS/400, back when they were pushing Windows Server. Reportedly they ran their entire business on one AS/400. Embarrassed, they got off the AS/400. Took about 20-23 Windows servers to match its performance and reliability. Told me what I needed to know. ;) But, yeah, they don't use it any more. Most mainframe users are legacy users. However, there's occasional conversions especially for consolidation. IBM's competitor, Group Bull, posted some case studies on that such as consolidating hundreds of desktops and servers onto 2 easily-managed mainframes with cost savings.
  - jussij 11y ago
    I do remember this event, when Microsoft was forced to admit they did not use any of their their own technology other than the desktop and it resulted in a big hit to the credibility of any the Microsoft server technologies. But the result actually meant Microsoft had to then eat their own dog food and in a strange way it forced Microsoft to greatly improve their server platforms to where they are today. So in a funny way it probably backfired on those early Microsoft detractors.
    - nickpsecurity 11y ago
      Never thought of it like that. Damn. Why did we have to run our mouths lol...
kjs3 11y ago
In my experience: Insurance, banking, credit card processing, airline reservations, payroll processing.
mod 11y ago
I worked for a company that does z/OS software and got to see their client list--it was pretty much a who's who of industry leaders in every industry, plus a bunch of colleges.
mfisher87 11y ago
Financial services are a good example. Lots of highly regulated legacy systems. For many big name banks and credit cards, it's pretty safe to assume that a Z/OS mainframe is doing some work with the data you've generated with every swipe.
- zmonkeyz 11y ago
  Nationwide did a big linux on z consolidation a couple of years back.
AnonNo15 11y ago
Royal Bank of Canada for sure. Other banks from Big 5, probably too.
surge 11y ago
Every large financial institution, airline, or large company that does millions of transactions an hour that isn't competing with IBM or young enough to implement a different platform that can scale to that level. There is a z/OS security talk I saw that partially dispels the myths and prejudices against mainframes as legacy. In some cases z/OS is more modern than what you might find on your desktop and OSs like Linux are implementing features in its kernel that have been in zOS for years.
- ghuntley 11y ago
  That would be Soldier of Fortran - https://www.youtube.com/watch?v=Xfl4spvM5DI https://www.youtube.com/watch?v=Xfl4spvM5DI
  - surge 11y ago
    Right I love his stuff. I was thinking of his talk at BSides LV 2014 which he gave the same talk but he also had talks this year also at BSides and DefCon. I also wanted to mention that where I've worked and many other places there is a crisis of brain drain at companies as the mainframe experts who are highly paid retire out and there are programs to recruit young people who are more attracted to other platforms. There is a lot of pay and job security in the field. I know of one situation where a guy past retirement is paid full salary to work 1/3 of full-time from his vacation home because the company could not afford to lose his knowledge and struggled to find someone competent enough to learn under him that was interested in mainframe. In other words, someone young with these skills would be highly paid and in higher demand at many companies.
    - DigitalJack 11y ago
      Is there any way to learn this sort of thing outside of being at a company with a z/architecture mainframe? I know about the Hercules project, but as far as I can tell, you can only run predecessors to Z/OS (due to licensing as I understand it), and zLinux.
      - 11y ago
coleca 11y ago
Many retail (brick and mortar chains) use them extensively. Way back in the day, that was what you did when you wanted to process "big data" in the 80's and 90's. Those systems became so ingrained into the business processes of the retailers that they are hard to remove. I've seen some retailers in the North East try and "get off the mainframe" by moving to a modern ERP from Oracle or SAP only to have the projects fail after spending $250m-$500m and to this day they are still running 20-40 year old backend software for things like financials, pricing, payroll and merchandising. I've seen production COBOL code running that was authored in the late 70's.
- Spooky23 11y ago
  I was on the outskirts of a conversation where someone wanted to get away from the "inflexibility of the mainframe", so they were moving to SAP. I missed the rest of the conversation, because I was choking on my beer.
  - nickpsecurity 11y ago
    Lmao. I would've been to. Like with person you replied to, retail companies in my area still use mainframes, AS/400, and even IBM's DOS variant. Few people even understand fully how it works and they're not young people wanting a ton of change to happen. Improvements are incremental and stay within the terminal. However, they do use newer technologies on GUI's etc that integrate with their mainframe solutions. People often have to keep working with two, inconsistent systems at once, though.

joshbaptiste 11y ago

Airline GDS such a Sabre are on such systems.

MyDogHasFleas 11y ago
AFAIK, SABRE is/was an airlines reservation system, not a GDS. SABRE evolved into TPF which is not related to z/OS except inasmuch as they are both operating systems that run on IBM z-series mainframes. See http://www-03.ibm.com/systems/z/os/ http://www-03.ibm.com/systems/z/os/ for a list of IBM z-series operating systems.
- ajdlinux 11y ago
  SABRE is a GDS - it's used by a whole bunch of US and non-US airlines.
- bdg 11y ago
  I worked with SABRE -- it's a GDS, and a reason I really want to avoid the entire industry.

DigitalJack 11y ago

If your software stack has per core licensing, a z/architecture mainframe can make a lot of sense. They are extremely efficient at running virtual machines (with linux guests for example). Where an Intel cluster can handle 10-20 VMs per core, a z/arch mainframe can do 100s per core. They have dedicated processors for offloading IO operations, which has a big impact on reducing pipeline stalls, and also part of why they are so much better at handling a multitude of VMs. They have 4 levels of cache, with gigabytes of memory at level 4, data is kept close to the processors. The hardware costs a lot, but if your software is licensed per core, there is a good chance you could save millions compared to an Intel solution. It depends on your usage scenario though. If everything is open source and you aren't paying per core fees, then there is probably no advantage for you.

Spooky23 11y ago
cough Oracle cough Just be careful with this strategy. When you could start offloading per-socket IBM licensed software to more performant multi-core Intel or POWER CPUs, IBM switched from socket to "Processor Value Units". (Aka "give us more money")
- savramescu 11y ago
  A single core is not equivalent to a hexa core so I think PVU's make more sense in today's world. What I would like from them is to be more transparent with the requirement to use ILMT.
MichaelGG 11y ago
Who licenses per core but then ignores virtualization? (For apps where you aren't throughput bound.)

wyclif 11y ago

What's the z/OS jobs scene like now?

NelsonMinar 11y ago
Good, in that many of the folks who know z/OS programming are retiring or dying.
hvs 11y ago
I'm not closely involved with it, but my understanding is that there is a big push to train software engineers in India on z/OS to make up for aging out of talent (i.e. retiring).
- webwanderings 11y ago
  The people in India are cheap compared to the aging IBMers in US. The push is economical and forced. They could easily make the same push here, but it won't happen in the current environment.
  - nickpsecurity 11y ago
    That's not true: there is a push here. There's been previous articles, college partnerships, and so on. I'm not going to pretend they'll put more effort here than on cheap labor. However, there's been colleges teaching COBOL, mainframe, etc, even community colleges, with almost nobody trying to get into those jobs outside people already working one. This is one of few times where there might be a real labor shortage as one generation retires or dies off without another to take the reins.
    - StillBored 11y ago
      I talked to a professor at a community college teaching mainframe skills a couple years ago. I asked him point blank at one point, what the average starting salary for a graduate from his school hired to do mainframe work was. The number he threw out was less than 1/2 of the starting salary of the average comp-sci graduate from the state school down the road. Granted the people from the CC had a two year degree, and were being hired for pretty low level positions. But unless the pay ramp was really steep anyone doing it was crazy not to just enroll in the comp-sci department at the state school and put in the additional two years. I considered if becoming a mainframe expert was a good career move, and from what I can tell midlevel mainframe guys don't make as much as midlevel PHP experts. The top end is ok, but i've heard of people doing vmware designs/installs for more money. Maybe some mainframe guys are paid by the gold bar, but I sure didn't meet any. That is probably why there is a "shortage". People my age look at it and decide its not a good move because while companies will pay 8 figures for a machine they won't pay a fraction of it to the guys running said machine.

renownedmedia 11y ago

"... a 13-volume collection that provides an introduction ..."

zubairsaif007 11y ago

http://googler700.blogspot.com/ http://googler700.blogspot.com/

nickpsecurity 11y ago

Looks like an uphill, learning experience.

StillBored 11y ago
And frustrating for anyone with extensive experience with modern software stacks. The zseries is a strange combination of semi-modern hardware with ancient software/hardware practices. I spent a year of my life with a recent mainframe as part of my previous job. During that time I discovered a long list of things that are frankly unbelievable to anyone familiar with modern computing. I should write a book about all the crazy crap, but I don't really have too. Just reading the install/setup/operating guides with a mainframe in hand should be sufficient for anyone with a clue. Bonus points if you install any industry standard benchmarks to measure things like memory/disk io/networking latency/bandwidth. But, in the end I get it. People talk about all kinds of crap that isn't true about the mainframe, but that doesn't matter. What companies are getting when they buy a zeries from IBM is a software stack with a 99.99% future compatibility statement. That means the millions you invest in developing software today will still work two decades from now. IBM isn't going to pull the rug out from under you when the latest OS/language/etc shows up. That is why its possible to run JCL/COBOL/etc written decades ago without change on modern machines. For companies that use computers to assist their main line of business (rather than being the main line itself) this is an extremely valuable feature.
- nickpsecurity 11y ago
  "What companies are getting when they buy a zeries from IBM is a software stack with a 99.99% future compatibility statement. " That is the greatest benefit. I think there's a potential market for a company using something modern (and maintainable) that offers same benefit. Especially if it can work side-by-side with mainframes and/or popular stacks.