7 ms·

They have a lot more software in the works, such as Cassanda, MongoDB and Spark [1]. It's a push to have good software support for the IBM LinuxONE machine [2],

They have a lot more software in the works, such as Cassanda, MongoDB and Spark [1]. It's a push to have good software support for the IBM LinuxONE machine [2], essentially an IBM z Systems (z13) mainframe that will only run Linux on KVM (i.e. without expensive z/OS or z/VM 'legacy'). I'm not sure about the costs, but it might make for a compelling platform as a 'private cloud' solution. A single mainframe is a lot easier to maintain than racks full of x86 boxes. Apparently one LinuxONE Emperor machine can run up to 8,000 Linux VMs [3] (not sure with what load). [1] https://github.com/linux-on-ibm-z https://github.com/linux-on-ibm-z [2] http://www-03.ibm.com/systems/z/os/linux/linux-one.html http://www-03.ibm.com/systems/z/os/linux/linux-one.html [3] http://www.zdnet.com/article/linuxone-ibms-new-linux-mainframes/ http://www.zdnet.com/article/linuxone-ibms-new-linux-mainfra...

bryanlarsen 11y ago
up to 8,000 Linux VMs (not sure with what load) Traditionally mainframes were superior for loads that were IO bound rather than CPU bound. I suspect that this is still the case.
- praseodym 11y ago
  Even with high CPU loads, due to license costs for expensive proprietary software, the advantage can shift towards mainframes when dedicated Application Assist Processors (zAAP) are employed, because usually only the generic CPUs have to be licensed. By offloading work to the 'license-free' zAAPs, overall throughput can be greatly increased with equal license costs. Of course, this scenario isn't what LinuxONE boxes are made for; they are more focused towards open source deployments in the enterprise, with low license costs (usually only support contract costs).
  - mitchty 11y ago
    I used to work at a place that did this. The only issue we ran into was memory was HELLA expensive on the z series boxes. It was to the point that it was cheaper to buy an hp server with 1TiB ram than get 16g of ram from the zaaps. That and getting software to run on it was almost impossible. In the end we ended up moving everything to HP servers for a fraction of the cost.
    - barkingcat 11y ago
      When deployments like this happen, where does the z series box then go? Back to IBM (because it was on lease?) Is the z series modular enough that it's easy for ibm to reconfigure to lease to someone else?
      - kev009 11y ago
        If it was a lease through something like IBM Global Services they will take it back and possibly redeploy it. Otherwise, they can kind of get lost in the noise, either ending up at auction or straight to an edisposal service. They are usually parted out at that point to sell spare parts, because the licensing makes it nearly impossible to use one of these w/o involving and paying IBM, usually monthly. Otherwise they are crushed for precious metal recovery. Kind of sad ends for such works of art. The Linux dichotomy IBM created is pretty stupid. I don't know why you'd want to run a tire fire on such expensive HW, sure maybe utility VMs, but not as the raison d'etat. Instead they should be a lot more willing to provide cheap z/VM, z/OS, z/TPF licenses for smaller shops to build up an ISV, admin, and porting community. But they charge thousands of dollars for an emulator when Hercules is freely available and ADCD is quite restricted. I recently bought an older one for the cost of a soda https://twitter.com/kevinbowling1/status/685240077161598976 https://twitter.com/kevinbowling1/status/685240077161598976. I would warn against doing this for a number of reasons, namely FICON storage and licensing, but if you are determined it is a lot of fun.
        nickpsecurity 11y ago
        Good for you haha. How much power do one of those things use, btw?
        kev009 11y ago
        This is the baby class z800, it's 3kw max which isn't too bad for a full cab.
        11y ago
- vram22 11y ago
  Interesting. Never worked on mainframes myself, but have on Unix (and Linux) a lot. Had read some years ago that mainframes had some powerful features that were often not available on Unixes. One such feature that was available, could be HP's PRM (Process Resource Manager), which, IIRC (from my time at a HP joint venture), could ensure allocation of a guaranteed minimum percentage of system resources (such as RAM and CPU time) to a specific process (say a payroll job running at the end of the month). I remember being impressed by the concept at the time. There was also HP MC/Service Guard, but didn't have a good idea what it did, just remember the name. P.S. Had also tried out rtprio and prioc(n)tl on some Unix boxes, maybe one was an HP one, the other a SVR4. They allowed assigning (quite) higher priorities to specified processes, IIRC. P.P.S. Both those product names could have been changed by now, or been merged into bigger products, as sometimes happens. Edited to say that HP's PRM _was_ an example of a mainframe-like feature.
  - pjmlp 11y ago
    Another one I like very much is how OS/400 works. Written originally in PL/I and Assembly, it contains a kernel level JIT. All user space applications target a general purpose abstract Assembly and they are AOT to native either at installation time or on-demand. Other mainframes had similar approaches to code portability. Sounds familiar? Another cool thing are the catalog based filesystem and OO Assembly.
    - nickpsecurity 11y ago
      The JIT scheme worked well for future-proofing for decades. That means it's an approach others should consider emulating if they're not yet. Only thing I couldn't figure out, due to IBM terminology, is if the compiled output is actual microcode or just CPU instructions. Or if they both create a optimized, microcode implementation and a JIT to move things to it. Integrating DB-like functionality into the system was ahead of time. The original, System/38, had capability security in the hardware enforcing an object-based view of memory with permissible operations on different types of objects: processes, I/O, arrays, stacks, and so on. The security implications of that means I'd recommend one to a business in a heartbeat if they hadn't ditched that for POWER. https://homes.cs.washington.edu/~levy/capabook/Chapter8.pdf https://homes.cs.washington.edu/~levy/capabook/Chapter8.pdf All in all, System/38 was a very, forward-thinking and practical design that continues to prove itself year after year. If it doesn't exist, I'd like to see a Lesson's Learned report on that project and how the team decided what would be in it.
- rodgerd 11y ago
  This has not been my experience running Linux on mainframes. The 8000 VM tests I've seen picked apart are typically "a webserver" type load, not meaningful work.
- StillBored 11y ago
  The mainframe hardware isn't anything special (I've said this before, I still have access to a z114). You can probably host that many lxc containers on a decent x86 server too. Not that you probably want to. The z114 CPU doesn't keep up with anything > midrange x86, and its IO capabilities are limited by the inifiniband links it uses to communicate with the IO drawers. A pretty basic dual socket E5 v3 has more PCIe bandwidth available than you can get out of the z114. It appears that is the case for larger machines as well (aka there are E7 configurations with significantly more bandwidth than the z13). And yes, I've heard all about SC's and all that. But what most people fail to realize is that most storage adapters have some form of ARM/etc cpu onboard that does most of what the SC does on a mainframe, and people have been fighting the CPU vs offload engine battle for years for network adapters. So, a lot of the mainframe advantage is in zOS/zTPF and not the hardware. Running linux with its inefficient block layer, misses the point. Heck, your probably better off running linux on POWER8.
  - unixhero 11y ago
    Can you do an AMA, or umm, tell us more? +1 Insightful
  - Spooky23 11y ago
    A few years ago our management had a healthy portion of kool aid and demanded that we land a few systems to Linux zOS containers. It was awful and we discovered why a few days in -- running our workload on a few VMs on my crappy corporate laptop in VMWare smoked the mainframe. I guess it's better now, but it seems more like an Oracle license laundering scheme that something anyone would choose to run.
  - neopallium 11y ago
    A company I worked for about 10 years ago had an IBM s/390 mainframe for about a month. Our manager at that time had previously worked for IBM and was trying to move the company over to hardware managed by IBM. The biggest selling point was redundancy in the hardware being able to handle CPU/RAM/Disk/PSU failures without taking down the Linux VMs running on it. Our workload was network bound, not much CPU or Disk IO load. We ended up using generic Intel PCs (some custom built, some Dell and some IBM Netfinities). We had more down time with the IBM managed Netfinities, then with the custom built rack mount servers. Back then we also had some Sun UltraSparc E3500 & E6500s, but we found that the Java VM (from Sun!) was much slower on the UltraSparc CPUs then on Intel CPUs. I find it interesting that we had much better results from generic no brand rack mount servers then most of the expensive "enterprise" class servers. Note: This was over 10 years ago. Anyone have more recent experience with servers in a self-managed DC or co-location? It seems most companies/developers now use cloud/vps/hosted servers.
SixSigma 11y ago
I know it's not on a mainframe but if you think 8,000 vms is a lot (which it is!) Ron Minnich ran 1,000,000 Linux VMs on the Thunderbird supercomputing cluster with Dell supplied 4,480-nodes. https://share.sandia.gov/news/resources/news_releases/sandia-computer-scientists-successfully-boot-one-million-linux-kernels-as-virtual-machines/ https://share.sandia.gov/news/resources/news_releases/sandia...
- deleted 11y ago
  [deleted]
- unixhero 11y ago
  But still. If I understand this correctly 8000vms is one of these mainframes. That's one machine versus a supercomluting cluster of 4480 nodes.
  - SixSigma 11y ago
    yes, that's why I qualified it and gave the numbers so it gives a kind of measure of the relative power if these things. ~5000 dell nodes = 1,000,000 vms = 200 per node 1 mainframe = 8000 vms per mainframe so 1 mainframe approx. 40 dell nodes
jacques_chester 11y ago
> I'm not sure about the costs, but it might make for a compelling platform as a 'private cloud' solution. My assumption is that they will port Cloud Foundry and sell it under a name like "BlueMix/Z". Disclaimer: I work for Pivotal, the major donor of engineering time to Cloud Foundry. I believe IBM is the second biggest donor of engineering time.

bryanlarsen 11y ago

up to 8,000 Linux VMs (not sure with what load) Traditionally mainframes were superior for loads that were IO bound rather than CPU bound. I suspect that this is still the case.

praseodym 11y ago
Even with high CPU loads, due to license costs for expensive proprietary software, the advantage can shift towards mainframes when dedicated Application Assist Processors (zAAP) are employed, because usually only the generic CPUs have to be licensed. By offloading work to the 'license-free' zAAPs, overall throughput can be greatly increased with equal license costs. Of course, this scenario isn't what LinuxONE boxes are made for; they are more focused towards open source deployments in the enterprise, with low license costs (usually only support contract costs).
- mitchty 11y ago
  I used to work at a place that did this. The only issue we ran into was memory was HELLA expensive on the z series boxes. It was to the point that it was cheaper to buy an hp server with 1TiB ram than get 16g of ram from the zaaps. That and getting software to run on it was almost impossible. In the end we ended up moving everything to HP servers for a fraction of the cost.
  - barkingcat 11y ago
    When deployments like this happen, where does the z series box then go? Back to IBM (because it was on lease?) Is the z series modular enough that it's easy for ibm to reconfigure to lease to someone else?
    - kev009 11y ago
      If it was a lease through something like IBM Global Services they will take it back and possibly redeploy it. Otherwise, they can kind of get lost in the noise, either ending up at auction or straight to an edisposal service. They are usually parted out at that point to sell spare parts, because the licensing makes it nearly impossible to use one of these w/o involving and paying IBM, usually monthly. Otherwise they are crushed for precious metal recovery. Kind of sad ends for such works of art. The Linux dichotomy IBM created is pretty stupid. I don't know why you'd want to run a tire fire on such expensive HW, sure maybe utility VMs, but not as the raison d'etat. Instead they should be a lot more willing to provide cheap z/VM, z/OS, z/TPF licenses for smaller shops to build up an ISV, admin, and porting community. But they charge thousands of dollars for an emulator when Hercules is freely available and ADCD is quite restricted. I recently bought an older one for the cost of a soda https://twitter.com/kevinbowling1/status/685240077161598976 https://twitter.com/kevinbowling1/status/685240077161598976. I would warn against doing this for a number of reasons, namely FICON storage and licensing, but if you are determined it is a lot of fun.
      - nickpsecurity 11y ago
        Good for you haha. How much power do one of those things use, btw?
        kev009 11y ago
        This is the baby class z800, it's 3kw max which isn't too bad for a full cab.
        11y ago
vram22 11y ago
Interesting. Never worked on mainframes myself, but have on Unix (and Linux) a lot. Had read some years ago that mainframes had some powerful features that were often not available on Unixes. One such feature that was available, could be HP's PRM (Process Resource Manager), which, IIRC (from my time at a HP joint venture), could ensure allocation of a guaranteed minimum percentage of system resources (such as RAM and CPU time) to a specific process (say a payroll job running at the end of the month). I remember being impressed by the concept at the time. There was also HP MC/Service Guard, but didn't have a good idea what it did, just remember the name. P.S. Had also tried out rtprio and prioc(n)tl on some Unix boxes, maybe one was an HP one, the other a SVR4. They allowed assigning (quite) higher priorities to specified processes, IIRC. P.P.S. Both those product names could have been changed by now, or been merged into bigger products, as sometimes happens. Edited to say that HP's PRM _was_ an example of a mainframe-like feature.
- pjmlp 11y ago
  Another one I like very much is how OS/400 works. Written originally in PL/I and Assembly, it contains a kernel level JIT. All user space applications target a general purpose abstract Assembly and they are AOT to native either at installation time or on-demand. Other mainframes had similar approaches to code portability. Sounds familiar? Another cool thing are the catalog based filesystem and OO Assembly.
  - nickpsecurity 11y ago
    The JIT scheme worked well for future-proofing for decades. That means it's an approach others should consider emulating if they're not yet. Only thing I couldn't figure out, due to IBM terminology, is if the compiled output is actual microcode or just CPU instructions. Or if they both create a optimized, microcode implementation and a JIT to move things to it. Integrating DB-like functionality into the system was ahead of time. The original, System/38, had capability security in the hardware enforcing an object-based view of memory with permissible operations on different types of objects: processes, I/O, arrays, stacks, and so on. The security implications of that means I'd recommend one to a business in a heartbeat if they hadn't ditched that for POWER. https://homes.cs.washington.edu/~levy/capabook/Chapter8.pdf https://homes.cs.washington.edu/~levy/capabook/Chapter8.pdf All in all, System/38 was a very, forward-thinking and practical design that continues to prove itself year after year. If it doesn't exist, I'd like to see a Lesson's Learned report on that project and how the team decided what would be in it.
rodgerd 11y ago
This has not been my experience running Linux on mainframes. The 8000 VM tests I've seen picked apart are typically "a webserver" type load, not meaningful work.
StillBored 11y ago
The mainframe hardware isn't anything special (I've said this before, I still have access to a z114). You can probably host that many lxc containers on a decent x86 server too. Not that you probably want to. The z114 CPU doesn't keep up with anything > midrange x86, and its IO capabilities are limited by the inifiniband links it uses to communicate with the IO drawers. A pretty basic dual socket E5 v3 has more PCIe bandwidth available than you can get out of the z114. It appears that is the case for larger machines as well (aka there are E7 configurations with significantly more bandwidth than the z13). And yes, I've heard all about SC's and all that. But what most people fail to realize is that most storage adapters have some form of ARM/etc cpu onboard that does most of what the SC does on a mainframe, and people have been fighting the CPU vs offload engine battle for years for network adapters. So, a lot of the mainframe advantage is in zOS/zTPF and not the hardware. Running linux with its inefficient block layer, misses the point. Heck, your probably better off running linux on POWER8.
- unixhero 11y ago
  Can you do an AMA, or umm, tell us more? +1 Insightful
- Spooky23 11y ago
  A few years ago our management had a healthy portion of kool aid and demanded that we land a few systems to Linux zOS containers. It was awful and we discovered why a few days in -- running our workload on a few VMs on my crappy corporate laptop in VMWare smoked the mainframe. I guess it's better now, but it seems more like an Oracle license laundering scheme that something anyone would choose to run.
- neopallium 11y ago
  A company I worked for about 10 years ago had an IBM s/390 mainframe for about a month. Our manager at that time had previously worked for IBM and was trying to move the company over to hardware managed by IBM. The biggest selling point was redundancy in the hardware being able to handle CPU/RAM/Disk/PSU failures without taking down the Linux VMs running on it. Our workload was network bound, not much CPU or Disk IO load. We ended up using generic Intel PCs (some custom built, some Dell and some IBM Netfinities). We had more down time with the IBM managed Netfinities, then with the custom built rack mount servers. Back then we also had some Sun UltraSparc E3500 & E6500s, but we found that the Java VM (from Sun!) was much slower on the UltraSparc CPUs then on Intel CPUs. I find it interesting that we had much better results from generic no brand rack mount servers then most of the expensive "enterprise" class servers. Note: This was over 10 years ago. Anyone have more recent experience with servers in a self-managed DC or co-location? It seems most companies/developers now use cloud/vps/hosted servers.

SixSigma 11y ago

I know it's not on a mainframe but if you think 8,000 vms is a lot (which it is!) Ron Minnich ran 1,000,000 Linux VMs on the Thunderbird supercomputing cluster with Dell supplied 4,480-nodes. https://share.sandia.gov/news/resources/news_releases/sandia-computer-scientists-successfully-boot-one-million-linux-kernels-as-virtual-machines/ https://share.sandia.gov/news/resources/news_releases/sandia...

deleted 11y ago
[deleted]
unixhero 11y ago
But still. If I understand this correctly 8000vms is one of these mainframes. That's one machine versus a supercomluting cluster of 4480 nodes.
- SixSigma 11y ago
  yes, that's why I qualified it and gave the numbers so it gives a kind of measure of the relative power if these things. ~5000 dell nodes = 1,000,000 vms = 200 per node 1 mainframe = 8000 vms per mainframe so 1 mainframe approx. 40 dell nodes

jacques_chester 11y ago

> I'm not sure about the costs, but it might make for a compelling platform as a 'private cloud' solution. My assumption is that they will port Cloud Foundry and sell it under a name like "BlueMix/Z". Disclaimer: I work for Pivotal, the major donor of engineering time to Cloud Foundry. I believe IBM is the second biggest donor of engineering time.