I bought an IBM mainframe for personal use. I am doing this for learning and figuring out how it works. If you are curious about what goes into this process, I hope this post will interest you.
I am not the first one by far to do something like this. There are some people on the internet that I know have their own personal mainframes, and I have drawn inspiration from each and every one of them. You should follow them if you are interested in these things:
Buying a mainframe takes time. I never spent so much time on a purchase before. In fact - I purchased my first apartment with probably less planning and research. Compared to buying an apartment you have no guard rails. You are left to your own devices to ensure the state of whatever thing you are buying as it likely is sold as-is. Unless you are buying refurbished the seller will likely have no idea of how to verify the item and might even lack the equipment to power it on.
You might end up having to read up on what items are supposed to be in the frame of your mainframe and request heaps of pictures of parts in order to verify the exact configuration. That was the case for me, which ended up being loads of fun - but something I had not expected when starting the search.
The important thing with the Support Elements is that they contain the license for running the mainframe. When a customer buys the mainframe from IBM they put a lot more hardware in the frame than the customer bought, generally called "Plan Ahead". The licenses installed in the SE ensures that the customer only has access to what they have paid for. This means no SE = no mainframe. This is also called LICCC (Licensed Internal Code Configuration Control).
Likewise, if you cannot login to the SE you cannot use the mainframe. IBM is rumored to have services to password reset the SE but I would not want to depend on that fact. On the other hand, anything can be hacked with local access to it so if you are up for a (bigger) challenge it should be possible and might even get you a discount on the mainframe. Luckily it seems common people are running with the default passwords so asking the seller to verify the login credentials of the ACSADMIN and SYSPROG user should be good enough.
In my case photos from inside the CPC drawer showed 10x 45D8410 modules (picture 3). After some internet searching one can figure out that this is the feature 1605 for 4 GB RAM sticks and it had 10 of them - 40 GB raw, or 24 GB usable after accounting for overhead and redundancies (see Table 2-4 z114 memory offerings, z114 Technical Guide).
There is also a significant difference in power usage if you have Balanced versus Unbalanced Power. You can figure out if you have this by looking at pictures of how many Bulk Power Regulators (BPR) there are. If you only have one you have unbalanced single phase power. If you have all three, you have balanced power as far as I know. If you have two I assume you have unbalanced two-phased power. Balanced power does sound like a good thing, but if you only care about power consumption it adds something like 500 W to the power consumption according to IBM.
IBM has a nice power planner utility available for its mainframes, but it requires you to register and get approved for access to their Resource Link knowledge base.
This particular unit is configured with a single BPR, one I/O cage and one PCIe cage. The PCIe cage contains the FICON interfaces, and Gigabit Ethernet cards while the I/O cage contains only ESCON and some additional Gigabit Ethernet. ESCON is the predecessor to FICON and as I have no plans of running it I plan to disconnect the I/O cage, saving 700 W in the process.
My estimate is that the continuous consumption will be 1.7 kW. Pretty comfortable, and well within specification for attaching a 16A C20 connector straight to the mainframe.
I found a power cable used for z114 with bare wires (picture 4) which I will attach a C20 connector on. This should allow running the z114 without any issues more or less anywhere with a 16A outlet.
Items generally have three numbers associated with them. The Part Number (P/N), the Customer Card Identification Numbers (CCIN), and the Feature Code (FC). Not all items have all three, for example port fillers may only have a part number and devices that are required in the base version of the unit may not have a feature code. If an item has a feature code I assume it is likely that you will need to have that feature activated in your LICCC for the item to work. However, there are exceptions: mainframe memory banks on the z114 are not enforced - only the resulting capacity is capped if it is above the licensed maximum. The z114 System Overview documentation states:
To find what items you can purchase one good source is the mainframe's part catalog. For z114 this is GC28-6908-00. It contains breakdowns of what optional features exist, how they are mounted, and where they would be present if they are already available. This makes it possible to determine if the feature is already present based on pictures. Granted, the document is far from rich in detail but it helped me loads to understand what different parts go where and how I would go about to e.g. source the battery backup feature if I so wanted.
When you have found an item you wish to purchase your best bet is eBay, AliExpress, or even Taobao. Refurbished sellers work as well but they of course charge premiums. Taobao has some otherwise hard to find parts. d-parts.ru and allegro.pl are two other sites that have some part numbers that you might need - I only ended up buying from the latter so I cannot say anything about the former.
As one would expect for something that is mainframe specific, FICON storage with ECKD support is expensive. I found one provider that quoted me a bit north of $30,000 for their smallest entry-level unit, while I am being told that the larger vendors start at $100,000+.
There is a saving grace however. There are plenty of IBM DS6800 arrays floating around in the second hand market at $6,000 - $8,000. If you are a bit more adventurous there are even more in China on Taobao for $1,000 - $2,000 without disks, and $15 per disk. I managed to buy one on AliExpress for a significant discount as the seller did not have the equipment to ensure the system was working and offered me the item for $1,800 + shipping. Expensive, yes, but DASD is probably the most important accessory to your mainframe so it makes sense to splurge a little.
The industry appears to have embraced the cheap and comparatively fast disks as a contender to tapes in modern times, which has spawned a whole industry of virtual tape libraries. These can be found on the previously mentioned places on the internet. I got my hands on parts of an EMC DLm2000 environment that I am hoping to pick apart as well as some extra FICON cards. This particular solution works using NFS servers to store the tapes, so I am hopeful I can integrate it with FUSE and store my tapes directly on Google Cloud Storage or equivalent.
I am not the first one by far to do something like this. There are some people on the internet that I know have their own personal mainframes, and I have drawn inspiration from each and every one of them. You should follow them if you are interested in these things:
This post is about buying an IBM z114 mainframe (picture 1) but should translate well to any of the IBM mainframes from z9 to z14.
 |
| Picture 1: An IBM z114 mainframe in all its glory Source: IBM |
What to expect of the process
You might end up having to read up on what items are supposed to be in the frame of your mainframe and request heaps of pictures of parts in order to verify the exact configuration. That was the case for me, which ended up being loads of fun - but something I had not expected when starting the search.
Important items to investigate
There are some crucial facts you will want to verify in order to ensure that 1) you are not buying a bricked mainframe 2) the mainframe will fit the purpose.What model is it?
This should hopefully be pretty self-explanatory but you should verify the claimed model. My mainframe is 2818-M05 which is one CPC drawer, the M10 has two. Easy enough to spot in a picture.
Are the Support Elements (SE) present? Are the credentials known?
The important thing with the Support Elements is that they contain the license for running the mainframe. When a customer buys the mainframe from IBM they put a lot more hardware in the frame than the customer bought, generally called "Plan Ahead". The licenses installed in the SE ensures that the customer only has access to what they have paid for. This means no SE = no mainframe. This is also called LICCC (Licensed Internal Code Configuration Control).
Likewise, if you cannot login to the SE you cannot use the mainframe. IBM is rumored to have services to password reset the SE but I would not want to depend on that fact. On the other hand, anything can be hacked with local access to it so if you are up for a (bigger) challenge it should be possible and might even get you a discount on the mainframe. Luckily it seems common people are running with the default passwords so asking the seller to verify the login credentials of the ACSADMIN and SYSPROG user should be good enough.
Is there a Hardware Management Console (HMC) included?
The Hardware Management Console is mostly to connect many mainframes into what is called an Ensemble, which if you're reading this is probably not something you will ever do with your mainframe, but it does have some other neat features like being able to be accessed from a browser remotely.
You do not need an HMC if you only wish to operate the mainframe when locally present. Otherwise getting an HMC might be a good idea. Sadly it requires Java in the z114 version but that is a price you have to pay. Maybe someday somebody writes a replacement for hobbyists to run without the need to use Java.
You do not need an HMC if you only wish to operate the mainframe when locally present. Otherwise getting an HMC might be a good idea. Sadly it requires Java in the z114 version but that is a price you have to pay. Maybe someday somebody writes a replacement for hobbyists to run without the need to use Java.
You can buy an HMC separately quite easily as long as you have the drivers needed. My z114 came with a 93G driver installation DVD to install HMCs with. I have been told it is important that the driver version of the HMC is the same as the mainframe. I have no idea what happens if it is not.
What optics are in the FICON/OSA cards?
 |
| Picture 2: OSA card with CCIN 57E7 Photo by Vibrant Technologies |
You will want to connect your mainframe to a bunch of various equipment. For that you need to figure out what optics are present in your mainframe.
If you look at the OSA card in picture 2 you can see it says 57E7 at the right side, that is called a Custom Card Identification Number (CCIN). With the CCIN in hand one can use the handy reference table called "IBM System z IO Feature Reference Table" to translate this to a product name. In the case for 57E7 we learn that it is a "OSA-Express4S GbE SX (PCIe)" with 2 ports using multi-mode (MM) and LC connectors.
If you look at the OSA card in picture 2 you can see it says 57E7 at the right side, that is called a Custom Card Identification Number (CCIN). With the CCIN in hand one can use the handy reference table called "IBM System z IO Feature Reference Table" to translate this to a product name. In the case for 57E7 we learn that it is a "OSA-Express4S GbE SX (PCIe)" with 2 ports using multi-mode (MM) and LC connectors.
How much storage (memory)?
Note: In the mainframe world what is usually referred to as memory in the PC world is called storage for historical reasons.
 |
| Picture 3: 4 GB DIMM module Source: Vibrant Technologies |
What CPU (CP) capacity, configuration, and other features?
As mentioned above in the section about Support Elements, the IBM mainframes come with more physical capacity than probably is licensed so figuring out what capacity is licensed is in your interest. The 2818-M05 model is measured from A01 to Z05 where A-Z is the speed rating of the CPs and the number afterwards is the number of activated cores. Z05 being the fastest, A01 being the slowest. This number is a big deal as Z05 provides 3139 MIPS while A01 only provides 26 - i.e. less than 1% of the full capacity!
Figuring this out can be hard unless the seller can power on the machine and logon into the SE and check. If you are lucky there are some printed documentation for the mainframe that tells you at least what it used to have, even though it might have been upgraded (maybe even downgraded to reduce software licensing costs?) in the time since.
IBM can also check what the latest report they have on file says about the configured capacity, but that of course requires you to have a contact with IBM in place.
In addition to this the mainframe will also have a certain number IFL, ICF, zAAP, and/or zIIF cores licensed as well as a myriad of other features which may be of interest to know about. This again most likely requires logging in to the SE to check what those are.
What is the shipping going to cost?
The 2818-M05 weighs at least 700 kg and is the size of a full ordinary server rack. Shipping is going to be a significant part of the purchase unless you are able to handle it yourself. I was quoted $8,000 for shipping between the US and Europe, and $3,500 inside Europe.
Fun facts about power
A good thing to know is that the power supplies in these machines do not care that much about the incoming voltage. If you look at the PSU and read the z114 Physical Planning Manual (IMPP, GC28-6907-00) one can deduce that the mainframe is willing to accept practically any AC/DC between 200V-600V.There is also a significant difference in power usage if you have Balanced versus Unbalanced Power. You can figure out if you have this by looking at pictures of how many Bulk Power Regulators (BPR) there are. If you only have one you have unbalanced single phase power. If you have all three, you have balanced power as far as I know. If you have two I assume you have unbalanced two-phased power. Balanced power does sound like a good thing, but if you only care about power consumption it adds something like 500 W to the power consumption according to IBM.
IBM has a nice power planner utility available for its mainframes, but it requires you to register and get approved for access to their Resource Link knowledge base.
This particular unit is configured with a single BPR, one I/O cage and one PCIe cage. The PCIe cage contains the FICON interfaces, and Gigabit Ethernet cards while the I/O cage contains only ESCON and some additional Gigabit Ethernet. ESCON is the predecessor to FICON and as I have no plans of running it I plan to disconnect the I/O cage, saving 700 W in the process.
My estimate is that the continuous consumption will be 1.7 kW. Pretty comfortable, and well within specification for attaching a 16A C20 connector straight to the mainframe.
 |
| Picture 4: Power cable 41U0108 Source: Allegro |
Sourcing items
Odds are that you need some extra equipment like an storage array or maybe an HMC.Items generally have three numbers associated with them. The Part Number (P/N), the Customer Card Identification Numbers (CCIN), and the Feature Code (FC). Not all items have all three, for example port fillers may only have a part number and devices that are required in the base version of the unit may not have a feature code. If an item has a feature code I assume it is likely that you will need to have that feature activated in your LICCC for the item to work. However, there are exceptions: mainframe memory banks on the z114 are not enforced - only the resulting capacity is capped if it is above the licensed maximum. The z114 System Overview documentation states:
Larger capacity cards may be used for repair actions and manufacturing substitution. LICCC will dial down to ordered size.The z114 Technical Guide also says:
Each DIMM has a size of 4 GB, 8 GB, or 16 GB, and there is no mixing of DIMM sizes on a processor drawer.This means that there are some leeway for some aspects. Be careful before you know what features you have and try to make an informed decision about what the risks are that the item you are thinking of purchasing will not work in the mainframe.
To find what items you can purchase one good source is the mainframe's part catalog. For z114 this is GC28-6908-00. It contains breakdowns of what optional features exist, how they are mounted, and where they would be present if they are already available. This makes it possible to determine if the feature is already present based on pictures. Granted, the document is far from rich in detail but it helped me loads to understand what different parts go where and how I would go about to e.g. source the battery backup feature if I so wanted.
When you have found an item you wish to purchase your best bet is eBay, AliExpress, or even Taobao. Refurbished sellers work as well but they of course charge premiums. Taobao has some otherwise hard to find parts. d-parts.ru and allegro.pl are two other sites that have some part numbers that you might need - I only ended up buying from the latter so I cannot say anything about the former.
Disk storage
If all you want to run is zLinux and z/VM then you should be fine with a normal Fiber Channel SAN running FCP and SCSI. In IBM mainframe world these are called FB-512 or 9336 type Direct-Access Storage Device (DASD). However, z/OS does not work on these type of DASDs so if you want a mainframe capable of running z/OS you need what is called ECKD based storage. ECKD uses channels which are only available over FICON.As one would expect for something that is mainframe specific, FICON storage with ECKD support is expensive. I found one provider that quoted me a bit north of $30,000 for their smallest entry-level unit, while I am being told that the larger vendors start at $100,000+.
There is a saving grace however. There are plenty of IBM DS6800 arrays floating around in the second hand market at $6,000 - $8,000. If you are a bit more adventurous there are even more in China on Taobao for $1,000 - $2,000 without disks, and $15 per disk. I managed to buy one on AliExpress for a significant discount as the seller did not have the equipment to ensure the system was working and offered me the item for $1,800 + shipping. Expensive, yes, but DASD is probably the most important accessory to your mainframe so it makes sense to splurge a little.
Tape library
Commonly IBM mainframes use formats such as 3480, 3490, or 3590 devices for tape. A lot of mainframe concepts still revolve around tapes and as such it may be an interesting addition to your mainframe.The industry appears to have embraced the cheap and comparatively fast disks as a contender to tapes in modern times, which has spawned a whole industry of virtual tape libraries. These can be found on the previously mentioned places on the internet. I got my hands on parts of an EMC DLm2000 environment that I am hoping to pick apart as well as some extra FICON cards. This particular solution works using NFS servers to store the tapes, so I am hopeful I can integrate it with FUSE and store my tapes directly on Google Cloud Storage or equivalent.
Future ideas
The itch that itches the most is the fact that hobbyists need to pay a lot for FICON accessories like DASD. I understand that in the mainframe production environment DASD is extremely important and reliability and performance is key - but that is not the case for hobbyists.
Going forward one of the projects I am most excited about is to use the extra FICON cards from the tape library to build a Linux-based and open-source ECKD DASD array. I have spent some time looking at how FICON works and it is very similar to Fibre Channel, so similar in fact that I think it might be possible to use normal FC HBAs to interface with FICON. It will not be hardware accelerated but for hobbyists it should be more than enough. Expect more blog posts about FICON and running channel controllers in Linux going forward.
If the mainframe arrives safely and even powers on that is ;-).
 |
| Picture 5: IBM z114 getting packed for transport |
Final Bill-of-Materials
This is what the purchase list contained in the end. I have removed some trivial details like LC-LC cables and power plugs from the otherwise complete list.| Amount | Product | Price |
|---|---|---|
| 1 | IBM 2818-M05 z114 | $12,000 |
| 1 | Datacenter Setup | $3,800 |
| 1 | IBM DS6800 | $2,710 |
| 2 | Brocade 5100 w/ FICON CUP | $1,750 |
| 1 | FICON Virtual Tape Library (DLm2000) | $1,315 |
| 2 | 41T8417 OSA-Express4S 10GbE LR (PCIe) | $640 |
| 1 | Arista DCS-7050S-52 | $529 |
| 1 | IBM System X3200 M3 (HMC) | $350 |
| 2 | Power cord | $220 |
| 24 | Brocade 8G FC SFP SM | $74 |
| 20 | Brocade 8G FC SFP MM | $58 |
| Total | $23,446 | |
Loved this article! Reminds me of my good old days with IBM's 360 and 370 series. I started with the IBM 1401, GE-1xx series and Burroughs machines with water-cooled CPU's on raised floors. Programming in a few languages such as Basic Assembler, Fortran, Cobol was fun too. Long live the Mainframe!!
ReplyDeleteThanks for reading! The mainframe certainly has a very interesting history and hopefully have some more stories to tell in the coming decades :-).
DeleteAny chance you might timeshare this thing? I'd love to learn more about mainframes, but not interested in purchasing one. :)
ReplyDeleteAbsolutely, when/if I get it running somewhat stable I was thinking of investigating something like that - mainly to allow Linux distributions to have somewhere to build/test on but also for other hobbyists.
DeleteDM me on twitter or email me at christian _at_ cmd.nu and we can discuss it further.
If you know any IBM employees or contractors, it is actually quite easy to find out what it is officially equipped with (both hardware and LIC/LFA), including any upgrades done while still serviced by IBM. Simply ask your contact to access "Inventory Services", and all they'll need is the four-digit machine type, and the full serial number of the unit. (NOTE: Sometimes when looking up systems, you need to not enter the first two S/N digits, but I can't remember off the top of my head if mainframes have that quirk.)
ReplyDeleteOf note is that the system is accessible to anybody with an IBM Intranet ID; they need know nothing about mainframes. This system works for all non-x86 IBM H/W, so it'll be useful for a DASD or tape purchase too.
Very useful information, thanks! I know some folks over at IBM in other departments. I'll see if they can verify some questions I have.
DeleteVery nice information, thanks for posting. One site I have in my mind for managed mainframe services- visit maintec.com. Services include
ReplyDelete1. System Admin (z/OS, DB2, CICS, Storage, Security) & Operations resources to manage your environment remotely.
2. Application Development & Maintenance services including COBOL, PL/1 & Assembler.
3. App Development on Mobile & Web Platform.
Hi Christian,
ReplyDeleteI'm the president of the Academic Mainframe Consortium (https://amc-ev.org/), located as a non-profit in Germany. We support universities with mainframe know-how. From a customer site we got two z114 M05 (very simular to your configuration) as a gift and replaced an old z9 EC at Leipzig University and another z9 BC at Tübingen University. The z9 EC is the machine that Sebastian Wind has assembled in his basement. From another customer we got two DS8K (also as a gift) and have know a very comfortable configuration for doing basic mainframe education at universities.
Cheers
Wolfram
Hi Wolfram! Happy that you found your way to the blog :-). I know Sebastian, he helps me a lot on these projects. I hope to join your IBM tour in September so maybe we will meet then!
DeleteTake care,
This is awesome! I have three so far. z890/2086, Z9/2096, Z10/2098. I have been working currently on my z9 for about two months because it was a running machine at time of removal. I am a 30 yr vet on AS/400 and just want to learn something new. It has been a real challenge on these trying to learn on your own from trial and error. Not too many people out there that know about these anymore. I had lost all data on my DS6800 and finally last night got it back online and initialized. The issue I am up against now is, I cant find anyone that knows how to restore z/VSE 5.2 from tape. I have searched the world over with no luck. I think it was last saved from a utility within ICCF. I have a DOSRES, SYSWK1, ICCF, SYSTEM LIB, SYSWK2 and SHRA02 tape. I can get it to IPL without IPL Parameters from the SYSTEM LIB tape to a z/VSE screen, but not sure what it is. Maybe ICKDSK? Does anyone know how to restore this OS?
ReplyDeleteThank You!
Hello I know a little bit about the installation and operation of zVSE 4.1 in fact I have Install tapes for it. I recently purchased a z114 M05 as well and will be setting it up soon it will be for academic use Feel free to message me on email vinatron2@gmail.com also I know Sebastian and Connor very well. Sebastian and I have been working on a few projects I’m in the US but if @Wolfram can contact me about software and so on I wanted to save the university’s old z114 but they had to scrap it very unfortunate. Been helping with the move to the z12EC it’s been fun so far looking forward to hearing from you. I also as a side note have zVM 6.2 media if anyone needs assistance with that.
DeleteKind regards,
Vincent
If you want my tapes for zVSE they work I can also send you zVM6.2 media if you want also if you want to give me a copy of your tapes I may be able to figure it out. Linux works well under zVM in fact it’s the preferred method however you can do a direct LPAR installation of zLinux as well I also have an eval version of zVM 5.4 as well witch also works well for that purpose but it’s pretty restricted since I don’t know what machine you have I can’t advise you further but running zVSE 5.4 sounds like it’s pretty modern. Please let me know if you require further assistance.
DeleteKind regards,
Vincent
Very interesting. I'm an old hacker and know most of the people you suggest following! I've mostly played with the older stuff: I have a System/370 and have or have had most of the smaller 390 systems:
ReplyDelete- Multiprise 2000
- S/390 Application StarterPak, a very rare beast (basically a Multiprise 2000 in a half-height rack, with the addition of a built in DAT tape unit, and, uniquely as far as I know, is the only IBM product that came with a *bundled* OS/390 license, allowing you to run OS legally at home! Internal IBM codename was 'Warthog' IIRC)
- S/390 Integrated Server 3006 (pretty much a Multiprise 3000 chassis, but containing a P/390e card as the S/390 CPU)
- Multiprise 3000, my current 'daily driver'.
I've often contemplated acquiring a zSeries machine, but looking back at my above machines, the great advantage for home use is that they all contained DASD integrated in the rack. Have you figured out a way to put a drawer of DASD in the rack of a zSeries machine? Like, in your example, you have a model M05, which only has one CPC drawer, and presumably a blank space where the second CPC drawer would be in an M10; do you think you could fit DASD in that space?
Obviously IBM never intended it to be used that way, but for the home user having the entire system in the footprint of a single rack is a great advantage - why I went the Multiprise route rather than buying a G5 or similar, back in the day. Interested in your thoughts on this!
Hi! Interesting idea! There should be ample of space (and for the z15 ZR1 that's not an issue since it's now 19") however the PSUs are very custom so you'd end up pulling in custom power. Besides, it wouldn't look very sleek unless you make some cover panels :-).
Delete