Skip to main content

Fibre Channel research

The last couple of weeks I've been traveling for work, and I spent a long flight reading some publications I had saved up about mainframe and Fibre Channel. I will talk about some good sources of information later, but for now know that FICON runs over Fibre Channel, and indeed a lot of the reasons behind why Fibre Channel is known to be really reliable is that it more or less came out of IBMs requirements for accessing mainframe disk storage.

Enterprise disk storage has this annoying issue that the disk arrays are:
  • Heavy
  • Power-hungry
  • Ridiculously expensive
An example: DS6800 is widely known to be an unreliable platform, and I would certainly never run a mainframe on it based on my experiences repairing mine. However it is as small as disk arrays get, but it is still 50 kg empty, and consumes 500W of power!

However the DS6800 is kind of a black sheep in reliability. What if we look at something bigger and better? The big brother DS8000 is known to be rock solid for example. The downside is of course that it weighs a bit more than one metric ton and consumes 6 kW of power. Not great for hobbyists.

Ideally I would like to have something that allows me to use the power efficiency of x86, the speed of SSD, at the cost of reliability. I do not need 99.999% uptime, I'm OK to use a conventional server, and I think fellow hobbyists would agree with me.

What we need is a storage system that runs in software, using the economies of scale to our advantage. That's why I started Fikonfarm. The goal is to implement Fibre Channel and FICON in software and somehow hook the mainframe to it. The keyword being "somehow".

I mentioned before that I have 2x Brocade 7800 switches. They are like a conventional Fibre Channel switch, but some of the ports are internally routed to an FPGA that converts the FC traffic (FICON or SCSI, doesn't matter) to TCP/IP. The idea is to figure out how the protocol they talk works so I can make Fikonfarm look like one of those switches.

That brings me to what I've been up to the last couple of weekends. Just that - figuring out the Brocade 7800 protocol. I've come so far to be able to establish an FCIP tunnel with the switch and keep it up. I can also receive FC traffic, as well as send some very very basic frames. Nothing usable yet, but it's pretty exciting :-).


Brocade 7800 protocol dissector in wireshark
Figure 1: Wireshark dissector

What you see in Figure 1 is the Wireshark dissector plugin for the Brocade 7800 protocol that I wrote to be able to map my thesis to real traffic. While I could implement a thesis in Fikonfarm straight away and see what happens, it is useful to be able to take a working session dump and apply the thesis on straight away. That way the outliers are visible, and it is harder to overfit a thesis. E.g. if a byte is 0x10 somewhere in the protocol - you could assume it is static, but it could also be e.g. the hour of the day. Without an extensive dump to test against it is hard to see what values never change, which do, and which seem interesting.

Anyway. When I've come so far as to be able to send and receive frames somewhat reliably I will write up the protocol details and hopefully contribute the dissector upstream. For now it's available on Github if you want it, you can try it on this PCAP if you're curious. Of course, any help is always welcome :-).

Now then, where would one go to learn more about these things? I have two sources I can recommend.

Brocade Reference


Brocade's reference for mainframe connecitivty
Figure 2: Brocade's reference for mainframe connecitivty
One is this practical book on mainframe connectivity by Dr. Guendert (former Brocade, now IBM). Even if you have no background in Fibre Channel this book will teach you how things work, and what options there are. The FC over IP (FCIP) discussed above is even mentioned.

The only downside I can think of is that it is only talking about Brocade solutions when discussing implementations. Keep that salt handy when it speaks about how good Brocade is compared to other vendors.

You can download the book legally for free here.

INCITS Standard Documents

I was about to add the number of pages in "Brocade Mainframe Connectivity Solutions" as a con, but given that my second recommendation is the standard documents themselves which are 10x as long, that seemed silly.

The Fibre Channel standard is divided in many many substandards. The owner of these standards have changed over the passage of time, so it can be hard to figure out what the documents are called. Right now it seems INCITS is the main owner of the standard however.

Here are some good standards to look at:

  • FC-SB-3 - this is classical FICON (before zHPF entered the game)
  • FC-FC-5 - this is the layer over the physical media, it explains how FC is framed and assembled
  • FC-SW-6 - explains how FC switches connect to eachother
  • FCP-4 - covers SCSI for FC
The list goes on, but that will give you an idea of how many standards there are. The number in the end is the version of the standard and while the latest probably gives you the most details about the protocol, it can also omit references to obsolete features that may well be in use in system that you would like to integrate with. FC-SW-6 for example contains 4 fields that are not described more than that they have been obsoleted in previous standards.

Another issue is that a single PDF of these standards, e.g. FC-SB-3, costs $60 to get. If you look at the Brocade Standard Compliance page you will find a lot of references to various versions of these standards.

However, there is another way. One trick I wish to share is that in a lot of cases you can search for the INCITS number as well as "public review" (e.g. "incits 545-2019 public review"). This will in the majority of cases give you a drafted version of the standard that was close to being finalized. It is nothing to base a commercial product on, but it is absolutely fine to use to study the behavior of the protocols as well as e.g. maintaining the Wireshark dissectors.

Summary

I certainly have a lot of work in front of me to build Fikonfarm and getting mainframe DASD up and running without extra hardware. I expect this to take a few years realistically, which means this will not block my normal mainframe tinkering - it is merely my long-term strategy to allow myself to not have to care about broken disk controllers, and focus on what I enjoy: unraveling the mysteries of the mainframe.

Thanks for reading,





https://storage.googleapis.com/bluecmd/fcip-reset-w-target.pcap

Comments

Popular posts from this blog

Buying an IBM Mainframe

I bought an IBM mainframe for personal use. I am doing this for learning and figuring out how it works. If you are curious about what goes into this process, I hope this post will interest you.

I am not the first one by far to do something like this. There are some people on the internet that I know have their own personal mainframes, and I have drawn inspiration from each and every one of them. You should follow them if you are interested in these things:
@connorkrukosky@sebastian_wind@faultywarrior@kevinbowling1 This post is about buying an IBM z114 mainframe (picture 1) but should translate well to any of the IBM mainframes from z9 to z14.

What to expect of the process Buying a mainframe takes time. I never spent so much time on a purchase before. In fact - I purchased my first apartment with probably less planning and research. Compared to buying an apartment you have no guard rails. You are left to your own devices to ensure the state of whatever thing you are buying as it likely…

Powering a mainframe

The last few days have been eventful. I was contacted by the datacenter that the mainframe's cage is now ready for moving in, and the power has been made available. Very exciting! I grabbed my home-made power cables (more on that later) and my best screwdrivers and set off to the datacenter.


The datacenter staff, not needing a forklift in their day-to-day, had managed to solicit the services of a forklift, the associated operator, and some very handy folks to help navigate the mainframe from the storage space to its final location.



After some intense period of fighting the inclination of the road between the storage facility and the cage (and a door that was a bit too small) it was finally in place. Incidentally we were forced to trust the wheels on this pretty rough floor. I did not expect it to roll that well on raw concrete, I was pleasantly surprised. This thing is a tank!

Now, everybody wanted to see if it was working. My machine did not come with a power cable so I had to so…

Open Datacenter Hardware - Leopard Server

Introduction The Leopard is an OpenRack v1 compliant 12V server commissioned by Facebook to offer compute power. It consists of 2x Intel Xeon E5-2678 v3 and is available with either DDR3 or DDR4 memory. The model is manufactured by two vendors primarily: Quanta and Wiwynn.

Leopard features a 24x PCIe slot which can fit either a PCIe card with low profile, or a riser card with 1x 16x and 1x 8x slots. The server also supports a 3.5" SATA drive as well as either an mSATA or an M.2 drive mounted on the motherboard.

Connectivity wise Leopard has a mezzanine card slot allowing for example 10Gb/s or 25Gb/s Ethernet.

Figure 1 and figure 2 shows the server layout. The server is made to fit inside an OpenRack v1 enclosure, at which point it looks something like figure 3. Due to power constraints an OpenRack v1 can fit 30 of these servers before power starts to become an issue. The Leopard servers that the organization Serverfarm Upstate provides are all fitted with 256GiB DDR3 RAM and 2x …