Skip to main content

Fibre Channel research

The last couple of weeks I've been traveling for work, and I spent a long flight reading some publications I had saved up about mainframe and Fibre Channel. I will talk about some good sources of information later, but for now know that FICON runs over Fibre Channel, and indeed a lot of the reasons behind why Fibre Channel is known to be really reliable is that it more or less came out of IBMs requirements for accessing mainframe disk storage.

Enterprise disk storage has this annoying issue that the disk arrays are:
  • Heavy
  • Power-hungry
  • Ridiculously expensive
An example: DS6800 is widely known to be an unreliable platform, and I would certainly never run a mainframe on it based on my experiences repairing mine. However it is as small as disk arrays get, but it is still 50 kg empty, and consumes 500W of power!

However the DS6800 is kind of a black sheep in reliability. What if we look at something bigger and better? The big brother DS8000 is known to be rock solid for example. The downside is of course that it weighs a bit more than one metric ton and consumes 6 kW of power. Not great for hobbyists.

Ideally I would like to have something that allows me to use the power efficiency of x86, the speed of SSD, at the cost of reliability. I do not need 99.999% uptime, I'm OK to use a conventional server, and I think fellow hobbyists would agree with me.

What we need is a storage system that runs in software, using the economies of scale to our advantage. That's why I started Fikonfarm. The goal is to implement Fibre Channel and FICON in software and somehow hook the mainframe to it. The keyword being "somehow".

I mentioned before that I have 2x Brocade 7800 switches. They are like a conventional Fibre Channel switch, but some of the ports are internally routed to an FPGA that converts the FC traffic (FICON or SCSI, doesn't matter) to TCP/IP. The idea is to figure out how the protocol they talk works so I can make Fikonfarm look like one of those switches.

That brings me to what I've been up to the last couple of weekends. Just that - figuring out the Brocade 7800 protocol. I've come so far to be able to establish an FCIP tunnel with the switch and keep it up. I can also receive FC traffic, as well as send some very very basic frames. Nothing usable yet, but it's pretty exciting :-).

Brocade 7800 protocol dissector in wireshark
Figure 1: Wireshark dissector

What you see in Figure 1 is the Wireshark dissector plugin for the Brocade 7800 protocol that I wrote to be able to map my thesis to real traffic. While I could implement a thesis in Fikonfarm straight away and see what happens, it is useful to be able to take a working session dump and apply the thesis on straight away. That way the outliers are visible, and it is harder to overfit a thesis. E.g. if a byte is 0x10 somewhere in the protocol - you could assume it is static, but it could also be e.g. the hour of the day. Without an extensive dump to test against it is hard to see what values never change, which do, and which seem interesting.

Anyway. When I've come so far as to be able to send and receive frames somewhat reliably I will write up the protocol details and hopefully contribute the dissector upstream. For now it's available on Github if you want it, you can try it on this PCAP if you're curious. Of course, any help is always welcome :-).

Now then, where would one go to learn more about these things? I have two sources I can recommend.

Brocade Reference

Brocade's reference for mainframe connecitivty
Figure 2: Brocade's reference for mainframe connecitivty
One is this practical book on mainframe connectivity by Dr. Guendert (former Brocade, now IBM). Even if you have no background in Fibre Channel this book will teach you how things work, and what options there are. The FC over IP (FCIP) discussed above is even mentioned.

The only downside I can think of is that it is only talking about Brocade solutions when discussing implementations. Keep that salt handy when it speaks about how good Brocade is compared to other vendors.

You can download the book legally for free here.

INCITS Standard Documents

I was about to add the number of pages in "Brocade Mainframe Connectivity Solutions" as a con, but given that my second recommendation is the standard documents themselves which are 10x as long, that seemed silly.

The Fibre Channel standard is divided in many many substandards. The owner of these standards have changed over the passage of time, so it can be hard to figure out what the documents are called. Right now it seems INCITS is the main owner of the standard however.

Here are some good standards to look at:

  • FC-SB-3 - this is classical FICON (before zHPF entered the game)
  • FC-FC-5 - this is the layer over the physical media, it explains how FC is framed and assembled
  • FC-SW-6 - explains how FC switches connect to eachother
  • FCP-4 - covers SCSI for FC
The list goes on, but that will give you an idea of how many standards there are. The number in the end is the version of the standard and while the latest probably gives you the most details about the protocol, it can also omit references to obsolete features that may well be in use in system that you would like to integrate with. FC-SW-6 for example contains 4 fields that are not described more than that they have been obsoleted in previous standards.

Another issue is that a single PDF of these standards, e.g. FC-SB-3, costs $60 to get. If you look at the Brocade Standard Compliance page you will find a lot of references to various versions of these standards.

However, there is another way. One trick I wish to share is that in a lot of cases you can search for the INCITS number as well as "public review" (e.g. "incits 545-2019 public review"). This will in the majority of cases give you a drafted version of the standard that was close to being finalized. It is nothing to base a commercial product on, but it is absolutely fine to use to study the behavior of the protocols as well as e.g. maintaining the Wireshark dissectors.


I certainly have a lot of work in front of me to build Fikonfarm and getting mainframe DASD up and running without extra hardware. I expect this to take a few years realistically, which means this will not block my normal mainframe tinkering - it is merely my long-term strategy to allow myself to not have to care about broken disk controllers, and focus on what I enjoy: unraveling the mysteries of the mainframe.

Thanks for reading,


Popular posts from this blog

Buying an IBM Mainframe

I bought an IBM mainframe for personal use. I am doing this for learning and figuring out how it works. If you are curious about what goes into this process, I hope this post will interest you. I am not the first one by far to do something like this. There are some people on the internet that I know have their own personal mainframes, and I have drawn inspiration from each and every one of them. You should follow them if you are interested in these things: @connorkrukosky @sebastian_wind @faultywarrior @kevinbowling1 This post is about buying an IBM z114 mainframe (picture 1) but should translate well to any of the IBM mainframes from z9 to z14. Picture 1: An IBM z114 mainframe in all its glory Source: IBM What to expect of the process Buying a mainframe takes time. I never spent so much time on a purchase before. In fact - I purchased my first apartment with probably less planning and research. Compared to buying an apartment you have no guard rails. You are left

Brocade Fabric OS downloads

Fabric OS is what runs on the SAN switches I will be using for the mainframe. It has a bit of annoying upgrade path as the guldmyr blog can attest to. TL;DR is that you need to do minor upgrades (6.3 -> 6.4 -> 7.0 -> ... > 7.4) which requires you to get all  Fabric OS images for those versions. Not always easy. So, let's make it a bit easier. Hopefully this will not end up with the links being taken down, but at least it helped somebody I hope. These downloads worked for me and are hash-verified when I could find a hash to verify against. Use at your own risk etc. The URLs are: ftp://ftp.hp.c

System z on contemporary zLinux

IBM System z supports a handful of operating systems; z/VM, z/VSE, z/OS, z/TPF, and finally zLinux. All the earlier mentioned OSes are proprietary except for zLinux which is simply Linux with a fancy z in the name. zLinux is the term used to describe a Linux distribution compiled for S390 (31 bit) or S390X (64 bit). As we are talking about modern mainframes I will not be discussing S390, only S390X. There is a comfortable amount of distributions that support S390X - more or less all of the popular distributions do. In this  list  we find distributions like Debian, Ubuntu, Gentoo, Fedora, and RHEL. Noticeably Arch is missing but then again they only have an official port for x86-64. This is great - this means that we could download the latest Ubuntu, boot the DVD, and be up and running in no time, right? Well, sadly no. The devil is, as always, in the details. When compiling high level code like C/C++/Go the compiler needs to select an instruction set to use for the compiled binar