Skip to main content

FICON HBAs: The mysterious Bus-tech

Commonly when talking about Fibre Channel the assumption is that you are talking about attaching some sort of storage device to some resource that wants to use that storage. However, Fibre Channel itself does not care about the storage part. Fibre Channel only facilitates opening a channel between two devices, which is referred to as nodes in Fibre Channel lingo - then what you do after that is up to you. Most commonly you will use a protocol called FCP (protocol 0x08), which confusingly stands for Fibre Channel Protocol, which is the protocol responsible to moving SCSI commands over Fibre Channel. Another relatively new protocol is FC-NVMe (protocol 0x28) which is for, wll, NVMe over Fibre Channel without going through any SCSI layers. Another one is FICON which uses protocol 0x1B or 0x1C depending if the frame is from or to a control unit.

A host bus adapter (HBA) is a device that enables a computer to access a network, more or less always some sort of storage network and most commonly an FCP SAN. A common manufacturer of HBAs is QLogic whos HBA QLE2562 is probably one of the most frequently used in the world.

Normal FC HBAs can be found very easily, either new or second hand. Depending on which speed you want they can set you back between $10 to $1,000. A hobbyist would be closer to the $10-$30 range depending if you want 4Gbit/s or 8Gbit/s. The really expensive ones are for 32 Gbit/s which is the current generation.

In mainframe lingo when referring to an HBA we call it a channel. Different word, same function as is the case for many things in the mainframe world given that they evolved in parallel with the PC. N.B: This means that the blog title should have been "FICON channels" but given the blog audience is mostly non-mainframe people I chose to go with the HBA term instead.

Any form of network card, which HBA is part of, has a hardware accelerated part and a software part. A key take-away is that data that is handled in the hardware accelerated part never reaches the OS. If you are familiar with the OSI model, for Ethernet the hardware part is Layer-1, while software commonly takes over processing at Layer-2. This is a bit simplistic but more or less how it works.

For FC the HBA handles FC-0 to FC-2 but possibly even more. I say possibly because this is not easy information to come across. When looking at QLogics various HBA controllers we see that they list protocols like "FCP (SCSI-FCP), IP (FC-IP), FICON (FC-SB-2), FC-TAPE (FCP-2), FC-VI" [QLogic ISP2432 datasheet]. This means that they at least claim that the FC-4 layer is hardware accelerated fully or partly. They do list FICON which is interesting, but without access to any driver or documentation there is a near impossibility of accessing that functionality. The Linux kernel only implements FCP and FC-NVME for these controllers, and QLogic has unsurprisingly not responded to my requests for documentation.

This means that while some common FC-HBAs seems to be able to handle FICON, it is locked away under non-documented APIs. We need an alternative.

DLm virtual tape library

Dell EMC has a product they call DLm which is an virtual tape library. The latest version is called DLm8500 and works by providing FICON connections to your mainframe SAN and presenting itself as e.g. 3590 tape drives. This allows you to migrate a pre-existing tape oriented workflow to using e.g. cloud storage or hard drives without changing the workflow itself. However, the DLm8xxx series is a huge rack filled with NASes and servers, not something a hobbyist would like to run.

The cool thing is that the servers are normal x86 and they use a PCIe card that talks FICON - i.e. a FICON HBA, exactly what we have been looking for (picture 1). The part that takes incoming FICON and translates it to NAS accesses is called an virtual tape engine (VTE).

FICON HBA from DLm8000 VTE
Picture 1: FICON HBA from a DLm8000  virtual tape engine (VTE)
This seems to be the HBA card that Connor found 2016 and he documents the frustration around having the card but no software to use with the card. Luckily, since I purchased the whole VTE I also have the software and drivers to run the card. For good measure, I bought some extra cards for experimentation - happy to borrow them to fellow hobbyists if you have a cool project in mind.

Each card of this particular model have a 4 Gbit/s FICON connection, which is well enough for a hobbyist system.

However, is this card really made by Dell EMC? The answer is legally yes, but it is part of a company they acquired back in 2010 - Bus-tech. Actually, the whole DLm solution is from Bus-tech which becomes evident when looking at the system utilities that are part of the system.

The card itself does not hide this fact:
05:00.0 Network controller: IBM Unknown device 02d6
        Subsystem: Bus-Tech, Inc. Unknown device 0403
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 211
        Region 0: Memory at b2300000 (64-bit, non-prefetchable) [size=1M]
        Region 2: I/O ports at 6000 [size=1K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] Message Signalled Interrupts: Mask- 64bit+ Queue=0/2 Enable+
                Address: 00000000fee00000  Data: 40d3
        Capabilities: [58] Express Legacy Endpoint IRQ 0
                Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
                Device: Latency L0s <64ns, L1 <1us
                Device: AtnBtn- AtnInd- PwrInd-
                Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
                Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
                Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s L1, Port 0
                Link: Latency L0s <256ns, L1 <2us
                Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
                Link: Speed 2.5Gb/s, Width x4
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [1f8] Unknown (11)

When using the software suite Virtuent to interrogate the card this is what we get:

DLm056I: Channel driver version is 4.4.15,
DLm075I: Interface #0: 197888 (0x030500) bus:5 slot:0 type:15 (PEFA-LP) media:3 (FiCon)
DLm076I: Interface #0: hardware s/n: 000D13086098
DLm077I: Interface #0: Firmware emulation type: TRANS, version: 1320 2013/10/28
DLm070I: Interface #0: TRANSX emulation version set to 3
DLm081I: Interface #0: Current state: not started; Desired state: not started; Media Down, Loop Down

So, is this just an FC HBA with FICON support in it? No, turns out this card is a bit more. While SCSI uses quite straight foward commands to communicate and manipulate the storage device, FICON is more complicated. FICON sends small programs called channel command word (CCW) to the controller unit (CU) in charge of the device. This means that for Linux to be a provider of a FICON device, it needs to implement the CUs CCW. From reading the specifications of FICON and 3590 tape drive systems this involves quite a lot of work. However, this HBA in combination with the provided drivers implement all this for us, so that's pretty nice.

This is the limit of how much I have explored the DLm and the FICON HBAs to date, but as you probably can guess these systems seem to have a number of stories left to tell.

One of the things I would like to figure out what it would take to run these cards in a virtualized environment with VT-d or equivalent. That should provide a nice way to experiment as well as running a stable environment next to each other while not costing twice the electricity.

As always, thanks for reading and let me know if you have any questions in the comments below!

Comments

Popular posts from this blog

Buying an IBM Mainframe

I bought an IBM mainframe for personal use. I am doing this for learning and figuring out how it works. If you are curious about what goes into this process, I hope this post will interest you.

I am not the first one by far to do something like this. There are some people on the internet that I know have their own personal mainframes, and I have drawn inspiration from each and every one of them. You should follow them if you are interested in these things:
@connorkrukosky@sebastian_wind@faultywarrior@kevinbowling1 This post is about buying an IBM z114 mainframe (picture 1) but should translate well to any of the IBM mainframes from z9 to z14.

What to expect of the process Buying a mainframe takes time. I never spent so much time on a purchase before. In fact - I purchased my first apartment with probably less planning and research. Compared to buying an apartment you have no guard rails. You are left to your own devices to ensure the state of whatever thing you are buying as it likely…

Powering a mainframe

The last few days have been eventful. I was contacted by the datacenter that the mainframe's cage is now ready for moving in, and the power has been made available. Very exciting! I grabbed my home-made power cables (more on that later) and my best screwdrivers and set off to the datacenter.


The datacenter staff, not needing a forklift in their day-to-day, had managed to solicit the services of a forklift, the associated operator, and some very handy folks to help navigate the mainframe from the storage space to its final location.



After some intense period of fighting the inclination of the road between the storage facility and the cage (and a door that was a bit too small) it was finally in place. Incidentally we were forced to trust the wheels on this pretty rough floor. I did not expect it to roll that well on raw concrete, I was pleasantly surprised. This thing is a tank!

Now, everybody wanted to see if it was working. My machine did not come with a power cable so I had to so…

Open Datacenter Hardware - Leopard Server

Introduction The Leopard is an OpenRack v1 compliant 12V server commissioned by Facebook to offer compute power. It consists of 2x Intel Xeon E5-2678 v3 and is available with either DDR3 or DDR4 memory. The model is manufactured by two vendors primarily: Quanta and Wiwynn.

Leopard features a 24x PCIe slot which can fit either a PCIe card with low profile, or a riser card with 1x 16x and 1x 8x slots. The server also supports a 3.5" SATA drive as well as either an mSATA or an M.2 drive mounted on the motherboard.

Connectivity wise Leopard has a mezzanine card slot allowing for example 10Gb/s or 25Gb/s Ethernet.

Figure 1 and figure 2 shows the server layout. The server is made to fit inside an OpenRack v1 enclosure, at which point it looks something like figure 3. Due to power constraints an OpenRack v1 can fit 30 of these servers before power starts to become an issue. The Leopard servers that the organization Serverfarm Upstate provides are all fitted with 256GiB DDR3 RAM and 2x …