Skip to main content

Open Datacenter Hardware - Leopard Server


Introduction

The Leopard is an OpenRack v1 compliant 12V server commissioned by Facebook to offer compute power. It consists of 2x Intel Xeon E5-2678 v3 and is available with either DDR3 or DDR4 memory. The model is manufactured by two vendors primarily: Quanta and Wiwynn.

Leopard features a 24x PCIe slot which can fit either a PCIe card with low profile, or a riser card with 1x 16x and 1x 8x slots. The server also supports a 3.5" SATA drive as well as either an mSATA or an M.2 drive mounted on the motherboard.

Connectivity wise Leopard has a mezzanine card slot allowing for example 10Gb/s or 25Gb/s Ethernet.
Wiwynn Leopard DDR4 from the front
Figure 1: Wiwynn Leopard DDR4 from the front
Source: Wiwynn OCP design package

Wiwynn Leopard DDR4 from above
Figure 2: Wiwynn Leopard DDR4 from above
Source: Wiwynn OCP design package
Figure 1 and figure 2 shows the server layout. The server is made to fit inside an OpenRack v1 enclosure, at which point it looks something like figure 3. Due to power constraints an OpenRack v1 can fit 30 of these servers before power starts to become an issue. The Leopard servers that the organization Serverfarm Upstate provides are all fitted with 256GiB DDR3 RAM and 2x CPUs and 1x 10Gb/s Ethernet interface, which means that a single rack hosts 60x CPUs and 7.5 TiB RAM for a total network capacity of 300 Gb/s.

Leopard in OCP v1 Rack
Figure 3: Leopard in OCP v1 Rack
Source: Self
The Leopard server family has been superseeded by Tioga Pass, which has resulted in that these servers have now entered the hobbyist market which is how this article came to be.

Review

As Leopard is a family of servers that fullfills the OCP FB Server Intel Motherboard v3 requirements, the name "Leopard" refers to more than one server model - even though they are similar.

In this review the subject is the Quanta DDR3 model also called F06C. As Quanta is a bit secretive with information around this model the exact naming is not known, but F06C is an educated guess extracted from BIOS and BMC verision data.

From the previous article "What is OCP?" the point was made that OCP works with open specifications, so how come that Quanta can choose to keep the model information secret? Doesn't that defeat the purpose of OCP? It's certainly not ideal, but remember that OCP works by defining common interfaces - and really only enforces that those interfaces are compliant with the specification. If a manufacturer choses to keep their implementaiton secret, that's within their rights and up to them - while of course from an open source culture perspective releasing the implemention specifications would be welcomed.

In the end, what we have to work on here is a black-box implementation of the earlier mentioned FB Server v3 specification - which is already pretty good and a great development, as we will see.

The review aims to make it clear what is a design choice of the F06C model by Quanta and what is generic for all Leopards by calling the server either explicitly "F06C" for the former, or "Leopard family" for the latter

Hardware

Digging around in the firmware and following breadcrumbs, it seems the F06C started to be finalized for production around end 2013, start 2014. One of the machines reports a BIOS build date of 2015-08-27, while the BMC software hints towards a relase of it at 2014-09-23. Given that the final release of the FB Server v3 specification is dated November, 2013 the hypothesis seems to make sense.

The chipset used in F06C is C610/X99 series.

Storage

From the specification we can make out that the board should have either an M.2 (referred to by the earlier name NGFF in the specification) or an mSATA slot. The F06C has an M.2 2260 (see figure 4). Previously this article stated that this port was an mSATA, but it has later been discovered that it's an M.2 2260. To date it is unclear if the port is PCIe or SATA backed.

Onboard mSATA storage slot
Figure 4: Onboard M.2 2260 storage slot
Source: Photography by Markus Häll


The F06C also hosts one SATA port on the motherboard hosted by the chipset SATA Controller.

PCIe Expansion

There are two types of PCIe expansion ports on in the Leopard family, the PCIe x24 motherboard slot that can be split into x16 + x8 by using a raiser card, and the PCIe x8 mezzanine card slot. In addition the x24 port can be used as a normal PCIe slot if no raiser card is available.

Network Connectivity

The Leopard uses the mezzanine port as the primary network interface allowing for example a 2x 25Gbit/s Ethernet card to be installed. The equipment reviewed however uses a Mellanox ConnectX-3 EN - a card with one 10Gbit/s SFP+ slot, see figure 5.

Mellanox ConnectX-3 EN Mezzanine card for 10Gbit/s Ethernet
Figure 5: Mellanox ConnectX-3 EN Mezzanine card for 10Gbit/s Ethernet
Source: Mellanox product page
Leopard also has the option for a I210 onboard 10/100/1000 Ethernet interface, however the F06C is without this interface.

Worth noting is that the Mellanox interface available for this platform is surprisingly easy to work with. It is not uncommon for network vendors to enforce that only SFP+ modules with their own brand are to be used, but this ConnectX-3 EN interface appears to be working with the vast majority of SFP+ as well as plain SFP modules. Indeed, when testing we have not found any SFP+ or SFP module that did not work with this interface and not for lack of trying.

I/O Ports

The Leopard family does not come with any type of VGA adapter, and while it is fully feasible to put a graphics card into the PCIe x24 slot the server operator might want to be able to access a local console without adding an extra card to the server. For this, the Leopard family offers a serial port in the front of the server as part of a debug header.

This debug header multiplexes three UARTS into one physical UART. The first is a normal serial port on the host, nothing odd with that. The second is connected to the baseboard management controller, and the third one for something called the midplane. The operator can select which of these three UARTs should be routed to the physical UART by pulsing a specific pin inside the debug header. When doing this review there was no matching header available so the function of these ports have not been tested in any thorough manner.

Finally, the Leopard servers offers two USB connectors, one accessible from the front of the server and one on the motherboard.

Baseboard Management Controller

The baseboard management controller (BMC) is a separate server that lives inside the main server enclosure that offers the server administrator remote possibility to for example power-on/power-off the server.

The system-on-chip where the BMC lives is an Aspeed AST1250 ARM CPU with 128 MiB RAM and 32 MiB replaceable SPI flash, see figure 6. The BMC shares the network with the mezzanine card using network controller sideband interface (NC-SI) allowing a single network cable to be run from the server enclosure. The AST1250 does not come with any VGA/iKVM support, but it can attach USB devices to the host.

The BMC is the driver of the system fans and is the bus master for all sensors in the server. For a detailed list of what it can measure, see appendix A. It is also connected to the low-pin-bus (LPC) bus as part of it being the SuperIO controller for the server.

Aspeed AST1250 system-on-chip, baseboard management controller (BMC)
Figure 6: Aspeed AST1250 system-on-chip, baseboard management controller (BMC)
Source: Photography by Markus Häll

Other

The motherboard on the F06C has clear printing on the motherboard, helping to identify what chip does what, see figure 7.

The Leopard family is required to have dual, easily replaceable, BIOS chips (see figure 8). This makes the server friendly for companies or hobbyists that want to have control over the firmware. For example, by modifying the BIOS flash it is possible to disable the critizied Intel Management Engine (ME) or replace the shipped BIOS with LinuxBoot. More about this in the management section below.

Finally, the Leopard family comes with a removable TPM module attached to the low-pin-count (LPC) bus.


Motherboard overview picture for the Quanta F06C Leopard
Figure 7: Motherboard overview picture for the Quanta F06C Leopard
Source: Photography by Markus Häll


Dual BIOS chips in sockets
Figure 8: Dual BIOS chips in sockets
Source: Photography by Markus Häll

Installation and boot

The Leopard family uses UEFI BIOS and defaults to booting EFI payloads using PXE over IPv6. The boot order for F06C is something like this:
  • Boot PXE over IPv6
  • Boot PXE over IPv4
  • Boot harddrive
This boot order makes the boot take quite a while (5-7 min from power-on), but not anything extraordinary when compared to any other server platform. 

The BIOS for F06C is written by American Megatrends (AMI) and sadly contains a few bugs in the UEFI boot implementation. These issues are not uncommon, but they can be painful and are annoying. In this case the file EFI/BOOT/BOOTX64.EFI needs to exist on the EFI system partition even if it is not the one that will be booted. For the interested reader, see discussions in this Ubuntu bug. The workaround is easy enough however, just make sure you copy an EFI executable (possibly an empty file is enough) to EFI/BOOT/BOOTX64.EFI if your chosen OS does not do this for you (e.g. Ubuntu and Fedora does, Debian does not).

Management

The BMC software is written by American Megatrends and consists essentially of an IPMI server. While other functionalities around virtual media and SSH access has been included, they seem rushed and non-functional. IPv6 is supported for IPMI access.

The software on the BMC is sadly very outdated (2.6.28 kernel, Busybox 1.13.2) but it does work reasonably well if you constrain yourself to using IPMI. On SSH there lives a placeholder for something called System Management Architecture for Server Hardware (SMASH) but trying to access it using the default credentials sysadmin/superuser causes the session to be instantly terminated.

Using ipmitool it is possible to access serial-over-LAN (SOL) functionality to get remote console access to the server. This is extremely useful in theory but due to bugs in the implementation it does not generally survive interface flaps, which is quite common given that the interface is shared. The SOL must according to the Leopard family specification support buffering a fair amount of data, probably to ensure data is not lost, but that feature seems to not work with network connection issues.

For a demo how SOL looks like when the machine boots, click here. The digits in brackets that show in the beginning are POST codes used to display where the BIOS is currently executing.

IPMI allows vendors and OEMs to add additional commands that are not part of the standard. Sadly, these extra commands are not self-describing and if the vendor has not published any documentation things get very hard fast. As part of this review some of the many vendor and OEM specific commands where found and have been included in appendix B and appendix C.

Interestingly, the AST1250 allows arbitrary bus access from the host CPU by default, and the AMI software does not disable this. That means that if you have root on the host CPU, you can execute anything on the BMC or the BMC bus. This means that you can trivially root the BMC CPU, potentially install a root kit in the BMC or even in the BIOS, and then root the server even across reinstalls.

If you're curious on how to dump the BMC flash from the host CPU, this is how you do.
# Make sure you have iomem=relaxed as the kernel boot option
$ sudo ./socflash_x64 of=bmc.bin option=l lpcport=0x2e
ASPEED SOC Flash Utility v.1.18.00 
Static Memory Controller Information: 
CS0 Flash Type is SPI 
CS1 Flash Type is SPI 
CS2 Flash Type is NOR 
CS3 Flash Type is NOR 
CS4 Flash Type is NOR 
Boot CS is 0 
Option Information: 
CS: 0 
Flash Type: SPI 
[Warning] Don't AC OFF or Reboot System During BMC Firmware Update!! 
[SOCFLASH] Flash ID : 1920c2 
Find Flash Chip #1: MXIC MX25L256/257

After a while you will now have a bmc.bin file that contains most parts of the BMC firmware. Note that socflash skips some segments, but you can write your own flasher that allows for arbitrary flash access given that the host CPU has full bus access over LPC.

If this sounds scary, which it probably should, you can rest assured that it is possible to disable bus access over LPC from the BMC, but AMIs software does not appear to export that knob. For more information on how this can be solved, see the Future work section in the end.

Intel Management Engine (ME)

While the Intel ME facility is present on the Leopard family (and indeed any modern Intel CPU) no review of the functionality in the built-in Intel ME has been done. Documentation on what the listed features in figure 9 does is hard to come by. Given that there is already plenty of sensors and that the BMC offers a platform for managing the server, one can make the argument that Intel ME appears redundant.
|  Operational Firmware    3.0.7.145                 |meters) of the platform  |
|  Version                                           |location above the sea   |
|  Recovery Firmware       3.0.7.145                 |level is a signed value  |
|  Version                                           |in hex.                  |
|  ME Vendor Label         0x18791010                |Provide the 80000000     |
|  ME Firmware Features    SiEn+NM+PECIProxy+ICC     |value if the altitude    |
|  ME Firmware Status #1   0x001F0345                |is unknown.              |
|  ME Firmware Status #2   0x38008000                |                         |
|    Current State         Operational               |-------------------------|
|    Error Code            No Error                  |><: Select Screen        |
|  Altitude                12c                       |^v: Select Item          |
|  MCTP Bus Owner          0                         |Enter: Select            |
Figure 9: Intel Management Engine (ME) information from F06C
Source: BIOS screen on bootup

As the BIOS chips are accessible and that the Leopard family requires the BMC to have access to flashing these chips, disabling Intel ME by using a utility called me_cleaner is straight forward and has been done successfully on the F06C model.

Note that the BMC by default will use sensors provided by ME which means that disabling ME will cause the fan controller to enter emergency mode and put fans to 100%. That behavior can be overridden, but it's not trivial.

Physical and Operational

The Open Compute guide site has excellent online guides on how to execute common and uncommon physical operations on various OCP servers, and the Leopard family is included among those guides. Given that F06C is just one model in the family the steps in the guides sometimes mentions steps that should be omitted, like removing the 10/100/1000 Gigabit Ethernet connection - which the F06C does not have.

One noteworthy fact about the F06C is that it is almost noiseless when it is running. This is most likely due to the optimized airflow design as part OpenRack and the use of high quality fans. You will have no issue having a normal conversation with your co-workers even when standing next to a full rack of powered on F06C servers.

A slightly hidden fact is that there are ATX 12V connectors (2x P4, 1x EPS12V) on the back of the server (see figure 10) allowing the F06C to be powered using any normal ATX PSU as long as it has enough power to supply the full server on its 12V rail. Experiments has shown that while 47A is the max the hotswap controller (HSC) will report, running with a PSU with a 40A 12V rail works just fine as long as it is a single rail and not 2x 12V 20A for example.

Figure 10: Power back plane showing the 3x 12V ATX connectors
Source: Photography by Markus Häll
Effectively what this means is that even if you do not have space for a full OpenRack, or you just want to get started with OCP hardware, you can by running them using a normal server PSU. Include the fact that the network card runs fine with many SFP modules, it is even feasible to use one of these servers at home given the low noise level.

Review Summary and Future Work

Overall the F06C is an excellent server that shows that good quality was part of its success criteria. The fact that the firmware chips for BIOS and BMC are replaceable, that BIOS has a backup chip, as well as that they are flashable from the BMC makes the server exciting for the company that wants to have the option of maintaining their own firmware.

Incidentally, the BIOS and the BMC firmware is the major pain-points of the F06C model. Since the expected lifetime for these servers in the Serverfarm Upstate organization is to at least 2028 we are planning on replacing our BIOS and BMC firmware with custom open source components using LinuxBoot and a project we're calling u-bmc. That should make the servers more manageable for us, add more features (USB iKVM for example is something we're looking into) and allow us to keep them reasonable up to date for the coming decade. We're also watching closely the security reports that come out around Intel ME to guide us whether or not it is worth the hassle for us to disable it on all our servers.

That is all for this review, do let me know if you have any questions or corrections in the comments.
Thanks for reading!

Appendix: IMPI Sensors

Output of "ipmitool sensors" for the F06C server.
P0 Temp          | 28.000     | degrees C  | ok    | na        | na        | na        | na        | 91.000    | na        
P1 Temp          | 28.000     | degrees C  | ok    | na        | na        | na        | na        | 91.000    | na        
P0 DTSmax        | 95.000     | degrees C  | ok    | na        | na        | na        | na        | na        | na        
P1 DTSmax        | 95.000     | degrees C  | ok    | na        | na        | na        | na        | na        | na        
P0 Therm Margin  | -67.000    | degrees C  | ok    | na        | na        | na        | na        | -4.000    | na        
P1 Therm Margin  | -67.000    | degrees C  | ok    | na        | na        | na        | na        | -4.000    | na        
P3V3             | 3.355      | Volts      | ok    | na        | 2.973     | na        | na        | 3.625     | na        
P5V              | 5.058      | Volts      | ok    | na        | 4.501     | na        | na        | 5.493     | na        
P12V             | 12.420     | Volts      | ok    | na        | 11.340    | na        | na        | 13.200    | na        
P1V05_STBY       | 1.058      | Volts      | ok    | na        | 0.951     | na        | na        | 1.147     | na        
P1V8_AUX         | 1.813      | Volts      | ok    | na        | 1.627     | na        | na        | 1.980     | na        
P3V3_AUX         | 3.371      | Volts      | ok    | na        | 2.973     | na        | na        | 3.625     | na        
P5V_AUX          | 5.082      | Volts      | ok    | na        | 4.501     | na        | na        | 5.493     | na        
P3V_BAT          | 3.161      | Volts      | ok    | na        | 2.726     | na        | na        | 3.596     | na        
Inlet Temp       | 31.000     | degrees C  | ok    | na        | na        | na        | na        | 40.000    | na        
Outlet Temp      | 28.000     | degrees C  | ok    | na        | na        | na        | na        | 80.000    | na        
PCH Temp         | 40.000     | degrees C  | ok    | na        | na        | na        | na        | 66.000    | na        
HSC Input Volt   | 12.200     | Volts      | ok    | na        | 11.300    | na        | na        | 13.200    | na        
HSC Input Power  | 60.000     | Watts      | ok    | na        | na        | na        | na        | 501.000   | na        
HSC Temp         | 26.000     | degrees C  | ok    | na        | na        | na        | na        | 86.000    | na        
HSC Sts Low      | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
HSC Sts High     | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
HSC Output Curr  | 4.600      | Amps       | ok    | na        | na        | na        | na        | 47.800    | na        
SYS FAN0         | 2100.000   | RPM        | ok    | na        | 500.000   | na        | na        | 9000.000  | na        
SYS FAN1         | 2100.000   | RPM        | ok    | na        | 500.000   | na        | na        | 9000.000  | na        
P0 VR Temp       | 32.000     | degrees C  | ok    | na        | na        | na        | na        | 92.000    | na        
P0 core VR Vol   | 1.820      | Volts      | ok    | na        | 1.350     | na        | na        | 1.960     | na        
P0 core VR Curr  | 3.000      | Amps       | ok    | na        | na        | na        | na        | 108.000   | na        
P0 core VR POUT  | 5.000      | Watts      | ok    | na        | na        | na        | na        | 240.000   | na        
P0 core VR PIN   | 10.000     | Watts      | ok    | na        | na        | na        | na        | 240.000   | na        
P1 VR Temp       | 31.000     | degrees C  | ok    | na        | na        | na        | na        | 92.000    | na        
P1 core VR Vol   | 1.820      | Volts      | ok    | na        | 1.350     | na        | na        | 1.960     | na        
P1 core VR Curr  | 3.500      | Amps       | ok    | na        | na        | na        | na        | 108.000   | na        
P1 core VR POUT  | 6.000      | Watts      | ok    | na        | na        | na        | na        | 240.000   | na        
P1 core VR PIN   | 10.000     | Watts      | ok    | na        | na        | na        | na        | 240.000   | na        
P0 DIMM VR0 Temp | 31.000     | degrees C  | ok    | na        | na        | na        | na        | 83.000    | na        
P0 DIMM VR0 Vol  | 1.510      | Volts      | ok    | na        | 1.430     | na        | na        | 1.570     | na        
P0 DIMM VR0 Curr | 0.500      | Amps       | ok    | na        | na        | na        | na        | 69.000    | na        
P0 DIMM VR0 POUT | 1.000      | Watts      | ok    | na        | na        | na        | na        | 47.000    | na        
P0 DIMM VR0 PIN  | 3.000      | Watts      | ok    | na        | na        | na        | na        | 47.000    | na        
P0 DIMM VR1 Temp | 31.000     | degrees C  | ok    | na        | na        | na        | na        | 83.000    | na        
P0 DIMM VR1 Vol  | 1.510      | Volts      | ok    | na        | 1.430     | na        | na        | 1.570     | na        
P0 DIMM VR1 Curr | 1.500      | Amps       | ok    | na        | na        | na        | na        | 69.000    | na        
P0 DIMM VR1 POUT | 1.000      | Watts      | ok    | na        | na        | na        | na        | 47.000    | na        
P0 DIMM VR1 PIN  | 4.000      | Watts      | ok    | na        | na        | na        | na        | 47.000    | na        
P1 DIMM VR0 Temp | 29.000     | degrees C  | ok    | na        | na        | na        | na        | 83.000    | na        
P1 DIMM VR0 Vol  | 1.510      | Volts      | ok    | na        | 1.430     | na        | na        | 1.570     | na        
P1 DIMM VR0 Curr | 1.000      | Amps       | ok    | na        | na        | na        | na        | 69.000    | na        
P1 DIMM VR0 POUT | 3.000      | Watts      | ok    | na        | na        | na        | na        | 47.000    | na        
P1 DIMM VR0 PIN  | 4.000      | Watts      | ok    | na        | na        | na        | na        | 47.000    | na        
P1 DIMM VR1 Temp | 27.000     | degrees C  | ok    | na        | na        | na        | na        | 83.000    | na        
P1 DIMM VR1 Vol  | 1.510      | Volts      | ok    | na        | 1.430     | na        | na        | 1.570     | na        
P1 DIMM VR1 Curr | 1.000      | Amps       | ok    | na        | na        | na        | na        | 69.000    | na        
P1 DIMM VR1 POUT | 2.000      | Watts      | ok    | na        | na        | na        | na        | 47.000    | na        
P1 DIMM VR1 PIN  | 4.000      | Watts      | ok    | na        | na        | na        | na        | 47.000    | na        
P0 Package Power | 9.000      | Watts      | ok    | na        | na        | na        | na        | na        | na        
P1 Package Power | 9.000      | Watts      | ok    | na        | na        | na        | na        | na        | na        
P0 DIMM01 Temp   | 27.000     | degrees C  | ok    | na        | na        | na        | na        | 81.000    | na        
P0 DIMM23 Temp   | 26.000     | degrees C  | ok    | na        | na        | na        | na        | 81.000    | na        
P1 DIMM01 Temp   | 24.000     | degrees C  | ok    | na        | na        | na        | na        | 81.000    | na        
P1 DIMM23 Temp   | 25.000     | degrees C  | ok    | na        | na        | na        | na        | 81.000    | na        
C1 Local Temp    | na         | degrees C  | na    | na        | na        | na        | na        | 94.000    | na        
C1 Remote Temp   | na         | degrees C  | na    | na        | na        | na        | na        | 83.000    | na        
C2 Local Temp    | na         | degrees C  | na    | na        | na        | na        | na        | 94.000    | na        
C2 Remote Temp   | na         | degrees C  | na    | na        | na        | na        | na        | 83.000    | na        
C3 Local Temp    | na         | degrees C  | na    | na        | na        | na        | na        | 94.000    | na        
C3 Remote Temp   | na         | degrees C  | na    | na        | na        | na        | na        | 83.000    | na        
C4 Local Temp    | na         | degrees C  | na    | na        | na        | na        | na        | 94.000    | na        
C4 Remote Temp   | na         | degrees C  | na    | na        | na        | na        | na        | 83.000    | na        
CPU0 Error       | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
CPU1 Error       | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P0_CH0DIMM0_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P0_CH0DIMM1_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P0_CH1DIMM0_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P0_CH1DIMM1_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P0_CH2DIMM0_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P0_CH2DIMM1_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P0_CH3DIMM0_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P0_CH3DIMM1_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P1_CH0DIMM0_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P1_CH0DIMM1_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P1_CH1DIMM0_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P1_CH1DIMM1_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P1_CH2DIMM0_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P1_CH2DIMM1_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P1_CH3DIMM0_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
P1_CH3DIMM1_Sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
SEL Status       | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
DCMI Watchdog    | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
NTP Status       | 0x0        | discrete   | 0x0180| na        | na        | na        | na        | na        | na        
Chassis Pwr Sts  | 0x0        | discrete   | 0x0480| na        | na        | na        | na        | na        | na        
VR HOT           | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
CPU_DIMM HOT     | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
Airflow          | na         | CFM        | na    | na        | na        | na        | na        | na        | na        
Sys booting sts  | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
System Status    | 0x0        | discrete   | 0x0380| na        | na        | na        | na        | na        | na        
Processor Fail   | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
C2 NVMe Status   | na         | discrete   | na    | na        | na        | na        | na        | na        | na        
C2 NVMe Warn     | na         | discrete   | na    | na        | na        | na        | na        | na        | na        
C2 NVMe CTemp    | na         | degrees C  | na    | na        | na        | na        | na        | 83.000    | na        
C2 NVMe PDLU     | na         | unspecified | na    | na        | na        | na        | na        | 101.000   | na        
C2 NVMe Power    | na         | Watts      | na    | na        | na        | na        | na        | 167.000   | na        
C3 NVMe Status   | na         | discrete   | na    | na        | na        | na        | na        | na        | na        
C3 NVMe Warn     | na         | discrete   | na    | na        | na        | na        | na        | na        | na        
C3 NVMe CTemp    | na         | degrees C  | na    | na        | na        | na        | na        | 83.000    | na        
C3 NVMe PDLU     | na         | unspecified | na    | na        | na        | na        | na        | 101.000   | na        
C3 NVMe Power    | na         | Watts      | na    | na        | na        | na        | na        | 167.000   | na        
C4 NVMe Status   | na         | discrete   | na    | na        | na        | na        | na        | na        | na        
C4 NVMe Warn     | na         | discrete   | na    | na        | na        | na        | na        | na        | na        
C4 NVMe CTemp    | na         | degrees C  | na    | na        | na        | na        | na        | 83.000    | na        
C4 NVMe PDLU     | na         | unspecified | na    | na        | na        | na        | na        | 101.000   | na        
C4 NVMe Power    | na         | Watts      | na    | na        | na        | na        | na        | 167.000   | na        

Appendix B: IPMI OEM Commands

This is the list of the known Quanta specific OEM commands for the F06C Leopard model.
NetFn: 0x30
0x18 QuantaSetSystemMac
0x19 QuantaGetSystemMac
0x1A QuantaSetProcessorInfo
0x1B QuantaGetProcessorInfo
0x1C QuantaSetDimmInfo
0x1D QuantaGetDimmInfo
0x73 QuantaSendPostStart
0x74 QuantaSendPostEnd
0x77 QuantaSetPPIN
0x78 QuantaGetPPIN
0xEE QuantaSetDeviceGuid
0xEF QuantaSetSystemGuid
0x32 QuantaGetBmcInfo
0x33 QuantaSetChassisType
0x7D QuantaGetMBPosition
0x99 QuantaSetBiosUpdateConfig
0xA1 QuantaSetSysNicInfo
0xA2 QuantaGetSysNicInfo
0xE0 QuantaSetGPIO
0xE1 QuantaGetGPIO
0xD8 QuantaSetMezzProtocolPriority
0xD9 QuantaGetMezzProtocolPriority
0xDA QuantaSetPowerCapping
0xDB QuantaGetPowerCapping
0xE4 QuantaSetNTP
0xE5 QuantaGetNTP
0x48 QuantaReplaySolBuffer
0x90 QuantaPidReload
0x91 QuantaPidDebugMsg
0x97 QuantaSetSysFanDuty
0xD0 QuantaGetFastPROCHOT
0xD1 QuantaSetFastPROCHOT
0x47 QuantaGetSELFilter
0x46 QuantaSetSELFilter
0x41 QuantaPerformPrimaryBiosRecovery
0x42 QuantaPerformGoldBiosRecovery
0x43 QuantaSetDualBiosMux
0x44 QuantaSetDualBiosRecovery
0x45 QuantaGetDualBiosStatus
0x40 QuantaFirstTimeBiosBoot
0x54 QuantaSetBIOSVersionObtain
0xDE QuantaSolDump
0x49 QuantaGetPostCode
0x50 QuantaGetDualBiosVersion
0x51 QuantaGetVRVersion
0xD3 QuantaStoreVRVersionToBMC
0x52 QuantaSetBIOSBootOrder
0x53 QuantaGetBIOSBootOrder
0xD2 QuantaSetVRMonitor
0xD4 QuantaGetVRMonitor
0x17 QuantaGetCpldInfo
0x58 QuantaSetBiosVersion
0x83 QuantaGetBiosVersion
0xDC QuantaSetEINCollection
0xDD QuantaGetEINCollection
0xDF QuantaGetPIN
0x55 QuantaGetBIOSFlashInfo
0x21 QuantaEnterFSCUpdateMode
0x22 QuantaWriteFSCConfigData
0x23 QuantaExitFSCUpdateMode
0x24 QuantaReadFSCConfigData
0x25 QuantaRestoreFSCControl
0x26 QuantaGetPWMValue
0x27 QuantaSetPWMValue
0x29 QuantaGetFSCVersion
0x2A QuantaIntelFSCDebugMsg
0x61 QuantaOCPEnterFSCUpdateMode
0x62 QuantaOCPExitFSCUpdateMode
0x63 QuantaOCPReadFSCProfileHeader
0x64 QuantaOCPWriteFSCProfileHeader
0x65 QuantaOCPReadFSCProfile
0x66 QuantaOCPWriteFSCProfile
0x67 QuantaOCPReadZoneControlSetting
0x68 QuantaOCPWriteZoneControlSetting
0x69 QuantaOCPGetPWMValue
0x70 QuantaOCPSetPWMValue
0x71 QuantaOCPGetPWMNum
0x6A QuantaOCPSetMachineConfig
0x6B QuantaOCPGetMachineConfig
0x6C QuantaOCPSetCFMProfileID
0x6D QuantaOCPGetCFMProfileID
0x72 QuantaOCPRestoreFSC
0x6E QuantaBIOSSetPPR
0x6F QuantaBIOSGetPPR
0x30 QuantaSetPHYResetStatus
0x31 QuantaGetPHYResetStatus
0xD6 QuantaClearMemErrInject
0xFC QuantaTestTrap
0xB0 QuantaSetNetworkSequence
0xB1 QuantaGetNetworkSequence
0xEA QuantaEnableDmsg
0xD7 QuantaIERRLogDump
0xFD QuantaNetworkDebugDump

NetFn: 0x2e
0x21 QuantaSetUbootEthaddr
0x22 QuantaSetUbootEth1addr
0x24 QuantaGetUbootEthaddr
0x25 QuantaGetUbootEth1addr

Appendix C: Example of OEM commands

These commands have been reverse engineered through educated guesses and trial and error. It is possible to do the same for all other commands listed in Appendix B if one is willing to spend the time.

QuantaSetBIOSVersionObtain (0x30 0x54)
Will read BIOS versions from flash. This requires the board to be powered off to not have multiple SPI masters on the same bus.
Argument 1: 0x00 = Only execute if already powered off
            0x01 = Power off and execute
            Else: Invalid argument

QuantaSetDualBiosMux (0x30 0x43)
Set the mux which BIOS ROM to use primarily when booting.
Will fail if the BIOS update thread is currently running, try again in that case.
Argument 1: 0x00 = Only execute if already powered off
            0x01 = Power off and execute
            Else: Invalid argument
Argument 2: 0x00 = ROM 1
            0x01 = ROM 2
            Else: Invalid argument

QuantaSetPWMValue (0x30 0x27)
Set fan speed as a percentage of max RPM. Disables BMC fan logic.
Argument 1: 0x00 = Fan 1
            0x01 = Fan 2
            Else: Invalid argument
Argument 2: 0x00 = 0%
            ..
            0x16 = 22% (~2600RPM)
            ..
            0x64 = 100% (~9000 RPM)
            Else: Invalid argument

QuantaRestoreFSCControl (0x30 0x25)
Restore fan control to BMC fan logic.
No arguments.

Comments

  1. Hi, friend. Could you tell me what is the username and password of the ipmi? I searched the web and tried many times but can't login ipmi, and can't use ipmitool

    ReplyDelete
    Replies
    1. Hi! On these servers, and in some other platforms I've seen, it's USERID / PASSWORD.
      Note that when you're on the local machine and talking to the BMC usually that's a trusted connection (over KCS and not IP) and no username and password is required.

      Delete
    2. Thank you, you are so cool.

      Delete

Post a Comment

Popular posts from this blog

Buying an IBM Mainframe

I bought an IBM mainframe for personal use. I am doing this for learning and figuring out how it works. If you are curious about what goes into this process, I hope this post will interest you. I am not the first one by far to do something like this. There are some people on the internet that I know have their own personal mainframes, and I have drawn inspiration from each and every one of them. You should follow them if you are interested in these things: @connorkrukosky @sebastian_wind @faultywarrior @kevinbowling1 This post is about buying an IBM z114 mainframe (picture 1) but should translate well to any of the IBM mainframes from z9 to z14. Picture 1: An IBM z114 mainframe in all its glory Source: IBM What to expect of the process Buying a mainframe takes time. I never spent so much time on a purchase before. In fact - I purchased my first apartment with probably less planning and research. Compared to buying an apartment you have no guard rails. You are left

System z on contemporary zLinux

IBM System z supports a handful of operating systems; z/VM, z/VSE, z/OS, z/TPF, and finally zLinux. All the earlier mentioned OSes are proprietary except for zLinux which is simply Linux with a fancy z in the name. zLinux is the term used to describe a Linux distribution compiled for S390 (31 bit) or S390X (64 bit). As we are talking about modern mainframes I will not be discussing S390, only S390X. There is a comfortable amount of distributions that support S390X - more or less all of the popular distributions do. In this  list  we find distributions like Debian, Ubuntu, Gentoo, Fedora, and RHEL. Noticeably Arch is missing but then again they only have an official port for x86-64. This is great - this means that we could download the latest Ubuntu, boot the DVD, and be up and running in no time, right? Well, sadly no. The devil is, as always, in the details. When compiling high level code like C/C++/Go the compiler needs to select an instruction set to use for the compiled binar

Brocade Fabric OS downloads

Fabric OS is what runs on the SAN switches I will be using for the mainframe. It has a bit of annoying upgrade path as the guldmyr blog can attest to. TL;DR is that you need to do minor upgrades (6.3 -> 6.4 -> 7.0 -> ... > 7.4) which requires you to get all  Fabric OS images for those versions. Not always easy. So, let's make it a bit easier. Hopefully this will not end up with the links being taken down, but at least it helped somebody I hope. These downloads worked for me and are hash-verified when I could find a hash to verify against. Use at your own risk etc. The URLs are: ftp://ftp.hp.com/pub/softlib/software13/COL59674/co-168954-1/v7.3.2a.zip ftp://ftp.hp.com/pub/softlib/software13/COL59674/co-157071-1/v7.2.1g.zip ftp://ftp.hp.com/pub/softlib/software13/COL59674/co-150357-1/v7.1.2b.zip ftp://ftp.hp.com/pub/softlib/software12/COL38684/co-133135-1/v7.0.2e.zip ftp://ftp.hp.com/pub/softlib/software13/COL22074/co-155018-1/v6.4.3h.zip ftp://ftp.hp.c