Transparent PCIe Hot add support

Dolphin eXpressWare enables customers to utilize PCI Express hot add

Dolphin eXpressWare adds PCIe hot add support with Linux systems. Cabled PCIe expansion systems and IO devices can dynamically be added, hot swapped or removed from the system without the need to reboot the host.

SmartIO Dynamic Device add and remove

PCIe NTB technology enables software based resource management and enumeration of standard PCIe devices. The process of adding or removing devices can automatic or managed through some basic command line utilities.

The main difference from the traditional setup is that the PCIe host uplink card needs to be replaced with one of Dolphins PCIe NTB cards. The solution works with directly attached PCIe devices as well as PCIe devices installed into PCI Express expansion systems.

The solution does not require any special BIOS version. Hot adding many devices or devices that requires a large PCIe address space may require the BIOS to support 64bit encoding and large PCIe BARs.

Performance

The SmartIO software does not add any overhead to the standard IO performance.

Availability

The flexible device management functionality is currently available with Linux and the MXH830, PXH820/PXH824, PXH830 and PXH840 cards using eXpressWare 5.13.1 or newer. Windows support is planned and will be available with the PCIe 4.0 MXH930, MXH940 and MXH950 cards.

Supported Configurations

The figures below shows some of the supported configurations. The main requirement is to have a Dolphin NTB enabled PCIe card in the host running Linux. Please contact Dolphin for details and requirements if you are connecting the Dolphin NTB card to a non Dolphin target (e.g. an FPGA or expansion box without a Dolphin card installed).

Hot Add PCIe single device

Hot Add PCIe JBOF.

Hot Add PCIe MXS824

How it works

Automatic mode

The Hot Add functionality is easy to use. If Autoborrow mode is enabled, all devices found after a scan operation will automatically be added to the system and behave as if they was present during boot. The typical commands to set up Autoborrow and scan for devices can be seen in the screenshot below; Every time a device is attached, you only need to redo the scan operations.

SmartIO hot add autoborrow

Manual mode

The Hot Add functionality can be used in a selective way if you only want to attach selected devices.

1. Scan for the devices

Smartio Hot add scan list

2. Attache the selected device to the Linux device tree by specifying the Fabric Device Id found using the smartio_tool list command.

Hot Add PCIe SmartIO

3. List the local NVMe drives.

Hot add

4. Gracefully detach the device from the system by using the smartio_tool return command. Use the same Fabric Device Id as used with the borrow command.