PCIe Intel NTB Networking
Some Intel Xeon server CPUs include a PCI Express Non Transparent Bridge (NTB) function. This can in combination with the Crystal Beach DMA and Intel VT-d technology be used to create a super low latency PCIe clustering solution. PCIe re-driver cards are used to reliably connect the servers using standard PCIe cables.
Dolphin has ported its eXpressWare software stack to natively support Intel NTB enabled CPUs. The main benefits are reduced latency and hardware cost.
Dolphins standard SISCI API PIO latency benchmark scipp implements a full two way memory to memory ping pong transfer between two applications running on the servers. The full round trip latency for a 4 byte message is below 600 nanoseconds which means that the one way latency is below 300 nanoseconds.
Figure 1: scipp latency using Intel NTB.
Crystal Beach DMA transfers are also supported and well integrated with the eXpressWare software stack. DMA can be used to transfer larger blocks of data between local and remote memories and IO addresses.
To connect two servers, you need a transparent PCIe re-driver card that support this configuration. More systems can be interconnected using Dolphins IXS600 Gen3 PCIe switch. Please contact Dolphin for more details.
The Linux Intel NTB eXpressWare software is currently available for beta testers. Please e-mail firstname.lastname@example.org if you would like to test the new capabilities. Please note that you need an Intel NTB enabled server to use this software. Please consult your system vendor or the BIOS users guide to ensure the server supports Intel NTB.