Dolphin SISCI Developer's Kit

System architects seeking to maximize distributed application performance are exploring PCI Express® networking. Dolphin’s Software Infrastructure Shared-Memory Cluster Interconnect (SISCI) API makes developing PCI Express® Network applications faster and easier. The SISCI API is a well established API for shared memory environments and can be used for both intra and inter system communication.

In a multiprocessing architecture with PCI Express®, the SISCI API enables PCI Express® based applications to use distributed resources like CPUs, I/O, and memory. The resulting application features reduced system latency and increased data throughput. The Dolphin SISCI Developers kit (Software Infrastructure Shared-Memory Cluster Interconnect) consists of driver and API software, tools, documentation and source needed to develop your own embedded application utilizing the low latency and high performance of a Dolphin PCI Express Network. The development kit provides a C system call interface to ease customer integration to the PCI Express Network. SISCI enables customer application to easily bypass the limitations of traditional network solutions, avoiding time consuming operating system calls, and network protocol software overhead. The SISCI software supports clusters of hundreds of nodes.

The SISCI API has been defined in the European Esprit project 23174 as an industry standard Application Programming Interface (API) for shared memory based clustering. The SISCI API is available for the Dolphin Express PX, IX, DX and SCI technology from Dolphin.

Features

• Shared memory API for clusters and SMP's
• Reflective Memory / PCIe Multicast
• PCI Express Peer to Peer support
• PCI Express replicated/reflective memory support
• Distributed shared memory
• Easy deployment of DMA transfers
• Low latency MPICH2 - NMPI
• Create and trigger remote application level interrupts
• Direct remote CPU load and stores, memcpy()
• Windows, RTX, VxWorks and Linux OS support
• Caching and error checking support
• Events and callbacks

Data transfers

The SISCI API supports data transfers between local and remote memory addresses and PCIe devices. The SISCI API makes it easy to write a program that can communicate with other SISCI programs independent if they are running on a local or remote system. You can easily move your SISCI application from a large SMP to a PCI Express network or a mix of SMP's and PCIe networks.

PIO Transfers: The local CPU does a load or store operation or memcpy() between:

• Two local memory addresses
• Local memory address and remote memory address
• Local memory address and local PCIe device
• Local memory address and remote PCIe device

DMA transfers: The PCI Express DMA engine can transfer data between:

• Two local memory addresses
• Local memory address and remote memory address
• Local memory address and local PCIe device
• Local memory address and remote PCIe device

A PCIe master device (e.g. GPU, FPGA) can transfer data between:

• PCIe device and local memory
• PCIe device access to remote memory
• PCIe device access to another local PCIe device (peer to peer transfer)
• PCIe device access to remote PCIe device (remote peer to peer transfer)

Hard Real-Time Behavior

PCI Express is well suited for (hard) real time applications - the data transfers are implemented in hardware (ASICS) based upon PCIe protocols that provides a fair sharing of bandwidth and resources. Even if you create communication hot spots (all nodes communicating to one node, max throughput) the underlying hardware will cause the available bandwidth to be shared almost perfectly.

The most optimal way to preserve the maximum real-time behavior all the way up to the application is to only use SISCI PIO transfers in combination with a real-time operating system that will allow the application to run without being interrupted by the operating system. SISCI PIO transfers will be performed within the application address space and by the CPU running the application, the data transfer does not depend on any operating system functionality or communication library.

More details on SISCI used with Real-Time operating systems:

• RedHawk
• VxWorks

Typical Applications for SISCI

Many applications has successfully migrated to the powerful SISCI environment after it initially become available in 1999. The following application areas are known to Dolphin:

• Low latency, low jitter real-time simulators, electronic trading and industry control systems
• Bus bridging PCIe-PCI-XMC-VME-PXI-VPX-cPCI using carrier cards
• Remote access to PCIe IO devices
• Direct device to device peer to peer transfers, direct GPU and FPGA communication
• Reflective memory functionality
• High Availability servers / Fast fail-over
• Fat pipes / low latency messaging
• Bridging between operating systems and endianness

Shared Memory Programming Made Easy

Using SISCI you can easily create an application or library that can:

• Send and receive interrupt from remote user applications
• Send and receive messages
• Access local and remote memory directly
• Access to local and remote device memory
• Use DMA to move large amounts of data between nodes
• Set up and manage PCIe peer to peer  device transfers (GPU, Intel Phi, FPGA, NVMe etc.)

SISCI Documentation

The following SISCI API documentation is available:

• SISCI Users Guide (pdf)
• SISCI Functional Specification (html)
• SISCI Functional Specification (pdf)
• SISCI API Presentation (pdf)

Examples and Test Programs

The developers kit includes source for several small test and example applications that can be used as a basis or reference for new application development.

Success Stories

The SISCI API has been an important building block for various embedded projects and products since 1999. Take a look at some of the success stories and white papers.

Availability

Dolphin SISCI Developers kit are currently available for Dolphin Express MX PX, IX, DX and SCI cluster interconnect. The API is agnostic to the underlying hardware and enables applications to easily migrate to faster and newer products without any code changes.

The list of currently supported operating systems and platforms are:

Interconnect	Architecture	Supported Operating System
Dolphin Express MX	Intel x86, x64	Linux, Windows, VxWorks 6.9 / 7 SMP
Dolphin Express MX	ARM32, ARM64v8 X-Gene, Tegra K1, X1, X2	Linux
Dolphin Express PX	Intel x86, x64	Linux, Windows, RTX, VxWorks 6.9 / 7 SMP
Dolphin Express PX	ARM 32, ARM64v8 X-Gene, Tegra K1, X1, X2	Linux
Dolphin Express IX	Intel x86, x64	Linux, Windows, RTX, VxWorks 6.9 / 7 SMP
Dolphin Express IX	ARM32, ARM64v8 X-Gene, Tegra K1, X1, X2	Linux
Dolphin Express DX	Intel x86, x64	Linux, Windows
Dolphin Express SCI	Intel x86, x64	Linux, Windows, VxWorks

The SISCI software can also be licensed to Microchip, PLX/Broadcom/ Avago tech and IDT OEM customers building their own hardware products based on PCI Express Gen2 and Gen3 NTB chip sets. Please contact Dolphin for more information.

Support

SISCI is fully supported and maintained by Dolphin. Support for SISCI related issues can be requested by sending an e-mail to pci-support@dolphinics.com.

Dolphin or one of our recommended consulting companies can assist you to migrate to a SISCI based solutions. Please contact Dolphin for more information.