Benefits of using PCI Express with VxWorks

Reduced communication jitter

The SISCI API provides true kernel bypass communication, enabling user space applications to safely access remote memory or PCIe devices. The Dolphin Express products are implemented using standard PCI Express bridges and switches; the data flow is managed by PCI Express ASICs. The hardware will instantly and automatically forward data to the addressed remote memory. The transfer does not depend on extensive software communication protocols, regular DMA operations or system interrupts that will add an undeterministic communication delay and data delivery jitter.

The average communication latency between two systems connected using e.g. Dolphins Gen3 x8 PXH810 cards is typically around 540 nanoseconds on all supported operating systems. As long as there is enough free CPU resources in a system, the applications will be allocated to a free CPU and enabled to run virtually uninterrupted until it completes its tasks.

In practice when using a regular operating system like Windows or Linux, the application will be re-scheduled by the system scheduler (another application is given the ability to run for some time) or interrupted by the the operating system when it issues an internal inter-CPU interrupt etc. These execution delays are hardly noticeable for regular applications, but will cause hard real-time applications to fail to meet their timing requirements.

CPU shielding

VxWorks from Wind River is a real-time operating system that enables the user to run an application without being interrupted by the operating system or other applications. The benefits are obvious for real-time processes - the application will be allowed to always use 100% of the allocated CPU(s) to execute its real time process.

Combined with a PCI Express network, the software and hardware combination will ensure a very low and deterministic max communication latency. This again will enable customers to spend less time on the communication protocol and more time on the actual real-time process.

Performance

The SISCI ping pong benchmark scipp is included in the Dolphin software distribution and can be used to measure best, average and worst case latency for a sequence of ping pong data transfer operations.

The following results was found when the benchmark utility was executed with VxWorks 7.0 and with standard Linux (Fedora 26). The latency is for a half-way full round-trip between two systems (the time it takes to send a message of "size" bytes to the remote system, receive it and send back a similar size message divided by two). A high precision timer is used to measure the round-trip latency for each transfer:

The minimum half way round-trip latency is 510 nanoseconds. The average latency is slightly higher 540 nanoseconds for both VxWorks and Linux - this means that the application often is able to run without being interrupted. The worst case latency for VxWorks is 1.24 microseconds, but the measured worst case latency using Fedora 26 is significantly higher - up to 35 microseconds in these measurements. Significant higher Linux worst case latency may occur. The VxWorks numbers above was measured over a 24 hour continuous run.

By using VxWorks - the worst case latency is reduced down to slightly above the average latency.

The graph below shows the same performance results as above, but without the Linux worst case.

By using the combination of Dolphin PCIe products and VxWorks, the worst case communication latency and jitter can be reduced down to an absolute minimum. This enables hard real-time applications to meet their real-time constraint.

You can easily verify these real-time numbers on your own system. After installing and configuring the SISCI software, just run the included 'scipp' benchmark using the -hist option.