Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization

PCI Express (PCIe) is today widely used for static local input/output (I/O) expansion and is gaining momentum as a host-to-host high speed interconnect.  However these services are currently not able to integrate onto a single hardware infrastructure, two separate PCIe networks are needed.

Dolphin has developed interconnect solutions since the early 1990s and its current product line relies exclusively on PCIe-based interface cards and switches and competes with alternative high-speed communication technologies. A great advantage of Dolphin’s PCIe products is the reduced protocol overhead compared to technologies like 10Gb Ethernet, InfiniBand and other proprietary interconnect technologies. Dolphin is the market leader in providing fast, optimized and easy to use software and hardware products enabling PCIe to be used as a high speed interconnect.

I/O devices are typically statically assigned to a single root complex (host), hot-add, device migration, device sharing and remote access are not supported in flexible way.

In principle, all I/O devices on a shared PCIe fabric can be accessed directly using the PCIe Non Transparent Bridging (NTB) addressing techniques by any connected remote computer, but the required software structure, OS interfaces and implementations do not exist.

Device sharing functionality is addressed in the PCIe Single Root I/O Virtualization (SR-IOV) and Multi Root I/O Virtualization (MR-IOV) specifications but this is still a static approach and requires PCIe devices to be developed according to these specifications.

As I/O devices are normally owned by one compute node, I/O data are normally relayed to remote servers though the host using traditional networking services. Using PCIe NTB techniques, it is possible to enable I/O devices to directly transfer data to a remote node. This will significantly reduce latency and overhead during data transfers.

Final goal:

The goal with this project is to develop a new framework for the operating system and virtual machines that will enable remote discover, addressing, access and use of standard PCIe devices. The framework will enable standard PCIe devices to be re-allocated and shared by computer nodes in the PCIe network with no or minimum changes to the applications and device drivers. On top of this framework we will develop services to validate and demonstrate the framework such as a fast cluster file system, legacy device driver access to I/O devices on a remote node, and clustering of accelerator cards such as the Intel Xeon Phi co-processor and Nvidia Tesla graphics processing units.\

Funding source:

The Research Council of Norway

All partners:

Dolphin Interconnect Solutions