DopplerLog: A Performance Measurement Tool for Composable PCIe Infrastructures

Advancements in PCIe hardware have enabled the development of low-latency, high-bandwidth commodity interconnect technologies such as PCIe 6.0 and CXL 2.0. These advances have opened up new research opportunities, such as composable disaggregated infrastructure (CDI), which utilizes shared memory for memory-mapped I/O devices, allowing them to appear in the host’s local device tree even though they are physically remote. In the deployment of I/O-bound services, this capability provides system engineers with increased flexibility, enabling higher I/O utilization, reduced power consumption, lower operating costs, and improved failover mechanisms. As a result, CDIs present a promising alternative to traditional, direct-attached PCIe server solutions.

While PCIe is both extensively deployed and well-studied in intra-host scenarios, less is known about its behavior as an interconnect. In particular, the traffic generated by different classes of applications and device drivers when crossing PCIe switches, or when devices communicate without host intervention, is not well understood. Correspondingly, few mature tools exist to assess these traffic patterns.

To overcome this problem, we design a proof-of-concept benchmarking tool, named after the Doppler log, an instrument used for measuring an object’s relative speed to water, drawing inspiration from the aquatic brand theme of Dolphin Interconnect Solutions AS (Dolphin ICS).

Goal

The primary goal of this thesis is to explore and gain insight into the performance benchmarking of composable infrastructures and the traffic profile of PCIe interconnects. This can be achieved through a combination of the following objectives:

Literature review. Conduct a comprehensive survey of existing approaches to benchmarking intra-host PCIe, CXL, and RDMA networks, as well as prior attempts to profile PCIe as an interconnect.
Tool development. Adapt an existing approach or implement a new benchmarking tool and deploy it as a proof-of-concept on Dolphin ICS software and hardware.
Benchmark design. Design a range of application-independent microbenchmarks and/or find macrobenchmarks relevant to a specific domain.
Benchmark execution. Successfully execute benchmarking runs, collect data, and document their results.
Analysis and recommendations. Discuss the findings, analyze, draw conclusions, and derive specific system design tips (such as topology setups, device and workload allocation, etc.) based on the observations.

Learning outcome

Upon successful completion of this thesis, the student will have gained:

Advanced knowledge of modern interconnect technologies, including CXL, PCIe, and RDMA, and familiarity with various I/O device classes such as NVMe storage, NICs, GPUs, and FPGAs.
Expertise in operating systems and hardware development: Hands-on experience in designing, implementing, and optimizing device drivers. Practical knowledge of low-level interactions between hardware and software.
Problem-solving in cutting-edge technologies: Experience in tackling open research problems that span applications and hardware. The ability to analyze system behavior from device-level to overall system performance.
Strong research and analytical skills: The ability to conduct independent research, critically evaluate scientific literature, design and execute complex experiments, analyze data, and present findings clearly and concisely.
Software development: Practical experience with Dolphin ICS’s hardware and driver stack, as well as other open- and closed-source hardware and software platforms.

Qualifications

The scope of this topic is both broad and deep, and a thesis is not expected to cover all the questions above. Therefore, the scope of the project can and should be narrowed down. However, candidates should possess:

Required:

BSc or equivalent in Computer Science, Electrical Engineering, Applied Physics, or a related field.
Strong motivation for hands-on experimental work with hardware.
Proficiency in C programming and experience with developer tools such as Git.
Experience in working with computer architecture and operating systems.
Basic understanding of networking concepts such as routing and flow control.
General understanding of computer system principles.

Highly desired (but can be learned during the thesis or scoped out):

Prior exposure to device drivers and Linux kernel development.
Experience with FPGA development or CUDA development.
The ability to consume academic literature and understand advanced topics in computer systems rapidly.

Collaboration partners

Dolphin Interconnect Solutions

References