Masters-projects
Non-cache-coherent Memory Tiering in PCI Express using dSwap

Non-cache-coherent Memory Tiering in PCI Express using dSwap

dSwap is an implementation of swap memory that utilizes the memory on other nodes as a lower-latency layer on top of storage. It builds on the PCIe protocol for communication among nodes, known as routable PCIe. The goal of this project is to design, implement, and test dSwap. This project is a collaboration among UiO, Simula Research Laboratory, and Dolphin ICS, and the implementation of dSwap will run in conjunction with Dolphin ICS's hardware and software. Before scheduling a meeting about this project, please review the project description and the Infiniswap paper found at the bottom of the project description.

Being a master’s student
List of projects
Register as a master’s student

Asymmetry in technological advancements in processor and memory architectures has led processors to generate traffic 2-10x higher than the memory interconnect bandwidth. This is known as the memory wall. To overcome the memory wall, memory-intensive applications, such as in-memory databases, graph processing, and artificial intelligence, split their working set across memory tiers because they cannot fully fit in local memory. A memory tiering system is a hierarchy ordered by the access latency of each medium. The hierarchy has local memory as the highest tier, networked memory as the middle tiers, and storage as the lowest tier. Due to the wide performance-characterization differences in networked memory, it's the area with the most active research.

While most research focuses on RDMA and CXL, there is an unexplored area in memory tiering over routable PCIe. Routable PCIe is a cluster interconnect used to build composable infrastructure, in which computer hardware resources (memory and I/O peripherals) are no longer tied to a single node but are available to all nodes in the pod. A pod is a small cluster, typically consisting of 2 to 16 nodes. In terms of performance, routable PCIe access latency is on the order of hundreds of nanoseconds - far below RDMA but higher than CXL.

Additionally, routable PCIe lacks cache coherency, making reads significantly slower than on cache-coherent interconnects such as CXL. However, CXL memory devices are still in a research stage and not widely deployed. Thus, there is a place for memory tiering over routable PCIe.

We present dSwap, the first memory tiering system backed by routable PCIe interconnects. dSwap leverages the low latency of routable PCIe for synchronous accesses on the critical path, and migrates remote memory with little CPU involvement on the off-path. It harvests free memory in remote hosts without disturbing remote CPUs. Lastly, dSwap requires no application or operating system modifications.

Goals

The overarching goal of this thesis is to explore and gain insight into memory tiering over routable PCIe. In particular, we aim to answer the following research questions (RQ):

RQ1: How can we implement a page swapping system using routable PCIe and memory of remote hosts?

RQ2: How can dSwap mitigate the non-cache-coherency of routable PCIe?

RQ3: What is the performance of end-to-end applications on systems using dSwap, compared to the state of the art?

Successful project completion may result in the publication of the thesis work at a research conference or in a journal.

Learning outcomes

Upon successful completion of this thesis, the student will have gained:

  • Advanced knowledge of memory tiering systems, and basic knowledge about modern interconnect technologies, including CXL, PCIe, and RDMA.
  • Expertise in operating systems and hardware development: Hands-on experience in designing, implementing, and optimizing device drivers. Practical knowledge of low-level interactions between hardware and software.
  • Problem-solving in cutting-edge technologies: Experience in tackling open research problems that span applications and hardware. The ability to analyze system behavior from device-level to overall system performance.
  • Strong research and analytical skills. The ability to conduct independent research, critically evaluate scientific literature, design and execute complex experiments, analyze data, and present findings clearly and concisely.
  • Software development: Practical experience with Dolphin ICS’s hardware and driver stack, as well as other open- and closed-source hardware and software platforms.

Qualifications

Required:

  • BSc or equivalent in Computer Science, Electrical Engineering, Applied Physics
  • Good results in IN3000/IN4000 - Operating Systems
  • Proficiency in C programming, experience working with computer architecture, and a general understanding of computer system principles.

Desired:

  • IN5060 - Quantitative Performance Analysis.
  • IN5050 - Programming heterogeneous multi-core architectures.
  • Prior exposure to device drivers and Linux kernel development.

Supervisor

  • Håkon Kvale Stensland

Collaboration partner

  • Dolphin Interconnect Solutions

References

Associated contact