Systems Research and Architecture System und Rechnerarchitektur SRA
a SRA Theses Developing a low-level NVMe driver for microbenchmarks

Developing a low-level NVMe driver for microbenchmarks

image

[Generated with AI]

Context

In recent years SSDs have evolved into highly optimized hardware capable of delivering bandwidths up to 14 GB/s. Modern SSD controllers are based on full-fledged ARM cores that perform complex tasks such as advanced bookkeeping, caching, wear leveling and health monitoring. However, their inner workings and performance characteristics are largely hidden from application and driver developers. Aside from unspecific best-case performance metrics in the marketing materials, SSDs only offer scarce performance hints via their NVMe interface. This makes it increasingly harder to take full advantage of the growing bandwidth of modern SSDs. Even for a simple sequential read our initial measurements showed large differences (~1.5x) in throughput for varying command submission scenarios.

Problem

To precisely analyze the micro-architectural behavior of modern NVMe SSDs a specialized setup is required. The operating systems runtime overhead has to be reduced to minimum, eliminating syscall and scheduling overhead during latency benchmarks. Furthermore, fine-grained control of the device is required. Ideally, the benchmarks would directly interface with the SSD controller, which requires a suitable open source driver. While some open source NVMe drivers exist, they are either to complex (DPDK4), do not expose the SSD controller directly (io_uring5, libnvme6), or are incomplete and hard to extend (unvme7).

Goal

To enable precise measurements in the future, the goal of this thesis is to develop a low-level NVMe driver for the Linux operating system. By utilizing VFIO2 and iommufd3, the driver shall be implemented in userspace, bypassing the kernel completely. Besides exposing full control over the device to the benchmark application, the codebase should be kept small and easy to extend. Configuring the controller and implementing new features should be easy. Since the driver runs as a “normal” application, it does not need to be implemented in C. C++ or even Rust may also be possible.

The NVMe standard1 contains a large set of optional features and commands a controller may support. It is therefore not required to implement a complete driver in this thesis. However, the implementation must at least contain:

There are also plenty optional features for very motivated students:

To dampen the learning curve, students should first try to control a SSD using an existing driver implementation before diving into the actual implementation.

Single commands (such as identify) can also be sent using the nvme-cli8 utility without writing any code.

Topics: NVMe, VFIO, iommufd, Linux, Driver, C/C++/Rust

References

  1. https://nvmexpress.org/specifications/ ↩

  2. https://www.kernel.org/doc/html/latest/driver-api/vfio.html ↩

  3. https://docs.kernel.org/next/userspace-api/iommufd.html ↩

  4. https://www.dpdk.org/ ↩

  5. https://man.archlinux.org/man/io_uring.7 ↩

  6. https://github.com/linux-nvme/libnvme ↩

  7. https://github.com/zenglg/unvme ↩

  8. https://github.com/linux-nvme/nvme-cli ↩