From Global to Local Quiescence: Wait-Free Code Patching of Multi-Threaded Processes

OSDI Conference A*
From Global to Local Quiescence: Wait-Free Code Patching of Multi-Threaded Processes

Florian Rommel, Christian Dietrich, Birte Friesel, Marcel Köppen, Christoph Borchert, Michael Müller, Olaf Spinczyk, Daniel Lohmann14th Symposium on Operating System Design and Implementation (OSDI '20)2020.
PDF Details Video [BibTex]

×

BibTex Entry

@inproceedings{rommel:20:osdi,
author = {Florian Rommel and Christian Dietrich and Birte Friesel and Marcel K{ö}ppen and Christoph Borchert and Michael M{ü}ller and Olaf Spinczyk and Daniel Lohmann},
booktitle = {14th Symposium on Operating System Design and Implementation (OSDI '20)},
entrysubtype = {Conference},
isbn = {978-1-939133-19-9},
month = {November},
pages = {651-666},
title = {From Global to Local Quiescence: Wait-Free Code Patching of Multi-Threaded Processes},
year = {2020},
}

Close

Abstract

Live patching has become a common technique to keep long-running system services secure and up-to-date without causing downtimes during patch application. However, to safely apply a patch, existing live-update methods require the entire process to enter a state of quiescence, which can be highly disruptive for multi-threaded programs: Having to halt all threads (e.g., at a global barrier) for patching not only hampers quality of service, but can also be tremendously difficult to implement correctly without causing deadlocks or other synchronization issues.

In this paper, we present WfPatch, a wait-free approach to inject code changes into running multi-threaded programs. Instead of having to stop the world before applying a patch, WfPatch can gradually apply it to each thread individually at a local point of quiescence, while all other threads can make uninterrupted progress. We have implemented WfPatch as a kernel service and user-space library for Linux 5.1 and evaluated it with OpenLDAP, Apache, Memcached, Samba, Node.js, and MariaDB on Debian 10 (“buster”). In total, we successfully applied 33 different binary patches into running programs while they were actively servicing requests; 15 patches had a CVE number or were other critical updates. Applying a patch with WfPatch did not lead to any noticeable increase in request latencies – even under high load – while applying the same patch after reaching global quiescence increases tail latencies by a factor of up to 41× for MariaDB.

Source Code

The source code of the WfPatch kernel extension is available on Github (luhsra/linux-wfpatch).

Please note: This is the original source code used for the evaluation in the paper.

An improved version of the kernel extension intended for a broader range of applications is available at Github (luhsra/linux-mmview). Have a look at the ATLAS project for more information.

Artifacts

We provide a a virtual machine image with the evoultion artifacts. You will need QEMU to run the virtual machine.

Download (7.5 GB)