Building Blocks for Containerized Ceph

Originally, Kubernetes PersistentVolumes (PVs) could only present storage to containers as filesystems. However, some applications prefer to use block storage, usually for reasons of performance, and have no need for a full filesystem. Several such applications have had to go as far as directly accessing local system directories to get the functionality they need in Kubernetes. Raw block PVs are a relatively new feature that went beta in Kubernetes 1.13. They allow Kubernetes to present storage to containers as block devices, removing the need for provisioners to format filesystems on top of them. This not only allows for greater performance to the applications that expect it, it also helps improve security by reducing the level of permissions such an application's containers require to run. Rook-Ceph is the Ceph operator for the Rook project. It provides resilient storage by running the various Ceph components as containers and managing them via Kubernetes. Originally, it would bind-mount system directories to manipulate the storage devices it consumed. It now leverages raw block PVs to store its data, expanding the types of storage it can consume. In particular, cloud environments are now a space where its storage Pods can migrate in response to node failures and have the storage devices move with their Pods. For the uninitiated, this presentation will start with an overview of how storage is modeled and presented in Kubernetes. It will then describe how that storage was originally consumed by Rook-Ceph, what we changed about it, and the consequences (both good and bad) of those changes.