Storage Solutions

Container local storage (no persistent volume):

Pros: Simple
Cons: No fault tolerance

Network attached storage (e.g. EBS/EFS on AWS or PD on GCP)

Pros: Cheap; Support for Snapshots; Simple
Cons: Slower I/O
How: Neither of EBS or PD support ReadWriteMany. In order to leverage them, we can either (1) manually attach an EBS/PD to the nodes and use HostPath with ReadWriteMany, or (2) Setup an NFS server (with Rook) backed by the EBS/PD and use this in ReadWriteMany mode. The example below demonstrates the latter.

Local SSD storage:

Pros: Fast
Cons: Expensive

Notes:

If every node has an additional SSD attached as a block device, then we can setup a distributed filesystem using them using Ceph (with Rook). This is a very performant solution. However, production maintenance would require devops engineers to learn some ceph tools. We’ll add a guide for this solution shortly.