johanneskueber.com

Per-PVC Encryption with Longhorn and CSI Secret Templates

#longhorn #csi #kubernetes #encryption #luks #storage #talos

This article documents how to configure a Longhorn StorageClass that encrypts every PVC with its own per-volume key, derived through CSI’s secret-template parameters, and how to provision the matching secrets so the keys are scoped to the application namespace.

1. What encryption Longhorn actually does

Longhorn 1.4+ supports LUKS encryption at the block device layer. When a PVC’s StorageClass declares encrypted: "true", Longhorn calls cryptsetup luksFormat on the underlying replica devices using a passphrase pulled from a Kubernetes Secret. The PVC is then exposed to the consuming Pod as an unencrypted filesystem — the kernel handles the encryption transparently through the device-mapper layer.

The data on disk, in Longhorn replica state, and in any Longhorn-driven backup is ciphertext. The kernel’s dm-crypt is the only piece that ever holds the plaintext-deriving passphrase.

The natural shape is one passphrase per cluster — simple, but a key compromise leaks every PVC. A more interesting shape is one passphrase per PVC. The CSI standard supports it through template parameters in the StorageClass, and Longhorn honours them.

2. The encrypted StorageClass

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: longhorn-encrypted
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880"
  fromBackup: ""
  encrypted: "true"
  fsType: "xfs"

  csi.storage.k8s.io/provisioner-secret-name:       ${pvc.name}-longhorn
  csi.storage.k8s.io/provisioner-secret-namespace:  ${pvc.namespace}
  csi.storage.k8s.io/node-publish-secret-name:      ${pvc.name}-longhorn
  csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
  csi.storage.k8s.io/node-stage-secret-name:        ${pvc.name}-longhorn
  csi.storage.k8s.io/node-stage-secret-namespace:   ${pvc.namespace}

mountOptions:
  - discard

Field reference:

  • provisioner: driver.longhorn.io — the CSI driver name registered by Longhorn.
  • numberOfReplicas: "3" — three replicas across nodes. Encrypted volumes carry the same replication semantics as plain ones; each replica is encrypted independently with the same passphrase.
  • staleReplicaTimeout: "2880" — minutes (48 hours) before a disconnected replica is considered stale and garbage-collected. Larger values protect against transient node outages at the cost of slower rebuild on permanent loss.
  • encrypted: "true" — the toggle. Without this, the secret references are ignored.
  • fsType: "xfs" — XFS performs better than ext4 on the trim-aware dm-crypt path with mountOptions: discard. The combination causes the kernel to TRIM the encrypted layer when the application unlinks files, returning space to the underlying Longhorn replica.
  • csi.storage.k8s.io/*-secret-name: ${pvc.name}-longhorn — the CSI template. ${pvc.name} and ${pvc.namespace} are expanded at PVC provisioning time, so each PVC reads a uniquely-named Secret in its own namespace. The three role-specific keys (provisioner, node-publish, node-stage) all point to the same Secret here; CSI permits independent secrets per phase, but for LUKS there is one passphrase shared across them.
  • mountOptions: - discard — propagates TRIM through dm-crypt to the underlying Longhorn replica. Critical for thin-provisioned storage backing the replicas.

3. The per-PVC Secret

For a PVC named gitea-data in namespace gitea, the Secret is gitea-data-longhorn in gitea:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

apiVersion: v1
kind: Secret
metadata:
  name: gitea-data-longhorn
  namespace: gitea
type: Opaque
stringData:
  CRYPTO_KEY_VALUE: "<random-32-byte-base64>"
  CRYPTO_KEY_PROVIDER: "secret"
  CRYPTO_KEY_CIPHER: "aes-xts-plain64"
  CRYPTO_KEY_HASH: "sha256"
  CRYPTO_KEY_SIZE: "256"
  CRYPTO_PBKDF: "argon2id"

Field reference:

  • CRYPTO_KEY_VALUE — the passphrase. Longhorn passes this to cryptsetup luksFormat --key-file -. Generate with openssl rand -base64 32 or a Vault-issued one-time value.
  • CRYPTO_KEY_PROVIDER: "secret" — Longhorn reads the value from this Secret directly. The alternative kms provider is reserved for future KMS integrations and not implemented.
  • CRYPTO_KEY_CIPHER: "aes-xts-plain64" — the LUKS cipher mode. XTS is the modern default for full-disk encryption; CBC is legacy and worse on every axis.
  • CRYPTO_KEY_HASH: "sha256" — passphrase hashing algorithm. Used during PBKDF, not for the data itself.
  • CRYPTO_KEY_SIZE: "256" — key size in bits. XTS uses two keys, so 256 means a 512-bit XTS keyset under the hood.
  • CRYPTO_PBKDF: "argon2id" — the password-based key derivation function. argon2id is the modern default; pbkdf2 is also accepted but slower to brute-force evaluation per unit of legitimate work, hence weaker.

The Secret must exist before the PVC is created. The provisioner will not create one; it will fail with failed to get secret gitea-data-longhorn in the Longhorn manager logs.

4. Bootstrapping the Secret

A plaintext passphrase in Git is obviously not acceptable. The shape of the answer is to materialise the Secret at apply-time rather than commit-time — SOPS, External Secrets, or a Composition step that emits the Secret alongside the PVC all work. Which one fits depends on the rest of the stack; pick whichever already handles the other namespace-scoped Secrets in the cluster.

5. Provisioning a PVC against the StorageClass

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gitea-data
  namespace: gitea
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-encrypted
  resources:
    requests:
      storage: 10Gi

Longhorn provisions three replicas, runs luksFormat on each using the passphrase from gitea-data-longhorn, and mounts the resulting dm-crypt device into the consuming Pod. The Pod sees an XFS filesystem on a 10 GiB volume — the encryption is invisible above the kernel block layer.

Volume expansion works through the encryption: allowVolumeExpansion: true on the StorageClass plus kubectl patch pvc to a larger size resizes the underlying Longhorn replicas, then cryptsetup resize on the dm-crypt mapping, then XFS online grow. Longhorn orchestrates all three transparently.

6. Verifying the result

Volume is encrypted:

1
2
3
4
5
6

kubectl get pvc -n gitea gitea-data -o yaml | yq .spec.volumeName
# pvc-3e72b1...

kubectl exec -n longhorn-system -it $(
  kubectl get pod -n longhorn-system -l app=longhorn-manager -o name | head -1
) -- bash -c "ls -la /var/lib/longhorn/replicas/pvc-3e72b1*"

The presence of a replica directory whose volume.meta records EncryptionEnabled: true confirms encryption is on at the Longhorn layer.

dm-crypt device is in place on the consuming node:

1
2
3
4

# from a debug shell on the Longhorn-attaching node
ls /dev/mapper/ | grep crypt
# pvc-3e72b1...-crypt
cryptsetup status /dev/mapper/pvc-3e72b1...-crypt

type: LUKS2, cipher: aes-xts-plain64, and a device pointing at a Longhorn /dev/longhorn/... path confirm the encryption layer is active and on top of the replica.

7. Practical notes

  • Secret rotation does not re-encrypt the volume. LUKS binds the data-encryption key on volume create; rotating the passphrase post-hoc requires cryptsetup luksChangeKey against every replica, which Longhorn does not automate. Treat the passphrase as immutable per-volume.
  • mountOptions: discard is not the default. Without it, deleted file space remains allocated at the dm-crypt and Longhorn layers. Storage utilisation balloons until the PVC is detached and reattached.
  • Per-PVC keys vs. cluster key. Per-PVC isolates blast radius but multiplies the number of secrets to track. For homelabs or air-gapped clusters with a strong workstation-key story, a cluster-wide passphrase is acceptable. For multi-tenant production, per-PVC is the right default.