Soo broken

Kubernetes - Dynamic NFS provisioning

Published On: 2020-07-23, Reading Time: 5 minutes


Introduction

We previously configured NFS volumes in Kubernetes for reliable free fault-tolerant networked storage.

The downside of manually defining volumes is that only Kubernetes administrators can create volumes; this might be desired in some cases to implement strict control of NFS usage.

For organizations using agile and self-service where application owners do NFS provisioning in a decentralized way, Kubernetes allows dynamic volume provisioning. In this model, PersistanceVolumeClaims will trigger volume creation.

Before we start.

Make sure you have a working NFS server and connectivity is sorted out, Refer to Using NFS in Kubernetes for help. and the destination path in your NAS exists (in my case casa:/nfs-managed)

Install NFS Dynmaic provisioner

Once a resource provisioner for storageClass is defined, VolumeClaims (PVCs) with matching the class will dynamically create volumes the class provisioner. We will use NFS-Client Provisioner in our Kubernetes cluster. If you are interested here is the source code

  1. Create a Service Account NFS provisioner needs a service account to manage NFS volumes. Note: namespace: kube-custom is where we will be running nfs-provisioner resources. (Kubernetes RBAC is outside out of scope from this blog). This RBAC role gives serviceAccount nfs-client-provisioner ability to manage volumes.

File:rbac-nfs-client-provisioner.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: kube-custom
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: kube-custom
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: kube-custom
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: kube-custom
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: kube-custom
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

Apply it

$ kubectl apply -f rbac-nfs-client-provisioner.yaml
serviceaccount/nfs-client-provisioner created
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
  1. Create a namespace to hold custom global resources.
kubectl create namespace kube-custom
namespace/kube-custom created

File: nfs-provisioner.yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nfs-client-provisioner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: nfs-storage
            - name: NFS_SERVER
              value: casa
            - name: NFS_PATH
              value: "/nfs-managed"
      volumes:
        - name: nfs-client-root
          nfs:
            server: casa
            path: "/nfs-managed"
  1. Create a StorageClass
    StorageClasses are a way to map Claims with volumes or the way to provision them. In this case, we are using a provisioner named nfs-storage
    File: managed-nfs-storage.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: nfs-storage
parameters:
  archiveOnDelete: "false"

Tip setting archiveOnDelete: “true”. Causes provisioner to archive volumens upon deletetion of the Claim (PVC).

Apply it

$ kubectl -n kube-custom apply -f managed-nfs-storage.yaml
storageclass.storage.k8s.io/managed-nfs-storage created

$ kubectl get storageclass/managed-nfs-storage
NAME                  PROVISIONER   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage   nfs-storage   Delete          Immediate           false                  38s

Test dynamic volumes

  1. Create a VolumeClaim using managed-nfs-storage storageClass. It will auto-select the NFS provisioner and dynamically create the requested volume.

File: test-managed-claim.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-managed-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi
  storageClassName: managed-nfs-storage

Apply it

$ kubectl -n playground apply -f test-managed-claim.yaml
persistentvolumeclaim/test-managed-claim created

$ kubectl -n playground get pvc/test-managed-claim
NAME                 STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS          AGE
test-managed-claim   Pending                                      managed-nfs-storage   91s
  1. Run pod to use your new NFS volume claim
    File: test-pod-nfs-managed-pvc.yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-managed-nfs-pvc
  namespace: playground
spec:
  containers:
  - name: busybox
    image: busybox:latest
    command: ["sh"]
    args: ["-c", "while [ 1 ]; do echo 'Hello World'>/data/nfs-managed-data.dat && sleep 10; done"]
    resources:
      limits:
        cpu: 100m
        memory: 50Mi
    volumeMounts:
    - name: test-store
      mountPath: /data
  volumes:
  - name: test-store
    persistentVolumeClaim:
      claimName: test-managed-claim

Apply it

$ kubectl -n playground apply -f test-pod-nfs-managed-pvc.yaml
pod/test-managed-nfs-pvc created

$ kubectl -n playground get pods/test-managed-nfs-pvc
NAME                   READY   STATUS    RESTARTS   AGE
test-managed-nfs-pvc   1/1     Running   0          16s

$ kubectl -n playground exec test-managed-nfs-pvc -- cat /data/nfs-managed-data.dat
Hello World

$ kubectl -n playground get pv | { head -1 ; grep test-managed-claim; }
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS          REASON   AGE
pvc-1b6ac5c6-dbcf-477a-a17e-1ec0c48e032d   1Mi        RWX            Delete           Bound    playground/test-managed-claim   managed-nfs-storage            31s

Verify on your NAS

You can now verify on your NAS (Network Attached Storage) dynamic NFS provisioning is in effect. If you list the contents of /nfs-managed, you will see a folder for each PersistentVolumeClaim

$ ls -al nfs-managed/
total 70
drwxr-xr-x  3 luis  wheel  3 Jul 22 17:38 .
drwxr-xr-x  5 luis  wheel  6 Jul 22 16:23 ..
drwxrwxrwx  2 root  wheel  3 Jul 22 17:38 playground-test-managed-claim-pvc-1b6ac5c6-dbcf-477a-a17e-1ec0c48e032d

Note of caution with EFS

If you are using dynamic provisioning with AWS EFS or other NFS providers with IO quotas. The dynamic provisioner keeps all claims as directories under the same share; This results in NFS volumes sharing the same IO and disk quotas and limits. I have seen this create cascading outages in all applications using EFS when the io quota is exhausted. Use one or more of the following workarounds when dealing with IO quotas:

  • Store a large image file in the root EFS volume to warranty a minimum throughput. (EFS gives more IO as you add more storage).
fallocate -l 500G do-not-delete-io-limiter.img
  • Use provisioned EFS storage. (Can be expensive).
  • Create multiple NFS provisioners and storage classes; each one using a different EFS resource:

managed-nfs-provisioned-io
managed-nfs-bust-io
managed-nfs-provisioned-prod-db.

Final thoughts

Empowering applications to dynamically create NFS volumes is what Kubernetes is all about! Removes complexity while at the same time allowing you to have centralized resource management, developers, and admins love it. It frees your applications to run across nodes and have shorter TTR (Time to recover) in a failure scenario. Finally, we have all the groundwork for launching a real-life persistent service. Next, we will implement MySQL using our newly configured NFS resources.