feat(e2e-test): Add e2e-tests for zfs-localpv (#298)

Signed-off-by: w3aman <aman.gupta@mayadata.io>

e2e-tests/experiments/functional/csi-volume-resize/README.md

@@ -0,0 +1,72 @@
+## About this experiment
+
+This experiment verifies the csi volume resize feature of zfs-localpv. For resizing the volume we just need to update the pvc yaml with desired size and apply it. We can directly edit the pvc by ```kubectl edit pvc <pvc_name> -n <namespace>``` command and update the spec.resources.requests.storage field with desired volume size. One thing need to be noted that volume resize can only be done from lower pvc size to higher pvc size. We can not resize the volume from higher pvc size to lower one, in-short volume shrink is not possible. zfs driver supports online volume expansion, so that for using the resized volume, application pod restart is not required. For resize, storage-class which will provision the pvc should have `allowVolumeExpansion: true` field.
+
+for e.g.
+```
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: zfspv-sc
+allowVolumeExpansion: true
+parameters:
+  poolname: "zfs-test-pool"
+provisioner: zfs.csi.openebs.io
+```
+
+## Supported platforms:
+
+K8s : 1.18+
+
+OS : Ubuntu, CentOS
+
+ZFS : 0.7, 0.8
+
+## Entry-criteria
+
+- K8s cluster should be in healthy state including desired worker nodes in ready state.
+- zfs-controller and csi node-agent daemonset pods should be in running state.
+- storage class with `allowVolumeExpansion: true` enable should be present.
+- Application should be deployed succesfully consuming the zfs-localpv storage.
+
+## Exit-criteria
+
+- Volume should be resized successfully and application should be accessible seamlessly.
+- Application should be able to use the new resize volume space.
+
+## Steps performed
+
+- Check pvc status, it should be Bound. Get storage class name and capacity size of this pvc.
+- Update the pvc size with desired volume size, which should not be lesser than previous volume size because volume shrink is not supported.
+- Check the updated size in pvc spec.
+- Since it is online volume expansion, we don't need to restart the application pod but here we restart intentionally to validate that resized space is available after restart of application pod.
+- To use the resized space this test will dump some dummy data at application mount point. This dummy data size will be previous volume size + 1 Gi. So make sure we have this much enough space.
+- At last this test will delete the dummy data files to free the space.
+
+## How to run
+
+- This experiment accepts the parameters in form of kubernetes job environmental variables.
+- For running this experiment of csi volume resize, clone openens/zfs-localpv[https://github.com/openebs/zfs-localpv] repo and then first apply rbac and crds for e2e-framework.
+```
+kubectl apply -f zfs-localpv/e2e-tests/hack/rbac.yaml
+kubectl apply -f zfs-localpv/e2e-tests/hack/crds.yaml
+```
+then update the needed test specific values in run_e2e_test.yml file and create the kubernetes job.
+```
+kubectl create -f run_e2e_test.yml
+```
+All the env variables description is provided with the comments in the same file.
+
+After creating kubernetes job, when the job’s pod is instantiated, we can see the logs of that pod which is executing the test-case.
+
+```
+kubectl get pods -n e2e
+kubectl logs -f <csi-volume-resize-xxxxx-xxxxx> -n e2e
+```
+To get the test-case result, get the corresponding e2e custom-resource `e2eresult` (short name: e2er ) and check its phase (Running or Completed) and result (Pass or Fail).
+
+```
+kubectl get e2er
+kubectl get e2er csi-volume-resize -n e2e --no-headers -o custom-columns=:.spec.testStatus.phase
+kubectl get e2er csi-volume-resize -n e2e --no-headers -o custom-columns=:.spec.testStatus.result
+```

e2e-tests/experiments/functional/csi-volume-resize/run_e2e_test.yml

@@ -0,0 +1,43 @@
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  generateName: csi-volume-resize-
+  namespace: e2e
+spec:
+  template:
+    metadata:
+      name: csi-volume-resize
+      labels:
+        test: csi-volume-resize
+
+    spec:
+      serviceAccountName: e2e
+      restartPolicy: Never
+      containers:
+      - name: ansibletest
+        image: openebs/zfs-localpv-e2e:ci
+        imagePullPolicy: IfNotPresent
+
+        env:
+          - name: ANSIBLE_STDOUT_CALLBACK
+            value: default
+
+            # This is the namespace where application pod is running
+          - name: APP_NAMESPACE
+            value: ''
+
+            # Name of the application pvc
+          - name: APP_PVC
+            value: '' 
+
+            # Application pod label in `key=value` format
+          - name: APP_LABEL
+            value: ''
+
+            # Resized PVC size (for eg. 10Gi)
+          - name: NEW_PV_CAPACITY
+            value: ''
+
+        command: ["/bin/bash"]
+        args: ["-c", "ansible-playbook ./e2e-tests/experiments/functional/csi-volume-resize/test.yml -i /etc/ansible/hosts -vv; exit 0"]

e2e-tests/experiments/functional/csi-volume-resize/test.yml

@@ -0,0 +1,172 @@
+---
+- hosts: localhost
+  connection: local
+  gather_facts: False
+
+  vars_files:
+    - test_vars.yml
+
+  tasks:
+    - block:
+
+          ## Generating the testname for csi volume resize test
+        - include_tasks: /e2e-tests/hack/create_testname.yml
+    
+          ## Record SOT (start of test) in e2e result e2e-cr (e2e-custom-resource)
+        - include_tasks: /e2e-tests/hack/update_e2e_result_resource.yml
+          vars:
+            status: 'SOT'
+
+        - block:
+
+            - name: Check if the pvc {{ app_pvc }} is bound
+              shell: >
+                kubectl get pvc {{ app_pvc }} -n {{ app_ns }} --no-headers
+                -o custom-columns=:.status.phase
+              args:
+                executable: /bin/bash
+              register: pvc_status
+              failed_when: "'Bound' not in pvc_status.stdout"
+
+            - name: Get the storage class name used for provisioning {{ app_pvc }} pvc
+              shell: >
+                kubectl get pvc {{ app_pvc }} -n {{ app_ns }} --no-headers
+                -o custom-columns=:.spec.storageClassName
+              args:
+                executable: /bin/bash
+              register: storage_class
+
+            - name: Get the present capacity size of pvc {{ app_pvc }}
+              shell: >
+                kubectl get pvc {{ app_pvc }} -n {{ app_ns }} --no-headers
+                -o custom-columns=:.status.capacity.storage
+              args:
+                executable: /bin/bash
+              register: vol_size
+
+              # This test will work with one pod at a time retrieved by app_label in app_namespace
+              # If there are multiple pods with same label within one namespace, it takes one
+              # pod randomly. (for e.g. shared mount pods)
+            - name: Get the application pod name which is consuming {{ app_pvc }} pvc
+              shell: >
+                kubectl get pod -n {{ app_ns }} -l {{ app_label }} --no-headers
+                -o custom-columns=:.metadata.name | shuf -n1
+              args:
+                executable: /bin/bash
+              register: app_pod
+
+            - name: Obtain the mount path for the application
+              shell: > 
+                  kubectl get pod {{ app_pod.stdout }} -n {{ app_ns }}
+                  -o custom-columns=:.spec.containers[].volumeMounts[].mountPath --no-headers
+              args: 
+                  executable: /bin/bash
+              register: mount
+
+            - name: Fetch the value part from storage capacity
+              shell: echo "{{ vol_size.stdout }}" | grep -o -E '[0-9]+'
+              args:
+                executable: /bin/bash
+              register: value_str
+
+            - name: Obtain the PVC spec
+              shell: >
+                kubectl get pvc {{ app_pvc }} -n {{ app_ns }}
+                --no-headers -o yaml > pvc.yml
+              args:
+                executable: /bin/bash
+
+            - name: Update the desired capacity in PVC spec
+              replace:
+                path: pvc.yml
+                before: 'storageClassName: {{ storage_class.stdout }}'
+                regexp: "storage: {{ vol_size.stdout }}"
+                replace: "storage: {{ desired_vol_size }}"
+
+            - name: Configure PVC with the new capacity
+              shell: kubectl apply -f pvc.yml
+              args:
+                executable: /bin/bash
+              register: result
+              failed_when: "result.rc != 0"
+
+            - name: Check if the update PVC is bound
+              shell: >
+                kubectl get pvc {{ app_pvc }} -n {{ app_ns }} --no-headers
+                -o custom-columns=:.status.phase
+              args:
+                executable: /bin/bash
+              register: pvc_status
+              failed_when: "'Bound' not in pvc_status.stdout"
+
+            - name: Check if the storage capacity is updated in PVC
+              shell: >
+                 kubectl get pvc {{ app_pvc }} -n {{ app_ns }} --no-headers 
+                 -o custom-columns=:status.capacity.storage
+              args:
+                executable: /bin/bash
+              register: capacity
+              until: "desired_vol_size in capacity.stdout"
+              delay: 3
+              retries: 60
+
+            - name: Restart the application pod after resizing the volume
+              shell: kubectl delete pod {{ app_pod.stdout }} -n {{ app_ns }}
+              args:
+                executable: /bin/bash
+              register: app_pod_status
+              failed_when: app_pod_status.rc != 0
+
+            - name: Verify that application pod is deleted successfully.
+              shell: >
+                kubectl get pods -n {{ app_ns }}
+              args:
+                executable: /bin/bash
+              register: app_pod_list
+              until: '"{{ app_pod.stdout }}" not in app_pod_list.stdout'
+              delay: 3
+              retries: 30
+
+            - name: Get the name of application pod after Restart
+              shell: >
+                kubectl get pod -n {{ app_ns }} -l {{ app_label }} --no-headers
+                -o custom-columns=:.metadata.name | shuf -n1
+              args:
+                executable: /bin/bash
+              register: app_pod_name
+
+             ## Here we will dump +1Gi data than to previous pvc size
+            - set_fact:
+                value_num: '{{ ( (value_str.stdout | int + 1 | int) * 262144) |  int }}'
+
+            - name: Dump some more dummy data in the application mount point for using resized volume
+              shell: >
+                  kubectl exec -it "{{ app_pod_name.stdout }}" -n "{{ app_ns }}" 
+                  -- sh -c "cd {{ mount.stdout }} && dd if=/dev/urandom of=volume.txt bs=4k count={{ value_num }}"
+              args:
+                  executable: /bin/bash
+              register: load
+              failed_when: "load.rc != 0"
+
+            - name: Delete the test file from application mount point
+              shell: >
+                  kubectl exec -it "{{ app_pod_name.stdout }}" -n "{{ app_ns }}" 
+                  -- sh -c "cd {{ mount.stdout }} && rm -f volume.txt"
+              args:
+                  executable: /bin/bash
+              register: testfile
+              failed_when: "testfile.rc != 0"
+
+        - set_fact:
+            flag: "Pass"
+
+      rescue:
+        - set_fact:
+            flag: "Fail"
+
+      always:
+
+          # RECORD END-OF-TEST IN e2e RESULT CR
+        - include_tasks: /e2e-tests/hack/update_e2e_result_resource.yml
+          vars:
+            status: 'EOT'

e2e-tests/experiments/functional/csi-volume-resize/test_vars.yml

@@ -0,0 +1,14 @@
+---
+test_name: csi-volume-resize
+
+app_ns: "{{ lookup('env','APP_NAMESPACE') }}"
+
+app_label: "{{ lookup('env','APP_LABEL') }}"
+
+app_pvc: "{{ lookup('env','APP_PVC') }}"
+
+vol_size: "{{ lookup('env','OLD_PV_CAPACITY') }}"
+
+desired_vol_size: "{{ lookup('env','NEW_PV_CAPACITY') }}"
+
+storage_class: "{{ lookup('env','STORAGE_CLASS') }}"