MicroK8S 1.27.2 breaks OpenEBS

[sjerman]

Summary

When you install the openebs local-lvm helm chart on MicrosK8S 1.27.2 revision 5372. PVC provisioned using a local-lvm storage class get stuck in 'pending'.

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  102s (x2 over 7m5s)  default-scheduler  0/1 nodes are available: 1 node(s) did not have enough free storage. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..

if you change the version of MicroK8S to 1.26.2 everything starts working again..

snap install microk8s --classic  --channel=1.26/stable

What Should Happen Instead?

PVC should be provisioned.

Reproduction Steps

We have duplicated the issue on multiple systems.

Introspection Report

attached

Can you suggest a fix?

Are you interested in contributing with a fix?

inspection-report-20230607_094320.tar.gz

[sachinkumarsingh092]

Hey @sjerman, thanks for reporting this.
Here's what I did and was able to provision PVC using local-lvm storage class. Do tell me what you did differently here:

• Installed microk8s v1.27:

sudo snap install microk8s --classic --channel=1.27

• Created Volume Group for local testing using the official instructions.
• Installed the openebs localPV provisioned using the docs. Make a note on the step when installing on microk8s to change the kubelet directory path.
• Created and applied the storage class and PVC from the instruction.
  After all this I get a PVC which works fine:

$ k get pvc -A
NAMESPACE   NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
default     csi-lvmpv   Bound    pvc-08e2c003-ae88-4f40-867a-b4d9f5432487   4Gi        RWO            openebs-lvmpv   5s

What steps and/or configuration did you follow that were different?

[sjerman]

Hi,

spent a little time on it. It seems at though the issue happens with nfs-provisioning via the lvm provisioner.

Steve

So :)

snap install microk8s --classic  --channel=1.27/stable
microk8s helm repo add openebs https://openebs.github.io/charts
microk8s helm install --create-namespace -n openebs --wait -f openebs.yaml openebs openebs/openebs
microk8s kubectl install -f config.yaml

And you get symptoms like:

jerman@steve-nuc:~/Projects/microk8s-bug$ kubectl describe pod -n openebs nfs-pvc-7ae6fe71-1dcd-43bc-876e-9fe7f6cf966c-645597df89-dwftx 
Name:             nfs-pvc-7ae6fe71-1dcd-43bc-876e-9fe7f6cf966c-645597df89-dwftx
Namespace:        openebs
...
Labels:           nfs.openebs.io/nfs-pvc-name=nfs-rwx-pvc
                  nfs.openebs.io/nfs-pvc-namespace=default
                  nfs.openebs.io/nfs-pvc-uid=7ae6fe71-1dcd-43bc-876e-9fe7f6cf966c
                  openebs.io/nfs-server=nfs-pvc-7ae6fe71-1dcd-43bc-876e-9fe7f6cf966c
                  pod-template-hash=645597df89
...
Volumes:
  exports-dir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  nfs-pvc-7ae6fe71-1dcd-43bc-876e-9fe7f6cf966c
    ReadOnly:   false

Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  3m12s  default-scheduler  0/1 nodes are available: 1 node(s) did not have enough free storage. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..

Doesn't happen on 1.26.

Relevant files....

files.zip

[sjerman]

Oh.. one thing.. you will need to edit the config.yaml file to set the volume group to use. The default is vgubuntu .. and it assigns 2x 10G volumes...

[sachinkumarsingh092]

Hi @sjerman,
Thanks for the steps, I was able to replicate your error. The problem probably occurs during the external provisioning of PVs. The logs are full with

waiting for a volume to be created, either by external provisioner "openebs.io/nfsrwx" or manually created by system administrator

I tried to isolate the entire thing to see if the problem was with the NFS provisioned. We deploy the NFS provisioner while deploying the helm chart via the configs so that's already done. Then I created the nfs rwx storage class. But after that, when I create a sample PVC, it remains stuck in the pending state with logs:

│   Type     Reason                Age                   From                                                                                             Message                                                                                                                                               │
│   ----     ------                ----                  ----                                                                                             -------                                                                                                                                               │
│   Normal   Provisioning          4m45s (x7 over 17m)   openebs.io/nfsrwx_openebs-nfs-provisioner-74f4f7cffd-2z72t_a8d3d535-64be-40af-a297-904bbef73094  External provisioner is provisioning volume for claim "default/nfs-pvc"                                                                               │
│   Warning  ProvisioningFailed    3m45s (x7 over 16m)   openebs.io/nfsrwx_openebs-nfs-provisioner-74f4f7cffd-2z72t_a8d3d535-64be-40af-a297-904bbef73094  failed to provision volume with StorageClass "nfs-rwx-storage": failed to deploy NFS Server: timed out waiting for PVC{openebs/nfs-pvc-05a685ce-ec56- │
│ 45f7-bcd3-83be452ffdc6} to bound                                                                                                                                                                                                                                                                              │
│   Normal   ExternalProvisioning  2m24s (x63 over 17m)  persistentvolume-controller                                                                      waiting for a volume to be created, either by external provisioner "openebs.io/nfsrwx" or manually created by system administrator                    │
│                                                                                                                                                                                                                                                                                                 ```

Searchin a bit more, I found this issue in openEBS dynamic nfs provisioner which is similar to our issue. I've commented there to get the discussion started around this issue for microk8s.

[ergleb78]

Hi @sachinkumarsingh092 By any chance don't you know - is there a solution for that?
I'm sort of stuck with the problem when new PVCscan't be created after the upgrade.
Getting:
waiting for a volume to be created, either by external provisioner "openebs.io/local" or manually created by system administrator

The problem is that I can't even provision volumes using standard hostpath class.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.