[Bug]: openebs-zfs-plugin: Operation cannot be fulfilled on zfsnodes.zfs.openebs.io

[Subetov]

What steps did you take and what happened:
A large number of similar errors in the logs of zfs-localpv-node pod.

openebs-zfs-plugin E0517 11:56:59.878576       1 zfsnode.go:279] error syncing 'kube-system/some-node': update zfs node kube-system/some-node: Operation cannot be fulfilled on zfsnodes.zfs.openebs.io "some-node": the object has been modified; please apply your changes to the latest version and try again, requeuing
openebs-zfs-plugin E0517 11:57:16.284629       1 zfsnode.go:279] error syncing 'kube-system/some-node': update zfs node kube-system/some-node: Operation cannot be fulfilled on zfsnodes.zfs.openebs.io "some-node": the object has been modified; please apply your changes to the latest version and try again, requeuing
openebs-zfs-plugin E0517 11:57:32.716366       1 zfsnode.go:279] error syncing 'kube-system/some-node': update zfs node kube-system/some-node: Operation cannot be fulfilled on zfsnodes.zfs.openebs.io "some-node": the object has been modified; please apply your changes to the latest version and try again, requeuing
openebs-zfs-plugin E0517 11:57:48.973489       1 zfsnode.go:279] error syncing 'kube-system/some-node': update zfs node kube-system/some-node: Operation cannot be fulfilled on zfsnodes.zfs.openebs.io "some-node": the object has been modified; please apply your changes to the latest version and try again, requeuing
openebs-zfs-plugin E0517 11:58:13.618259       1 zfsnode.go:279] error syncing 'kube-system/some-node': update zfs node kube-system/some-node: Operation cannot be fulfilled on zfsnodes.zfs.openebs.io "some-node": the object has been modified; please apply your changes to the latest version and try again, requeuing

I'm not 100% sure, but it seems to me that the error is related to the fact that there are too many requests to change the "zfsnodes.zfs.openebs.io" resource. (constantly changes the amount of free disk space)
Perhaps it depends on the load of the system.

What did you expect to happen:
No such errors.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other Pastebin is fine.)

• kubectl logs -f openebs-zfs-controller-0 -n kube-system -c openebs-zfs-plugin
  controller-openebs-zfs-plugin.logs.txt
• kubectl logs -f openebs-zfs-node-[xxxx] -n kube-system -c openebs-zfs-plugin
  node-openebs-zfs-plugin.logs.txt
• kubectl get pods -n kube-system
  openebs-zfs-zfs-localpv-node-pxz28 2/2 Running 0 91d
  openebs-zfs-zfs-localpv-controller-0 5/5 Running 0 91d
• kubectl get zv -A -o yaml
  zfs-volumes.yaml.txt

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:

• ZFS-LocalPV version: chart: zfs-localpv-1.9.7 appVer: 1.9.1
• Kubernetes version (use kubectl version): v1.24.8+k3s1

[Abhinandan-Purkait]

@niladrih Do you have any update on this. Seems like you have self assigned yourself.

[niladrih]

Hi @Subetov. Does this issue still persist? Were there instances of manual changes to any of the ZfsNode CustomResources?