AKS Azure Fileshare CSI PV/PVC Suddenly producing "Read-Only" errors

[wanzhuM]

What happened:
We have a PV and associated PVC, shown below. We have a pod which mounts it and writes to it. It has been working for a long time with no issues, and recently it started to throw the following error:
---> System.IO.IOException: Read-only file system :

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: {{ .Values.fileshare.micsiName }}
provisioner: file.csi.azure.com
allowVolumeExpansion: true
parameters:
  shareName: {{ .Values.fileshare.shareName }}
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=0
  - gid=0
  - mfsymlinks
  - cache=strict # https://linux.die.net/man/8/mount.cifs
  - nosharesock
  - actimeo=30
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: {{ .Values.fileshare.geospatial.mipvName }}
spec:
  capacity:
    storage: {{ .Values.fileshare.capacity }}
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: {{ .Values.fileshare.micsiName }}
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - uid=0
    - gid=0
    - mfsymlinks
    - cache=strict # https://linux.die.net/man/8/mount.cifs
    - nosharesock
    - actimeo=30
  csi:
    driver: file.csi.azure.com
    readOnly: false
    volumeHandle: {{ .Values.fileshare.geospatial.mivolumeHandleName }}
    volumeAttributes:
      resourceGroup: {{ .Values.environment.resourcegroup }}
      storageAccount: {{ .Values.fileshare.storageAccountName }}
      shareName: {{ .Values.fileshare.shareName }}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {{ .Values.fileshare.geospatial.mipvcName }}
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: {{ .Values.fileshare.micsiName }}
  resources:
    requests:
      storage: {{ .Values.fileshare.capacity }}
  volumeName: {{ .Values.fileshare.geospatial.mipvName }}

What you expected to happen:
We expect the pod to continue to write no problem.

How to reproduce it:
Not sure at this point, failure seemed to occur randomly between regions.

Anything else we need to know?:
This is running fine in certain regions, and producing this error in others. It is not consistent.
Allow Blob anonymous access is enabled
Allow storage account key access is enabled
The fileshare is SMB.

Environment:

• CSI Driver version: image: mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.30.5
• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.2", GitCommit:"fc04e732bb3e7198d2fa44efa5457c7c6f8c0f5b", GitTreeState:"clean", BuildDate:"2023-02-22T13:39:03Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"29", GitVersion:"v1.29.8", GitCommit:"234bc63696ad15dcf62584b6ba48671bf0f25fb6", GitTreeState:"clean", BuildDate:"2024-08-15T17:13:33Z", GoVersion:"go1.22.5", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):

  Kernel Version:             5.15.164.1-1.cm2
  OS Image:                   CBL-Mariner/Linux

• Install tools: Followed docs at: https://learn.microsoft.com/en-us/azure/aks/azure-files-csi
• Others:

[andyzhangx]

the data path does not go through csi driver, it's the issue between cifs driver and azure file server, pls file a support ticket to azure file team, thx

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale