pod-io-stress.md

id	title	sidebar_label
pod-io-stress	Pod IO Stress Details	Pod IO Stress

---

Experiment Metadata

Type	Description	Tested K8s Platform
Generic	Inject IO stress on the application container	GKE, Packet(Kubeadm), Minikube, AKS

Prerequisites

• Ensure that Kubernetes Version > 1.16
• Ensure that the Litmus Chaos Operator is running by executing kubectl get pods in operator namespace (typically, litmus). If not, install from here
• Ensure that the pod-io-stress experiment resource is available in the cluster by executing kubectl get chaosexperiments in the desired namespace. If not, install from here
• Cluster must run docker container runtime

Entry Criteria

• Application pods are healthy on the respective nodes before chaos injection

Exit Criteria

• Application pods are healthy on the respective nodes post chaos injection

Details

• This experiment causes disk stress on the application pod. The experiment aims to verify the resiliency of applications that share this disk resource for ephemeral or persistent storage purposes

Integrations

• Pod IO Stress can be effected using the chaos library: litmus and pumba

Steps to Execute the Chaos Experiment

• This Chaos Experiment can be triggered by creating a ChaosEngine resource on the cluster. To understand the values to provide in a ChaosEngine specification, refer Getting Started

• Follow the steps in the sections below to create the chaosServiceAccount, prepare the ChaosEngine & execute the experiment.

Prepare chaosServiceAccount

Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment.

Sample Rbac Manifest

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-io-stress-sa
  namespace: default
  labels:
    name: pod-io-stress-sa
    app.kubernetes.io/part-of: litmus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-io-stress-sa
  namespace: default
  labels:
    name: pod-io-stress-sa
    app.kubernetes.io/part-of: litmus
rules:
- apiGroups: [""]
  resources: ["pods","events"]
  verbs: ["create","list","get","patch","update","delete","deletecollection"]
- apiGroups: [""]
  resources: ["pods/exec","pods/log","replicationcontrollers"]
  verbs: ["create","list","get"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["create","list","get","delete","deletecollection"]
- apiGroups: ["apps"]
  resources: ["deployments","statefulsets","daemonsets","replicasets"]
  verbs: ["list","get"]
- apiGroups: ["apps.openshift.io"]
  resources: ["deploymentconfigs"]
  verbs: ["list","get"]
- apiGroups: ["argoproj.io"]
  resources: ["rollouts"]
  verbs: ["list","get"]
- apiGroups: ["litmuschaos.io"]
  resources: ["chaosengines","chaosexperiments","chaosresults"]
  verbs: ["create","list","get","patch","update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-io-stress-sa
  namespace: default
  labels:
    name: pod-io-stress-sa
    app.kubernetes.io/part-of: litmus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-io-stress-sa
subjects:
- kind: ServiceAccount
  name: pod-io-stress-sa
  namespace: default

Note: In case of restricted systems/setup, create a PodSecurityPolicy(psp) with the required permissions. The chaosServiceAccount can subscribe to work around the respective limitations. An example of a standard psp that can be used for litmus chaos experiments can be found here.

Prepare ChaosEngine

• Provide the application info in spec.appinfo
• Provide the auxiliary applications info (ns & labels) in spec.auxiliaryAppInfo
• Override the experiment tunables if desired in experiments.spec.components.env
• To understand the values to provided in a ChaosEngine specification, refer ChaosEngine Concepts

Supported Experiment Tunables

Variables	Description	Type	Notes
FILESYSTEM_UTILIZATION_PERCENTAGE	Specify the size as percentage of free space on the file system	Optional	Default to 10%
FILESYSTEM_UTILIZATION_BYTES	Specify the size in GigaBytes(GB). FILESYSTEM_UTILIZATION_PERCENTAGE & FILESYSTEM_UTILIZATION_BYTES are mutually exclusive. If both are provided, FILESYSTEM_UTILIZATION_PERCENTAGE is prioritized.	Optional
NUMBER_OF_WORKERS	It is the number of IO workers involved in IO disk stress	Optional	Default to 4
TOTAL_CHAOS_DURATION	The time duration for chaos (seconds)	Optional	Default to 120s
VOLUME_MOUNT_PATH	Fill the given volume mount path	Optional
LIB	The chaos lib used to inject the chaos	Optional	Default to litmus. Available litmus and pumba.
LIB_IMAGE	Image used to run the stress command	Optional	Default to litmuschaos/go-runner:latest
TARGET_PODS	Comma separated list of application pod name subjected to pod io stress chaos	Optional	If not provided, it will select target pods randomly based on provided appLabels
PODS_AFFECTED_PERC	The Percentage of total pods to target	Optional	Defaults to 0 (corresponds to 1 replica), provide numeric value only
CONTAINER_RUNTIME	container runtime interface for the cluster	Optional	Defaults to docker, supported values: docker, containerd and crio for litmus and only docker for pumba LIB
SOCKET_PATH	Path of the containerd/crio/docker socket file	Optional	Defaults to `/var/run/docker.sock`
RAMP_TIME	Period to wait before and after injection of chaos in sec	Optional
SEQUENCE	It defines sequence of chaos execution for multiple target pods	Optional	Default value: parallel. Supported: serial, parallel
INSTANCE_ID	A user-defined string that holds metadata/info about current run/instance of chaos. Ex: 04-05-2020-9-00. This string is appended as suffix in the chaosresult CR name.	Optional	Ensure that the overall length of the chaosresult CR is still < 64 characters

Sample ChaosEngine Manifest

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: nginx-chaos
  namespace: default
spec:
  # It can be active/stop
  engineState: 'active'
  appinfo:
    appns: 'default'
    applabel: 'app=nginx'
    appkind: 'deployment'
  chaosServiceAccount: pod-io-stress-sa
  experiments:
    - name: pod-io-stress
      spec:
        components:
          env:
            # set chaos duration (in sec) as desired
            - name: TOTAL_CHAOS_DURATION
              value: '120'

            ## specify the size as percentage of free space on the file system
            - name: FILESYSTEM_UTILIZATION_PERCENTAGE
              value: '10'

             ## provide the cluster runtime
            - name: CONTAINER_RUNTIME
              value: 'docker'   

            # provide the socket file path
            - name: SOCKET_PATH
              value: '/var/run/docker.sock'     

Create the ChaosEngine Resource

• Create the ChaosEngine manifest prepared in the previous step to trigger the Chaos.

  kubectl apply -f chaosengine.yml

• If the chaos experiment is not executed, refer to the troubleshooting section to identify the root cause and fix the issues.

Watch Chaos progress

• View the status of the pods as they are subjected to IO disk stress.

  watch -n 1 kubectl get pods -n <application-namespace>

• Monitor the capacity filled up on the host filesystem

  watch du -h

Abort/Restart the Chaos Experiment

• To stop the pod-io-stress experiment immediately, either delete the ChaosEngine resource or execute the following command:

  kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"stop"}}'

• To restart the experiment, either re-apply the ChaosEngine YAML or execute the following command:

  kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"active"}}'

Check Chaos Experiment Result

• Check whether the application stack is resilient to IO stress on the app replica, once the experiment (job) is completed. The ChaosResult resource name is derived like this: <ChaosEngine-Name>-<ChaosExperiment-Name>.

  kubectl describe chaosresult nginx-chaos-pod-io-stress -n <application-namespace>

Pod IO Stress Experiment Demo

• The Demo Video will be Added soon.