cassandra-pod-delete.md

id	title	sidebar_label
cassandra-pod-delete	Cassandra Pod Delete Experiment Details	Cassandra Pod Delete

---

Experiment Metadata

Type	Description	Tested K8s Platform
Cassandra	Fail the Cassandra statefulset pod	GKE, Konvoy(AWS), Packet(Kubeadm), Minikube, EKS

Prerequisites

• Ensure that the Litmus Chaos Operator is running by executing kubectl get pods in operator namespace (typically, litmus).If not, install from here
• Ensure that the cassandra-pod-delete experiment resource is available in the cluster by executing kubectl get chaosexperiments in the desired namespace. If not, install from here

Entry Criteria

• Cassandra pods are healthy before chaos injection

• The load should be distributed on the each replicas.

Exit Criteria

• Cassandra pods are healthy post chaos injection

• The load should be distributed on the each replicas.

Details

• Causes (forced/graceful) pod failure of specific/random replicas of an cassandra statefulset
• Tests cassandra sanity (replica availability & uninterrupted service) and recovery workflow of the cassandra statefulset.

Integrations

• Pod failures can be effected by setting LIB env as litmus

Steps to Execute the Chaos Experiment

• This Chaos Experiment can be triggered by creating a ChaosEngine resource on the cluster. To understand the values to provide in a ChaosEngine specification, refer Getting Started

• Follow the steps in the sections below to create the chaosServiceAccount, prepare the ChaosEngine & execute the experiment.

Prepare chaosServiceAccount

• Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment.

Sample Rbac Manifest

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cassandra-pod-delete-sa
  namespace: default
  labels:
    name: cassandra-pod-delete-sa
    app.kubernetes.io/part-of: litmus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cassandra-pod-delete-sa
  namespace: default
  labels:
    name: cassandra-pod-delete-sa
    app.kubernetes.io/part-of: litmus
rules:
- apiGroups: [""]
  resources: ["pods","events","services"]
  verbs: ["create","list","get","patch","update","delete","deletecollection"]
- apiGroups: [""]
  resources: ["pods/exec","pods/log"]
  verbs: ["create","list","get"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["create","list","get","delete","deletecollection"]
- apiGroups: ["apps"]
  resources: ["deployments","statefulsets"]
  verbs: ["list","get"]
- apiGroups: ["litmuschaos.io"]
  resources: ["chaosengines","chaosexperiments","chaosresults"]
  verbs: ["create","list","get","patch","update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cassandra-pod-delete-sa
  namespace: default
  labels:
    name: cassandra-pod-delete-sa
    app.kubernetes.io/part-of: litmus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cassandra-pod-delete-sa
subjects:
- kind: ServiceAccount
  name: cassandra-pod-delete-sa
  namespace: default

Prepare ChaosEngine

• Provide the application info in spec.appinfo
• Override the experiment tunables if desired in experiments.spec.components.env
• To understand the values to provided in a ChaosEngine specification, refer ChaosEngine Concepts

Supported Experiment Tunables

Variables	Description	Specify In ChaosEngine	Notes
CASSANDRA_SVC_NAME	Cassandra Service Name	Mandatory	Defaults value: cassandra
KEYSPACE_REPLICATION_FACTOR	Value of the Replication factor for the cassandra liveness deploy	Mandatory	It needs to create keyspace while checking the livenss of cassandra
CASSANDRA_PORT	Port of the cassandra statefulset	Mandatory	Defaults value: 9042
CASSANDRA_LIVENESS_CHECK	It allows to check the liveness of the cassandra statefulset	Optional	It can be`enabled` or `disabled`
CASSANDRA_LIVENESS_IMAGE	Image of the cassandra liveness deployment	Optional	Default value: litmuschaos/cassandra-client:latest
SEQUENCE	It defines sequence of chaos execution for multiple target pods	Optional	Default value: parallel. Supported: serial, parallel
TOTAL_CHAOS_DURATION	The time duration for chaos insertion (seconds)	Optional	Defaults to 15s
PODS_AFFECTED_PERC	The Percentage of total pods to target	Optional	Defaults to 0% (corresponds to 1 replica)
CHAOS_INTERVAL	Time interval b/w two successive pod failures (sec)	Optional	Defaults to 5s
LIB	The chaos lib used to inject the chaos	Optional	Defaults to litmus. Supported litmus only
FORCE	Application Pod deletion mode. `False` indicates graceful deletion with default termination period of 30s. 'True' indicates an immediate forceful deletion with 0s grace period	Optional	Default to `true`, With `terminationGracePeriodSeconds=0`
RAMP_TIME	Period to wait before injection of chaos in sec	Optional
INSTANCE_ID	A user-defined string that holds metadata/info about current run/instance of chaos. Ex: 04-05-2020-9-00. This string is appended as suffix in the chaosresult CR name.	Optional	Ensure that the overall length of the chaosresult CR is still < 64 characters

Sample ChaosEngine Manifest

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: cassandra-chaos
  namespace: default
spec:
  appinfo:
    appns: 'default'
    applabel: 'app=cassandra'
    appkind: 'statefulset'
  # It can be active/stop
  engineState: 'active'
  #ex. values: ns1:name=percona,ns2:run=nginx
  auxiliaryAppInfo: ''
  chaosServiceAccount: cassandra-pod-delete-sa
  experiments:
    - name: cassandra-pod-delete
      spec:
        components:
          env:
            # set chaos duration (in sec) as desired
            - name: TOTAL_CHAOS_DURATION
              value: '15'

            # set chaos interval (in sec) as desired
            - name: CHAOS_INTERVAL
              value: '15'
              
            # pod failures without '--force' & default terminationGracePeriodSeconds
            - name: FORCE
              value: 'false'

            # provide cassandra service name
            # default service: cassandra
            - name: CASSANDRA_SVC_NAME
              value: 'cassandra'

            # provide the keyspace replication factor
            - name: KEYSPACE_REPLICATION_FACTOR
              value: '3'

            # provide cassandra port 
            # default port: 9042
            - name: CASSANDRA_PORT
              value: '9042'

            # SET THE CASSANDRA_LIVENESS_CHECK
            # IT CAN BE `enable` OR `disable`
            - name: CASSANDRA_LIVENESS_CHECK
              value: ''

             ## percentage of total pods to target
            - name: PODS_AFFECTED_PERC
              value: ''
              

Create the ChaosEngine Resource

• Create the ChaosEngine manifest prepared in the previous step to trigger the Chaos.

  kubectl apply -f chaosengine.yml

• If the chaos experiment is not executed, refer to the troubleshooting section to identify the root cause and fix the issues.

Watch Chaos progress

• View pod terminations & recovery by setting up a watch on the pods in the application namespace

  watch -n 1 kubectl get pods -n <application-namespace>

Abort/Restart the Chaos Experiment

• To stop the pod-delete experiment immediately, either delete the ChaosEngine resource or execute the following command:

  kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"stop"}}'

• To restart the experiment, either re-apply the ChaosEngine YAML or execute the following command:

  kubectl patch chaosengine <chaosengine-name> -n <namespace> --type merge --patch '{"spec":{"engineState":"active"}}'

Check Chaos Experiment Result

• Check whether the cassandra statefulset is resilient to the pod failure, once the experiment (job) is completed. The ChaosResult resource name is derived like this: <ChaosEngine-Name>-<ChaosExperiment-Name>.

  kubectl describe chaosresult cassandra-chaos-cassandra-pod-delete -n <cassandra-namespace>

Cassandra Pod Failure Demo

• It will be added soon.