Pod Scenarios

This scenario disrupts the pods matching the label, excluded label or pod name in the specified namespace on a Kubernetes/OpenShift cluster.

Why pod scenarios are important:

Modern applications demand high availability, low downtime, and resilient infrastructure. Kubernetes provides building blocks like Deployments, ReplicaSets, and Services to support fault tolerance, but understanding how these interact during disruptions is critical for ensuring reliability. Pod disruption scenarios test this reliability under various conditions, validating that the application and infrastructure respond as expected.

Use cases of pod scenarios

kubectl delete pod <pod-name> -n <namespace>
kubectl get pods -n <namespace> -w # watch for new pods

How to know if it is highly available

• Multiple Replicas Exist: Confirmed by checking kubectl get deploy -n <namespace> and seeing atleast 1 replica.
• Pods Distributed Across Nodes/availability zones: Using topologySpreadConstraints or observing pod distribution in kubectl get pods -o wide. See Health Checks for real time visibility into the impact of chaos scenarios on application availability and performance
• Service Uptime Remains Unaffected: During chaos test, verify app availability (synthetic probes, Prometheus alerts, etc).
• Recovery Is Automatic: No manual intervention needed to restore service.
• Krkn Telemetry Indicators: End of run data includes recovery times, pod reschedule latency, and service downtime which are vital metrics for assessing HA.

Excluding Pods from Disruption

Employ exclude_label to designate the safe pods in a group, while the rest of the pods in a namespace are subjected to chaos. Some frequent use cases are:

• Turn off the backend pods but make sure the database replicas that are highly available remain untouched.
• Inject the fault in the application layer, do not stop the infrastructure/monitoring pods.
• Run a rolling disruption experiment with the control-plane or system-critical components that are not affected.

Format:

exclude_label: "key=value"

Mechanism:

1. Pods are selected based on namespace_pattern + label_selector or name_pattern.
2. Before deletion, the pods that match exclude_label are removed from the list.
3. Rest of the pods are subjected to chaos.

Example: Have the Leader Protected While Different etcd Replicas Are Killed

- id: kill_pods
    config:
        namespace_pattern: ^openshift-etcd$
        label_selector: k8s-app=etcd
        exclude_label: role=etcd-leader
        krkn_pod_recovery_time: 120
        kill: 1

Example: Disrupt Backend, Skip Monitoring

- id: kill_pods
    config:
        namespace_pattern: ^production$
        label_selector: app=backend
        exclude_label: component=monitoring
        krkn_pod_recovery_time: 120
        kill: 2

Targeting Pods on Specific Nodes

By default, pod scenarios target all pods matching the namespace and label selectors regardless of which node they run on. However, you can narrow down the scope to only affect pods running on specific nodes using two options:

Option 1: Using Node Label Selector

Target pods running on nodes with specific labels (e.g., control-plane nodes, worker nodes, nodes in a specific zone).

Format:

node_label_selector: "key=value"

Use Cases:

• Test resilience of control-plane workloads by disrupting pods only on master/control-plane nodes
• Simulate zone-specific failures by targeting nodes in a particular availability zone
• Test worker node failures without affecting control-plane components

Example: Target Pods on Control-Plane Nodes

- id: kill_pods
  config:
    namespace_pattern: ^kube-system$
    label_selector: k8s-app=kube-scheduler
    node_label_selector: node-role.kubernetes.io/control-plane=
    krkn_pod_recovery_time: 120

Example: Target Pods in a Specific Availability Zone

- id: kill_pods
  config:
    namespace_pattern: ^production$
    label_selector: app=backend
    node_label_selector: topology.kubernetes.io/zone=us-east-1a
    krkn_pod_recovery_time: 120

Option 2: Using Node Names

Target pods running on explicitly named nodes. This is useful for testing specific node scenarios or mixed node type environments.

Format:

node_names:
  - node-name-1
  - node-name-2

Use Cases:

• Test failures on specific nodes (e.g., nodes with known hardware issues)
• Simulate scenarios involving mixed node types (e.g., GPU nodes, high-memory nodes)
• Validate pod distribution and failover between specific nodes

Example: Target Pods on Specific Nodes

- id: kill_pods
  config:
    namespace_pattern: ^kube-system$
    label_selector: k8s-app=kube-scheduler
    node_names:
      - ip-10-0-31-8.us-east-2.compute.internal
      - ip-10-0-48-188.us-east-2.compute.internal
    krkn_pod_recovery_time: 120

Mechanism:

1. Pods are selected based on namespace_pattern + label_selector or name_pattern
2. The selection is further filtered to only include pods running on the specified nodes
3. If exclude_label is also specified, it’s applied after node filtering
4. The remaining pods are subjected to chaos

Recovery Time Metrics in Krkn Telemetry

Krkn tracks three key recovery time metrics for each affected pod:

1. pod_rescheduling_time - The time (in seconds) that the Kubernetes cluster took to reschedule the pod after it was killed. This measures the cluster’s scheduling efficiency and includes the time from pod deletion until the replacement pod is scheduled on a node.

2. pod_readiness_time - The time (in seconds) the pod took to become ready after being scheduled. This measures application startup time, including container image pulls, initialization, and readiness probe success.

3. total_recovery_time - The total amount of time (in seconds) from pod deletion until the replacement pod became fully ready and available to serve traffic. This is the sum of rescheduling time and readiness time.

These metrics appear in the telemetry output under PodsStatus.recovered for successfully recovered pods. Pods that fail to recover within the timeout period appear under PodsStatus.unrecovered without timing data.

Example telemetry output:

{
  "recovered": [
    {
      "pod_name": "backend-7d8f9c-xyz",
      "namespace": "production",
      "pod_rescheduling_time": 2.3,
      "pod_readiness_time": 5.7,
      "total_recovery_time": 8.0
    }
  ],
  "unrecovered": []
}

How to Run Pod Scenarios

Choose your preferred method to run pod scenarios:

• Krkn
• Krkn-hub
• Krknctl

kraken:
  chaos_scenarios:
    - pod_disruption_scenarios:
      - path/to/scenario.yaml

kraken:
  chaos_scenarios:
    - pod_disruption_scenarios:
      - path/to/scenario1.yaml
      - path/to/scenario2.yaml
      - path/to/scenario3.yaml

kraken:
  chaos_scenarios:
    - pod_disruption_scenarios:
      - scenarios/pod-kill.yaml
      - scenarios/etcd-kill.yaml
    - container_scenarios:
      - scenarios/container-kill.yaml
    - node_scenarios:
      - scenarios/node-reboot.yaml
    - pod_disruption_scenarios:  # Same type can appear multiple times
      - scenarios/pod-kill-2.yaml

# yaml-language-server: $schema=../plugin.schema.json
- id: kill-pods
  config:
    namespace_pattern: ^kube-system$
    label_selector: k8s-app=kube-scheduler
    krkn_pod_recovery_time: 120
    #Not needed by default, but can be used if you want to target pods on specific nodes
    # Option 1: Target pods on nodes with specific labels [master/worker nodes]
    node_label_selector: node-role.kubernetes.io/control-plane=      # Target control-plane nodes (works on both k8s and openshift)
    exclude_label: 'critical=true' # Optional - Pods matching this label will be excluded from the chaos
    # Option 2: Target pods of specific nodes (testing mixed node types)
    node_names:
      - ip-10-0-31-8.us-east-2.compute.internal      # Worker node 1
      - ip-10-0-48-188.us-east-2.compute.internal    # Worker node 2
      - ip-10-0-14-59.us-east-2.compute.internal     # Master node 1

python run_kraken.py --config config/config.yaml

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pod-scenarios
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

$ docker run $(./get_docker_params.sh) \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pod-scenarios
$ docker run \
  -e <VARIABLE>=<value> \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pod-scenarios

$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

export <parameter_name>=<value>

-e <VARIABLE>=<value>

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml \
  -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:container-scenarios

krknctl run pod-scenarios (optional: --<parameter>:<value> )

krknctl run pod-scenarios --help

Demo

See a demo of this scenario: