Getting Started with Running Scenarios

• 1: Running a Chaos Scenario with Krkn
• 2: Krkn Use cases

1 - Running a Chaos Scenario with Krkn

Getting Started Running Chaos Scenarios

Config

Instructions on how to setup the config and all the available options supported can be found at Config.

In all the examples below you’ll replace the scenario_type with the scenario plugin type that can be found in the second column here

Running a Single Scenario

To run a single scenario, you’ll edit the krkn config file and only have 1 item in the list of chaos_scenarios

kraken:
    ...
    chaos_scenarios:
        - <scenario_type>:
            - scenarios/<scenario_file>
    ...

Running Multiple Scnearios

To run multiple scenarios, you’ll edit the krkn config file and add multiple scenarios into chaos_scenarios. If you want to run multiple scenario files that are the same scenario type you can add multiple items under the scenario_type. If you want to run multiple different scenario types you can add those under chaos_scenarios

kraken:
    ...
    chaos_scenarios:
        - <scenario_type>:
            - scenarios/<scenario_file_1>
            - scenarios/<scenario_file_2>
        - <scenario_type_2>:
            - scenarios/<scenario_file_3>
            - scenarios/<scenario_file_4>

Creating a Scenario File

You can either copy an existing scenario yaml file and make it your own, or fill in one of the templates below to suit your needs.

Common Scenario Edits

If you just want to make small changes to pre-existing scenarios, feel free to edit the scenario file itself.

Example of Quick Pod Scenario Edit:

If you want to kill 2 pods instead of 1 in any of the pre-existing scenarios, you can either edit the iterations number located at config or edit the kill count in the scenario file

- id: kill-pods
  config:
    namespace_pattern: ^kube-system$
    name_pattern: .*
    kill: 1 -> 2
    krkn_pod_recovery_time: 120

Example of Quick Nodes Scenario Edit:

If your cluster is build on GCP instead of AWS, just change the cloud type in the node_scenarios_example.yml file.

node_scenarios:
  - actions:
    - node_reboot_scenario
    node_name:
    label_selector: node-role.kubernetes.io/worker
    instance_count: 1
    timeout: 120
    cloud_type: aws -> gcp
    parallel: true
    kube_check: true

Templates

Pod Scenario Yaml Template

For example, for adding a pod level scenario for a new application, refer to the sample scenario below to know what fields are necessary and what to add in each location:

# yaml-language-server: $schema=../plugin.schema.json
- id: kill-pods
  config:
    namespace_pattern: ^<namespace>$
    label_selector: <pod label>
    kill: <number of pods to kill>
    krkn_pod_recovery_time: <expected time for the pod to become ready>

Node Scenario Yaml Template

node_scenarios:
  - actions:  # Node chaos scenarios to be injected.
    - <chaos scenario>
    - <chaos scenario>
    node_name: <node name>  # Can be left blank.
    label_selector: <node label>
    instance_kill_count: <number of nodes on which to perform action>
    timeout: <duration to wait for completion>
    cloud_type: <cloud provider>

Time Chaos Scenario Template

time_scenarios:
  - action: 'skew_time' or 'skew_date'
    object_type: 'pod' or 'node'
    label_selector: <label of pod or node>

RBAC

Based on the type of chaos test being executed, certain scenarios may require elevated privileges. The specific RBAC Authorization needed for each Krkn scenario are outlined in detail at the following link: Krkn RBAC

2 - Krkn Use cases

Network Chaos NG

Node Network Filter

AWS EFS (Elastic File System) disruption

Description

This scenario creates an outgoing firewall rule on specific nodes in your cluster, chosen by node name or a selector. This rule blocks connections to AWS EFS, leading to a temporary failure of any EFS volumes mounted on those affected nodes.

podman

podman run -v ~/.kube/config:/home/krkn/.kube/config:z -e TEST_DURATION="60" -e INGRESS="false" -e EGRESS="true" -e PROTOCOLS="tcp,udp" -e PORTS="2049" -e NODE_NAME="kind-control-plane" quay.io/krkn-chaos/krkn-hub:node-network-filter

krknctl

krknctl run node-network-filter \
 --chaos-duration 60 \
 --node-name kind-control-plane \
 --ingress false \
 --egress true \
 --protocols tcp,udp \
 --ports 2049

etcd split brain

Description

This scenario isolates an etcd node by blocking its network traffic. This action forces an etcd leader re-election. Once the scenario concludes, the cluster should temporarily exhibit a split-brain condition, with two etcd leaders active simultaneously. This is particularly useful for testing the etcd cluster’s resilience under such a challenging state.

podman

podman run -v ~/.kube/config:/home/krkn/.kube/config:z -e TEST_DURATION="60" -e INGRESS="false" -e EGRESS="true" -e PROTOCOLS="tcp" -e PORTS="2379,2380" -e NODE_NAME="kind-control-plane" quay.io/krkn-chaos/krkn-hub:node-network-filter

krknctl

krknctl run node-network-filter \
 --chaos-duration 60 \
 --node-name kind-control-plane \
 --ingress false \
 --egress true \
 --protocols tcp \
 --ports 2379,2380

Pod Network Filter

Pod DNS outage

Description

This scenario blocks all outgoing DNS traffic from a specific pod, effectively preventing it from resolving any hostnames or service names.

podman

podman run -v ~/.kube/config:/home/krkn/.kube/config:z -e TEST_DURATION="60" -e INGRESS="false" -e EGRESS="true" -e PROTOCOLS="tcp,udp" -e PORTS="53" -e POD_NAME="target-pod" quay.io/krkn-chaos/krkn-hub:pod-network-filter

krknctl

krknctl run pod-network-filter \
 --chaos-duration 60 \
 --pod-name target-pod \
 --ingress false \
 --egress true \
 --protocols tcp,udp \
 --ports 53

Pod AWS aurora Disruption

Description

This scenario blocks a pod’s outgoing MySQL and PostgreSQL traffic, effectively preventing it from connecting to any AWS Aurora SQL engine. It works just as well for standard MySQL and PostgreSQL connections too.

podman

podman run -v ~/.kube/config:/home/krkn/.kube/config:z -e TEST_DURATION="60" -e INGRESS="false" -e EGRESS="true" -e PROTOCOLS="tcp" -e PORTS="3306,5432" -e POD_NAME="target-pod" quay.io/krkn-chaos/krkn-hub:pod-network-filter

krknctl

krknctl run pod-network-filter \
 --chaos-duration 60 \
 --pod-name target-pod \
 --ingress false \
 --egress true \
 --protocols tcp \
 --ports 3306,5432