This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Service Disruption Scenarios

Using this type of scenario configuration one is able to delete crucial objects in a specific namespace, or a namespace matching a certain regex string.

How to Run Service Disruption Scenarios

Choose your preferred method to run service disruption scenarios:

Example scenario files from scenarios-hub:

Configuration Options:

namespace: Specific namespace or regex style namespace of what you want to delete. Gets all namespaces if not specified; set to "" if you want to use the label_selector field.

Set to ‘^.*$’ and label_selector to "" to randomly select any namespace in your cluster.

label_selector: Label on the namespace you want to delete. Set to "" if you are using the namespace variable.

delete_count: Number of namespaces to kill in each run. Based on matching namespace and label specified, default is 1.

runs: Number of runs/iterations to kill namespaces, default is 1.

sleep: Number of seconds to wait between each iteration/count of killing namespaces. Defaults to 10 seconds if not set

Refer to namespace_scenarios_example config file.

scenarios:
- namespace: "^.*$"
  runs: 1
- namespace: "^.*ingress.*$"
  runs: 1
  sleep: 15

Steps

This scenario will select a namespace (or multiple) dependent on the configuration and will kill all of the below object types in that namespace and will wait for them to be Running in the post action

Services
Daemonsets
Statefulsets
Replicasets
Deployments

How to Use Plugin Name

Add the plugin name to the list of chaos_scenarios section in the config/config.yaml file

kraken:
    kubeconfig_path: ~/.kube/config                     # Path to kubeconfig
    ..
    chaos_scenarios:
        - service_disruption_scenarios:
            - scenarios/<scenario_name>.yaml

Note

You can specify multiple scenario files of the same type by adding additional paths to the list:

kraken:
    chaos_scenarios:
        - service_disruption_scenarios:
            - scenarios/service-disruption-1.yaml
            - scenarios/service-disruption-2.yaml
            - scenarios/service-disruption-3.yaml

You can also combine multiple different scenario types in the same config.yaml file. Scenario types can be specified in any order, and you can include the same scenario type multiple times:

kraken:
    chaos_scenarios:
        - service_disruption_scenarios:
            - scenarios/service-disruption.yaml
        - pod_disruption_scenarios:
            - scenarios/pod-kill.yaml
        - container_scenarios:
            - scenarios/container-kill.yaml
        - service_disruption_scenarios:  # Same type can appear multiple times
            - scenarios/service-disruption-2.yaml

Run

python run_kraken.py --config config/config.yaml

This scenario deletes main objects within a namespace in your Kubernetes/OpenShift cluster. More information can be found here.

Run

If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED environment variable for the chaos injection container to autoconnect.

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:service-disruption-scenarios
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

Note

–env-host: This option is not available with the remote Podman client, including Mac and Windows (excluding WSL2) machines. Without the –env-host option you’ll have to set each environment variable on the podman command line like -e <VARIABLE>=<value>

$ docker run $(./get_docker_params.sh) \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:service-disruption-scenarios
$ docker run \
  -e <VARIABLE>=<value> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:service-disruption-scenarios

$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

Tip

Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands:

kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host --pull=always -v ~kubeconfig:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:<scenario>

Supported parameters

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

Example if –env-host is used:

export <parameter_name>=<value>

OR on the command line like example:

-e <VARIABLE>=<value>

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables

Parameter	Description	Default
LABEL_SELECTOR	Label of the namespace to target. Set this parameter only if NAMESPACE is not set	""
NAMESPACE	Name of the namespace you want to target. Set this parameter only if LABEL_SELECTOR is not set	“openshift-etcd”
SLEEP	Number of seconds to wait before polling to see if namespace exists again	15
DELETE_COUNT	Number of namespaces to kill in each run, based on matching namespace and label specified	1
RUNS	Number of runs to execute the action	1

Note

In case of using custom metrics profile or alerts profile when CAPTURE_METRICS or ENABLE_ALERTS is enabled, mount the metrics profile from the host on which the container is run using podman/docker under /home/krkn/kraken/config/metrics-aggregated.yaml and /home/krkn/kraken/config/alerts.

For example:

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml \
  -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:service-disruption-scenarios

krknctl run service-disruption-scenarios (optional: --<parameter>:<value> )

Can also set any global variable listed here

Scenario specific parameters:

Parameter	Description	Type	Default
`--namespace`	Targeted namespace in the cluster (required)	string	openshift-etcd
`--label-selector`	Label of the namespace to target. Set this parameter only if NAMESPACE is not set	string
`--delete-count`	Number of namespaces to kill in each run, based on matching namespace and label specified	number	1
`--runs`	Number of runs to execute the action	number	1

To see all available scenario options

krknctl run service-disruption-scenarios --help

Demo

You can find a link to a demo of the scenario here