This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Node Scenarios

1: Node Scenarios using Krkn
2: Node Scenarios using Krkn-Hub

This scenario disrupts the node(s) matching the label on a Kubernetes/OpenShift cluster.

1 - Node Scenarios using Krkn

The following node chaos scenarios are supported:

node_start_scenario: Scenario to stop the node instance.
node_stop_scenario: Scenario to stop the node instance.
node_stop_start_scenario: Scenario to stop and then start the node instance. Not supported on VMware.
node_termination_scenario: Scenario to terminate the node instance.
node_reboot_scenario: Scenario to reboot the node instance.
stop_kubelet_scenario: Scenario to stop the kubelet of the node instance.
stop_start_kubelet_scenario: Scenario to stop and start the kubelet of the node instance.
restart_kubelet_scenario: Scenario to restart the kubelet of the node instance.
node_crash_scenario: Scenario to crash the node instance.
stop_start_helper_node_scenario: Scenario to stop and start the helper node and check service status.

Note

If the node does not recover from the node_crash_scenario injection, reboot the node to get it back to Ready state.

Note

node_start_scenario, node_stop_scenario, node_stop_start_scenario, node_termination_scenario , node_reboot_scenario and stop_start_kubelet_scenario are supported on AWS, Azure, OpenStack, BareMetal, GCP , VMware and Alibaba.

AWS

Cloud setup instructions can be found here. Sample scenario config can be found here.

Baremetal

Sample scenario config can be found here.

Note

Baremetal requires setting the IPMI user and password to power on, off, and reboot nodes, using the config options bm_user and bm_password. It can either be set in the root of the entry in the scenarios config, or it can be set per machine.

If no per-machine addresses are specified, kraken attempts to use the BMC value in the BareMetalHost object. To list them, you can do ‘oc get bmh -o wide –all-namespaces’. If the BMC values are blank, you must specify them per-machine using the config option ‘bmc_addr’ as specified below.

For per-machine settings, add a “bmc_info” section to the entry in the scenarios config. Inside there, add a configuration section using the node name. In that, add per-machine settings. Valid settings are ‘bmc_user’, ‘bmc_password’, and ‘bmc_addr’. See the example node scenario or the example below.

Note

Baremetal requires oc (openshift client) be installed on the machine running Kraken.

Note

Baremetal machines are fragile. Some node actions can occasionally corrupt the filesystem if it does not shut down properly, and sometimes the kubelet does not start properly.

Docker

The Docker provider can be used to run node scenarios against kind clusters.

kind is a tool for running local Kubernetes clusters using Docker container “nodes”.

kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

GCP

Cloud setup instructions can be found here. Sample scenario config can be found here.

Openstack

How to set up Openstack cli to run node scenarios is defined here.

The supported node level chaos scenarios on an OPENSTACK cloud are node_stop_start_scenario, stop_start_kubelet_scenario and node_reboot_scenario.

Note

For stop_start_helper_node_scenario, visit here to learn more about the helper node and its usage.

To execute the scenario, ensure the value for ssh_private_key in the node scenarios config file is set with the correct private key file path for ssh connection to the helper node. Ensure passwordless ssh is configured on the host running Kraken and the helper node to avoid connection errors.

Azure

Cloud setup instructions can be found here. Sample scenario config can be found here.

Alibaba

How to set up Alibaba cli to run node scenarios is defined here.

Note

There is no “terminating” idea in Alibaba, so any scenario with terminating will “release” the node . Releasing a node is 2 steps, stopping the node and then releasing it.

VMware

How to set up VMware vSphere to run node scenarios is defined here

This cloud type uses a different configuration style, see actions below and example config file

vmware-node-terminate
vmware-node-reboot
vmware-node-stop
vmware-node-start

IBMCloud

How to set up IBMCloud to run node scenarios is defined here

This cloud type uses a different configuration style, see actions below and example config file

ibmcloud-node-terminate
ibmcloud-node-reboot
ibmcloud-node-stop
ibmcloud-node-start

General

Note

The node_crash_scenario and stop_kubelet_scenario scenario is supported independent of the cloud platform.

Use ‘generic’ or do not add the ‘cloud_type’ key to your scenario if your cluster is not set up using one of the current supported cloud types.

2 - Node Scenarios using Krkn-Hub

This scenario disrupts the node(s) matching the label on a Kubernetes/OpenShift cluster. Actions/disruptions supported are listed here

Run

If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED environment variable for the chaos injection container to autoconnect.

$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-scenarios
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

$ docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-scenarios
OR 
$ docker run -e <VARIABLE>=<value> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-scenarios

$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

Tip

Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands:

kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v ~kubeconfig:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:<scenario>

Supported parameters

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

example: export <parameter_name>=<value>

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables

Parameter	Description	Default
ACTION	Action can be one of the following	node_stop_start_scenario for aws and vmware-node-reboot for vmware, ibmcloud-node-reboot for ibmcloud
LABEL_SELECTOR	Node label to target	node-role.kubernetes.io/worker
NODE_NAME	Node name to inject faults in case of targeting a specific node; Can set multiple node names separated by a comma	""
INSTANCE_COUNT	Targeted instance count matching the label selector	1
RUNS	Iterations to perform action on a single node	1
CLOUD_TYPE	Cloud platform on top of which cluster is running, supported platforms - aws, vmware, ibmcloud, bm	aws
TIMEOUT	Duration to wait for completion of node scenario injection	180
DURATION	Duration to stop the node before running the start action - not supported for vmware and ibm cloud type	120
VERIFY_SESSION	Only needed for vmware - Set to True if you want to verify the vSphere client session using certificates	False
SKIP_OPENSHIFT_CHECKS	Only needed for vmware - Set to True if you don’t want to wait for the status of the nodes to change on OpenShift before passing the scenario	False
BMC_USER	Only needed for Baremetal ( bm ) - IPMI/bmc username	""
BMC_PASSWORD	Only needed for Baremetal ( bm ) - IPMI/bmc password	""
BMC_ADDR	Only needed for Baremetal ( bm ) - IPMI/bmc username	""

Note

In case of using custom metrics profile or alerts profile when CAPTURE_METRICS or ENABLE_ALERTS is enabled, mount the metrics profile from the host on which the container is run using podman/docker under /home/krkn/kraken/config/metrics-aggregated.yaml and /home/krkn/kraken/config/alerts.

For example:

$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:container-scenarios

The following environment variables need to be set for the scenarios that requires intereacting with the cloud platform API to perform the actions:

Amazon Web Services

$ export AWS_ACCESS_KEY_ID=<>
$ export AWS_SECRET_ACCESS_KEY=<>
$ export AWS_DEFAULT_REGION=<>

VMware Vsphere

$ export VSPHERE_IP=<vSphere_client_IP_address>

$ export VSPHERE_USERNAME=<vSphere_client_username>

$ export VSPHERE_PASSWORD=<vSphere_client_password>

Ibmcloud

$ export IBMC_URL=https://<region>.iaas.cloud.ibm.com/v1

$ export IBMC_APIKEY=<ibmcloud_api_key>

Baremetal

$ export BMC_USER=<bmc/IPMI user>
$ export BMC_PASSWORD=<bmc/IPMI password>
$ export BMC_ADDR=<bmc address>

Google Cloud Platform

TBD

Azure

$ export AZURE_TENANT_ID=<>
$ export AZURE_CLIENT_SECRET=<>
$ export AZURE_CLIENT_ID=<>

OpenStack

TBD

Demo

You can find a link to a demo of the scenario here