This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Node Scenarios

This scenario disrupts the node(s) matching the label or node name(s) on a Kubernetes/OpenShift cluster. These scenarios are performed in two different ways, either by the clusters cloud cli or by common/generic commands that can be performed on any cluster.

Actions

The following node chaos scenarios are supported:

  1. node_start_scenario: Scenario to start the node instance. Need access to cloud provider
  2. node_stop_scenario: Scenario to stop the node instance. Need access to cloud provider
  3. node_stop_start_scenario: Scenario to stop and then start the node instance. Not supported on VMware. Need access to cloud provider
  4. node_termination_scenario: Scenario to terminate the node instance. Need access to cloud provider
  5. node_reboot_scenario: Scenario to reboot the node instance. Need access to cloud provider
  6. stop_kubelet_scenario: Scenario to stop the kubelet of the node instance. Need access to cloud provider
  7. stop_start_kubelet_scenario: Scenario to stop and start the kubelet of the node instance. Need access to cloud provider
  8. restart_kubelet_scenario: Scenario to restart the kubelet of the node instance. Can be used with generic cloud type or when you don’t have access to cloud provider
  9. node_crash_scenario: Scenario to crash the node instance. Can be used with generic cloud type or when you don’t have access to cloud provider
  10. stop_start_helper_node_scenario: Scenario to stop and start the helper node and check service status. Need access to cloud provider
  11. node_block_scenario: Scenario to block inbound and outbound traffic from other nodes to a specific node for a set duration (only for Azure). Need access to cloud provider
  12. node_disk_detach_attach_scenario: Scenario to detach and reattach disks (only for baremetals).

Clouds

Supported cloud supported:

Recovery Times

In each node scenario, the end telemetry details of the run will show the time it took for each node to stop and recover depening on the scenario.

The details printed in telemetry:

  • node_name: Node name
  • node_id: Node id
  • not_ready_time: Amount of time the node took to get to a not ready state after cloud provider has stopped node
  • ready_time: Amount of time the node took to get to a ready state after cloud provider has become in started state
  • stopped_time: Amount of time the cloud provider took to stop a node
  • running_time: Amount of time the cloud provider took to get a node running
  • terminating_time: Amount of time the cloud provider took for node to become terminated

Example:

"affected_nodes": [
    {
        "node_name": "cluster-name-**.438115.internal",
        "node_id": "cluster-name-**",
        "not_ready_time": 0.18194103240966797,
        "ready_time": 0.0,
        "stopped_time": 140.74104499816895,
        "running_time": 0.0,
        "terminating_time": 0.0
    },
    {
        "node_name": "cluster-name-**-master-0.438115.internal",
        "node_id": "cluster-name-**-master-0",
        "not_ready_time": 0.1611928939819336,
        "ready_time": 0.0,
        "stopped_time": 146.72056317329407,
        "running_time": 0.0,
        "terminating_time": 0.0
    },
    {
        "node_name": "cluster-name-**.438115.internal",
        "node_id": "cluster-name-**",
        "not_ready_time": 0.0,
        "ready_time": 43.521320104599,
        "stopped_time": 0.0,
        "running_time": 12.305592775344849,
        "terminating_time": 0.0
    },
    {
        "node_name": "cluster-name-**-master-0.438115.internal",
        "node_id": "cluster-name-**-master-0",
        "not_ready_time": 0.0,
        "ready_time": 48.33336925506592,
        "stopped_time": 0.0,
        "running_time": 12.052034854888916,
        "terminating_time": 0.0
    }
]

How to Run Node Scenarios

Choose your preferred method to run node scenarios:

For any of the node scenarios, you’ll specify node_scenarios as the scenario type.

Example scenario files from scenarios-hub:

See example config here:

    chaos_scenarios:
        - node_scenarios: # List of chaos node scenarios to load
            - scenarios/***.yml
            - scenarios/***.yml # Can specify multiple files here

Sample scenario file, you are able to specify multiple list items under node_scenarios that will be ran serially

node_scenarios:
  - actions:                   # node chaos scenarios to be injected
    - <action>                 # Can specify multiple actions here
    node_name: <node_name>     # node on which scenario has to be injected; can set multiple names separated by comma
    label_selector: <label>    # when node_name is not specified, a node with matching label_selector is selected for node chaos scenario injection; can specify multiple by a comma separated list
    exclude_label: <label>     # if label_selector is set, will exclude nodes marked by this label from the chaos scenario
    instance_count: <instance_number> # Number of nodes to perform action/select that match the label selector
    runs: <run_int>            # number of times to inject each scenario under actions (will perform on same node each time)
    timeout: <timeout>         # duration to wait for completion of node scenario injection
    duration: <duration>       # duration to stop the node before running the start action
    cloud_type: <cloud>        # cloud type on which Kubernetes/OpenShift runs  
    parallel: <true_or_false>  # Run action on label or node name in parallel or sequential, defaults to sequential
    kube_check: <true_or_false> # Run the kubernetes api calls to see if the node gets to a certain state during the node scenario
    disable_ssl_verification: <true_or_false> # Disable SSL verification, to avoid certificate errors

AWS

Cloud setup instructions can be found here. Sample scenario config can be found here.

The cloud type in the scenario yaml file needs to be aws

Baremetal

Sample scenario config can be found here.

The cloud type in the scenario yaml file needs to be bm

Docker

The Docker provider can be used to run node scenarios against kind clusters.

kind is a tool for running local Kubernetes clusters using Docker container “nodes”.

kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

GCP

Cloud setup instructions can be found here. Sample scenario config can be found here.

The cloud type in the scenario yaml file needs to be gcp

Openstack

How to set up Openstack cli to run node scenarios is defined here.

The cloud type in the scenario yaml file needs to be openstack

The supported node level chaos scenarios on an OPENSTACK cloud are only: node_stop_start_scenario, stop_start_kubelet_scenario and node_reboot_scenario.

To execute the scenario, ensure the value for ssh_private_key in the node scenarios config file is set with the correct private key file path for ssh connection to the helper node. Ensure passwordless ssh is configured on the host running Kraken and the helper node to avoid connection errors.

Azure

Cloud setup instructions can be found here. Sample scenario config can be found here.

The cloud type in the scenario yaml file needs to be azure

Alibaba

How to set up Alibaba cli to run node scenarios is defined here.

The cloud type in the scenario yaml file needs to be alibaba

VMware

How to set up VMware vSphere to run node scenarios is defined here

The cloud type in the scenario yaml file needs to be vmware

IBMCloud

How to set up IBMCloud to run node scenarios is defined here

See a sample of ibm cloud node scenarios example config file

The cloud type in the scenario yaml file needs to be ibm

IBMCloud Power

How to set up IBMCloud Power to run node scenarios is defined here

See a sample of ibm cloud node scenarios example config file

The cloud type in the scenario yaml file needs to be ibmpower or ibmcloudpower

General

Use ‘generic’ or do not add the ‘cloud_type’ key to your scenario if your cluster is not set up using one of the current supported cloud types.

Run

python run_kraken.py --config config/config.yaml

This scenario disrupts the node(s) matching the label on a Kubernetes/OpenShift cluster. Actions/disruptions supported are listed here

Run

If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED environment variable for the chaos injection container to autoconnect.

$ podman run \
  --name=<container_name> \
  --net=host \
  --env-host=true \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:node-scenarios
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
$ docker run $(./get_docker_params.sh) \
  --name=<container_name> \
  --net=host \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:node-scenarios
$ docker run \
  -e <VARIABLE>=<value> \
  --net=host \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:node-scenarios

$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

Supported parameters

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

Example if –env-host is used:

export <parameter_name>=<value>

OR on the command line like example:

-e <VARIABLE>=<value>

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables

ParameterDescriptionTypeDefault
ACTIONAction can be one of the followingenumnode_stop_start_scenario
LABEL_SELECTORNode label to targetstringnode-role.kubernetes.io/worker
EXCLUDE_LABELNodes labeled with this value will be excluded from the chaosstring
NODE_NAMENode name to inject faults in case of targeting a specific node; Can set multiple node names separated by a commastring""
INSTANCE_COUNTTargeted instance count matching the label selectornumber1
RUNSIterations to perform action on a single nodenumber1
CLOUD_TYPECloud platform on top of which cluster is running, supported platforms - aws, vmware, ibmcloud, ibmcloudpower, bmenumaws
TIMEOUTDuration to wait for completion of node scenario injectionnumber180
DURATIONDuration to stop the node before running the start action - not supported for vmware and ibm cloud typenumber120
KUBE_CHECKConnect to the kubernetes api to see if the node gets to a certain state during the node scenario. Supported values: true, falseenumTrue
PARALLELRun action on label or node name in parallel or sequential. Supported values: true, falseenumFalse
DISABLE_SSL_VERIFICATIONDisable SSL verification, to avoid certificate errors. Supported values: true, falseenumFalse
VERIFY_SESSIONVerify the SSH session during node scenariosstringfalse
SKIP_OPENSHIFT_CHECKSSkip OpenShift-specific cluster checks (set to true for vanilla Kubernetes)stringfalse
BMC_USEROnly needed for Baremetal ( bm ) - IPMI/bmc usernamestring""
BMC_PASSWORDOnly needed for Baremetal ( bm ) - IPMI/bmc passwordstring""
BMC_ADDROnly needed for Baremetal ( bm ) - IPMI/bmc addressstring""
DISKSComma-separated list of disks for baremetal disk detach/attach scenariosstring""

For example:

$ podman run \
  --name=<container_name> \
  --net=host \
  --env-host=true \
  -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml \
  -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:node-scenarios

The following environment variables need to be set for the scenarios that requires interacting with the cloud platform API to perform the actions:

Amazon Web Services

$ export AWS_ACCESS_KEY_ID=<>
$ export AWS_SECRET_ACCESS_KEY=<>
$ export AWS_DEFAULT_REGION=<>

VMware Vsphere

$ export VSPHERE_IP=<vSphere_client_IP_address>

$ export VSPHERE_USERNAME=<vSphere_client_username>

$ export VSPHERE_PASSWORD=<vSphere_client_password>

Ibmcloud

$ export IBMC_URL=https://<region>.iaas.cloud.ibm.com/v1

$ export IBMC_APIKEY=<ibmcloud_api_key>

Baremetal
Check Bare Metal Documentation

Google Cloud Platform

$ export GOOGLE_APPLICATION_CREDENTIALS=<GCP Json>

Azure

$ export AZURE_TENANT_ID=<>
$ export AZURE_CLIENT_SECRET=<>
$ export AZURE_CLIENT_ID=<>

OpenStack

export OS_USERNAME=username
export OS_PASSWORD=password
export OS_TENANT_NAME=projectName
export OS_AUTH_URL=https://identityHost:portNumber/v2.0
export OS_TENANT_ID=tenantIDString
export OS_REGION_NAME=regionName
export OS_CACERT=/path/to/cacertFile
krknctl run node-scenarios [--<parameter> <value>]

Can also set any global variable listed here

Scenario specific parameters: (be sure to scroll to right)

ParameterDescriptionTypeRequiredDefaultPossible Values
--actionaction performed on the node, visit https://github.com/krkn-chaos/krkn/blob/main/docs/node_scenarios.md for more infosenumYesnode_start_scenario,node_stop_scenario,node_stop_start_scenario,node_termination_scenario,node_reboot_scenario,stop_kubelet_scenario,stop_start_kubelet_scenario,restart_kubelet_scenario,node_crash_scenario,stop_start_helper_node_scenario
--label-selectorNode label to targetstringNonode-role.kubernetes.io/worker
--exclude-labelexcludes nodes marked by this label from chaosstringNo
--node-nameNode name to inject faults in case of targeting a specific node; Can set multiple node names separated by a commastringNo
--instance-countTargeted instance count matching the label selectornumberNo1
--runsIterations to perform action on a single nodenumberNo1
--cloud-typeCloud platform on top of which cluster is running, supported platforms - aws, azure, gcp, vmware, ibmcloud, bmenumNoaws
--kube-checkConnecting to the kubernetes api to check the node status, set to False for SNOenumNotrue
--timeoutDuration to wait for completion of node scenario injectionnumberNo180
--durationDuration to wait for completion of node scenario injectionnumberNo120
--vsphere-ipvSphere IP addressstringNo
--vsphere-usernamevSphere IP addressstring (secret)No
--vsphere-passwordvSphere passwordstring (secret)No
--aws-access-key-idAWS Access Key Idstring (secret)No
--aws-secret-access-keyAWS Secret Access Keystring (secret)No
--aws-default-regionAWS default regionstringNo
--bmc-userOnly needed for Baremetal ( bm ) - IPMI/bmc usernamestring(secret)No
--bmc-passwordOnly needed for Baremetal ( bm ) - IPMI/bmc passwordstring(secret)No
--bmc-addressOnly needed for Baremetal ( bm ) - IPMI/bmc addressstringNo
--ibmc-addressIBM Cloud URLstringNo
--ibmc-api-keyIBM Cloud API Keystring (secret)No
--ibmc-power-addressIBM Power Cloud URLstringNo
--ibmc-cnrIBM Cloud Power Workspace CNRstringNo
--disable-ssl-verificationDisable SSL verification, to avoid certificate errorsenumYesfalse
--azure-tenantAzure TenantstringNo
--azure-client-secretAzure Client Secretstring(secret)No
--azure-client-idAzure Client IDstring(secret)No
--azure-subscription-idAzure Subscription IDstring (secret)No
--gcp-application-credentialsGCP application credentials file locationfileNo

NOTE: The secret string types will be masked when scenario is ran

Parameter Dependencies

  • --node-name vs --label-selector: When --node-name is set, --label-selector is ignored. The scenario targets the named node(s) directly.
  • --instance-count: Only applies when using --label-selector. It limits how many of the matched nodes are targeted.
  • Cloud credentials: The --vsphere-*, --aws-*, --bmc-*, --ibmc-*, --azure-*, and --gcp-* parameters are only required for their respective --cloud-type value. For example, --aws-access-key-id is only needed when --cloud-type is aws.

To see all available scenario options

krknctl run node-scenarios --help

Demo

See a demo of this scenario:

1 - Node Scenarios on Bare Metal

Disrupts node(s) on a bare metal Kubernetes/OpenShift cluster by driving power state through the host's BMC (IPMI). Unlike the cloud-provider node scenarios, this flow requires IPMI credentials (either default or per-machine) and the OpenShift `oc` CLI on the runner host. Supported actions are inherited from the parent [Node Scenarios](../_index.md) page (start, stop, stop_start, terminate, reboot, kubelet stop/restart, disk detach/attach, and so on).

How to Run Node Scenarios on Bare Metal

Choose your preferred method to run baremetal node scenarios:

Example scenario file: baremetal_node_scenarios.yml

Configuration

For baremetal, set cloud_type: bm and provide IPMI credentials either at the root of the scenario entry (bmc_user / bmc_password) or per-machine inside bmc_info. If bmc_addr is omitted, Krkn falls back to the BMC value found on the matching BareMetalHost (oc get bmh -o wide --all-namespaces).

node_scenarios:
  - actions:
      - node_stop_start_scenario           # any action listed on the parent Node Scenarios page
    label_selector: node-role.kubernetes.io/worker
    instance_count: 1
    runs: 1
    timeout: 360
    duration: 120
    parallel: false
    cloud_type: bm
    kube_check: true
    bmc_user: defaultuser                  # default IPMI user; optional if every machine sets its own
    bmc_password: defaultpass              # default IPMI password; optional if every machine sets its own
    bmc_info:                              # per-machine overrides (optional)
      node-1:
        bmc_addr: mgmt-machine1.example.com
      node-2:
        bmc_addr: mgmt-machine2.example.com
        bmc_user: user
        bmc_password: pass

For the full set of node-scenario fields shared with other cloud providers (actions, node_name, label_selector, instance_count, etc.) see the parent Node Scenarios page.

Baremetal-specific fields

  • cloud_type — must be bm.
  • bmc_user, bmc_password — default IPMI credentials. May also be supplied via environment variables (BMC_USER, BMC_PASSWORD) — Krkn falls back to env when the YAML keys are absent.
  • bmc_info — per-machine overrides keyed by node name. Each entry accepts bmc_addr, bmc_user, bmc_password, and (for node_disk_detach_attach_scenario) a disks list.
  • For node_disk_detach_attach_scenario, bmc_info.<node>.disks is required and bmc_addr is not used.

Disk detach / attach

node_scenarios:
  - actions:
      - node_disk_detach_attach_scenario
    node_name: node-1
    instance_count: 1
    runs: 1
    timeout: 360
    duration: 120
    parallel: false
    cloud_type: bm
    bmc_info:
      node-1:
        disks: ["sda", "sdb"]

Usage

Enable baremetal node scenarios by adding the YAML file under node_scenarios in your kraken config:

kraken:
    chaos_scenarios:
        - node_scenarios:
            - scenarios/openshift/baremetal_node_scenarios.yml

Run

python run_kraken.py --config config/config.yaml

Run

Unlike other krkn-hub scenarios, baremetal node scenarios require a base64-encoded scenario file rather than per-parameter env vars. Author your scenario locally following the scenario syntax, then pass it to the container via SCENARIO_BASE64.

If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED for the chaos injection container to auto-connect.

$ podman run --name=<container_name> --net=host --pull=always --env-host=true \
    -e SCENARIO_BASE64="$(base64 -w0 <scenario_file>)" \
    -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:node-scenarios-bm
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
$ docker run $(./get_docker_params.sh) --name=<container_name> --net=host --pull=always \
    -e SCENARIO_BASE64="$(base64 -w0 <scenario_file>)" \
    -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:node-scenarios-bm
OR
$ docker run -e SCENARIO_BASE64="$(base64 -w0 <scenario_file>)" \
    --net=host --pull=always -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:node-scenarios-bm
$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

TIP: Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container:

kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host --pull=always -e SCENARIO_BASE64="$(base64 -w0 <scenario_file>)" -v ~/kubeconfig:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:node-scenarios-bm

Supported parameters

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario-specific variables.

ParameterDescriptionTypeDefaultRequired
SCENARIO_BASE64Base64-encoded contents of a baremetal node scenario YAML (base64 -w0 baremetal_node_scenarios.yml)stringYes
KRKN_DEBUGWhen set to True, prints the decoded scenario and config files before running and enables --debug TrueboolFalseNo

The contents of SCENARIO_BASE64 are validated against the node-scenarios-bm JSON schema before Krkn starts — invalid scenarios fail fast with a schema error.

NOTE In case of using a custom metrics profile or alerts profile when CAPTURE_METRICS or ENABLE_ALERTS is enabled, mount the metrics/alerts files from the host under /home/krkn/kraken/config/metrics-aggregated.yaml and /home/krkn/kraken/config/alerts:

$ podman run --name=<container_name> --net=host --pull=always --env-host=true \
    -e SCENARIO_BASE64="$(base64 -w0 <scenario_file>)" \
    -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml \
    -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts \
    -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:node-scenarios-bm
krknctl run node-scenarios-bm --scenario-file-path <path-to-baremetal_node_scenarios.yml>

Can also set any global variable listed here.

Node Scenarios BM Parameters

ArgumentTypeDescriptionRequiredDefault
--scenario-file-pathfile_base64Absolute path to the baremetal node-scenarios YAML file. krknctl base64-encodes the file and supplies it as SCENARIO_BASE64 to the container.true

The scenario YAML must follow the baremetal node scenario schema. See the Krkn tab on this page for an annotated example and the list of supported actions.

Example

krknctl run node-scenarios-bm \
  --scenario-file-path ~/krkn/scenarios/openshift/baremetal_node_scenarios.yml

Demo

See a demo of this scenario: