Node Scenarios using Krkn-Hub

This scenario disrupts the node(s) matching the label on a Kubernetes/OpenShift cluster. Actions/disruptions supported are listed here

Run

If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED environment variable for the chaos injection container to autoconnect.

$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:node-scenarios
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
$ docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:node-scenarios
OR 
$ docker run -e <VARIABLE>=<value> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:node-scenarios

$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

Supported parameters

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

Example if –env-host is used:

export <parameter_name>=<value>

OR on the command line like example:

-e <VARIABLE>=<value> 

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables

ParameterDescriptionDefault
ACTIONAction can be one of the followingnode_stop_start_scenario
LABEL_SELECTORNode label to targetnode-role.kubernetes.io/worker
NODE_NAMENode name to inject faults in case of targeting a specific node; Can set multiple node names separated by a comma""
INSTANCE_COUNTTargeted instance count matching the label selector1
RUNSIterations to perform action on a single node1
CLOUD_TYPECloud platform on top of which cluster is running, supported platforms - aws, vmware, ibmcloud, bmaws
TIMEOUTDuration to wait for completion of node scenario injection180
DURATIONDuration to stop the node before running the start action - not supported for vmware and ibm cloud type120
«««< HEAD
KUBE_CHECKConnect to the kubernetes api to see if the node gets to a certain state during the node scenarioTrue

======= KUBE_CHECK | Connect to the kubernetes api to see if the node gets to a certain state during the node scenario | False | PARALLEL | Run action on label or node name in parallel or sequential, set to true for parallel | False |

9d89bb5 (Expose parallel parameter for node scenarios) BMC_USER | Only needed for Baremetal ( bm ) - IPMI/bmc username | "" | BMC_PASSWORD | Only needed for Baremetal ( bm ) - IPMI/bmc password | "" | BMC_ADDR | Only needed for Baremetal ( bm ) - IPMI/bmc username | "" |

For example:

$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:container-scenarios 

The following environment variables need to be set for the scenarios that requires intereacting with the cloud platform API to perform the actions:

Amazon Web Services

$ export AWS_ACCESS_KEY_ID=<>
$ export AWS_SECRET_ACCESS_KEY=<>
$ export AWS_DEFAULT_REGION=<>

VMware Vsphere

$ export VSPHERE_IP=<vSphere_client_IP_address>

$ export VSPHERE_USERNAME=<vSphere_client_username>

$ export VSPHERE_PASSWORD=<vSphere_client_password>

Ibmcloud

$ export IBMC_URL=https://<region>.iaas.cloud.ibm.com/v1

$ export IBMC_APIKEY=<ibmcloud_api_key>

Baremetal
Check Bare Metal Documentation

Google Cloud Platform

$ export GOOGLE_APPLICATION_CREDENTIALS=<GCP Json>

Azure

$ export AZURE_TENANT_ID=<>
$ export AZURE_CLIENT_SECRET=<>
$ export AZURE_CLIENT_ID=<>

OpenStack

export OS_USERNAME=username
export OS_PASSWORD=password
export OS_TENANT_NAME=projectName
export OS_AUTH_URL=https://identityHost:portNumber/v2.0
export OS_TENANT_ID=tenantIDString
export OS_REGION_NAME=regionName
export OS_CACERT=/path/to/cacertFile

Demo

See a demo of this scenario: