Node Scenarios on Bare Metal

Disrupts node(s) on a bare metal Kubernetes/OpenShift cluster by driving power state through the host's BMC (IPMI). Unlike the cloud-provider node scenarios, this flow requires IPMI credentials (either default or per-machine) and the OpenShift `oc` CLI on the runner host. Supported actions are inherited from the parent [Node Scenarios](../_index.md) page (start, stop, stop_start, terminate, reboot, kubelet stop/restart, disk detach/attach, and so on).

How to Run Node Scenarios on Bare Metal

Choose your preferred method to run baremetal node scenarios:

Example scenario file: baremetal_node_scenarios.yml

Configuration

For baremetal, set cloud_type: bm and provide IPMI credentials either at the root of the scenario entry (bmc_user / bmc_password) or per-machine inside bmc_info. If bmc_addr is omitted, Krkn falls back to the BMC value found on the matching BareMetalHost (oc get bmh -o wide --all-namespaces).

node_scenarios:
  - actions:
      - node_stop_start_scenario           # any action listed on the parent Node Scenarios page
    label_selector: node-role.kubernetes.io/worker
    instance_count: 1
    runs: 1
    timeout: 360
    duration: 120
    parallel: false
    cloud_type: bm
    kube_check: true
    bmc_user: defaultuser                  # default IPMI user; optional if every machine sets its own
    bmc_password: defaultpass              # default IPMI password; optional if every machine sets its own
    bmc_info:                              # per-machine overrides (optional)
      node-1:
        bmc_addr: mgmt-machine1.example.com
      node-2:
        bmc_addr: mgmt-machine2.example.com
        bmc_user: user
        bmc_password: pass

For the full set of node-scenario fields shared with other cloud providers (actions, node_name, label_selector, instance_count, etc.) see the parent Node Scenarios page.

Baremetal-specific fields

cloud_type — must be bm.
bmc_user, bmc_password — default IPMI credentials. May also be supplied via environment variables (BMC_USER, BMC_PASSWORD) — Krkn falls back to env when the YAML keys are absent.
bmc_info — per-machine overrides keyed by node name. Each entry accepts bmc_addr, bmc_user, bmc_password, and (for node_disk_detach_attach_scenario) a disks list.
For node_disk_detach_attach_scenario, bmc_info.<node>.disks is required and bmc_addr is not used.

Disk detach / attach

node_scenarios:
  - actions:
      - node_disk_detach_attach_scenario
    node_name: node-1
    instance_count: 1
    runs: 1
    timeout: 360
    duration: 120
    parallel: false
    cloud_type: bm
    bmc_info:
      node-1:
        disks: ["sda", "sdb"]

Usage

Enable baremetal node scenarios by adding the YAML file under node_scenarios in your kraken config:

kraken:
    chaos_scenarios:
        - node_scenarios:
            - scenarios/openshift/baremetal_node_scenarios.yml

Note

Baremetal requires oc (OpenShift client) installed on the host running Krkn. Some node actions can occasionally corrupt the filesystem if the node does not shut down cleanly — keep recovery procedures handy.

Run

python run_kraken.py --config config/config.yaml

Run

Unlike other krkn-hub scenarios, baremetal node scenarios require a base64-encoded scenario file rather than per-parameter env vars. Author your scenario locally following the scenario syntax, then pass it to the container via SCENARIO_BASE64.

If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED for the chaos injection container to auto-connect.

$ podman run --name=<container_name> --net=host --pull=always --env-host=true \
    -e SCENARIO_BASE64="$(base64 -w0 <scenario_file>)" \
    -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-scenarios-bm
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

$ docker run $(./get_docker_params.sh) --name=<container_name> --net=host --pull=always \
    -e SCENARIO_BASE64="$(base64 -w0 <scenario_file>)" \
    -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-scenarios-bm
OR
$ docker run -e SCENARIO_BASE64="$(base64 -w0 <scenario_file>)" \
    --net=host --pull=always -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-scenarios-bm
$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

TIP: Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container:

kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host --pull=always -e SCENARIO_BASE64="$(base64 -w0 <scenario_file>)" -v ~/kubeconfig:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-scenarios-bm

Supported parameters

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario-specific variables.

Parameter	Description	Type	Default	Required
SCENARIO_BASE64	Base64-encoded contents of a baremetal node scenario YAML (`base64 -w0 baremetal_node_scenarios.yml`)	string		Yes
KRKN_DEBUG	When set to `True`, prints the decoded scenario and config files before running and enables `--debug True`	bool	`False`	No

The contents of SCENARIO_BASE64 are validated against the node-scenarios-bm JSON schema before Krkn starts — invalid scenarios fail fast with a schema error.

NOTE In case of using a custom metrics profile or alerts profile when CAPTURE_METRICS or ENABLE_ALERTS is enabled, mount the metrics/alerts files from the host under /home/krkn/kraken/config/metrics-aggregated.yaml and /home/krkn/kraken/config/alerts:

$ podman run --name=<container_name> --net=host --pull=always --env-host=true \
    -e SCENARIO_BASE64="$(base64 -w0 <scenario_file>)" \
    -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml \
    -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts \
    -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-scenarios-bm

krknctl run node-scenarios-bm --scenario-file-path <path-to-baremetal_node_scenarios.yml>

Can also set any global variable listed here.

Node Scenarios BM Parameters

Argument	Type	Description	Required	Default
`--scenario-file-path`	file_base64	Absolute path to the baremetal node-scenarios YAML file. krknctl base64-encodes the file and supplies it as `SCENARIO_BASE64` to the container.	true

The scenario YAML must follow the baremetal node scenario schema. See the Krkn tab on this page for an annotated example and the list of supported actions.

Example

krknctl run node-scenarios-bm \
  --scenario-file-path ~/krkn/scenarios/openshift/baremetal_node_scenarios.yml

Note

krknctl handles the base64 encoding for you — pass a plain filesystem path. The validation step inside the container (against config-schema.json) still applies, so invalid YAML is rejected before Krkn runs.

Demo

See a demo of this scenario: