VMI Outage Scenario
This scenario enables the simulation of VM-level disruptions in clusters where KubeVirt or OpenShift Containerized Network Virtualization (CNV) is installed. It allows users to delete a Virtual Machine Instance (VMI) to simulate a VM crash and test recovery capabilities.
Table of Contents
- Purpose
- Prerequisites
- Parameters
- Expected Behavior
- Advanced Use Cases
- Recovery Strategies
- Limitations
- Troubleshooting
Purpose
The vmi_outage scenario deletes a specific KubeVirt Virtual Machine Instance (VMI) to simulate a VM crash or outage. This helps users:
- Test the resilience of applications running inside VMs
- Verify that VM monitoring and recovery mechanisms work as expected
- Validate high availability configurations for VM workloads
- Understand the impact of sudden VM failures on workloads and the overall system
Prerequisites
Before using this scenario, ensure the following:
- KubeVirt or OpenShift CNV is installed in your cluster
- The target VMI exists and is running in the specified namespace
- Your cluster credentials have sufficient permissions to delete and create VMIs
- For optional guest reachability MTTR (
wait_for_guest_ping): the Krkn ServiceAccount needspods/execcreate, and ICMP (default) needs a namespace that allowsNET_RAW(not OpenShift restricted-v2)
Parameters
See the parameter tables in the How to Run section below for the supported parameters and their defaults, which vary slightly by run method (Krkn-hub vs. Krknctl).
Expected Behavior
When executed, the scenario will:
- Validate that KubeVirt is installed and the target VMI exists
- Save the initial state of the VMI
- Optionally prepare an in-cluster helper probe pod (when
wait_for_guest_pingis enabled) - Delete the VMI
- Wait for the VMI to become running or hit the timeout
- Optionally wait until the guest answers ICMP ping (default) or TCP, and record
guest_reachable_time - Attempt to recover the VMI:
- If the VMI is managed by a VirtualMachine resource with runStrategy: Always, it will automatically recover
- If automatic recovery doesn’t occur, the plugin will manually recreate the VMI using the saved state
- Validate that the VMI is running again
Note
If the VM is managed by a VirtualMachine resource withrunStrategy: Always, KubeVirt will automatically try to recreate the VMI after deletion. In this case, the scenario will wait for this automatic recovery to complete.Guest reachability MTTR
KubeVirtRunning can mean the guest is still at UEFI. Set wait_for_guest_ping: true to measure time until the guest answers ICMP ping (default). Leave guest_ping_pod empty so Krkn creates an ephemeral helper with NET_RAW before chaos. Use guest_ping_method: tcp only when NET_RAW is unavailable.Validating VMI SSH Connection
While the VMI outage is running you can enable kube virt checks to check the ssh connection to a list of VMIs to test if an outage of one VMI effects any others become unready/unconnectable. See more details on how to enable these checks in kubevirt checks
Advanced Use Cases
Testing High Availability VM Configurations
This scenario is particularly useful for testing high availability configurations, such as:
- Clustered applications running across multiple VMs
- VMs with automatic restart policies
- Applications with cross-VM resilience mechanisms
Recovery Strategies
The plugin implements two recovery strategies:
Automated Recovery: If the VM is managed by a VirtualMachine resource with
runStrategy: Always, the plugin will wait for KubeVirt’s controller to automatically recreate the VMI.Manual Recovery: If automatic recovery doesn’t occur within the timeout period, the plugin will attempt to manually recreate the VMI using the saved state from before the deletion.
Recovery Time Metrics in Krkn Telemetry
Krkn tracks these recovery time metrics for each affected VMI:
vmi_rescheduling_time - The time (in seconds) that the Kubernetes cluster took to reschedule the VMI after it was deleted. This measures the cluster’s scheduling efficiency and includes the time from VMI deletion until the replacement VMI is scheduled on a node.
vmi_readiness_time - The time (in seconds) the VMI took to become ready after being scheduled. This measures VMI startup time, including container image pulls, VM boot process, and readiness probe success.
guest_reachable_time - When
wait_for_guest_pingis enabled, the time (in seconds) from VMI phase=Runninguntil the guest first answers ICMP ping (default) or TCP via the in-cluster helper. This is the gap after readiness, not wall-clock from deletion. Omitted when guest ping is disabled.total_recovery_time -
vmi_rescheduling_time+vmi_readiness_time+guest_reachable_time(guest term only whenwait_for_guest_pingis enabled).
These metrics appear in the telemetry output under VMIsStatus.recovered for successfully recovered VMIs. VMIs that fail to recover within the timeout period appear under VMIsStatus.unrecovered without timing data.
Example telemetry output (3.2 + 12.5 + 28.4 = 44.1):
{
"recovered": [
{
"vmi_name": "virt-launcher-fedora-vm-xyz",
"namespace": "default",
"vmi_rescheduling_time": 3.2,
"vmi_readiness_time": 12.5,
"guest_reachable_time": 28.4,
"total_recovery_time": 44.1
}
],
"unrecovered": []
}
Limitations
- The scenario currently supports deleting a single VMI at a time
- If VM spec changes during the outage window, the manual recovery may not reflect those changes
- The scenario doesn’t simulate partial VM failures (e.g., VM freezing) - only complete VM outage
Troubleshooting
If the scenario fails, check the following:
- Ensure KubeVirt/CNV is properly installed in your cluster
- Verify that the target VMI exists and is running
- Check that your credentials have sufficient permissions to delete and create VMIs
- Examine the logs for specific error messages
How to Run VMI Outage Scenarios
Choose your preferred method to run KubeVirt VM outage scenarios:
VMI Outage Scenario in Kraken
The vmi_outage scenario in Kraken enables users to simulate VM-level disruptions by deleting a Virtual Machine Instance (VMI) to test resilience and recovery capabilities.
Example scenario file: vmi-outage.yaml
Implementation
This scenario is implemented in Kraken’s core repository, with the following key functionality:
- Finding and validating the target VMI
- Deleting the VMI using the KubeVirt API
- Monitoring the recovery process
- Implementing fallback recovery if needed
Usage
You can use this scenario in your Kraken configuration file as follows:
scenarios:
- name: "vmi outage"
scenario: vmi_outage
parameters:
vm_name: <my-application-vm>
namespace: <vm-workloads>
timeout: 60
disable_auto_restart: false
kill_count: 3
# Optional: Running → first guest ICMP ping MTTR
wait_for_guest_ping: false
Detailed Parameters
| Parameter | Description | Required | Default | Example Values |
|---|---|---|---|---|
| vm_name | The name of the VMI to delete | Yes* | N/A | “database-vm”, “web-server-vm” |
| label_selector | Label selector to target VMIs; optional if vm_name is set | Yes* | N/A | “app=myvm” |
| namespace | The namespace where the VMI is located | No | “default” | “openshift-cnv”, “vm-workloads” |
| timeout | How long to wait (in seconds) for VMI to become running before attempting recovery | No | 60 | 30, 120, 300 |
| disable_auto_restart | When set to true, patches spec.running=False on the VM so it does not auto-restart after deletion. | No | false | true, false |
| kill_count | How many VMI’s to kill serially | No | 1 | 3 |
| wait_for_guest_ping | After VMI is Running, wait until the guest answers ICMP ping (default) and record guest_reachable_time | No | false | true, false |
| guest_ping_timeout | Seconds to wait for first guest ping success after VMI Running | No | 300 | 120, 300 |
| guest_ping_interval | Seconds between guest ping probes | No | 2 | 1, 2, 5 |
| guest_ping_method | Reachability probe: icmp (default; needs NET_RAW) or tcp | No | icmp | icmp, tcp |
| guest_ping_port | TCP port when guest_ping_method=tcp (ignored for icmp) | No | 22 | 22, 3389 |
| guest_ping_namespace | Namespace for the helper probe pod; empty defaults to the VMI namespace | No | scenario namespace | “default” |
| guest_ping_pod | Leave empty for icmp (ephemeral helper with NET_RAW). For tcp, optionally reuse CI/templates/guest-ping-diagnosis-pod.yaml | No | empty | krkn-guest-ping-diagnosis |
| guest_ping_image | Helper image for the ephemeral probe pod (must include ping/nc). Override for disconnected / mirrored registries | No | quay.io/krkn-chaos/krkn-hub-multiarch:workload-krkn-tools | registry.local/krkn-tools:latest |
| guest_ping_settle_seconds | Seconds to sleep after the helper probe pod is Ready, before deleting the VMI | No | 5 | 0, 5 |
* Either vm_name or label_selector is required.
Execution Flow
When executed, the scenario follows this process:
- Initialization: Validates KubeVirt is installed and configures the KubeVirt client
- VMI Validation: Checks if the target VMI exists and is in Running state
- State Preservation: Saves the initial state of the VMI
- Probe prep (optional): If
wait_for_guest_pingis true, creates or reuses an in-cluster helper pod and settles before chaos (create/pull stay off the MTTR path) - Chaos Injection: Deletes the VMI using the KubeVirt API
- Wait for Running: Waits for VMI to become running again, up to the timeout specified
- Guest reachability (optional): Polls ICMP (default) or TCP until the guest answers; sets
guest_reachable_time - Recovery Monitoring: Checks if the VMI is automatically restored
- Manual Recovery: If automatic recovery doesn’t occur, manually recreates the VMI
- Validation: Confirms the VMI is running correctly
Sample Configuration
Here’s an example configuration for using the vmi_outage scenario:
scenarios:
- name: "vmi outage test"
scenario: vmi_outage
parameters:
vm_name: my-vm
namespace: kubevirt
timeout: 60
disable_auto_restart: false
kill_count: 3
# Optional guest MTTR (off by default). ICMP needs NET_RAW (not
# OpenShift restricted-v2). For restricted SCC use method: tcp.
# wait_for_guest_ping: true
# guest_ping_method: icmp
# guest_ping_timeout: 300
For multiple VMs in different namespaces:
scenarios:
- name: "kubevirt outage test - app VM"
scenario: vmi_outage
parameters:
vm_name: app-vm
namespace: application
timeout: 120
disable_auto_restart: false
kill_count: 1
- name: "kubevirt outage test - database VM"
scenario: vmi_outage
parameters:
vm_name: db-vm
namespace: database
timeout: 180
disable_auto_restart: false
kill_count: 2
Combining with Other Scenarios
For more comprehensive testing, you can combine this scenario with other Kraken scenarios in the list of chaos_scenarios in the config file:
kraken:
kubeconfig_path: ~/.kube/config # Path to kubeconfig
...
chaos_scenarios:
- hog_scenarios:
- scenarios/kube/cpu-hog.yml
- vmi_outage:
- scenarios/kubevirt/vmi-outage.yaml
Note
You can specify multiple scenario files of the same type by adding additional paths to the list:
kraken:
chaos_scenarios:
- vmi_outage:
- scenarios/kubevirt/vmi-outage-1.yaml
- scenarios/kubevirt/vmi-outage-2.yaml
- scenarios/kubevirt/vmi-outage-3.yaml
You can also combine multiple different scenario types in the same config.yaml file. Scenario types can be specified in any order, and you can include the same scenario type multiple times:
kraken:
chaos_scenarios:
- vmi_outage:
- scenarios/kubevirt/vmi-outage.yaml
- pod_disruption_scenarios:
- scenarios/pod-kill.yaml
- node_scenarios:
- scenarios/node-reboot.yaml
- vmi_outage: # Same type can appear multiple times
- scenarios/kubevirt/vmi-outage-2.yaml
Run
python run_kraken.py --config config/config.yaml
This scenario deletes a VMI matching the namespace and name on a Kubernetes/OpenShift cluster.
Run
If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED environment variable for the chaos injection container to autoconnect.
$ podman run \
--name=<container_name> \
--net=host \
--pull=always \
--env-host=true \
-v <path-to-kube-config>:/home/krkn/.kube/config:Z \
-d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:vmi-outage
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> \
--format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
Note
–env-host: This option is not available with the remote Podman client, including Mac and Windows (excluding WSL2) machines. Without the –env-host option you’ll have to set each environment variable on the podman command line like-e <VARIABLE>=<value>$ docker run $(./get_docker_params.sh) \
--name=<container_name> \
--net=host \
--pull=always \
-v <path-to-kube-config>:/home/krkn/.kube/config:Z \
-d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:vmi-outage
$ docker run \
-e <VARIABLE>=<value> \
--net=host \
--pull=always \
-v <path-to-kube-config>:/home/krkn/.kube/config:Z \
-d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:vmi-outage
$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> \
--format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
Tip
Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands:kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host --pull=always -v ~/kubeconfig:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:<scenario>Supported parameters
The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:
Example if –env-host is used:
export <parameter_name>=<value>
OR on the command line like example:
-e <VARIABLE>=<value>
See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables
| Parameter | Description | Type | Required | Default |
|---|---|---|---|---|
| NAMESPACE | Targeted namespace in the cluster | string | Yes | default |
| VM_NAME | Name of the vm to delete. Optional if label-selector is set | string | No | "" |
| LABEL_SELECTOR | Label selector to target VMs (e.g. app=myvm). Optional if vm-name is set | string | No | "" |
| TIMEOUT | Time that scneario will wait for VM to come back | number | No | 60 |
| KILL_COUNT | Number of VMIs to kill (performed serially) | number | No | 1 |
| WAIT_FOR_GUEST_PING | After VMI is Running, wait until guest answers ICMP ping (default) or TCP and record guest_reachable_time MTTR | boolean | No | False |
| GUEST_PING_TIMEOUT | Seconds to wait for first guest ping/TCP success after VMI Running | number | No | 300 |
| GUEST_PING_INTERVAL | Seconds between guest ping/TCP probes | number | No | 2 |
| GUEST_PING_METHOD | Reachability probe: icmp (default system ping; needs NET_RAW) or tcp | string | No | icmp |
| GUEST_PING_PORT | TCP port used when guest-ping-method=tcp (ignored for icmp) | number | No | 22 |
| GUEST_PING_NAMESPACE | Namespace for the probe pod; empty defaults to the VMI namespace | string | No | "" |
| GUEST_PING_POD | Leave empty for icmp (ephemeral helper with NET_RAW). For tcp, optionally reuse CI/templates/guest-ping-diagnosis-pod.yaml | string | No | "" |
| GUEST_PING_IMAGE | Container image for the ephemeral guest probe helper (must include ping/nc). Override for disconnected or mirrored registries | string | No | quay.io/krkn-chaos/krkn-hub-multiarch:workload-krkn-tools |
| GUEST_PING_SETTLE_SECONDS | Seconds to sleep after the helper probe pod is Ready, before deleting the VMI | number | No | 5 |
| SCENARIO_TYPE | Plugin key krkn dispatches the scenario on. Fixed by the container image, not a knob to change. | - | No | vmi_outage |
| SCENARIO_FILE | Path to the scenario file baked into the container image. Fixed by the image, not a knob to change. | - | No | scenarios/vmi_scenario.yaml |
Guest reachability MTTR (optional)
SetWAIT_FOR_GUEST_PING=True to keep polling after the VMI is Running until the guest answers an ICMP ping (default) and record guest_reachable_time. Leave GUEST_PING_POD empty so Krkn creates an ephemeral helper with NET_RAW before chaos (create/pull stay off the MTTR path). The namespace must allow NET_RAW (not OpenShift restricted-v2). For restricted SCC, set GUEST_PING_METHOD=tcp and optionally reuse CI/templates/guest-ping-diagnosis-pod.yaml from krkn via GUEST_PING_POD. The Krkn ServiceAccount needs pods/exec create.Note
In case of using custom metrics profile or alerts profile whenCAPTURE_METRICS or ENABLE_ALERTS is enabled, mount the metrics profile from the host on which the container is run using podman/docker under /home/krkn/kraken/config/metrics-aggregated.yaml and /home/krkn/kraken/config/alerts.$ podman run \
--name=<container_name> \
--net=host \
--pull=always \
--env-host=true \
-v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml \
-v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts \
-v <path-to-kube-config>:/home/krkn/.kube/config:Z \
-d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:vmi-outage
krknctl run vmi-outage [--<parameter> <value>]
Can also set any global variable listed here
Scenario specific parameters: (be sure to scroll to right)
| Parameter | Description | Type | Required | Default |
|---|---|---|---|---|
| --namespace | Targeted namespace in the cluster | string | Yes | default |
| --vm-name | Name of the vm to delete. Optional if label-selector is set | string | No | "" |
| --label-selector | Label selector to target VMs (e.g. app=myvm). Optional if vm-name is set | string | No | "" |
| --timeout | Time that scneario will wait for VM to come back | number | No | 60 |
| --kill-count | Number of VMIs to kill (performed serially) | number | No | 1 |
| --wait-for-guest-ping | After VMI is Running, wait until guest answers ICMP ping (default) or TCP and record guest_reachable_time MTTR | boolean | No | False |
| --guest-ping-timeout | Seconds to wait for first guest ping/TCP success after VMI Running | number | No | 300 |
| --guest-ping-interval | Seconds between guest ping/TCP probes | number | No | 2 |
| --guest-ping-method | Reachability probe: icmp (default system ping; needs NET_RAW) or tcp | string | No | icmp |
| --guest-ping-port | TCP port used when guest-ping-method=tcp (ignored for icmp) | number | No | 22 |
| --guest-ping-namespace | Namespace for the probe pod; empty defaults to the VMI namespace | string | No | "" |
| --guest-ping-pod | Leave empty for icmp (ephemeral helper with NET_RAW). For tcp, optionally reuse CI/templates/guest-ping-diagnosis-pod.yaml | string | No | "" |
| --guest-ping-image | Container image for the ephemeral guest probe helper (must include ping/nc). Override for disconnected or mirrored registries | string | No | quay.io/krkn-chaos/krkn-hub-multiarch:workload-krkn-tools |
| --guest-ping-settle-seconds | Seconds to sleep after the helper probe pod is Ready, before deleting the VMI | number | No | 5 |
Behavior Notes
- VM recovery: After krkn deletes the VM, the KubeVirt controller automatically recreates the VMI unless
runStrategyis set toManual. The--timeoutparameter controls how long krkn waits for the VM to come back before reporting failure. - Guest reachability MTTR (optional):
--wait-for-guest-pingpolls after VMIRunninguntil the guest answers ICMP ping (default) and recordsguest_reachable_time. Leave--guest-ping-podempty for the ephemeralNET_RAWhelper. Use--guest-ping-method tcponly whenNET_RAWis unavailable.
To see all available scenario options
krknctl run vmi-outage --help