VMI Outage Scenario

This scenario enables the simulation of VM-level disruptions in clusters where KubeVirt or OpenShift Containerized Network Virtualization (CNV) is installed. It allows users to delete a Virtual Machine Instance (VMI) to simulate a VM crash and test recovery capabilities.

Table of Contents

Purpose

The vmi_outage scenario deletes a specific KubeVirt Virtual Machine Instance (VMI) to simulate a VM crash or outage. This helps users:

  • Test the resilience of applications running inside VMs
  • Verify that VM monitoring and recovery mechanisms work as expected
  • Validate high availability configurations for VM workloads
  • Understand the impact of sudden VM failures on workloads and the overall system

Prerequisites

Before using this scenario, ensure the following:

  1. KubeVirt or OpenShift CNV is installed in your cluster
  2. The target VMI exists and is running in the specified namespace
  3. Your cluster credentials have sufficient permissions to delete and create VMIs
  4. For optional guest reachability MTTR (wait_for_guest_ping): the Krkn ServiceAccount needs pods/exec create, and ICMP (default) needs a namespace that allows NET_RAW (not OpenShift restricted-v2)

Parameters

See the parameter tables in the How to Run section below for the supported parameters and their defaults, which vary slightly by run method (Krkn-hub vs. Krknctl).

Expected Behavior

When executed, the scenario will:

  1. Validate that KubeVirt is installed and the target VMI exists
  2. Save the initial state of the VMI
  3. Optionally prepare an in-cluster helper probe pod (when wait_for_guest_ping is enabled)
  4. Delete the VMI
  5. Wait for the VMI to become running or hit the timeout
  6. Optionally wait until the guest answers ICMP ping (default) or TCP, and record guest_reachable_time
  7. Attempt to recover the VMI:
    • If the VMI is managed by a VirtualMachine resource with runStrategy: Always, it will automatically recover
    • If automatic recovery doesn’t occur, the plugin will manually recreate the VMI using the saved state
  8. Validate that the VMI is running again

Validating VMI SSH Connection

While the VMI outage is running you can enable kube virt checks to check the ssh connection to a list of VMIs to test if an outage of one VMI effects any others become unready/unconnectable. See more details on how to enable these checks in kubevirt checks

Advanced Use Cases

Testing High Availability VM Configurations

This scenario is particularly useful for testing high availability configurations, such as:

  • Clustered applications running across multiple VMs
  • VMs with automatic restart policies
  • Applications with cross-VM resilience mechanisms

Recovery Strategies

The plugin implements two recovery strategies:

  1. Automated Recovery: If the VM is managed by a VirtualMachine resource with runStrategy: Always, the plugin will wait for KubeVirt’s controller to automatically recreate the VMI.

  2. Manual Recovery: If automatic recovery doesn’t occur within the timeout period, the plugin will attempt to manually recreate the VMI using the saved state from before the deletion.

Recovery Time Metrics in Krkn Telemetry

Krkn tracks these recovery time metrics for each affected VMI:

  1. vmi_rescheduling_time - The time (in seconds) that the Kubernetes cluster took to reschedule the VMI after it was deleted. This measures the cluster’s scheduling efficiency and includes the time from VMI deletion until the replacement VMI is scheduled on a node.

  2. vmi_readiness_time - The time (in seconds) the VMI took to become ready after being scheduled. This measures VMI startup time, including container image pulls, VM boot process, and readiness probe success.

  3. guest_reachable_time - When wait_for_guest_ping is enabled, the time (in seconds) from VMI phase=Running until the guest first answers ICMP ping (default) or TCP via the in-cluster helper. This is the gap after readiness, not wall-clock from deletion. Omitted when guest ping is disabled.

  4. total_recovery_time - vmi_rescheduling_time + vmi_readiness_time + guest_reachable_time (guest term only when wait_for_guest_ping is enabled).

These metrics appear in the telemetry output under VMIsStatus.recovered for successfully recovered VMIs. VMIs that fail to recover within the timeout period appear under VMIsStatus.unrecovered without timing data.

Example telemetry output (3.2 + 12.5 + 28.4 = 44.1):

{
  "recovered": [
    {
      "vmi_name": "virt-launcher-fedora-vm-xyz",
      "namespace": "default",
      "vmi_rescheduling_time": 3.2,
      "vmi_readiness_time": 12.5,
      "guest_reachable_time": 28.4,
      "total_recovery_time": 44.1
    }
  ],
  "unrecovered": []
}

Limitations

  • The scenario currently supports deleting a single VMI at a time
  • If VM spec changes during the outage window, the manual recovery may not reflect those changes
  • The scenario doesn’t simulate partial VM failures (e.g., VM freezing) - only complete VM outage

Troubleshooting

If the scenario fails, check the following:

  1. Ensure KubeVirt/CNV is properly installed in your cluster
  2. Verify that the target VMI exists and is running
  3. Check that your credentials have sufficient permissions to delete and create VMIs
  4. Examine the logs for specific error messages

How to Run VMI Outage Scenarios

Choose your preferred method to run KubeVirt VM outage scenarios:

VMI Outage Scenario in Kraken

The vmi_outage scenario in Kraken enables users to simulate VM-level disruptions by deleting a Virtual Machine Instance (VMI) to test resilience and recovery capabilities.

Example scenario file: vmi-outage.yaml

Implementation

This scenario is implemented in Kraken’s core repository, with the following key functionality:

  1. Finding and validating the target VMI
  2. Deleting the VMI using the KubeVirt API
  3. Monitoring the recovery process
  4. Implementing fallback recovery if needed

Usage

You can use this scenario in your Kraken configuration file as follows:

scenarios:
  - name: "vmi outage"
    scenario: vmi_outage
    parameters:
      vm_name: <my-application-vm>
      namespace: <vm-workloads>
      timeout: 60
      disable_auto_restart: false
      kill_count: 3
      # Optional: Running → first guest ICMP ping MTTR
      wait_for_guest_ping: false

Detailed Parameters

ParameterDescriptionRequiredDefaultExample Values
vm_nameThe name of the VMI to deleteYes*N/A“database-vm”, “web-server-vm”
label_selectorLabel selector to target VMIs; optional if vm_name is setYes*N/A“app=myvm”
namespaceThe namespace where the VMI is locatedNo“default”“openshift-cnv”, “vm-workloads”
timeoutHow long to wait (in seconds) for VMI to become running before attempting recoveryNo6030, 120, 300
disable_auto_restartWhen set to true, patches spec.running=False on the VM so it does not auto-restart after deletion.Nofalsetrue, false
kill_countHow many VMI’s to kill seriallyNo13
wait_for_guest_pingAfter VMI is Running, wait until the guest answers ICMP ping (default) and record guest_reachable_timeNofalsetrue, false
guest_ping_timeoutSeconds to wait for first guest ping success after VMI RunningNo300120, 300
guest_ping_intervalSeconds between guest ping probesNo21, 2, 5
guest_ping_methodReachability probe: icmp (default; needs NET_RAW) or tcpNoicmpicmp, tcp
guest_ping_portTCP port when guest_ping_method=tcp (ignored for icmp)No2222, 3389
guest_ping_namespaceNamespace for the helper probe pod; empty defaults to the VMI namespaceNoscenario namespace“default”
guest_ping_podLeave empty for icmp (ephemeral helper with NET_RAW). For tcp, optionally reuse CI/templates/guest-ping-diagnosis-pod.yamlNoemptykrkn-guest-ping-diagnosis
guest_ping_imageHelper image for the ephemeral probe pod (must include ping/nc). Override for disconnected / mirrored registriesNoquay.io/krkn-chaos/krkn-hub-multiarch:workload-krkn-toolsregistry.local/krkn-tools:latest
guest_ping_settle_secondsSeconds to sleep after the helper probe pod is Ready, before deleting the VMINo50, 5

* Either vm_name or label_selector is required.

Execution Flow

When executed, the scenario follows this process:

  1. Initialization: Validates KubeVirt is installed and configures the KubeVirt client
  2. VMI Validation: Checks if the target VMI exists and is in Running state
  3. State Preservation: Saves the initial state of the VMI
  4. Probe prep (optional): If wait_for_guest_ping is true, creates or reuses an in-cluster helper pod and settles before chaos (create/pull stay off the MTTR path)
  5. Chaos Injection: Deletes the VMI using the KubeVirt API
  6. Wait for Running: Waits for VMI to become running again, up to the timeout specified
  7. Guest reachability (optional): Polls ICMP (default) or TCP until the guest answers; sets guest_reachable_time
  8. Recovery Monitoring: Checks if the VMI is automatically restored
  9. Manual Recovery: If automatic recovery doesn’t occur, manually recreates the VMI
  10. Validation: Confirms the VMI is running correctly

Sample Configuration

Here’s an example configuration for using the vmi_outage scenario:

scenarios:
  - name: "vmi outage test"
    scenario: vmi_outage
    parameters:
      vm_name: my-vm
      namespace: kubevirt
      timeout: 60
      disable_auto_restart: false
      kill_count: 3
      # Optional guest MTTR (off by default). ICMP needs NET_RAW (not
      # OpenShift restricted-v2). For restricted SCC use method: tcp.
      # wait_for_guest_ping: true
      # guest_ping_method: icmp
      # guest_ping_timeout: 300

For multiple VMs in different namespaces:

scenarios:
  - name: "kubevirt outage test - app VM"
    scenario: vmi_outage
    parameters:
      vm_name: app-vm
      namespace: application
      timeout: 120
      disable_auto_restart: false
      kill_count: 1
  
  - name: "kubevirt outage test - database VM"
    scenario: vmi_outage
    parameters:
      vm_name: db-vm
      namespace: database
      timeout: 180
      disable_auto_restart: false
      kill_count: 2

Combining with Other Scenarios

For more comprehensive testing, you can combine this scenario with other Kraken scenarios in the list of chaos_scenarios in the config file:

kraken:
    kubeconfig_path: ~/.kube/config                     # Path to kubeconfig
    ...
    chaos_scenarios:
        - hog_scenarios:
            - scenarios/kube/cpu-hog.yml
        -  vmi_outage:
               - scenarios/kubevirt/vmi-outage.yaml

Run

python run_kraken.py --config config/config.yaml

This scenario deletes a VMI matching the namespace and name on a Kubernetes/OpenShift cluster.

Run

If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED environment variable for the chaos injection container to autoconnect.

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:vmi-outage
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
$ docker run $(./get_docker_params.sh) \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:vmi-outage
$ docker run \
  -e <VARIABLE>=<value> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:vmi-outage

$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

Supported parameters

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

Example if –env-host is used:

export <parameter_name>=<value>

OR on the command line like example:

-e <VARIABLE>=<value>

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables

ParameterDescriptionTypeRequiredDefault
NAMESPACETargeted namespace in the clusterstringYesdefault
VM_NAMEName of the vm to delete. Optional if label-selector is setstringNo""
LABEL_SELECTORLabel selector to target VMs (e.g. app=myvm). Optional if vm-name is setstringNo""
TIMEOUTTime that scneario will wait for VM to come backnumberNo60
KILL_COUNTNumber of VMIs to kill (performed serially)numberNo1
WAIT_FOR_GUEST_PINGAfter VMI is Running, wait until guest answers ICMP ping (default) or TCP and record guest_reachable_time MTTRbooleanNoFalse
GUEST_PING_TIMEOUTSeconds to wait for first guest ping/TCP success after VMI RunningnumberNo300
GUEST_PING_INTERVALSeconds between guest ping/TCP probesnumberNo2
GUEST_PING_METHODReachability probe: icmp (default system ping; needs NET_RAW) or tcpstringNoicmp
GUEST_PING_PORTTCP port used when guest-ping-method=tcp (ignored for icmp)numberNo22
GUEST_PING_NAMESPACENamespace for the probe pod; empty defaults to the VMI namespacestringNo""
GUEST_PING_PODLeave empty for icmp (ephemeral helper with NET_RAW). For tcp, optionally reuse CI/templates/guest-ping-diagnosis-pod.yamlstringNo""
GUEST_PING_IMAGEContainer image for the ephemeral guest probe helper (must include ping/nc). Override for disconnected or mirrored registriesstringNoquay.io/krkn-chaos/krkn-hub-multiarch:workload-krkn-tools
GUEST_PING_SETTLE_SECONDSSeconds to sleep after the helper probe pod is Ready, before deleting the VMInumberNo5
SCENARIO_TYPEPlugin key krkn dispatches the scenario on. Fixed by the container image, not a knob to change.-Novmi_outage
SCENARIO_FILEPath to the scenario file baked into the container image. Fixed by the image, not a knob to change.-Noscenarios/vmi_scenario.yaml

For example:

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml \
  -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:vmi-outage
krknctl run vmi-outage [--<parameter> <value>]

Can also set any global variable listed here

Scenario specific parameters: (be sure to scroll to right)

ParameterDescriptionTypeRequiredDefault
--namespaceTargeted namespace in the clusterstringYesdefault
--vm-nameName of the vm to delete. Optional if label-selector is setstringNo""
--label-selectorLabel selector to target VMs (e.g. app=myvm). Optional if vm-name is setstringNo""
--timeoutTime that scneario will wait for VM to come backnumberNo60
--kill-countNumber of VMIs to kill (performed serially)numberNo1
--wait-for-guest-pingAfter VMI is Running, wait until guest answers ICMP ping (default) or TCP and record guest_reachable_time MTTRbooleanNoFalse
--guest-ping-timeoutSeconds to wait for first guest ping/TCP success after VMI RunningnumberNo300
--guest-ping-intervalSeconds between guest ping/TCP probesnumberNo2
--guest-ping-methodReachability probe: icmp (default system ping; needs NET_RAW) or tcpstringNoicmp
--guest-ping-portTCP port used when guest-ping-method=tcp (ignored for icmp)numberNo22
--guest-ping-namespaceNamespace for the probe pod; empty defaults to the VMI namespacestringNo""
--guest-ping-podLeave empty for icmp (ephemeral helper with NET_RAW). For tcp, optionally reuse CI/templates/guest-ping-diagnosis-pod.yamlstringNo""
--guest-ping-imageContainer image for the ephemeral guest probe helper (must include ping/nc). Override for disconnected or mirrored registriesstringNoquay.io/krkn-chaos/krkn-hub-multiarch:workload-krkn-tools
--guest-ping-settle-secondsSeconds to sleep after the helper probe pod is Ready, before deleting the VMInumberNo5

Behavior Notes

  • VM recovery: After krkn deletes the VM, the KubeVirt controller automatically recreates the VMI unless runStrategy is set to Manual. The --timeout parameter controls how long krkn waits for the VM to come back before reporting failure.
  • Guest reachability MTTR (optional): --wait-for-guest-ping polls after VMI Running until the guest answers ICMP ping (default) and records guest_reachable_time. Leave --guest-ping-pod empty for the ephemeral NET_RAW helper. Use --guest-ping-method tcp only when NET_RAW is unavailable.

To see all available scenario options

krknctl run vmi-outage --help