Health Check Features on Krkn

Health Check Run Timing (run_during)

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Health checks in Krkn can run at three different timings using the run_during configuration:

Pre-checks (run_during: "pre"): Run once before chaos scenarios to verify baseline health
During checks (run_during: "during"): Run continuously during chaos to monitor degradation
Post-checks (run_during: "post"): Run once after chaos scenarios to verify recovery

You can combine timings using a list: run_during: ["pre", "post"] to run the same health check at multiple points.

Object State Health Checks

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The Object State Health Check plugin allows you to monitor the state of any Kubernetes resource by checking conditions on those objects. This is useful for verifying that critical infrastructure components maintain their desired state during chaos testing.

Features

Flexible Resource Monitoring: Check any Kubernetes resource type (Pod, Deployment, StatefulSet, DaemonSet, etc.)
Condition-Based Checks: Monitor specific conditions like Ready, Available, Progressing
Regex Pattern Matching: Use regex patterns to match multiple objects with one check
Label Selector Support: Filter objects by labels
Run Timing Control: Use run_during to run checks at pre, during, or post chaos stages
All Must Pass: When multiple objects match, ALL must be healthy for the check to pass

Configuration

Basic Structure

object_state_checks:
 interval: 5 # Check interval for continuous monitoring
 run_during: "during" # When to run: "pre", "during", "post", or ["pre", "post"]
 exit_on_failure: False # Fail on unhealthy objects
 config:
 - name: "check-name" # Descriptive name
 kind: "Pod" # Resource kind
 object_name: "my-pod-.*" # Name or regex pattern
 namespace: "default" # Namespace
 label_selector: "" # Optional label selector
 condition:
 type: "Ready" # Condition type
 status: "True" # Expected status

Examples

Check Pod Readiness

Verify that all etcd pods are Ready:

Prometheus Alert Health Checks

Mon, 01 Jan 0001 00:00:00 +0000

Prometheus alert health checks evaluate PromQL expressions from an alert profile during a Krkn run. They let you verify cluster SLOs and automatically identify health regressions caused by a chaos scenario.

Prerequisites

Prometheus must be reachable from the machine running Krkn.
On OpenShift, Krkn can discover the Prometheus route and bearer token. On Kubernetes, set prometheus_url and prometheus_bearer_token.
An alert profile must contain one or more PromQL expressions.

Configuration

Configure alert health checks under performance_monitoring:

Kube Virt Checks

Mon, 01 Jan 0001 00:00:00 +0000

Kube Virt Checks

Virt checks provide real-time visibility into the impact of chaos scenarios on VMI ssh connectivity and performance. Virt checks are configured in the config.yaml here

The system periodically checks the VMI’s in the provided namespace based on the defined interval and records the results in Telemetry. The checks will run continuously from the very beginning of krkn until all scenarios are done and wait durations are complete. The telemetry data includes: