This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Health Check Features

Advanced health check capabilities in Krkn

Krkn provides comprehensive health check capabilities to monitor your applications and infrastructure during chaos testing.

Health Check Types

Advanced Features

Quick Reference

Run Timing Options 

health_checks:
    run_during: "pre"                    # Before chaos only
    run_during: "during"                 # During chaos only (default)
    run_during: "post"                   # After chaos only
    run_during: ["pre", "post"]          # Before and after (not during)
    run_during: ["pre", "during", "post"]  # All three stages

Health Check Types in Config 

# HTTP endpoints
health_checks:
    interval: 5
    run_during: "during"
    config:
        - url: "http://my-app/health"

# VMI SSH connectivity
kubevirt_checks:
    interval: 10
    run_during: "during"
    namespace: "vms"

# Kubernetes object conditions
object_state_checks:
    interval: 5
    run_during: "during"
    config:
        - name: "etcd-ready"
          kind: "Pod"
          object_name: "etcd-.*"
          namespace: "kube-system"
          condition: {type: "Ready", status: "True"}

1 - Health Check Run Timing (run_during)

Control when health checks execute: pre-chaos, during chaos, or post-chaos

Overview

Health checks in Krkn can run at three different timings using the run_during configuration:

  1. Pre-checks (run_during: "pre"): Run once before chaos scenarios to verify baseline health
  2. During checks (run_during: "during"): Run continuously during chaos to monitor degradation
  3. Post-checks (run_during: "post"): Run once after chaos scenarios to verify recovery

You can combine timings using a list: run_during: ["pre", "post"] to run the same health check at multiple points.

Configuration

Configure health checks with the run_during option in your config.yaml:

health_checks:
    interval: 2                      # For continuous checks (run_during: "during")
    run_during: "during"             # When to run: "pre", "during", "post", or ["pre", "post"]
    exit_on_failure: False           # If True: pre-checks block chaos, post-checks fail the run
    config:
        - url: "http://my-app.example.com/health"
          bearer_token: "my-token"

kubevirt_checks:
    interval: 2                      # For continuous checks (run_during: "during")
    run_during: "during"             # When to run: "pre", "during", "post", or ["pre", "post"]
    exit_on_failure: False           # If True: pre-checks block chaos, post-checks fail the run
    namespace: "my-namespace"
    name: "my-vmi-.*"                # Regex pattern for VMI names

object_state_checks:
    interval: 5
    run_during: "during"
    exit_on_failure: False
    config:
        - name: "etcd-pods-ready"
          kind: "Pod"
          object_name: "etcd-.*"
          namespace: "kube-system"
          condition: {type: "Ready", status: "True"}

Supported Values

Value Behavior
"pre" Run once before chaos scenarios
"during" Run continuously during chaos (default)
"post" Run once after chaos scenarios
["pre", "post"] Run at multiple timings (any combination)

Use Cases

Pre-Check Only: Ensure Clean Baseline

Verify your system is healthy before starting chaos:

health_checks:
    run_during: "pre"
    exit_on_failure: True
    config:
        - url: "http://my-app.example.com/health"

Post-Check Only: Verify Recovery

Verify your system has recovered after chaos:

health_checks:
    run_during: "post"
    exit_on_failure: True
    config:
        - url: "http://my-app.example.com/health"

Pre and Post: Baseline and Recovery

Run the same check before and after chaos:

health_checks:
    run_during: ["pre", "post"]
    exit_on_failure: True
    config:
        - url: "http://my-app.example.com/health"

Complete Monitoring: Pre, During, and Post

Monitor health at all stages:

health_checks:
    interval: 5
    run_during: ["pre", "during", "post"]
    exit_on_failure: True
    config:
        - url: "http://my-app.example.com/health"

Exit Codes

Code Meaning
0 Success
1 Post-scenario failure
2 Critical Prometheus alerts
3 Continuous health check failure (run_during: "during")
4 Pre or post health check failure (when exit_on_failure: True)

Backward Compatibility

If run_during is not specified, health checks default to "during":

# This (existing configs)
health_checks:
    interval: 5
    config:
        - url: "http://my-app/health"

# Is equivalent to
health_checks:
    interval: 5
    run_during: "during"  # Default
    config:
        - url: "http://my-app/health"

2 - Object State Health Checks

Monitor Kubernetes resource conditions during chaos testing

Overview

The Object State Health Check plugin allows you to monitor the state of any Kubernetes resource by checking conditions on those objects. This is useful for verifying that critical infrastructure components maintain their desired state during chaos testing.

Features

  • Flexible Resource Monitoring: Check any Kubernetes resource type (Pod, Deployment, StatefulSet, DaemonSet, etc.)
  • Condition-Based Checks: Monitor specific conditions like Ready, Available, Progressing
  • Regex Pattern Matching: Use regex patterns to match multiple objects with one check
  • Label Selector Support: Filter objects by labels
  • Run Timing Control: Use run_during to run checks at pre, during, or post chaos stages
  • All Must Pass: When multiple objects match, ALL must be healthy for the check to pass

Configuration

Basic Structure 

object_state_checks:
    interval: 5                              # Check interval for continuous monitoring
    run_during: "during"                     # When to run: "pre", "during", "post", or ["pre", "post"]
    exit_on_failure: False                   # Fail on unhealthy objects
    config:
        - name: "check-name"                 # Descriptive name
          kind: "Pod"                        # Resource kind
          object_name: "my-pod-.*"          # Name or regex pattern
          namespace: "default"               # Namespace
          label_selector: ""                 # Optional label selector
          condition:
              type: "Ready"                  # Condition type
              status: "True"                 # Expected status

Examples

Check Pod Readiness

Verify that all etcd pods are Ready:

object_state_checks:
    interval: 5
    run_during: ["pre", "during", "post"]
    exit_on_failure: True
    config:
        - name: "etcd-pods-ready"
          kind: "Pod"
          object_name: "etcd-.*"
          namespace: "kube-system"
          condition:
              type: "Ready"
              status: "True"

Check Deployment Availability

Verify a deployment is Available:

object_state_checks:
    interval: 10
    run_during: "post"
    exit_on_failure: True
    config:
        - name: "myapp-deployment-available"
          kind: "Deployment"
          object_name: "myapp"
          namespace: "default"
          condition:
              type: "Available"
              status: "True"

Multiple Checks

Monitor different resource types:

object_state_checks:
    interval: 5
    run_during: ["pre", "during", "post"]
    exit_on_failure: True
    config:
        - name: "kube-apiserver-ready"
          kind: "Pod"
          object_name: "kube-apiserver-.*"
          namespace: "kube-system"
          condition: {type: "Ready", status: "True"}

        - name: "etcd-ready"
          kind: "Pod"
          object_name: "etcd-.*"
          namespace: "kube-system"
          condition: {type: "Ready", status: "True"}

        - name: "app-deployment-available"
          kind: "Deployment"
          object_name: "my-app"
          namespace: "production"
          condition: {type: "Available", status: "True"}

Multiple Object Behavior

IMPORTANT: When multiple objects match the pattern/labels, the health check requires ALL of them to pass.

How It Works 

- name: "etcd-pods-ready"
  kind: "Pod"
  object_name: "etcd-.*"           # Matches: etcd-0, etcd-1, etcd-2
  namespace: "kube-system"
  condition: {type: "Ready", status: "True"}

Behavior:

  • ✅ Check PASSES if: etcd-0 Ready=True AND etcd-1 Ready=True AND etcd-2 Ready=True
  • ❌ Check FAILS if: ANY pod has Ready=False (e.g., etcd-1 is not ready)

Common Condition Types

Pods

Condition Type Status Meaning
Ready True All containers are ready
PodScheduled True Pod has been scheduled to a node
Initialized True Init containers have completed
ContainersReady True All containers are ready

Deployments

Condition Type Status Meaning
Available True Minimum availability requirements met
Progressing True Deployment is progressing

StatefulSets / DaemonSets

Condition Type Status Meaning
Ready True All replicas/pods are ready
Available True Resource is available

Pattern Matching

Exact Match 

object_name: "my-pod"  # Matches only "my-pod"

Regex Pattern 

object_name: "etcd-.*"         # Matches etcd-0, etcd-1, etcd-2, etc.
object_name: ".*-worker-.*"    # Matches any name containing "-worker-"
object_name: "app-[0-9]+"      # Matches app-1, app-2, app-123, etc.

Label Selectors 

kind: "Pod"
namespace: "production"
label_selector: "app=myapp,version=v2"
# Checks only pods with both labels

Run Timing Examples

Pre-Check: Gate Chaos on Healthy State 

object_state_checks:
    run_during: "pre"
    exit_on_failure: True
    config:
        - name: "control-plane-healthy"
          kind: "Pod"
          namespace: "kube-system"
          label_selector: "tier=control-plane"
          condition: {type: "Ready", status: "True"}

During: Continuous Monitoring 

object_state_checks:
    interval: 5
    run_during: "during"
    exit_on_failure: False
    config:
        - name: "app-pods-status"
          kind: "Pod"
          namespace: "production"
          label_selector: "app=myapp"
          condition: {type: "Ready", status: "True"}

Post-Check: Verify Recovery 

object_state_checks:
    run_during: "post"
    exit_on_failure: True
    config:
        - name: "all-deployments-available"
          kind: "Deployment"
          namespace: "production"
          condition: {type: "Available", status: "True"}

Supported Resource Kinds

  • Pod
  • Deployment
  • StatefulSet
  • DaemonSet
  • ReplicaSet

See Also