Storage I/O Throttle

This scenario throttles storage I/O on a PVC-backed volume used by a target pod. It limits read/write IOPS and bandwidth (bytes per second) using Linux cgroup controllers, allowing you to observe how your application behaves under degraded disk performance.

• cgroups v2: writes to io.max
• cgroups v1: writes to blkio.throttle.*_device

The plugin automatically detects the cgroup version, container runtime (CRI-O or containerd), and the block device backing the volume. No manual host configuration is required.

Why Storage Throttle Scenarios Are Important

Many production incidents stem from slow or saturated storage, not complete outages. Storage I/O throttling lets you simulate realistic degraded-disk conditions and answer critical questions before they happen in production:

• Database slowdowns: Will your database handle reduced disk throughput gracefully, or will queries time out and cascade into application-level failures?
• Write-ahead log (WAL) backpressure: If writes to a WAL-backed volume slow down, does your system queue correctly or lose data?
• Logging and monitoring pressure: When log volumes become slow, do your applications block on writes or degrade silently?
• Storage class validation: Verify that your provisioned IOPS and throughput limits match application requirements under real workload conditions.
• Pod eviction thresholds: Test whether Kubernetes eviction policies trigger correctly when I/O-bound pods consume disproportionate resources.

How It Works

The storage throttle scenario follows these steps:

1. Resolve the target pod — If pvc_name is provided, the plugin looks up which pod has the PVC mounted. Otherwise, it uses the pod_name directly.
2. Find the PVC volume mount — The plugin identifies the container and mount path for the PVC-backed volume inside the target pod. If mount_path is specified, only that specific mount is targeted.
3. Discover the block device — The block device major:minor is extracted from /proc/self/mountinfo inside the target container, which is reliable for CSI-provisioned volumes.
4. Deploy a privileged helper pod — A short-lived privileged pod is deployed on the same node as the target workload. It mounts the host root filesystem and uses chroot to access host cgroup files.
5. Detect the cgroup version — The helper pod checks whether the node uses cgroups v1 or v2.
6. Discover the cgroup path — The plugin finds the real host cgroup path for the target container by its container ID, excluding CRI-O conmon (container monitor) paths.
7. Apply I/O throttle — IOPS and/or bandwidth limits are written to the appropriate cgroup files.
8. Hold for duration — The throttle remains active for the configured duration, with progress logged every 30 seconds.
9. Remove throttle — Limits are reset to their default (unlimited) values.
10. Cleanup — The privileged helper pod is deleted.

Configuration Parameters

Parameter Dependencies

• pvc_name vs pod_name: At least one is required. If both are set, pvc_name takes precedence and the pod is auto-resolved from the PVC.
• throttle_type controls which limits apply:
  • iops — only read_iops and write_iops are applied
  • bandwidth — only read_bps and write_bps are applied
  • both — all four limits are applied simultaneously

Supported Unit Suffixes

Byte values (read_bps, write_bps) accept Kubernetes-style unit suffixes:

Duration values (duration) accept time suffixes:

Prerequisites

• The target PVC must be in Bound state and mounted to a running pod.
• The privileged helper image (quay.io/krkn-chaos/krkn:tools by default) must be pullable on worker nodes.
• The cluster must allow privileged pods (required for writing to host cgroup files).
• Supported container runtimes: CRI-O and containerd.
• Supported cgroup versions: v1 and v2.

How to Run Storage I/O Throttle Scenarios

Choose your preferred method to run storage I/O throttle scenarios:

• Krkn
• Krkn-hub
• Krknctl

storage_throttle_scenario:
  pvc_name: ""                    # Target PVC name. If set, pod_name is auto-resolved from PVC.
  pod_name: my-app-pod            # Target pod name. Ignored if pvc_name is set.
  namespace: default              # Namespace of the target PVC/pod (required)
  mount_path: ""                  # Specific mount path to throttle. If empty, first PVC mount is used.
  throttle_type: bandwidth        # "iops", "bandwidth", or "both"
  read_iops: 100                  # Max read IOPS (used when throttle_type is "iops" or "both")
  write_iops: 50                  # Max write IOPS (used when throttle_type is "iops" or "both")
  read_bps: 1Mi                   # Max read bytes/sec (used when throttle_type is "bandwidth" or "both")
  write_bps: 512Ki                # Max write bytes/sec (used when throttle_type is "bandwidth" or "both")
  duration: 1m                    # How long to hold the throttle (supports suffixes: 30s, 2m, 1h)
  # image: quay.io/krkn-chaos/krkn:tools  # (optional) override helper pod image

storage_throttle_scenario:
  pvc_name: my-data-pvc
  namespace: production
  throttle_type: bandwidth
  read_bps: 1Mi                   # 1 MiB/s read limit
  write_bps: 512Ki                # 512 KiB/s write limit
  duration: 2m

storage_throttle_scenario:
  pod_name: postgres-0
  namespace: database
  throttle_type: iops
  read_iops: 50
  write_iops: 25
  duration: 90s

storage_throttle_scenario:
  pvc_name: wal-pvc
  namespace: database
  throttle_type: both
  read_iops: 100
  write_iops: 50
  read_bps: 5Mi
  write_bps: 2Mi
  duration: 3m

kraken:
    kubeconfig_path: ~/.kube/config
    ..
    chaos_scenarios:
        - storage_throttle_scenarios:
            - scenarios/kube/storage_throttle.yaml

kraken:
    chaos_scenarios:
        - storage_throttle_scenarios:
            - scenarios/kube/storage_throttle_bandwidth.yaml
            - scenarios/kube/storage_throttle_iops.yaml

kraken:
    chaos_scenarios:
        - storage_throttle_scenarios:
            - scenarios/kube/storage_throttle.yaml
        - pvc_scenarios:
            - scenarios/kube/pvc_scenario.yaml
        - pod_disruption_scenarios:
            - scenarios/kube/pod-kill.yaml
        - storage_throttle_scenarios:  # Same type can appear multiple times
            - scenarios/kube/storage_throttle_iops.yaml

python run_kraken.py --config config/config.yaml

$ podman run --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -e PVC_NAME=<target_pvc_name> \
  -e NAMESPACE=<target_namespace> \
  -e THROTTLE_TYPE=bandwidth \
  -e READ_BPS=1Mi \
  -e WRITE_BPS=512Ki \
  -e DURATION=1m \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:storage-throttle

$ podman logs -f <container_name or container_id>

$ podman inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}"

$ docker run $(./get_docker_params.sh) \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -e PVC_NAME=<target_pvc_name> \
  -e NAMESPACE=<target_namespace> \
  -e THROTTLE_TYPE=bandwidth \
  -e READ_BPS=1Mi \
  -e WRITE_BPS=512Ki \
  -e DURATION=1m \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:storage-throttle

$ docker logs -f <container_name or container_id>

$ docker inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}"

kubectl config view --flatten > ~/kubeconfig && \
chmod 444 ~/kubeconfig && \
docker run $(./get_docker_params.sh) \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v ~/kubeconfig:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:storage-throttle

export <parameter_name>=<value>

-e <VARIABLE>=<value>

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml \
  -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:storage-throttle

krknctl run storage-throttle [--<parameter> <value>]

krknctl run storage-throttle --help