PVC Scenario

Scenario to fill up a given PersistenVolumeClaim by creating a temp file on the PVC from a pod associated with it. The purpose of this scenario is to fill up a volume to understand faults caused by the application using this volume.

How to Run PVC Scenarios

Choose your preferred method to run PVC scenarios:

Sample scenario config
$$pvc_scenario: pvc_name: # Name of the target PVC. pod_name: # Name of the pod where the PVC is mounted. It will be ignored if the pvc_name is defined. namespace: # Namespace where the PVC is. fill_percentage: 50 # Target percentage to fill up the cluster. Value must be higher than current percentage. Valid values are between 0 and 99. duration: 60 # Duration in seconds for the fault.$$
Steps
  • Get the pod name where the PVC is mounted.
  • Get the volume name mounted in the container pod.
  • Get the container name where the PVC is mounted.
  • Get the mount path where the PVC is mounted in the pod.
  • Get the PVC capacity and current used capacity.
  • Calculate file size to fill the PVC to the target fill_percentage.
  • Connect to the pod.
  • Create a temp file kraken.tmp with random data on the mount path:
    • dd bs=1024 count=$file_size </dev/urandom > /mount_path/kraken.tmp
  • Wait for the duration time.
  • Remove the temp file created:
    • rm kraken.tmp

How to Use Plugin Name

Add the plugin name to the list of chaos_scenarios section in the config/config.yaml file

kraken:
    kubeconfig_path: ~/.kube/config                     # Path to kubeconfig
    .. 
    chaos_scenarios:
        - pvc_scenarios:
            - scenarios/<scenario_name>.yaml

Run

python run_kraken.py --config config/config.yaml

This scenario fills up a given PersistenVolumeClaim by creating a temp file on the PVC from a pod associated with it. The purpose of this scenario is to fill up a volume to understand faults cause by the application using this volume. For more information refer the following documentation.

Run

If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED environment variable for the chaos injection container to autoconnect.

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pvc-scenarios
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
$ docker run $(./get_docker_params.sh) \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pvc-scenarios
$ docker run \
  -e <VARIABLE>=<value> \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pvc-scenarios

$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

Supported parameters

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

Example if –env-host is used:

$$export =$$

OR on the command line like example:

$$-e = $$

If both PVC_NAME and POD_NAME are defined, POD_NAME value will be overridden from the Mounted By: value on PVC definition.

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables

ParameterDescriptionDefault
PVC_NAMETargeted PersistentVolumeClaim in the cluster (if null, POD_NAME is required)
POD_NAMETargeted pod in the cluster (if null, PVC_NAME is required)
NAMESPACETargeted namespace in the cluster (required)
FILL_PERCENTAGETargeted percentage to be filled up in the PVC50
DURATIONDuration in seconds with the PVC filled up60

For example:

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml \
  -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pvc-scenarios
krknctl run pvc-scenarios (optional: --<parameter>:<value> )

Can also set any global variable listed here

Scenario specific parameters:

ParameterDescriptionTypeDefault
--pvc-nameTargeted PersistentVolumeClaim in the cluster (if null, POD_NAME is required)string
--pod-nameTargeted pod in the cluster (if null, PVC_NAME is required)string
--namespaceTargeted namespace in the cluster (required)string
--fill-percentageTargeted percentage to be filled up in the PVCnumber50
--durationDuration to wait for completion of node scenario injectionnumber1200

To see all available scenario options

krknctl run pvc-scenarios --help