PVC Scenario
How to Run PVC Scenarios
Choose your preferred method to run PVC scenarios:
Sample scenario config
Steps
- Get the pod name where the PVC is mounted.
- Get the volume name mounted in the container pod.
- Get the container name where the PVC is mounted.
- Get the mount path where the PVC is mounted in the pod.
- Get the PVC capacity and current used capacity.
- Calculate file size to fill the PVC to the target fill_percentage.
- Connect to the pod.
- Create a temp file
kraken.tmpwith random data on the mount path:dd bs=1024 count=$file_size </dev/urandom > /mount_path/kraken.tmp
- Wait for the duration time.
- Remove the temp file created:
rm kraken.tmp
How to Use Plugin Name
Add the plugin name to the list of chaos_scenarios section in the config/config.yaml file
kraken:
kubeconfig_path: ~/.kube/config # Path to kubeconfig
..
chaos_scenarios:
- pvc_scenarios:
- scenarios/<scenario_name>.yaml
Run
python run_kraken.py --config config/config.yaml
This scenario fills up a given PersistenVolumeClaim by creating a temp file on the PVC from a pod associated with it. The purpose of this scenario is to fill up a volume to understand faults cause by the application using this volume. For more information refer the following documentation.
Run
If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED environment variable for the chaos injection container to autoconnect.
$ podman run \
--name=<container_name> \
--net=host \
--pull=always \
--env-host=true \
-v <path-to-kube-config>:/home/krkn/.kube/config:Z \
-d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pvc-scenarios
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> \
--format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
Note
–env-host: This option is not available with the remote Podman client, including Mac and Windows (excluding WSL2) machines. Without the –env-host option you’ll have to set each environment variable on the podman command line like-e <VARIABLE>=<value>$ docker run $(./get_docker_params.sh) \
--name=<container_name> \
--net=host \
--pull=always \
-v <path-to-kube-config>:/home/krkn/.kube/config:Z \
-d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pvc-scenarios
$ docker run \
-e <VARIABLE>=<value> \
--name=<container_name> \
--net=host \
--pull=always \
-v <path-to-kube-config>:/home/krkn/.kube/config:Z \
-d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pvc-scenarios
$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> \
--format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
Tip
Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands:kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host --pull=always -v ~kubeconfig:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:<scenario>Supported parameters
The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:
Example if –env-host is used:
OR on the command line like example:
If both PVC_NAME and POD_NAME are defined, POD_NAME value will be overridden from the Mounted By: value on PVC definition.
See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables
| Parameter | Description | Default |
|---|---|---|
| PVC_NAME | Targeted PersistentVolumeClaim in the cluster (if null, POD_NAME is required) | |
| POD_NAME | Targeted pod in the cluster (if null, PVC_NAME is required) | |
| NAMESPACE | Targeted namespace in the cluster (required) | |
| FILL_PERCENTAGE | Targeted percentage to be filled up in the PVC | 50 |
| DURATION | Duration in seconds with the PVC filled up | 60 |
Note
Set NAMESPACE environment variable toopenshift-.* to pick and disrupt pods randomly in openshift system namespaces, the DAEMON_MODE can also be enabled to disrupt the pods every x seconds in the background to check the reliability.Note
In case of using custom metrics profile or alerts profile whenCAPTURE_METRICS or ENABLE_ALERTS is enabled, mount the metrics profile from the host on which the container is run using podman/docker under /home/krkn/kraken/config/metrics-aggregated.yaml and /home/krkn/kraken/config/alerts.$ podman run \
--name=<container_name> \
--net=host \
--pull=always \
--env-host=true \
-v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml \
-v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts \
-v <path-to-kube-config>:/home/krkn/.kube/config:Z \
-d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pvc-scenarios
krknctl run pvc-scenarios (optional: --<parameter>:<value> )
Can also set any global variable listed here
Scenario specific parameters:
| Parameter | Description | Type | Default |
|---|---|---|---|
--pvc-name | Targeted PersistentVolumeClaim in the cluster (if null, POD_NAME is required) | string | |
--pod-name | Targeted pod in the cluster (if null, PVC_NAME is required) | string | |
--namespace | Targeted namespace in the cluster (required) | string | |
--fill-percentage | Targeted percentage to be filled up in the PVC | number | 50 |
--duration | Duration to wait for completion of node scenario injection | number | 1200 |
To see all available scenario options
krknctl run pvc-scenarios --help