GPU Device Plugin

Disrupt NVIDIA device plugin pods on GPU nodes and verify they recover

This scenario kills the NVIDIA device plugin pods on GPU nodes to simulate a device plugin crash, then verifies that the pods are rescheduled and become ready.

It runs as a Pod Scenario targeted at the device plugin pods, so you use the example config below with the tool of your choice.

Prerequisites

A cluster with GPU nodes.
The NVIDIA GPU Operator installed (device plugin runs in the nvidia-gpu-operator namespace by default).
GPU nodes labeled nvidia.com/gpu.present=true.

Note

Adjust the namespace and labels to match your install. Check with kubectl get pods -n nvidia-gpu-operator --show-labels.

How to Run

Choose your preferred method to run the GPU device plugin disruption scenario:

Example Config

Create the scenario file scenarios/kube/gpu_device_plugin.yaml:

- id: gpu-device-plugin-disruption
  config:
    namespace_pattern: ^nvidia-gpu-operator$                 # GPU Operator namespace
    label_selector: app=nvidia-device-plugin-daemonset       # device plugin pods
    node_label_selector: nvidia.com/gpu.present=true         # only target GPU nodes
    kill: 1                                                  # pods to delete
    krkn_pod_recovery_time: 120                              # seconds to wait for recovery

See Pod Scenarios for all available options.

Then add it to the chaos_scenarios section of config/config.yaml:

kraken:
  kubeconfig_path: ~/.kube/config
  chaos_scenarios:
    - pod_disruption_scenarios:
        - scenarios/kube/gpu_device_plugin.yaml

Run

python run_kraken.py --config config/config.yaml

Run

$ podman run \
  --name=gpu-device-plugin-disruption \
  --net=host \
  --pull=always \
  --env-host=true \
  -e NAMESPACE="nvidia-gpu-operator" \
  -e POD_LABEL="app=nvidia-device-plugin-daemonset" \
  -e NODE_LABEL_SELECTOR="nvidia.com/gpu.present=true" \
  -e DISRUPTION_COUNT="1" \
  -e EXPECTED_RECOVERY_TIME="120" \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pod-scenarios
$ podman logs -f gpu-device-plugin-disruption

See Pod Scenarios for all supported parameters.

krknctl run pod-scenarios \
  --namespace nvidia-gpu-operator \
  --pod-label app=nvidia-device-plugin-daemonset \
  --node-label-selector nvidia.com/gpu.present=true \
  --disruption-count 1 \
  --expected-recovery-time 120

See Pod Scenarios for all supported parameters, or run:

krknctl run pod-scenarios --help