This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Pod Network Scenarios

Pod outage

Scenario to block the traffic (Ingress/Egress) of a pod matching the labels for the specified duration of time to understand the behavior of the service/other services which depend on it during downtime. This helps with planning the requirements accordingly, be it improving the timeouts or tweaking the alerts etc. With the current network policies, it is not possible to explicitly block ports which are enabled by allowed network policy rule. This chaos scenario addresses this issue by using OVS flow rules to block ports related to the pod. It supports OpenShiftSDN and OVNKubernetes based networks.

Excluding Pods from Network Outage

The pod outage scenario now supports excluding specific pods from chaos testing using the exclude_label parameter. This allows you to target a namespace or group of pods with your chaos testing while deliberately preserving certain critical workloads.

Why Use Pod Exclusion?

This feature addresses several common use cases:

Testing resiliency of an application while keeping critical monitoring pods operational
Preserving designated “control plane” pods within a microservice architecture
Allowing targeted chaos without affecting auxiliary services in the same namespace
Enabling more precise pod selection when network policies require all related services to be in the same namespace

How to Use the `exclude_label` Parameter

The exclude_label parameter works alongside existing pod selection parameters (label_selector and pod_name). The system will:

Identify all pods in the target namespace
Exclude pods matching the exclude_label criteria (in format “key=value”)
Apply the existing filters (label_selector or pod_name)
Apply the chaos scenario to the resulting pod list

Example Configurations

Basic exclude configuration:

- id: pod_network_outage
  config:
    namespace: my-application
    label_selector: "app=my-service"
    exclude_label: "critical=true"
    direction:
      - egress
    test_duration: 600

In this example, network disruption is applied to all pods with the label app=my-service in the my-application namespace, except for those that also have the label critical=true.

Complete scenario example:

- id: pod_network_outage
  config:
    namespace: openshift-console
    direction:
      - ingress
    ingress_ports:
      - 8443
    label_selector: 'component=ui'
    exclude_label: 'excluded=true'
    test_duration: 600

This scenario blocks ingress traffic on port 8443 for pods matching component=ui label in the openshift-console namespace, but will skip any pods labeled with excluded=true.

The exclude_label parameter is also supported in the pod network shaping scenarios (pod_egress_shaping and pod_ingress_shaping), allowing for the same selective application of network latency, packet loss, and bandwidth restriction.

How to Run Pod Network Scenarios

Choose your preferred method to run pod network scenarios:

Example scenario file: pod_network_outage.yml

Sample scenario config (using a plugin)

- id: pod_network_outage
  config:
    namespace: openshift-console   # Required - Namespace of the pod to which filter need to be applied
    direction:                     # Optional - List of directions to apply filters
        - ingress                  # Blocks ingress traffic, Default both egress and ingress
    ingress_ports:                 # Optional - List of ports to block traffic on
        - 8443                     # Blocks 8443, Default [], i.e. all ports.
    label_selector: 'component=ui' # Blocks access to openshift console
    exclude_label: 'critical=true' # Optional - Pods matching this label will be excluded from the chaos
    image: quay.io/krkn-chaos/krkn:tools

Pod Network shaping

Scenario to introduce network latency, packet loss, and bandwidth restriction in the Pod’s network interface. The purpose of this scenario is to observe faults caused by random variations in the network.

Sample scenario config for egress traffic shaping (using plugin)

- id: pod_egress_shaping
  config:
    namespace: openshift-console   # Required - Namespace of the pod to which filter need to be applied.
    label_selector: 'component=ui' # Applies traffic shaping to access openshift console.
    exclude_label: 'critical=true' # Optional - Pods matching this label will be excluded from the chaos
    network_params:
        latency: 500ms             # Add 500ms latency to egress traffic from the pod.
    image: quay.io/krkn-chaos/krkn:tools

Sample scenario config for ingress traffic shaping (using plugin)

- id: pod_ingress_shaping
  config:
    namespace: openshift-console   # Required - Namespace of the pod to which filter need to be applied.
    label_selector: 'component=ui' # Applies traffic shaping to access openshift console.
    exclude_label: 'critical=true' # Optional - Pods matching this label will be excluded from the chaos
    network_params:
        latency: 500ms             # Add 500ms latency to egress traffic from the pod.
    image: quay.io/krkn-chaos/krkn:tools

Steps

Pick the pods to introduce the network anomaly either from label_selector or pod_name.
Identify the pod interface name on the node.
Set traffic shaping config on pod’s interface using tc and netem.
Wait for the duration time.
Remove traffic shaping config on pod’s interface.
Remove the job that spawned the pod.

How to Use Plugin Name

Add the plugin name to the list of chaos_scenarios section in the config/config.yaml file

kraken:
    kubeconfig_path: ~/.kube/config                     # Path to kubeconfig
    ..
    chaos_scenarios:
        - pod_network_scenarios:
            - scenarios/<scenario_name>.yaml

Note

You can specify multiple scenario files of the same type by adding additional paths to the list:

kraken:
    chaos_scenarios:
        - pod_network_scenarios:
            - scenarios/pod-network-1.yaml
            - scenarios/pod-network-2.yaml
            - scenarios/pod-network-3.yaml

You can also combine multiple different scenario types in the same config.yaml file. Scenario types can be specified in any order, and you can include the same scenario type multiple times:

kraken:
    chaos_scenarios:
        - pod_network_scenarios:
            - scenarios/pod-network.yaml
        - pod_disruption_scenarios:
            - scenarios/pod-kill.yaml
        - container_scenarios:
            - scenarios/container-kill.yaml
        - pod_network_scenarios:  # Same type can appear multiple times
            - scenarios/pod-network-2.yaml

Run

python run_kraken.py --config config/config.yaml

This scenario runs network chaos at the pod level on a Kubernetes/OpenShift cluster.

Run

If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED environment variable for the chaos injection container to autoconnect.

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pod-network-chaos
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

Note

–env-host: This option is not available with the remote Podman client, including Mac and Windows (excluding WSL2) machines. Without the –env-host option you’ll have to set each environment variable on the podman command line like -e <VARIABLE>=<value>

$ docker run $(./get_docker_params.sh) \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pod-network-chaos
$ docker run \
  -e <VARIABLE>=<value> \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pod-network-chaos

$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

Tip

Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands:

kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host --pull=always -v ~kubeconfig:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:<scenario>

Supported parameters

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

Example if –env-host is used:

export <parameter_name>=<value>

OR on the command line like example:

-e <VARIABLE>=<value>

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables

Parameter	Description	Default
NAMESPACE	Required - Namespace of the pod to which filter need to be applied	""
IMAGE	Image used to disrupt network on a pod	“quay.io/krkn-chaos/krkn:tools”
LABEL_SELECTOR	Label of the pod(s) to target	""
POD_NAME	When label_selector is not specified, pod matching the name will be selected for the chaos scenario	""
EXCLUDE_LABEL	Pods matching this label will be excluded from the chaos even if they match other criteria	""
INSTANCE_COUNT	Number of pods to perform action/select that match the label selector	1
TRAFFIC_TYPE	List of directions to apply filters - egress/ingress ( needs to be a list )	[ingress, egress]
INGRESS_PORTS	Ingress ports to block ( needs to be a list )	[] i.e all ports
EGRESS_PORTS	Egress ports to block ( needs to be a list )	[] i.e all ports
WAIT_DURATION	The duration (in seconds) that the network chaos (traffic shaping, packet loss, etc.) persists on the target pods. This is the actual time window where the network disruption is active. It must be longer than TEST_DURATION to ensure the fault is active for the entire test.	300
TEST_DURATION	Duration of the test run (e.g. workload or verification)	120

Note

For disconnected clusters, be sure to also mirror the helper image of quay.io/krkn-chaos/krkn:tools and set the mirrored image path properly

Note

In case of using custom metrics profile or alerts profile when CAPTURE_METRICS or ENABLE_ALERTS is enabled, mount the metrics profile from the host on which the container is run using podman/docker under /home/krkn/kraken/config/metrics-aggregated.yaml and /home/krkn/kraken/config/alerts.

For example:

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml \
  -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pod-network-chaos

krknctl run pod-network-chaos (optional: --<parameter>:<value> )

Can also set any global variable listed here

Scenario specific parameters:

Parameter	Description	Type	Default
`--namespace`	Namespace of the pod to which filter need to be applied	string
`--image`	Image used to disrupt network on a pod	string	quay.io/krkn-chaos/krkn:tools
`--label-selector`	When pod_name is not specified, pod matching the label will be selected for the chaos scenario	string
`--exclude-label`	Pods matching this label will be excluded from the chaos even if they match other criteria	string	""
`--pod-name`	When label_selector is not specified, pod matching the name will be selected for the chaos scenario	string
`--instance-count`	Targeted instance count matching the label selector	number	1
`--traffic-type`	List of directions to apply filters - egress/ingress ( needs to be a list )	string	“[ingress,egress]”
`--ingress-ports`	Ingress ports to block ( needs to be a list )	string
`--egress-ports`	Egress ports to block ( needs to be a list )	string
`--wait-duration`	Ensure that it is at least about twice of test_duration	number	300
`--test-duration`	Duration of the test run	number	120

To see all available scenario options

krknctl run pod-network-chaos --help

Pod Network Scenarios

Pod outage

Excluding Pods from Network Outage

Why Use Pod Exclusion?

How to Use the exclude_label Parameter

Example Configurations

How to Run Pod Network Scenarios

Sample scenario config (using a plugin)

Pod Network shaping

Sample scenario config for egress traffic shaping (using plugin)

Sample scenario config for ingress traffic shaping (using plugin)

Steps

How to Use Plugin Name

Note

Run

Run

Note

Tip

Supported parameters

Note

Note

How to Use the `exclude_label` Parameter