Pod Network Scenarios
Pod outage
Scenario to block the traffic (Ingress/Egress) of a pod matching the labels for the specified duration of time to understand the behavior of the service/other services which depend on it during downtime. This helps with planning the requirements accordingly, be it improving the timeouts or tweaking the alerts etc.
With the current network policies, it is not possible to explicitly block ports which are enabled by allowed network policy rule. This chaos scenario addresses this issue by using OVS flow rules to block ports related to the pod. It supports OpenShiftSDN and OVNKubernetes based networks.
Excluding Pods from Network Outage
The pod outage scenario now supports excluding specific pods from chaos testing using the exclude_label parameter. This allows you to target a namespace or group of pods with your chaos testing while deliberately preserving certain critical workloads.
Why Use Pod Exclusion?
This feature addresses several common use cases:
- Testing resiliency of an application while keeping critical monitoring pods operational
- Preserving designated “control plane” pods within a microservice architecture
- Allowing targeted chaos without affecting auxiliary services in the same namespace
- Enabling more precise pod selection when network policies require all related services to be in the same namespace
How to Use the exclude_label Parameter
The exclude_label parameter works alongside existing pod selection parameters (label_selector and pod_name). The system will:
- Identify all pods in the target namespace
- Exclude pods matching the
exclude_label criteria (in format “key=value”) - Apply the existing filters (
label_selector or pod_name) - Apply the chaos scenario to the resulting pod list
Example Configurations
Basic exclude configuration:
- id: pod_network_outage
config:
namespace: my-application
label_selector: "app=my-service"
exclude_label: "critical=true"
direction:
- egress
test_duration: 600
In this example, network disruption is applied to all pods with the label app=my-service in the my-application namespace, except for those that also have the label critical=true.
Complete scenario example:
- id: pod_network_outage
config:
namespace: openshift-console
direction:
- ingress
ingress_ports:
- 8443
label_selector: 'component=ui'
exclude_label: 'excluded=true'
test_duration: 600
This scenario blocks ingress traffic on port 8443 for pods matching component=ui label in the openshift-console namespace, but will skip any pods labeled with excluded=true.
The exclude_label parameter is also supported in the pod network shaping scenarios (pod_egress_shaping and pod_ingress_shaping), allowing for the same selective application of network latency, packet loss, and bandwidth restriction.
1 - Pod Network Chaos Scenarios using Krknctl
krknctl run pod-network-chaos (optional: --<parameter>:<value> )
Can also set any global variable listed here
Scenario specific parameters:
| Parameter | Description | Type | Default |
|---|
--namespace | Namespace of the pod to which filter need to be applied | string | |
--label-selector | When pod_name is not specified, pod matching the label will be selected for the chaos scenario | string | |
--pod-name | When label_selector is not specified, pod matching the name will be selected for the chaos scenario | string | |
--instance-count | Targeted instance count matching the label selector | number | 1 |
--traffic-type | List of directions to apply filters - egress/ingress ( needs to be a list ) | string | “[ingress,egress]” |
--ingress-ports | Ingress ports to block ( needs to be a list ) | string | |
--egress-ports | Egress ports to block ( needs to be a list ) | string | |
--wait-duration | Ensure that it is at least about twice of test_duration | number | 300 |
--test-duration | Duration of the test run | number | 120 |
To see all available scenario options
krknctl run pod-network-chaos --help
2 - Pod Scenarios using Krkn
Sample scenario config (using a plugin)
- id: pod_network_outage
config:
namespace: openshift-console # Required - Namespace of the pod to which filter need to be applied
direction: # Optional - List of directions to apply filters
- ingress # Blocks ingress traffic, Default both egress and ingress
ingress_ports: # Optional - List of ports to block traffic on
- 8443 # Blocks 8443, Default [], i.e. all ports.
label_selector: 'component=ui' # Blocks access to openshift console
exclude_label: 'critical=true' # Optional - Pods matching this label will be excluded from the chaos
Pod Network shaping
Scenario to introduce network latency, packet loss, and bandwidth restriction in the Pod’s network interface. The purpose of this scenario is to observe faults caused by random variations in the network.
Sample scenario config for egress traffic shaping (using plugin)
- id: pod_egress_shaping
config:
namespace: openshift-console # Required - Namespace of the pod to which filter need to be applied.
label_selector: 'component=ui' # Applies traffic shaping to access openshift console.
exclude_label: 'critical=true' # Optional - Pods matching this label will be excluded from the chaos
network_params:
latency: 500ms # Add 500ms latency to egress traffic from the pod.
Sample scenario config for ingress traffic shaping (using plugin)
- id: pod_ingress_shaping
config:
namespace: openshift-console # Required - Namespace of the pod to which filter need to be applied.
label_selector: 'component=ui' # Applies traffic shaping to access openshift console.
exclude_label: 'critical=true' # Optional - Pods matching this label will be excluded from the chaos
network_params:
latency: 500ms # Add 500ms latency to egress traffic from the pod.
Steps
- Pick the pods to introduce the network anomaly either from label_selector or pod_name.
- Identify the pod interface name on the node.
- Set traffic shaping config on pod’s interface using tc and netem.
- Wait for the duration time.
- Remove traffic shaping config on pod’s interface.
- Remove the job that spawned the pod.
3 - Pod Network Chaos Scenarios using Krkn-hub
This scenario runs network chaos at the pod level on a Kubernetes/OpenShift cluster.
Run
If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED environment variable for the chaos injection container to autoconnect.
$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pod-network-chaos
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
Note
–env-host: This option is not available with the remote Podman client, including Mac and Windows (excluding WSL2) machines.
Without the –env-host option you’ll have to set each enviornment variable on the podman command line like -e <VARIABLE>=<value>$ docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pod-network-chaos
OR
$ docker run -e <VARIABLE>=<value> --name=<container_name> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pod-network-chaos
$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
Tip
Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands:
kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v ~kubeconfig:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:<scenario>Supported parameters
The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:
Example if –env-host is used:
export <parameter_name>=<value>
OR on the command line like example:
-e <VARIABLE>=<value>
See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables
| Parameter | Description | Default |
|---|
| NAMESPACE | Required - Namespace of the pod to which filter need to be applied | "" |
| LABEL_SELECTOR | Label of the pod(s) to target | "" |
| POD_NAME | When label_selector is not specified, pod matching the name will be selected for the chaos scenario | "" |
| EXCLUDE_LABEL | Pods matching this label will be excluded from the chaos even if they match other criteria | "" |
| INSTANCE_COUNT | Number of pods to perform action/select that match the label selector | 1 |
| TRAFFIC_TYPE | List of directions to apply filters - egress/ingress ( needs to be a list ) | [ingress, egress] |
| INGRESS_PORTS | Ingress ports to block ( needs to be a list ) | [] i.e all ports |
| EGRESS_PORTS | Egress ports to block ( needs to be a list ) | [] i.e all ports |
| WAIT_DURATION | Ensure that it is at least about twice of test_duration | 300 |
| TEST_DURATION | Duration of the test run | 120 |
Note
In case of using custom metrics profile or alerts profile when CAPTURE_METRICS or ENABLE_ALERTS is enabled, mount the metrics profile from the host on which the container is run using podman/docker under /home/krkn/kraken/config/metrics-aggregated.yaml and /home/krkn/kraken/config/alerts.For example:
$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:pod-network-chaos
