Node Network Chaos

Injects network degradation (latency, packet loss, bandwidth) into a target node’s network interfaces using Linux tc rules.

Injects network degradation (latency, packet loss, bandwidth restriction) into a target node’s network interfaces using Linux tc (traffic control) rules. Unlike node-network-filter which blocks specific ports via iptables, this module shapes traffic at the interface level. Includes safety checks for existing tc rules on the node.

How to Run Node Network Chaos Scenarios

Choose your preferred method to run node network chaos scenarios:

Configuration

- id: node_network_chaos
  image: "quay.io/krkn-chaos/krkn-network-chaos:latest"
  wait_duration: 1
  test_duration: 60
  label_selector: ""
  service_account: ""
  instance_count: 1
  execution: parallel
  namespace: default
  # scenario specific settings
  target: "<node_name>"
  interfaces: []
  ingress: true
  egress: true
  latency: ""         # empty string to skip; or e.g. 100ms (units: us, ms, s)
  loss: 10           # percentage (no % symbol)
  bandwidth: 1gbit   # supported units: bit, kbit, mbit, gbit, tbit
  force: false
  taints: []

For the common module settings please refer to the documentation.

latency: network latency to inject. Format: integer followed by us (microseconds), ms (milliseconds), or s (seconds). Example: 100ms. Set to empty string to skip.
loss: packet loss percentage as a plain integer (no % symbol). Example: 10 means 10% packet loss. Set to empty string to skip.
bandwidth: bandwidth limit. Format: integer followed by bit, kbit, mbit, gbit, or tbit. Example: 100mbit. Set to empty string to skip.
interfaces: list of network interface names to target. Leave empty to auto-detect the node’s default interface.
ingress: apply rules to incoming traffic (default: true)
egress: apply rules to outgoing traffic (default: true)
target: the node name to target (used when label_selector is not set)
force: by default (false), if the target node already has tc rules configured, the scenario aborts with a warning to avoid damaging cluster networking. Set to true to override existing rules. A 10-second warning delay is inserted before proceeding. Use with caution.

Usage

To enable node network chaos scenarios edit the kraken config file, go to the section kraken -> chaos_scenarios of the yaml structure and add a new element to the list named network_chaos_ng_scenarios then add the desired scenario pointing to the scenario yaml file.

kraken:
    ...
    chaos_scenarios:
        - network_chaos_ng_scenarios:
            - scenarios/kube/node-network-chaos.yml

Note

You can specify multiple scenario files of the same type by adding additional paths to the list:

kraken:
    chaos_scenarios:
        - network_chaos_ng_scenarios:
            - scenarios/kube/node-network-chaos-1.yml
            - scenarios/kube/node-network-chaos-2.yml

You can also combine multiple different scenario types in the same config.yaml file. Scenario types can be specified in any order, and you can include the same scenario type multiple times:

kraken:
    chaos_scenarios:
        - network_chaos_ng_scenarios:
            - scenarios/kube/node-network-chaos.yml
        - pod_disruption_scenarios:
            - scenarios/pod-kill.yaml
        - node_scenarios:
            - scenarios/node-reboot.yaml

Warning

When force is set to false (default), the scenario will check if the target node already has complex tc queueing disciplines configured. If existing rules are detected, the scenario aborts to prevent damaging cluster networking. Only set force: true if you understand the implications of overriding existing traffic control rules.

Run

python run_kraken.py --config config/config.yaml

Run

$ podman run --name=<container_name> --net=host --pull=always --env-host=true -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-network-chaos
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

$ docker run $(./get_docker_params.sh) --name=<container_name> --net=host --pull=always -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-network-chaos
OR
$ docker run -e <VARIABLE>=<value> --net=host --pull=always -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-network-chaos
$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

TIP: Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands:

kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host --pull=always -v ~/kubeconfig:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-network-chaos

Supported parameters

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

ex.) export <parameter_name>=<value>

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables

Parameter	Description	Default
LABEL_SELECTOR	Label selector to target one or more nodes (e.g. `node-role.kubernetes.io/worker=`). If omitted, NODE_NAME is used instead	""
NODE_NAME	Exact node name to target when LABEL_SELECTOR is not specified	""
INSTANCE_COUNT	Number of nodes matching the selector to apply chaos on simultaneously	1
INTERFACES	YAML-style list of interface names to shape (e.g. `[br-ex]` or `[eth0,eth1]`). Leave empty to target the node’s default interface	“[]”
TRAFFIC_TYPE	Direction of traffic to shape. Accepted values: `[egress]`, `[ingress]`, or `[egress,ingress]`	“[egress]”
LATENCY	Artificial delay to add to matching packets. Format: `<number><unit>` where unit is `us`, `ms`, or `s` (e.g. `200ms`). Leave empty to skip	""
LOSS	Percentage of packets to drop. Digits only, no `%` symbol (e.g. `10` means 10%). Leave empty to skip	""
BANDWIDTH	Maximum throughput for matching traffic. Format: `<number><unit>` where unit is `bit`, `kbit`, `mbit`, `gbit`, or `tbit` (e.g. `100mbit`). Leave empty to skip	""
TEST_DURATION	How long (in seconds) to hold the tc rules before cleanup	120
WAIT_DURATION	Seconds to wait after chaos cleanup before the scenario exits	0
EXECUTION	Execution mode when targeting multiple nodes: `serial` or `parallel`	parallel
FORCE	When `true`, removes any pre-existing tc qdiscs on the interface before applying new rules. Use with caution	false
NAMESPACE	Kubernetes namespace where the scenario helper pod will be scheduled	default
IMAGE	Container image used for the chaos helper pod that applies tc rules on the node	quay.io/krkn-chaos/krkn-network-chaos:latest
SERVICE_ACCOUNT	Kubernetes service account to assign to the helper pod (leave empty to use the namespace default)	""
TAINTS	YAML-style list of node taints the helper pod should tolerate (e.g. `["node-role.kubernetes.io/master:NoSchedule"]`)	“[]”

NOTE In case of using custom metrics profile or alerts profile when CAPTURE_METRICS or ENABLE_ALERTS is enabled, mount the metrics profile from the host on which the container is run using podman/docker under /home/krkn/kraken/config/metrics-aggregated.yaml and /home/krkn/kraken/config/alerts. For example:

$ podman run --name=<container_name> --net=host --pull=always --env-host=true -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-network-chaos

krknctl run node-network-chaos [--<parameter> <value>]

Can also set any global variable listed here

Node Network Chaos Parameters

Argument	Type	Description	Required	Default Value
`--label-selector`	string	Label selector to target one or more nodes (e.g. `node-role.kubernetes.io/worker=`). If omitted, `--node-name` is used	false
`--node-name`	string	Exact node name to target when `--label-selector` is not specified	false
`--instance-count`	number	Number of nodes matching the selector to apply chaos on simultaneously	false	1
`--interfaces`	string	YAML-style list of interface names to shape (e.g. `[br-ex]` or `[eth0,eth1]`). Leave empty to target the default interface	false
`--traffic-type`	string	Direction of traffic to shape. Accepted values: `[egress]`, `[ingress]`, or `[egress,ingress]`	false	[egress]
`--latency`	string	Artificial delay to add to matching packets. Format: `<number><unit>` where unit is `us`, `ms`, or `s` (e.g. `200ms`). Leave empty to skip	false
`--loss`	string	Percentage of packets to drop. Digits only, no `%` symbol (e.g. `10` means 10%). Leave empty to skip	false
`--bandwidth`	string	Maximum throughput for matching traffic. Format: `<number><unit>` where unit is `bit`, `kbit`, `mbit`, `gbit`, or `tbit` (e.g. `100mbit`). Leave empty to skip	false
`--test-duration`	number	How long (in seconds) to hold the tc rules before cleanup	false	120
`--execution`	enum	Execution mode when targeting multiple nodes: `serial` or `parallel`	false	parallel
`--force`	enum	When `true`, removes any pre-existing tc qdiscs on the interface before applying new rules. Use with caution	false	false
`--namespace`	string	Kubernetes namespace where the scenario helper pod will be scheduled	false	default
`--image`	string	Container image used for the chaos helper pod that applies tc rules on the node	false	quay.io/krkn-chaos/krkn-network-chaos:latest
`--service-account`	string	Kubernetes service account to assign to the helper pod (leave empty to use the namespace default)	false
`--taints`	string	YAML-style list of node taints the helper pod should tolerate	false

Example scenario file: node-network-chaos.yml