This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Network Chaos NG Scenarios

This scenario introduce a new infrastructure to refactor and port the current implementation of the network chaos plugins

Available Scenarios

Network Chaos NG scenarios:

1 - Network Chaos API

AbstractNetworkChaosModule abstract module class

All the plugins must implement the AbstractNetworkChaosModule abstract class in order to be instantiated and ran by the Netwok Chaos NG plugin. This abstract class implements two main abstract methods:

  • run(self, target: str, kubecli: KrknTelemetryOpenshift, error_queue: queue.Queue = None) is the entrypoint for each Network Chaos module. If the module is configured to be run in parallel error_queue must not be None
    • target: param is the name of the resource (Pod, Node etc.) that will be targeted by the scenario
    • kubecli: the KrknTelemetryOpenshift needed by the scenario to access to the krkn-lib methods
    • error_queue: a queue that will be used by the plugin to push the errors raised during the execution of parallel modules
  • get_config(self) -> (NetworkChaosScenarioType, BaseNetworkChaosConfig) returns the common subset of settings shared by all the scenarios BaseNetworkChaosConfig and the type of Network Chaos Scenario that is running (Pod Scenario or Node Scenario)

BaseNetworkChaosConfig base module configuration

Is the base class that contains the common parameters shared by all the Network Chaos NG modules.

  • id is the string name of the Network Chaos NG module
  • wait_duration if there is more than one network module config in the same config file, the plugin will wait wait_duration seconds before running the following one
  • test_duration the duration in seconds of the scenario
  • label_selector the selector used to target the resource
  • instance_count if greater than 0 picks instance_count elements from the targets selected by the filters randomly
  • execution if more than one target are selected by the selector the scenario can target the resources both in serial or parallel.
  • namespace the namespace were the scenario workloads will be deployed
  • taints : List of taints for which tolerations need to created. Example: [“node-role.kubernetes.io/master:NoSchedule”]

2 - Node Network Filter

Creates iptables rules on one or more nodes to block incoming and outgoing traffic on a port in the node network interface. Can be used to block network based services connected to the node or to block inter-node communication.

2.1 - Node Network Filter using Krkn

Configuration

- id: node_network_filter
  wait_duration: 300
  test_duration: 100
  label_selector: "kubernetes.io/hostname=ip-10-0-39-182.us-east-2.compute.internal"
  instance_count: 1
  execution: parallel
  namespace: 'default'
  # scenario specific settings
  ingress: false
  egress: true
  target: node-name
  interfaces: []
  protocols:
   - tcp
  ports:
    - 2049
  taints: []

for the common module settings please refer to the documentation.

  • ingress: filters the incoming traffic on one or more ports. If set one or more network interfaces must be specified
  • egress : filters the outgoing traffic on one or more ports.
  • target: the node name (if label_selector not set)
  • interfaces: a list of network interfaces where the incoming traffic will be filtered
  • ports: the list of ports that will be filtered
  • protocols: the ip protocols to filter (tcp and udp)
  • taints : List of taints for which tolerations need to created. Example: [“node-role.kubernetes.io/master:NoSchedule”]

Usage

To enable hog scenarios edit the kraken config file, go to the section kraken -> chaos_scenarios of the yaml structure and add a new element to the list named network_chaos_ng_scenarios then add the desired scenario pointing to the hog.yaml file.

kraken:
    ...
    chaos_scenarios:
        - network_chaos_ng_scenarios:
            - scenarios/kube/node-network-filter.yml

Examples

Please refer to the use cases section for some real usage scenarios.

2.2 - Node Network Filter using Krkn-Hub

Run

$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-network-filter
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
$ docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-network-filter
OR 
$ docker run -e <VARIABLE>=<value> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-network-filter
$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

TIP: Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands: kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v ~kubeconfig:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:<scenario>

Supported parameters

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

ex.) export <parameter_name>=<value>

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables

ParameterDescriptionDefault
TOTAL_CHAOS_DURATIONset chaos duration (in sec) as desired60
NODE_SELECTORdefines the node selector for choosing target nodes. If not specified, one schedulable node in the cluster will be chosen at random. If multiple nodes match the selector, all of them will be subjected to stress.“node-role.kubernetes.io/worker=”
NODE_NAMEthe node name to target (if label selector not selected
INSTANCE_COUNTrestricts the number of selected nodes by the selector“1”
EXECUTIONsets the execution mode of the scenario on multiple nodes, can be parallel or serial“parallel”
INGRESSsets the network filter on incoming traffic, can be true or falsefalse
EGRESSsets the network filter on outgoing traffic, can be true or falsetrue
INTERFACESa list of comma separated names of network interfaces (eg. eth0 or eth0,eth1,eth2) to filter for outgoing traffic""
PORTSa list of comma separated port numbers (eg 8080 or 8080,8081,8082) to filter for both outgoing and incoming traffic""
PROTOCOLSa list of comma separated protocols to filter (tcp, udp or both)
TAINTSList of taints for which tolerations need to created. Example: [“node-role.kubernetes.io/master:NoSchedule”][]

NOTE In case of using custom metrics profile or alerts profile when CAPTURE_METRICS or ENABLE_ALERTS is enabled, mount the metrics profile from the host on which the container is run using podman/docker under /home/krkn/kraken/config/metrics-aggregated.yaml and /home/krkn/kraken/config/alerts. For example:

$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-network-filter

2.3 - Node Network Filter using krknctl

No problem! Here’s the data you provided, formatted as a Markdown table.


krknctl run node-network-filter (optional: --<parameter>:<value> )

Can also set any global variable listed here

Pod Network Filter Parameters

ArgumentTypeDescriptionRequiredDefault Value
--chaos-durationnumberChaos Durationfalse60
--pod-selectorstringPod Selectorfalse
--pod-namestringPod Namefalse
--namespacestringNamespacefalsedefault
--instance-countnumberNumber of instances to targetfalse1
--executionenumExecution modefalse
--ingressbooleanFilter incoming traffictrue
--egressbooleanFilter outgoing traffictrue
--interfacesstringNetwork interfaces to filter outgoing traffic (if more than one separated by comma)false
--portsstringNetwork ports to filter traffic (if more than one separated by comma)true
--imagestringThe network chaos injection workload container imagefalsequay.io/krkn-chaos/krkn-network-chaos:latest
--protocolsstringThe network protocols that will be filteredfalsetcp
--taintsStringList of taints for which tolerations need to createdfalse

3 - Pod Network Filter

Creates iptables rules on one or more pods to block incoming and outgoing traffic on a port in the pod network interface. Can be used to block network based services connected to the pod or to block inter-pod communication.

3.1 - Pod Network Filter Using Krkn

Configuration

- id: pod_network_filter
  wait_duration: 300
  test_duration: 100
  label_selector: "app=label"
  instance_count: 1
  execution: parallel
  namespace: 'default'
  # scenario specific settings
  ingress: false
  egress: true
  target: 'pod-name'
  interfaces: []
  protocols:
    - tcp
  ports:
    - 80
  taints: []

for the common module settings please refer to the documentation.

  • ingress: filters the incoming traffic on one or more ports. If set one or more network interfaces must be specified
  • egress : filters the outgoing traffic on one or more ports.
  • target: the pod name (if label_selector not set)
  • interfaces: a list of network interfaces where the incoming traffic will be filtered
  • ports: the list of ports that will be filtered
  • protocols: the ip protocols to filter (tcp and udp)
  • taints : List of taints for which tolerations need to created. Example: [“node-role.kubernetes.io/master:NoSchedule”]

Usage

To enable hog scenarios edit the kraken config file, go to the section kraken -> chaos_scenarios of the yaml structure and add a new element to the list named network_chaos_ng_scenarios then add the desired scenario pointing to the hog.yaml file.

kraken:
    ...
    chaos_scenarios:
        - network_chaos_ng_scenarios:
            - scenarios/kube/pod-network-filter.yml

Examples

Please refer to the use cases section for some real usage scenarios.

3.2 - Pod Network Filter Using Krkn-Hub

Run

$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-kube-config>:/home/krkn/.kube/config:z -d quay.io/krkn-chaos/krkn-hub:pod-network-filter
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
$ docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:z -d quay.io/krkn-chaos/krkn-hub:pod-network-filter
OR 
$ docker run -e <VARIABLE>=<value> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:z -d quay.io/krkn-chaos/krkn-hub:pod-network-filter
$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

TIP: Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands: kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v ~kubeconfig:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:<scenario>

Supported parameters

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

ex.) export <parameter_name>=<value>

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables

ParameterDescriptionDefault
TOTAL_CHAOS_DURATIONset chaos duration (in sec) as desired60
POD_SELECTORdefines the pod selector for choosing target pods. If multiple pods match the selector, all of them will be subjected to stress.“app=selector”
POD_NAMEthe pod name to target (if POD_SELECTOR not specified)
INSTANCE_COUNTrestricts the number of selected pods by the selector“1”
EXECUTIONsets the execution mode of the scenario on multiple pods, can be parallel or serial“parallel”
INGRESSsets the network filter on incoming traffic, can be true or falsefalse
EGRESSsets the network filter on outgoing traffic, can be true or falsetrue
INTERFACESa list of comma separated names of network interfaces (eg. eth0 or eth0,eth1,eth2) to filter for outgoing traffic""
PORTSa list of comma separated port numbers (eg 8080 or 8080,8081,8082) to filter for both outgoing and incoming traffic""
PROTOCOLSa list of comma separated network protocols (tcp, udp or both of them e.g. tcp,udp)“tcp”
TAINTSList of taints for which tolerations need to created. Example: [“node-role.kubernetes.io/master:NoSchedule”][]

NOTE In case of using custom metrics profile or alerts profile when CAPTURE_METRICS or ENABLE_ALERTS is enabled, mount the metrics profile from the host on which the container is run using podman/docker under /home/krkn/kraken/config/metrics-aggregated.yaml and /home/krkn/kraken/config/alerts. For example:

$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:pod-network-traffic

3.3 - Pod Network Filter Using Krknctl

krknctl run pod-network-filter (optional: --<parameter>:<value> )

Can also set any global variable listed here

ArgumentTypeDescriptionRequiredDefault Value
--chaos-durationnumberChaos Durationfalse60
--pod-selectorstringPod Selectorfalse
--pod-namestringPod Namefalse
--namespacestringNamespacefalsedefault
--instance-countnumberNumber of instances to targetfalse1
--executionenumExecution modefalse
--ingressbooleanFilter incoming traffictrue
--egressbooleanFilter outgoing traffictrue
--interfacesstringNetwork interfaces to filter outgoing traffic (if more than one separated by comma)false
--portsstringNetwork ports to filter traffic (if more than one separated by comma)true
--imagestringThe network chaos injection workload container imagefalsequay.io/krkn-chaos/krkn-network-chaos:latest
--protocolsstringThe network protocols that will be filteredfalsetcp
--taintsStringList of taints for which tolerations need to createdfalse