This scenario introduce a new infrastructure to refactor and port the current implementation of the network chaos plugins
This is the multi-page printable view of this section. Click here to print.
Network Chaos NG Scenarios
1 - Network Chaos API
AbstractNetworkChaosModule
abstract module class
All the plugins must implement the AbstractNetworkChaosModule
abstract class in order to be instantiated and ran by the Netwok Chaos NG plugin.
This abstract class implements two main abstract methods:
run(self, target: str, kubecli: KrknTelemetryOpenshift, error_queue: queue.Queue = None)
is the entrypoint for each Network Chaos module. If the module is configured to be run in parallelerror_queue
must not be Nonetarget
: param is the name of the resource (Pod, Node etc.) that will be targeted by the scenariokubecli
: theKrknTelemetryOpenshift
needed by the scenario to access to the krkn-lib methodserror_queue
: a queue that will be used by the plugin to push the errors raised during the execution of parallel modules
get_config(self) -> (NetworkChaosScenarioType, BaseNetworkChaosConfig)
returns the common subset of settings shared by all the scenariosBaseNetworkChaosConfig
and the type of Network Chaos Scenario that is running (Pod Scenario or Node Scenario)
BaseNetworkChaosConfig
base module configuration
Is the base class that contains the common parameters shared by all the Network Chaos NG modules.
id
is the string name of the Network Chaos NG modulewait_duration
if there is more than one network module config in the same config file, the plugin will waitwait_duration
seconds before running the following onetest_duration
the duration in seconds of the scenariolabel_selector
the selector used to target the resourceinstance_count
if greater than 0 picksinstance_count
elements from the targets selected by the filters randomlyexecution
if more than one target are selected by the selector the scenario can target the resources both inserial
orparallel
.namespace
the namespace were the scenario workloads will be deployedtaints
: List of taints for which tolerations need to created. Example: [“node-role.kubernetes.io/master:NoSchedule”]
2 - Node Network Filter
Creates iptables rules on one or more nodes to block incoming and outgoing traffic on a port in the node network interface. Can be used to block network based services connected to the node or to block inter-node communication.
2.1 - Node Network Filter using Krkn
Configuration
- id: node_network_filter
wait_duration: 300
test_duration: 100
label_selector: "kubernetes.io/hostname=ip-10-0-39-182.us-east-2.compute.internal"
instance_count: 1
execution: parallel
namespace: 'default'
# scenario specific settings
ingress: false
egress: true
target: node-name
interfaces: []
protocols:
- tcp
ports:
- 2049
taints: []
for the common module settings please refer to the documentation.
ingress
: filters the incoming traffic on one or more ports. If set one or more network interfaces must be specifiedegress
: filters the outgoing traffic on one or more ports.target
: the node name (if label_selector not set)interfaces
: a list of network interfaces where the incoming traffic will be filteredports
: the list of ports that will be filteredprotocols
: the ip protocols to filter (tcp and udp)taints
: List of taints for which tolerations need to created. Example: [“node-role.kubernetes.io/master:NoSchedule”]
Usage
To enable hog scenarios edit the kraken config file, go to the section kraken -> chaos_scenarios
of the yaml structure
and add a new element to the list named network_chaos_ng_scenarios
then add the desired scenario
pointing to the hog.yaml
file.
kraken:
...
chaos_scenarios:
- network_chaos_ng_scenarios:
- scenarios/kube/node-network-filter.yml
Examples
Please refer to the use cases section for some real usage scenarios.
2.2 - Node Network Filter using Krkn-Hub
Run
$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-network-filter
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
$ docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-network-filter
OR
$ docker run -e <VARIABLE>=<value> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-network-filter
$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
TIP: Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands:
kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v ~kubeconfig:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:<scenario>
Supported parameters
The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:
ex.)
export <parameter_name>=<value>
See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables
Parameter | Description | Default |
---|---|---|
TOTAL_CHAOS_DURATION | set chaos duration (in sec) as desired | 60 |
NODE_SELECTOR | defines the node selector for choosing target nodes. If not specified, one schedulable node in the cluster will be chosen at random. If multiple nodes match the selector, all of them will be subjected to stress. | “node-role.kubernetes.io/worker=” |
NODE_NAME | the node name to target (if label selector not selected | |
INSTANCE_COUNT | restricts the number of selected nodes by the selector | “1” |
EXECUTION | sets the execution mode of the scenario on multiple nodes, can be parallel or serial | “parallel” |
INGRESS | sets the network filter on incoming traffic, can be true or false | false |
EGRESS | sets the network filter on outgoing traffic, can be true or false | true |
INTERFACES | a list of comma separated names of network interfaces (eg. eth0 or eth0,eth1,eth2) to filter for outgoing traffic | "" |
PORTS | a list of comma separated port numbers (eg 8080 or 8080,8081,8082) to filter for both outgoing and incoming traffic | "" |
PROTOCOLS | a list of comma separated protocols to filter (tcp, udp or both) | |
TAINTS | List of taints for which tolerations need to created. Example: [“node-role.kubernetes.io/master:NoSchedule”] | [] |
NOTE In case of using custom metrics profile or alerts profile when CAPTURE_METRICS
or ENABLE_ALERTS
is enabled, mount the metrics profile from the host on which the container is run using podman/docker under /home/krkn/kraken/config/metrics-aggregated.yaml
and /home/krkn/kraken/config/alerts
. For example:
$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:node-network-filter
2.3 - Node Network Filter using krknctl
No problem! Here’s the data you provided, formatted as a Markdown table.
krknctl run node-network-filter (optional: --<parameter>:<value> )
Can also set any global variable listed here
Pod Network Filter Parameters
Argument | Type | Description | Required | Default Value |
---|---|---|---|---|
--chaos-duration | number | Chaos Duration | false | 60 |
--pod-selector | string | Pod Selector | false | |
--pod-name | string | Pod Name | false | |
--namespace | string | Namespace | false | default |
--instance-count | number | Number of instances to target | false | 1 |
--execution | enum | Execution mode | false | |
--ingress | boolean | Filter incoming traffic | true | |
--egress | boolean | Filter outgoing traffic | true | |
--interfaces | string | Network interfaces to filter outgoing traffic (if more than one separated by comma) | false | |
--ports | string | Network ports to filter traffic (if more than one separated by comma) | true | |
--image | string | The network chaos injection workload container image | false | quay.io/krkn-chaos/krkn-network-chaos:latest |
--protocols | string | The network protocols that will be filtered | false | tcp |
--taints | String | List of taints for which tolerations need to created | false |
3 - Pod Network Filter
Creates iptables rules on one or more pods to block incoming and outgoing traffic on a port in the pod network interface. Can be used to block network based services connected to the pod or to block inter-pod communication.
3.1 - Pod Network Filter Using Krkn
Configuration
- id: pod_network_filter
wait_duration: 300
test_duration: 100
label_selector: "app=label"
instance_count: 1
execution: parallel
namespace: 'default'
# scenario specific settings
ingress: false
egress: true
target: 'pod-name'
interfaces: []
protocols:
- tcp
ports:
- 80
taints: []
for the common module settings please refer to the documentation.
ingress
: filters the incoming traffic on one or more ports. If set one or more network interfaces must be specifiedegress
: filters the outgoing traffic on one or more ports.target
: the pod name (if label_selector not set)interfaces
: a list of network interfaces where the incoming traffic will be filteredports
: the list of ports that will be filteredprotocols
: the ip protocols to filter (tcp and udp)taints
: List of taints for which tolerations need to created. Example: [“node-role.kubernetes.io/master:NoSchedule”]
Usage
To enable hog scenarios edit the kraken config file, go to the section kraken -> chaos_scenarios
of the yaml structure
and add a new element to the list named network_chaos_ng_scenarios
then add the desired scenario
pointing to the hog.yaml
file.
kraken:
...
chaos_scenarios:
- network_chaos_ng_scenarios:
- scenarios/kube/pod-network-filter.yml
Examples
Please refer to the use cases section for some real usage scenarios.
3.2 - Pod Network Filter Using Krkn-Hub
Run
$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-kube-config>:/home/krkn/.kube/config:z -d quay.io/krkn-chaos/krkn-hub:pod-network-filter
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
$ docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:z -d quay.io/krkn-chaos/krkn-hub:pod-network-filter
OR
$ docker run -e <VARIABLE>=<value> --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:z -d quay.io/krkn-chaos/krkn-hub:pod-network-filter
$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario
TIP: Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands:
kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v ~kubeconfig:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:<scenario>
Supported parameters
The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:
ex.)
export <parameter_name>=<value>
See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables
Parameter | Description | Default |
---|---|---|
TOTAL_CHAOS_DURATION | set chaos duration (in sec) as desired | 60 |
POD_SELECTOR | defines the pod selector for choosing target pods. If multiple pods match the selector, all of them will be subjected to stress. | “app=selector” |
POD_NAME | the pod name to target (if POD_SELECTOR not specified) | |
INSTANCE_COUNT | restricts the number of selected pods by the selector | “1” |
EXECUTION | sets the execution mode of the scenario on multiple pods, can be parallel or serial | “parallel” |
INGRESS | sets the network filter on incoming traffic, can be true or false | false |
EGRESS | sets the network filter on outgoing traffic, can be true or false | true |
INTERFACES | a list of comma separated names of network interfaces (eg. eth0 or eth0,eth1,eth2) to filter for outgoing traffic | "" |
PORTS | a list of comma separated port numbers (eg 8080 or 8080,8081,8082) to filter for both outgoing and incoming traffic | "" |
PROTOCOLS | a list of comma separated network protocols (tcp, udp or both of them e.g. tcp,udp) | “tcp” |
TAINTS | List of taints for which tolerations need to created. Example: [“node-role.kubernetes.io/master:NoSchedule”] | [] |
NOTE In case of using custom metrics profile or alerts profile when CAPTURE_METRICS
or ENABLE_ALERTS
is enabled, mount the metrics profile from the host on which the container is run using podman/docker under /home/krkn/kraken/config/metrics-aggregated.yaml
and /home/krkn/kraken/config/alerts
. For example:
$ podman run --name=<container_name> --net=host --env-host=true -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:pod-network-traffic
3.3 - Pod Network Filter Using Krknctl
krknctl run pod-network-filter (optional: --<parameter>:<value> )
Can also set any global variable listed here
Argument | Type | Description | Required | Default Value |
---|---|---|---|---|
--chaos-duration | number | Chaos Duration | false | 60 |
--pod-selector | string | Pod Selector | false | |
--pod-name | string | Pod Name | false | |
--namespace | string | Namespace | false | default |
--instance-count | number | Number of instances to target | false | 1 |
--execution | enum | Execution mode | false | |
--ingress | boolean | Filter incoming traffic | true | |
--egress | boolean | Filter outgoing traffic | true | |
--interfaces | string | Network interfaces to filter outgoing traffic (if more than one separated by comma) | false | |
--ports | string | Network ports to filter traffic (if more than one separated by comma) | true | |
--image | string | The network chaos injection workload container image | false | quay.io/krkn-chaos/krkn-network-chaos:latest |
--protocols | string | The network protocols that will be filtered | false | tcp |
--taints | String | List of taints for which tolerations need to created | false |