Time Scenarios

Using this type of scenario configuration, one is able to change the time and/or date of the system for pods or nodes.

How to Run Time Scenarios

Choose your preferred method to run time scenarios:

Example scenario file: time_scenarios_example.yml

Configuration Options:

action: skew_time or skew_date.

object_type: pod or node.

namespace: namespace of the pods you want to skew. Needs to be set if setting a specific pod name.

label_selector: Label on the nodes or pods you want to skew.

container_name: Container name in pod you want to reset time on. If left blank it will randomly select one.

object_name: List of the names of pods or nodes you want to skew.

Refer to time_scenarios_example config file.

time_scenarios:
  - action: skew_time
    object_type: pod
    object_name:
      - apiserver-868595fcbb-6qnsc
      - apiserver-868595fcbb-mb9j5
    namespace: openshift-apiserver
    container_name: openshift-apiserver
  - action: skew_date
    object_type: node
    label_selector: node-role.kubernetes.io/worker

How to Use Plugin Name

Add the plugin name to the list of chaos_scenarios section in the config/config.yaml file

kraken:
    kubeconfig_path: ~/.kube/config                     # Path to kubeconfig
    ..
    chaos_scenarios:
        - time_scenarios:
            - scenarios/<scenario_name>.yaml

Note

You can specify multiple scenario files of the same type by adding additional paths to the list:

kraken:
    chaos_scenarios:
        - time_scenarios:
            - scenarios/time-skew-1.yaml
            - scenarios/time-skew-2.yaml
            - scenarios/time-skew-3.yaml

You can also combine multiple different scenario types in the same config.yaml file. Scenario types can be specified in any order, and you can include the same scenario type multiple times:

kraken:
    chaos_scenarios:
        - time_scenarios:
            - scenarios/time-skew.yaml
        - pod_disruption_scenarios:
            - scenarios/pod-kill.yaml
        - node_scenarios:
            - scenarios/node-reboot.yaml
        - time_scenarios:  # Same type can appear multiple times
            - scenarios/time-skew-2.yaml

Run

python run_kraken.py --config config/config.yaml

This scenario skews the date and time of the nodes and pods matching the label on a Kubernetes/OpenShift cluster. More information can be found here.

Run

If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED environment variable for the chaos injection container to autoconnect.

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:time-scenarios
$ podman logs -f <container_name or container_id> # Streams Kraken logs
$ podman inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

Note

–env-host: This option is not available with the remote Podman client, including Mac and Windows (excluding WSL2) machines. Without the –env-host option you’ll have to set each environment variable on the podman command line like -e <VARIABLE>=<value>

$ docker run $(./get_docker_params.sh) \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:time-scenarios
$ docker run \
  -e <VARIABLE>=<value> \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:time-scenarios

$ docker logs -f <container_name or container_id> # Streams Kraken logs
$ docker inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}" # Outputs exit code which can considered as pass/fail for the scenario

Tip

Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands:

kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host --pull=always -v ~/kubeconfig:/home/krkn/.kube/config:Z -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:<scenario>

Supported parameters

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

example:

export <parameter_name>=<value>

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables

Parameter	Description	Type	Default
ACTION	Action to run. Supported actions: skew_time, skew_date	enum	skew_date
OBJECT_TYPE	Object to target. Supported options: pod, node	enum	pod
LABEL_SELECTOR	Label of the container(s) or nodes to target	string	k8s-app=etcd
OBJECT_NAME	List of the names of pods or nodes you want to skew ( optional parameter )	string	[]
NAMESPACE	Namespace of the pods you want to skew, need to be set only if setting a specific pod name	string	`""`
CONTAINER_NAME	Container in the specified pod to target in case the pod has multiple containers running. Random container is picked if empty	string	`""`
SCENARIO_TYPE	Plugin key krkn dispatches the scenario on. Fixed by the container image, not a knob to change.	-	time_scenarios
SCENARIO_FILE	Path to the scenario file baked into the container image. Fixed by the image, not a knob to change.	-	scenarios/time_scenario.yaml

Note

In case of using custom metrics profile or alerts profile when CAPTURE_METRICS or ENABLE_ALERTS is enabled, mount the metrics profile from the host on which the container is run using podman/docker under /home/krkn/kraken/config/metrics-aggregated.yaml and /home/krkn/kraken/config/alerts.

For example:

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml \
  -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:time-scenarios

krknctl run time-scenarios [--<parameter> <value>]

Can also set any global variable listed here

Scenario specific parameters:

Parameter	Description	Type	Default	Possible Values
--object-type	Object to target. Supported options: pod or node	enum	pod	pod/node
--label-selector	Label of the container(s) or nodes to target	string	k8s-app=etcd
--action	Action to run. Supported actions: skew_time or skew_date	enum	skew_date	skew_time/skew_date
--object-names	List of the names of pods or nodes you want to skew	string	[]
--container-name	Container in the specified pod to target in case the pod has multiple containers running. Random container is picked if empty	string	`""`
--namespace	Namespace of the pods you want to skew, need to be set only if setting a specific pod name	string	`""`

To see all available scenario options

krknctl run time-scenarios --help

Demo

See a demo of this scenario: