HTTP Load Scenarios

This scenario generates distributed HTTP load against one or more target endpoints using Vegeta load testing pods deployed inside the Kubernetes cluster. It leverages the distributed nature of Kubernetes clusters to instantiate multiple load generator pods, significantly increasing the effectiveness of the load test.

The scenario supports multiple concurrent pods, configurable request rates, multiple HTTP methods (GET, POST, PUT, DELETE, PATCH, HEAD), custom headers, request bodies, and comprehensive results collection with aggregated metrics across all pods.

The configuration allows for the specification of multiple node selectors, enabling Kubernetes to schedule the attacker pods on a user-defined subset of nodes to make the test more realistic.

The attacker container source code is available here.

How to Run HTTP Load Scenarios

Choose your preferred method to run HTTP load scenarios:

Example scenario file: http_load_scenario.yml

Sample scenario config

- http_load_scenario:
    runs: 1                                            # number of times to execute the scenario
    number-of-pods: 2                                  # number of attacker pods instantiated
    namespace: default                                 # namespace to deploy load testing pods
    image: quay.io/krkn-chaos/krkn-http-load:latest    # http load attacker container image
    attacker-nodes:                                    # node affinity to schedule the attacker pods
      node-role.kubernetes.io/worker:                  # per each node label selector can be specified
        - ""                                           # multiple values so the kube scheduler will schedule
                                                       # the attacker pods in the best way possible
                                                       # set empty value `attacker-nodes: {}` to let kubernetes schedule the pods
    targets:                                           # Vegeta round-robins across all endpoints
      endpoints:                                       # supported methods: GET, POST, PUT, DELETE, PATCH, HEAD
        - url: "https://your-service.example.com/health"
          method: "GET"
        - url: "https://your-service.example.com/api/data"
          method: "POST"
          headers:
            Content-Type: "application/json"
            Authorization: "Bearer your-token"
          body: '{"key":"value"}'

    rate: "50/1s"                                      # request rate per pod: "50/1s", "1000/1m", "0" for max throughput
    duration: "30s"                                    # attack duration: "30s", "5m", "1h"
    workers: 10                                        # initial concurrent workers per pod
    max_workers: 100                                   # maximum workers per pod (auto-scales)
    connections: 100                                   # max idle connections per host
    timeout: "10s"                                     # per-request timeout
    keepalive: true                                    # use persistent HTTP connections
    http2: true                                        # enable HTTP/2
    insecure: false                                    # skip TLS verification (for self-signed certs)

How to Use Plugin Name

Add the plugin name to the list of chaos_scenarios section in the config/config.yaml file

kraken:
    kubeconfig_path: ~/.kube/config                     # Path to kubeconfig
    ..
    chaos_scenarios:
        - http_load_scenarios:
            - scenarios/<scenario_name>.yaml

Note

You can specify multiple scenario files of the same type by adding additional paths to the list:

kraken:
    chaos_scenarios:
        - http_load_scenarios:
            - scenarios/http-load-1.yaml
            - scenarios/http-load-2.yaml
            - scenarios/http-load-3.yaml

You can also combine multiple different scenario types in the same config.yaml file. Scenario types can be specified in any order, and you can include the same scenario type multiple times:

kraken:
    chaos_scenarios:
        - http_load_scenarios:
            - scenarios/http-load.yaml
        - pod_disruption_scenarios:
            - scenarios/pod-kill.yaml
        - syn_flood_scenarios:
            - scenarios/syn-flood.yaml
        - http_load_scenarios:  # Same type can appear multiple times
            - scenarios/http-load-2.yaml

Run

python run_kraken.py --config config/config.yaml

HTTP Load scenario

This scenario generates distributed HTTP load against one or more target endpoints using Vegeta load testing pods deployed inside the cluster.

Run

If enabling Cerberus to monitor the cluster and pass/fail the scenario post chaos, refer docs. Make sure to start it before injecting the chaos and set CERBERUS_ENABLED environment variable for the chaos injection container to autoconnect.

$ podman run --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -e TARGET_ENDPOINTS="GET https://myapp.example.com/health" \
  -e NAMESPACE=<target_namespace> \
  -e TOTAL_CHAOS_DURATION=30s \
  -e NUMBER_OF_PODS=2 \
  -e NODE_SELECTORS=<key>=<value>;<key>=<othervalue> \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:http-load

$ podman logs -f <container_name or container_id>

$ podman inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}"

Note

–env-host: This option is not available with the remote Podman client, including Mac and Windows (excluding WSL2) machines. Without the –env-host option you’ll have to set each environment variable on the podman command line like -e <VARIABLE>=<value>

$ docker run $(./get_docker_params.sh) \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -e TARGET_ENDPOINTS="GET https://myapp.example.com/health" \
  -e NAMESPACE=<target_namespace> \
  -e TOTAL_CHAOS_DURATION=30s \
  -e NUMBER_OF_PODS=2 \
  -e NODE_SELECTORS=<key>=<value>;<key>=<othervalue> \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:http-load

$ docker logs -f <container_name or container_id>

$ docker inspect <container-name or container-id> \
  --format "{{.State.ExitCode}}"

TIP: Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands:

kubectl config view --flatten > ~/kubeconfig && \
chmod 444 ~/kubeconfig && \
docker run $(./get_docker_params.sh) \
  --name=<container_name> \
  --net=host \
  --pull=always \
  -v ~/kubeconfig:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:http-load

Supported parameters

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

Example if –env-host is used:

export <parameter_name>=<value>

OR on the command line like example:

-e <VARIABLE>=<value>

See list of variables that apply to all scenarios here that can be used/set in addition to these scenario specific variables

Parameter	Description	Default
TARGET_ENDPOINTS	Semicolon-separated list of target endpoints. Format: METHOD URL;METHOD URL HEADER1:VAL1,HEADER2:VAL2 BODY. Example: GET https://myapp.example.com/health;POST https://myapp.example.com/api Content-Type:application/json {"key":"value"}	Required
RATE	Request rate per pod (e.g. 50/1s, 1000/1m, 0 for max throughput)	50/1s
TOTAL_CHAOS_DURATION	Duration of the load test (e.g. 30s, 5m, 1h)	30s
NAMESPACE	The namespace where the attacker pods will be deployed	default
NUMBER_OF_PODS	The number of attacker pods that will be deployed	2
WORKERS	Initial number of concurrent workers per pod	10
MAX_WORKERS	Maximum number of concurrent workers per pod (auto-scales)	100
CONNECTIONS	Maximum number of idle open connections per host	100
TIMEOUT	Per-request timeout (e.g. 10s, 30s)	10s
IMAGE	The container image that will be used to perform the scenario	quay.io/krkn-chaos/krkn-http-load:latest
INSECURE	Skip TLS certificate verification (for self-signed certs)	false
NODE_SELECTORS	The node selectors are used to guide the cluster on where to deploy attacker pods. You can specify one or more labels in the format key=value;key=value2 (even using the same key) to choose one or more node categories. If left empty, the pods will be scheduled on any available node, depending on the cluster’s capacity.

NOTE In case of using custom metrics profile or alerts profile when CAPTURE_METRICS or ENABLE_ALERTS is enabled, mount the metrics profile from the host on which the container is run using podman/docker under /home/krkn/kraken/config/metrics-aggregated.yaml and /home/krkn/kraken/config/alerts. For example:

$ podman run \
  --name=<container_name> \
  --net=host \
  --pull=always \
  --env-host=true \
  -v <path-to-custom-metrics-profile>:/home/krkn/kraken/config/metrics-aggregated.yaml \
  -v <path-to-custom-alerts-profile>:/home/krkn/kraken/config/alerts \
  -v <path-to-kube-config>:/home/krkn/.kube/config:Z \
  -d containers.krkn-chaos.dev/krkn-chaos/krkn-hub:http-load

krknctl run http-load (optional: --<parameter>:<value> )

Can also set any global variable listed here

Scenario specific parameters:

Parameter	Description	Type	Default
`--target-endpoints`	Semicolon-separated list of target endpoints. Format: METHOD URL;METHOD URL HEADER1:VAL1,HEADER2:VAL2 BODY. Example: GET https://myapp.example.com/health;POST https://myapp.example.com/api Content-Type:application/json {"key":"value"}	string	Required
`--rate`	Request rate per pod (e.g. 50/1s, 1000/1m, 0 for max throughput)	string	50/1s
`--chaos-duration`	Duration of the load test (e.g. 30s, 5m, 1h)	string	30s
`--namespace`	The namespace where the attacker pods will be deployed	string	default
`--number-of-pods`	The number of attacker pods that will be deployed	number	2
`--workers`	Initial number of concurrent workers per pod	number	10
`--max-workers`	Maximum number of concurrent workers per pod (auto-scales)	number	100
`--connections`	Maximum number of idle open connections per host	number	100
`--timeout`	Per-request timeout (e.g. 10s, 30s)	string	10s
`--image`	The container image that will be used to perform the scenario	string	quay.io/krkn-chaos/krkn-http-load:latest
`--insecure`	Skip TLS certificate verification (for self-signed certs)	string	false
`--node-selectors`	The node selectors are used to guide the cluster on where to deploy attacker pods. You can specify one or more labels in the format key=value;key=value2 (even using the same key) to choose one or more node categories. If left empty, the pods will be scheduled on any available node, depending on the cluster s capacity.	string

To see all available scenario options

krknctl run http-load --help