Randomized chaos testing

The random subcommand is valuable for generating chaos tests on a large scale with ease and speed. The random scaffold command, when used with the --seed-file and --number-of-scenarios flags, allows you to expand a pre-existing random or graph plan as a template (or seed). The tool randomly distributes scenarios from the seed-file to meet the specified number-of-scenarios. The resulting output is compatible exclusively with the random run command, which generates a random graph from it.

Warning

graph scaffolded scenarios can serve as input for random scaffold --seed-file and random run, as dependencies are simply ignored. However, the reverse is not true. To address this, graphs generated by the random run command are saved (with the path and file name configurable via the --graph-dump flag) and can be replayed using the graph run command.

Example

Let’s start from the following chaos test graph called graph.json:

{
  "application-outages-1-1": {
    "image": "quay.io/krkn-chaos/krkn-hub:application-outages",
    "name": "application-outages",
    "env": {
      "BLOCK_TRAFFIC_TYPE": "[Ingress, Egress]",
      "DURATION": "30",
      "NAMESPACE": "dittybopper",
      "POD_SELECTOR": "{app: dittybopper}",
      "WAIT_DURATION": "1",
      "KRKN_DEBUG": "True"
    },
  },
  "application-outages-1-2": {
    "image": "quay.io/krkn-chaos/krkn-hub:application-outages",
    "name": "application-outages",
    "env": {
      "BLOCK_TRAFFIC_TYPE": "[Ingress, Egress]",
      "DURATION": "30",
      "NAMESPACE": "default",
      "POD_SELECTOR": "{app: nginx}",
      "WAIT_DURATION": "1",
      "KRKN_DEBUG": "True"
    },
    "depends_on": "root-scenario"
  },
  "root-scenario-1": {
    "_comment": "I'm the root Node!",
    "image": "quay.io/krkn-chaos/krkn-hub:dummy-scenario",
    "name": "dummy-scenario",
    "env": {
      "END": "10",
      "EXIT_STATUS": "0"
    }
  }
}

Note

The larger the seed file, the more diverse the resulting output file will be.

Step 1: let’s expand it to 100 scenarios with the command krknctl random scaffold --seed-file graph.json --number-of-scenarios 100 > big-random-graph.json This will produce a file containing 100 compiled replicating the three scenarios above a random amount of times per each:

{
  "application-outages-1-1--6oJCqST": {
    "image": "quay.io/krkn-chaos/krkn-hub:application-outages",
    "name": "application-outages",
    "env": {
      "BLOCK_TRAFFIC_TYPE": "[Ingress, Egress]",
      "DURATION": "30",
      "KRKN_DEBUG": "True",
      "NAMESPACE": "dittybopper",
      "POD_SELECTOR": "{app: dittybopper}",
      "WAIT_DURATION": "1"
    }
  },
  "application-outages-1-1--JToAFrk": {
    "image": "quay.io/krkn-chaos/krkn-hub:application-outages",
    "name": "application-outages",
    "env": {
      "BLOCK_TRAFFIC_TYPE": "[Ingress, Egress]",
      "DURATION": "30",
      "KRKN_DEBUG": "True",
      "NAMESPACE": "dittybopper",
      "POD_SELECTOR": "{app: dittybopper}",
      "WAIT_DURATION": "1"
    }
  },
  "application-outages-1-1--ofb4iMD": {
    "image": "quay.io/krkn-chaos/krkn-hub:application-outages",
    "name": "application-outages",
    "env": {
      "BLOCK_TRAFFIC_TYPE": "[Ingress, Egress]",
      "DURATION": "30",
      "KRKN_DEBUG": "True",
      "NAMESPACE": "dittybopper",
      "POD_SELECTOR": "{app: dittybopper}",
      "WAIT_DURATION": "1"
    }
  },
  "application-outages-1-1--tLPY-MZ": {
    "image": "quay.io/krkn-chaos/krkn-hub:application-outages",
    "name": "application-outages",
    "env": {
      "BLOCK_TRAFFIC_TYPE": "[Ingress, Egress]",
      "DURATION": "30",
      "KRKN_DEBUG": "True",
      "NAMESPACE": "dittybopper",
      "POD_SELECTOR": "{app: dittybopper}",
      "WAIT_DURATION": "1"
    }
  },

  .... (and other 96 scenarios)

Step 2: run the randomly generated chaos test using the command krknctl random run big-random-graph.json --max-parallel 50 --graph-dump big-graph.json. This instructs krknctl to orchestrate the scenarios in the specified file within a graph, allowing up to 50 scenarios to run in parallel per step, while ensuring all scenarios listed in the JSON input file are executed.The generated random graph will be saved to a file named big-graph.json.

Warning

The max-parallel value should be tuned according to machine resources, as it determines the number of parallel krkn instances executed simultaneously on the local machine via containers on podman or docker

Step 3: if you found the previous chaos run disruptive and you want to re-execute it periodically you can store the big-graph.jsonsomewhere and replay it with the command krknctl graph run big-graph.json

Last modified May 5, 2025: added krknctl section (#46) (a462821)