Develop a scenario once, then expose it through a container image that Krkn’s
clients can discover and configure. This guide follows the full path from a
Python plugin to an execution managed by Krkn Operator.
The example is deliberately small: wait for a configurable number of seconds,
list pods in a namespace through krkn-lib, and report success or failure. It
lets you verify the integration without introducing a disruptive experiment.
Companion repository:krkn-chaos/krkn-dev-demo
contains the plugin skeleton, container files, and Kind and registry configuration
used throughout the lab.
Implement the plugin. Complete the two abstract methods and bind a
scenario type to a YAML configuration.
Package and test the image. Add input metadata, render the configuration,
and run the container with Podman.
Run through the operator. Install a private registry and Krkn Operator
in Kind, publish the image, and execute it against a registered target.
Diagnose the integration. Recognize discovery, image pull, permission,
kubeconfig, and plugin failures independently.
The lab builds a standalone scenario image from a custom Krkn base image. It
does not require a krkn-hub checkout or a rebuild of krknctl or the operator.
Target registration is supplied by the participant and is outside the lab.
Understand the contracts
Read Scenario architecture
for the responsibilities of the registry, operator, node runtime, image startup
script, Python plugin, and target cluster.
Scenario Plugin API defines
discovery rules, method signatures, configuration binding, and return values.
Scenario image contract explains OCI
labels and the typed input descriptors used by krknctl and the console.
Follow a scenario from registry metadata and typed inputs to execution against a target cluster.
A custom scenario connects two contracts: a container image contract for
discovery and execution, and a Python plugin contract for the Krkn engine.
The image carries the metadata needed to present a scenario to users; the
plugin implements the behavior that Krkn runs.
The same image connects discovery, configuration, and execution. Select the diagram to view it at full size.
Discovery reads the image
The registry stores both the image layers and the image configuration containing
its labels. Clients use registry APIs to read tags, manifests, and configuration
blobs. krknctl.is_a_scenario="true" identifies a scenario, and
krknctl.input_fields describes its inputs.
There is no separate scenario catalog database to update. Kubernetes resources
still store operator configuration, credentials, and run state. A registry
providing the catalog does not mean the platform is stateless.
The input metadata is a JSON array of Krkn field descriptors. Its name,
variable, and type properties connect a user-facing input to a container
environment variable. It is not a general-purpose JSON Schema document. See
the image contract reference.
The operator schedules the runner
The console reads scenario metadata through the operator API, presents inputs,
and submits a run. The operator resolves the scenario image, checks its
signature according to the configured policy, and creates the scenario Pod
in the operator’s management cluster.
For the operator version used in the lab, the container receives:
The image reference resolved to an immutable digest.
Environment variables derived from the chosen parameters.
The selected target’s kubeconfig and any configured file mounts.
An image pull secret when the scenario uses a saved private registry.
The console calls these executions jobs. In this implementation the controller
creates Pods directly; a console job does not imply a Kubernetes Job object.
The node’s kubelet asks its container runtime to pull the image. Registry access
from the operator and image pulls from containerd are separate connections.
Enabling HTTP in the operator’s registry settings does not change containerd.
This distinction explains why a scenario can appear in the console while its
Pod remains in ImagePullBackOff.
The image turns inputs into scenario configuration
In the demo, the user-facing wait input maps to WAIT_SECONDS; namespace
maps to NAMESPACE. The image’s /home/krkn/run.sh renders these variables into
a scenario YAML file and selects that file in the Krkn configuration:
The Krkn entrypoint starts the engine with that configuration. The plugin
factory discovers Python classes, and get_scenario_types() maps
dev_demo_scenarios to DevDemoScenarioPlugin. For each configured scenario
file, the engine passes the path to run() together with the telemetry objects.
The plugin uses lib_telemetry.get_lib_kubernetes() to interact with the target
cluster. The mounted kubeconfig determines that target; the location of the
scenario Pod does not.
The container contract also supports other implementations
A scenario can implement its behavior in a shell script or another language.
It still needs the metadata and runtime paths expected by its client. In
particular, the operator used in this lab invokes /home/krkn/run.sh through
/bin/bash -c, so an arbitrary image entrypoint alone is insufficient.
The Python path adds Krkn’s plugin discovery, telemetry integration, and
scenario lifecycle. A plugin should use existing krkn-lib helpers, contain
its exceptions, and return 0 on success or 1 on failure. Scenarios that
modify resources also need a recovery strategy; the demo only reads resources.
What changes when you publish an update
Editing Python source does not change an already built image. Rebuild the
custom base image after plugin edits, then rebuild and push the scenario image.
Changes to input metadata, startup scripts, or templates require rebuilding the
scenario image as well.
Existing Pods retain the image digest they were created with. Start a new run
after publishing a new tag value, and confirm the resolved image in the run
details. Do not assume that a retry used the updated image without checking
its digest.
Build, package, publish, and run a small custom Krkn plugin with Podman, Kind, and Krkn Operator.
Build a scenario that waits, lists pods in a namespace, and returns a clear
result. Start with a plugin skeleton in its own repository, add two methods,
package the plugin in a Krkn image, and expose it to Krkn Operator through a
private registry running inside Kind.
Get the lab files:krkn-chaos/krkn-dev-demo.
The commands below pin revision bf2ea8d,
including the executable-script and containerd configuration fixes used in this
walkthrough.
This lab focuses on the integration contract. The scenario reads Kubernetes
resources and does not disrupt workloads. You provide the target cluster and
register it with the operator separately; this lab creates only the Kind
management cluster that hosts the operator and registry.
What you will build
DevDemoScenarioPlugin, discovered as dev_demo_scenarios.
A custom Krkn base image containing that plugin.
A scenario image exposing wait and namespace through OCI labels.
An authenticated registry, installed with Helm inside Kind.
An operator run whose logs show the development banner and target pod names.
The plugin is built directly into the custom base image. There is no krkn-hub
checkout in this workflow, and the image metadata supplies the client-facing
parameter definitions.
Before you begin
Install Git, Podman, Kind, kubectl, Helm 3, jq, envsubst from GNU gettext,
OpenSSL, and an htpasswd implementation supporting bcrypt (-B). Have Python
3 available for the optional source syntax check. On macOS, start Podman Machine
and use a rootful Podman connection for this Kind lab.
You also need:
Internet access to clone the source and pull the published images and charts.
A running cluster and kubeconfig for the local container test. If it is a
Kind cluster, know its name from kind get clusters.
Free host ports 5001 for the registry and 3000 for the console forward.
An operator target whose Kubernetes API is reachable from the management
cluster’s scenario Pods. Registering that target is outside this lab.
The examples use Kind node image kindest/node:v1.32.2 with containerd 2,
Docker Registry chart 3.0.0, and Krkn Operator chart 1.1.2. These are the lab’s
version choices, not a claim that they are the latest releases. The companion
repository is pinned below so the permission and registry fixes demonstrated
here are included.
Prepare the workspace
Run these commands in one shell and keep its exported variables for the whole
lab. Paths are absolute, so later commands do not depend on the current working
directory. Start from /tmp in case a previous shell was left inside a removed
checkout.
export makes the paths and image names available to child processes. Quoting
the paths preserves them as single arguments. git -C runs Git inside the
plugin checkout without changing your shell’s directory, and checking out a
commit selects the lab’s exact files in a detached HEAD. Create your own branch
there if you want to commit your implementation. KIND_EXPERIMENTAL_PROVIDER
selects Podman for Kind; it does not start Podman Machine or change its connection.
Use an unused KRKN_DIR. If /tmp/krkn already contains work, choose another
absolute path before cloning. Do not remove an existing checkout to make the
command succeed.
The pinned plugin repository is a separate checkout nested inside Krkn’s plugin
directory. It contains:
dev_demo/
├── __init__.py
├── dev_demo_scenario_plugin.py # helper implemented; abstract methods left for you
├── scenario.yaml # example plugin input
├── scenario.yaml.template # environment-to-YAML template
├── env.sh # container defaults
├── run.sh # render configuration and start Krkn
├── krknctl-input.json # user-facing input descriptors
├── Dockerfile.base # add the plugin to a published Krkn runtime
├── Dockerfile.template # package the scenario contract
├── kind-config.yml # registry port and containerd hosts directory
└── registry-values.yml # authenticated registry Helm values
The cloned Krkn source lets you inspect its interface and loader. The base image
build copies the plugin into the published Krkn runtime; it does not rebuild
the rest of the cloned engine. Pin the published base image as well when turning
this workshop into a repeatable release pipeline.
Keep Troubleshooting
open while working. It covers the failures encountered while developing this
demo, including successful discovery followed by an HTTPS image pull error.
2.1 - Implement the Plugin
Complete the abstract interface and connect a scenario YAML file to krkn-lib operations.
Starting point: complete the workspace setup.
Result: a concrete plugin class that Krkn can discover and execute.
Read the prepared behavior
Open $KRKN_DEV_PLUGIN_DIR/dev_demo_scenario_plugin.py. Its
DevDemoScenarioPlugin class already implements execute_demo(scenario, lib_telemetry). The method:
Reads the YAML file identified by scenario.
Requires wait to be an integer from 0 to 60, and namespace to be a
nonempty string. A YAML boolean is not accepted as an integer.
Sleeps for the requested duration.
Obtains the Kubernetes helper from lib_telemetry and checks namespace
existence with list_namespaces().
Lists pods with list_pods(namespace) and logs a distinctive banner.
Returns 0 on success and 1 on a missing namespace or another failure.
The input file is intentionally small:
wait:2namespace:default
The helper uses the cluster client already initialized by Krkn:
kubernetes=lib_telemetry.get_lib_kubernetes()ifnamespacenotinkubernetes.list_namespaces():logging.error("Development scenario: namespace '%s' does not exist",namespace)return1pods=kubernetes.list_pods(namespace)
Checking the namespace list avoids intentionally triggering the library’s
verbose read_namespace_status error path for a missing namespace. This demo
therefore requires permission to list namespaces as well as pods. An existing
namespace with no pods is a successful result and is logged as (none).
Implement the two abstract methods
Ensure these imports are present once at the top of the file:
Paste the following methods inside DevDemoScenarioPlugin, at the same
indentation level as execute_demo(). The four-space class indentation is
included for copying:
get_scenario_types() supplies the configuration key that selects this class.
run() delegates to the helper, which contains the validation and exception
handling. The framework supplies the run identifier and telemetry record even
though this small helper does not need to use them directly.
The class remains abstract until both methods are implemented. Avoid renaming
the package to dev_demo_plugin or dev_demo_scenario: those directory names
violate the factory’s naming rules. The module is
dev_demo_scenario_plugin.py and its class is DevDemoScenarioPlugin.
Understand the configuration binding
The main Krkn configuration selects the type and its scenario file:
The engine calls run(scenario="scenarios/dev_demo.yaml", ...). The plugin
opens that file and reads wait and namespace. The scenario type, file path,
and field names serve different purposes; changing one does not automatically
change the others.
For a source-based Krkn run, copy the sample and derive a dedicated main config:
Set the kraken.chaos_scenarios section in dev_demo_config.yaml to the example
above. The next part packages this same binding in the image’s startup script,
so a source-based run is optional.
py_compile compiles the file without importing it or contacting Kubernetes.
Successful execution normally produces no output and writes bytecode under
__pycache__. It catches syntax and indentation errors, but does not prove
abstract-method completeness, plugin discovery, or Kubernetes permissions.
Those checks follow in the container tests.
The plugin keeps a nonzero result meaningful: an invalid input or missing
namespace must not be converted into success just to keep the container running.
For a real chaos action, add restoration and rollback behavior appropriate to
the resources it changes.
Tip: Dockerfile path and build context are different
-f selects the Dockerfile; the final argument selects the directory available
to its COPY instructions. -t assigns a local image name. Keeping both paths
absolute makes this command work from any directory. Nothing is pushed to a
registry at this stage.
This build context is the plugin repository. Using Krkn’s upstream container
Dockerfile unchanged would clone Krkn from GitHub during the build and omit the
local plugin. If you change the core engine or krkn-lib, build a runtime that
includes those changes explicitly.
Inspect the scenario contract
The image has two user inputs in krknctl-input.json:
[{"name":"wait","short_description":"Wait (seconds)","description":"Seconds to wait before listing pods in the namespace (0 to 60).","variable":"WAIT_SECONDS","type":"number","required":"false"},{"name":"namespace","short_description":"Namespace","description":"Kubernetes namespace whose pod names will be logged.","variable":"NAMESPACE","type":"string","required":"true","validator":"^[a-z0-9]([-a-z0-9]*[a-z0-9])?$","validation_message":"namespace must be a Kubernetes namespace name"}]
name identifies the input presented by a client; variable is the environment
variable injected into the container. The string values "true" and "false"
match krknctl’s field descriptor format. The plugin still enforces its own
integer range: a generic number field alone does not restrict input to an
integer from 0 to 60.
Dockerfile.template embeds the JSON in krknctl.input_fields, together with
the other labels included in the companion image:
LABEL krknctl.is_a_scenario="true"LABEL krknctl.privileged="false"LABEL krknctl.has_rollback="false"LABEL krknctl.kubeconfig_path="/home/krkn/.kube/config"LABEL krknctl.title="Dev Demo"LABEL krknctl.description="Waits, then logs pod names from one Kubernetes namespace."LABEL krknctl.input_fields='$KRKNCTL_INPUT'
Tip: an image label does not configure a volume mount
krknctl.kubeconfig_path is metadata in this template; krknctl and the operator
do not use it to select the mount destination. The operator uses the run’s
kubeconfigPath or its default, /home/krkn/.kube/config, which this lab keeps.
With a custom path, align the run configuration, the mount, and the startup
script’s KUBECONFIG value. See Kubeconfig mount path.
The same Dockerfile copies the startup script with an explicit executable bit:
This permission is required because the operator invokes /home/krkn/run.sh
directly. A local entrypoint that passes the script to bash can conceal a
missing executable bit.
Follow the runtime configuration
env.sh supplies defaults of 2 seconds and namespace default for direct
container execution. The console still requires the namespace input declared
in the metadata.
scenario.yaml.template contains:
wait:${WAIT_SECONDS}namespace:${NAMESPACE}
run.sh changes to /home/krkn/kraken and uses an explicit substitution list:
It derives config/dev_demo_config.yaml from the base configuration, sets
kraken.kubeconfig_path from KUBECONFIG, and selects only
dev_demo_scenarios. For this focused demo, telemetry export, event backup,
reports, resiliency scoring, and the inter-scenario stabilization delay are
disabled. These are demo settings, not a requirement for custom plugins.
Finally it hands control to Krkn while preserving the process exit status:
Compact the JSON, expand only its build-time placeholder, and select the custom
base image in the generated Dockerfile. The runtime placeholders in the YAML
and shell files remain untouched.
Tip: substitute build metadata without expanding runtime inputs
jq -c converts the descriptor array into one line. The single-quoted argument
to envsubst allows only KRKNCTL_INPUT to be expanded; it does not expand
runtime fields such as WAIT_SECONDS. sed replaces the template’s first
FROM line with your custom base image. fromjson decodes the label back into
JSON so you can inspect the actual built artifact. The executable check must
exit successfully as UID and GID 1001, matching the operator’s runner.
Run against an existing Kind cluster
Set KRKN_TEST_CLUSTER to a running cluster you supply. This test does not
create or register an operator target.
Kind’s host kubeconfig normally uses a loopback address. A container on the
kind network needs the internal API address instead. Save that kubeconfig
under your home directory so Podman Machine on macOS can mount it. The temporary
directory is private; the mounted file is readable by container UID 1001.
--network kind makes the internal Kind API address reachable. Each -e
sets a container environment variable, while -v host:container:ro mounts the
kubeconfig read-only at the container path supplied in that argument.
KUBECONFIG tells the startup script where to read it. --user 1001:1001
tests non-root access. Overriding the entrypoint with bash -c 'exec ...'
executes run.sh directly, exposing a missing executable bit. --rm removes
the finished container; the subshell’s trap removes only the generated
kubeconfig directory when the test exits.
On a Linux host with SELinux enforcing, use :ro,Z for this dedicated temporary
kubeconfig mount if a label denial prevents access.
For a non-Kind target, use a kubeconfig whose API endpoint is reachable from
the container. Replace kind get kubeconfig ... with
kubectl --kubeconfig "$KUBECONFIG" config view --raw --flatten --minify and
omit --network kind. Flattening embeds certificate files; an external
credential plugin still needs to be available in the image.
Check both outcomes
For an existing namespace, expect container exit status 0 and a log block:
##### DEVELOPMENT SCENARIO #####
Namespace: default
Pods:
(none)
##### END DEVELOPMENT SCENARIO #####
Pod names replace (none) when the namespace contains pods. Set
KRKN_TEST_NAMESPACE=dev-demo-missing-namespace to a namespace you have
confirmed does not exist and repeat the test. Expect a nonzero exit status and
a concise missing-namespace error. Do not create that namespace as part of
this failure-path test.
Host an authenticated registry inside Kind, configure containerd, and run the custom scenario through Krkn Operator.
Starting point: the scenario image passes its local tests.
Result: the operator discovers the image, renders its two inputs, and runs
it against a target you have already registered.
The management cluster hosts the registry, operator, and scenario Pods. The
target’s kubeconfig is mounted into each scenario Pod. You do not need to place
the private registry on the target or configure the target’s runtime to pull
this scenario image.
One registry, two client configurations. Select the diagram to view it at full size.
1. Create the management cluster
Use a dedicated name so an existing Kind cluster is preserved:
Tip: keep management operations on the intended cluster
kind create cluster --name creates the node container and a kubeconfig context
named kind-<name>. Explicit --context and Helm --kube-context arguments
keep later commands on that management cluster even if your current context
changes. --config reads the port mapping and containerd settings at creation
time; it is not a configuration file watched by the running cluster.
The port mapping is created with the node container. Editing this file does
not change an existing cluster. The containerd plugin key above is for the
containerd 2 runtime in this node image.
Tip: follow the registry's three ports
Podman publishes host port 5001 to port 30500 on the Kind node container.
The registry’s Kubernetes NodePort forwards 30500 to service port 5000.
listenAddress: 127.0.0.1 restricts the workstation listener to loopback.
Inside the cluster, clients use the node hostname and 30500, not the
workstation’s localhost:5001.
2. Install the authenticated registry
Keep generated credentials outside the checkout. Run the following in the same
shell so the registry credentials remain available for the push and UI setup:
openssl rand -hex 16 generates the demo password. htpasswd -Bbn writes a
bcrypt password entry to standard output, which is redirected into the private
state directory. Helm’s --set-file reads that file’s contents into the chart
value; it does not mount the workstation path. upgrade --install installs the
release or updates an existing one, and --wait checks readiness before
continuing. Keep the exported password for the Podman login and registry form.
The Twuni chart deploys Docker
Distribution. The demo values select NodePort 30500, service port 5000,
and persistence.enabled: false. Storage is disposable: replacing the registry
Pod or deleting the cluster can remove the pushed images. Use TLS and durable
storage for a registry outside this local lab.
Expect {}. An unauthenticated request should receive HTTP 401.
3. Configure image pulls on the management node
Use the node’s name on the Podman kind network as the registry address inside
the cluster. This avoids embedding an IP that can change after a Podman Machine
restart. The host continues to push through localhost:5001.
exportKRKN_MGMT_NODE="$(kind get nodes --name "$KRKN_MGMT_CLUSTER"| head -n 1)"exportKRKN_REGISTRY_URL="$KRKN_MGMT_NODE:30500"printf'Registry URL for the operator: %s\n'"$KRKN_REGISTRY_URL"for node in $(kind get nodes --name "$KRKN_MGMT_CLUSTER");doregistry_dir="/etc/containerd/certs.d/$KRKN_REGISTRY_URL" podman exec"$node" mkdir -p "$registry_dir"printf'server = "http://%s"\n[host."http://%s"]\n capabilities = ["pull", "resolve"]\n'\
"$KRKN_REGISTRY_URL""$KRKN_REGISTRY_URL"| podman exec -i "$node" sh -c "cat > '$registry_dir/hosts.toml'"done
Tip: configure the runtime inside the Kind node
Each Kind node is a container. podman exec writes its containerd configuration
inside that node, not on the workstation. The loop configures every management
node, and -i passes the generated TOML through standard input. The directory
name must exactly match the registry host and port used in the image reference.
The pull and resolve capabilities allow fetching image content and resolving
tags. Credentials remain in the operator’s pull secret, not this file.
The fresh cluster already has config_path enabled. Changes to the hosts files
do not require a containerd restart. For a cluster created with an older config,
follow Repair an existing node
before retrying a pull.
The registry URL, hosts.toml directory name, and operator pull secret must use
the same host and port. The operator’s Allow Insecure HTTP option affects
its registry API requests; this node configuration controls the actual pull.
Skip TLS Verification only changes certificate checks and does not enable
HTTP. See the containerd registry documentation.
Check that cluster Pods can resolve and reach the chosen hostname:
Expect HTTP 401: this probe intentionally sends no credentials. If hostname
resolution fails, fix the Kind/Podman network before registering the URL.
Tip: interpret the network probe
kubectl run --restart=Never creates a one-off Pod. --attach=true shows its
output and --rm removes it afterward. Curl discards the body and prints the
HTTP status. A 401 demonstrates DNS resolution, TCP connectivity, HTTP
transport, and an authentication challenge; it does not yet prove that the
image or supplied credentials are valid.
podman tag adds another local name to the already built image; it does not
rebuild it. push uploads that content to the registry. --password-stdin
reads the login secret from the pipe, and --tls-verify=false allows this local
HTTP endpoint. The operator uses a different address to reach the same registry,
so its credentials are configured separately in the next steps.
The response should list dev-demo. The repository is demo/krkn-hub; the tag
is the scenario identity. The operator accesses the same content as
$KRKN_REGISTRY_URL/demo/krkn-hub:dev-demo, using the internal address.
The first 3000 is the workstation port; the second is the service port.
kubectl port-forward selects a backing Pod and keeps the tunnel open while
the command runs. It does not create an Ingress or permanently expose the
console. Stop it with Ctrl+C when you finish the session.
Open http://localhost:3000, create the first administrator account, and sign
in. Keep the forward running. If you changed the cluster or namespace names,
use those names in the second terminal as well.
6. Register the image source
In Settings → Private Registries, add:
Name:dev-demo-registry.
Registry URL: the value of $KRKN_REGISTRY_URL, without http://.
Scenario Repository:demo/krkn-hub.
Authentication Type: Username & Password.
Username and password: the values of $REGISTRY_USER and
$REGISTRY_PASSWORD, not their variable names.
Allow Insecure HTTP: enabled.
Skip TLS Verification: disabled; this local registry serves plain HTTP.
Available to All Users: enabled for this dedicated demo installation.
For the default cluster name, the URL is
krkn-dev-demo-mgmt-control-plane:30500. The operator stores the credentials in
a Kubernetes pull secret. If you edit the URL later, use the registry form so
the stored URL and authentication host stay aligned.
The new custom image is unsigned. On the same settings tab, disable Image
signature verification for this dedicated lab installation. This setting is
operator-wide. Normal deployments should require signatures that the operator
trusts; adding an arbitrary signature is not sufficient. For the official
scenario collection, see mirroring signed images.
7. Run against your registered target
Provide a target through your usual operator setup. Its API must be reachable
from a scenario Pod in the management cluster, and its credentials must allow
listing namespaces and pods. This lab does not create or register the target.
In Run Scenario:
Select your registered target.
Choose the private registry dev-demo-registry.
Select dev-demo (display title Dev Demo).
Set Wait (seconds) to 2 and Namespace to an existing namespace.
Review the configuration and start the run.
In Jobs, inspect the resolved image, status, and logs. Expect the
##### DEVELOPMENT SCENARIO ##### banner, the selected namespace’s pod list,
and a successful result. An empty existing namespace is also successful.
Inspect the Kubernetes resources separately:
kubectl --context "$KRKN_MGMT_CONTEXT"\
-n "$KRKN_OPERATOR_NAMESPACE" get krknscenarioruns,pods
For the selected krkn-job-... Pod, use kubectl describe pod to inspect pull
events and kubectl logs to inspect plugin output. The current controller
creates Pods directly; kubectl get jobs can legitimately return no resources.
Tip: events and logs answer different questions
kubectl describe pod <name> shows scheduling, image pull, and container
startup events. kubectl logs <name> reads the process output after the
container has started. Use the management context and operator namespace for
both. An ImagePullBackOff failure usually has useful events but no scenario
logs; a missing target namespace is reported in the scenario logs.
Run again with a namespace you have confirmed is absent. Expect the plugin to
return 1 with a concise error. That failure proves the input and exit status
reached the plugin; it is different from a Pod that never started.
Iterate on the scenario
After editing plugin Python, rebuild the base and scenario images. For a
startup-script, template, or metadata change, rebuild the scenario image.
Repeat the tag and push commands, then start a new run and inspect its
resolved digest. Existing Pods retain the digest they were created with.
Diagnose discovery, registry transport, image startup, kubeconfig, and plugin failures at the layer that owns them.
First determine whether the scenario was discovered, whether its image was
pulled, and whether its process started. These are separate checkpoints.
Kubernetes events explain scheduling and pull failures; container logs explain
startup and plugin failures.
The scenario is visible, but the pull attempts HTTPS
Typical event:
http: server gave HTTP response to HTTPS client
The operator’s registry client successfully used HTTP to list metadata. Kubelet
then asked containerd to pull the image, and containerd defaulted to HTTPS.
Krknctl’s insecure setting does not configure the node runtime.
Check all three items on the management node where the scenario Pod runs:
The effective containerd registry config_path is /etc/containerd/certs.d.
/etc/containerd/certs.d/$KRKN_REGISTRY_URL/hosts.toml exists and uses
http:// for both the server and host entry.
The registry host and port exactly match the Pod’s image reference.
Containerd 1 uses plugins."io.containerd.grpc.v1.cri".registry instead. Match
the runtime you actually run; copying the older plugin key into a containerd 2
configuration can leave the intended registry configuration inactive.
Repair an existing node
If the containerd 2 registry section above is absent, append it once and
restart containerd on the affected demo node:
If the section already exists with a different value, edit that section rather
than appending a duplicate TOML table. Then write the
registry hosts file.
Hosts-file edits are read dynamically once config_path is active.
A Pod already in ImagePullBackOff may take several minutes to retry. During
the lab, restarting kubelet on the dedicated node cleared that wait:
Transport is now working; authentication still needs checking. Verify the
saved registry credentials and the Pod’s imagePullSecrets. The authentication
key in the secret must match the registry host and port in the image reference.
A login to localhost:5001 in Podman does not automatically configure a
Kubernetes pull secret for the node hostname.
Update credentials through Settings → Private Registries. Do not put
passwords in hosts.toml or paste decoded pull secrets into logs. The operator
supplies authentication through the pull secret; the node configuration
supplies the transport.
/home/krkn/run.sh: Permission denied
The operator invokes /home/krkn/run.sh directly. A script with mode 0644
can work when passed to bash locally and still fail in the operator Pod.
Use this Dockerfile instruction, then rebuild and push the scenario image:
Start a new run and inspect its digest. Fixing the file on the host does not
change an already built image, and pushing a new tag value does not rewrite
an existing Pod’s image reference.
Krkn looks for /root/.kube/config
Setting KUBECONFIG does not override an explicit
kraken.kubeconfig_path in the Krkn YAML configuration. The demo’s startup
script sets that configuration key from KUBECONFIG and defaults it to
/home/krkn/.kube/config.
Align the mounted path, the environment variable, and the main Krkn
configuration. In the operator, the mount destination comes from the run’s
spec.kubeconfigPath or the configured default, /home/krkn/.kube/config in
this lab. The image’s krknctl.kubeconfig_path label does not control the mount.
For a custom path, update the run configuration and the startup script’s
KUBECONFIG value together. Running the container as root can change a
home-directory fallback, which is why the demo uses an explicit path and tests
startup as UID 1001.
The kubeconfig works on the host but not in the container
127.0.0.1 inside a container refers to that container. For a Kind target,
generate the kubeconfig with kind get kubeconfig --internal --name ... and
attach the test container to the kind network. For another target, use an API
endpoint reachable from the container and include the required certificates.
If the host API port changed after a Podman Machine restart, start the stopped
Kind node and refresh the host kubeconfig:
Use the stable node hostname for the registry URL. If an old registry entry
contains a numeric IP, update its URL in the console and write a matching
hosts.toml. Do not change the target’s kubeconfig just to repair a registry URL.
The plugin is not discovered
Inspect Krkn’s factory logs. Verify the dev_demo package and __init__.py,
the dev_demo_scenario_plugin.py module, the DevDemoScenarioPlugin class,
and both abstract methods. The configuration type must be
dev_demo_scenarios and must not collide with another plugin.
If source edits appear to have no effect, rebuild the base image containing
the Python plugin before rebuilding the scenario image. The upstream Krkn
Dockerfile clones its own source and does not automatically include your local
plugin checkout.
The image is pushed but missing from scenario selection
Check the selected registry, repository (demo/krkn-hub), and tag (dev-demo).
Inspect the image for krknctl.is_a_scenario="true" and valid JSON in
krknctl.input_fields. A repository path is not a scenario name.
An unsigned custom image can also be blocked by signature verification. The
lab’s override is configured on the operator, not in the plugin or image label.
Keep it scoped to the dedicated lab installation.
The process starts and reports a missing namespace
This is the demo’s expected failure path: image discovery, pull, startup, and
configuration injection have already succeeded. Check the selected target and
namespace. Namespace existence is evaluated in the target cluster, not
necessarily in the management cluster hosting the Pod.
For an existing namespace, verify that the target credentials allow listing
namespaces and pods. An existing namespace with no pods should return success.
3 - Scenario Plugin API
Naming, discovery, method signatures, and configuration binding for Krkn scenario plugins.
A scenario plugin extends AbstractScenarioPlugin and implements
get_scenario_types() and run(). The factory discovers it under
krkn/scenario_plugins/; adding a valid plugin does not require editing a
central registration list.
Include __init__.py so the plugin directory is a Python package.
The plugin directory must not contain scenario or plugin in its name.
The module containing the plugin class ends with _scenario_plugin.py.
The class ends with ScenarioPlugin and uses the module’s corresponding
CamelCase name: dev_demo_scenario_plugin.py becomes DevDemoScenarioPlugin.
Each scenario type returned by the plugin must be unique across loaded plugins.
The factory instantiates classes while discovering their types. Keep
constructors free of cluster operations. If you override __init__, preserve
the base class’s scenario_type argument and initialization.
Abstract methods
The current interface is:
fromkrkn_lib.models.telemetryimportScenarioTelemetryfromkrkn_lib.telemetry.ocpimportKrknTelemetryOpenshiftfromkrkn.scenario_plugins.abstract_scenario_pluginimportAbstractScenarioPluginclassDevDemoScenarioPlugin(AbstractScenarioPlugin):defget_scenario_types(self)->list[str]:return["dev_demo_scenarios"]defrun(self,run_uuid:str,scenario:str,lib_telemetry:KrknTelemetryOpenshift,scenario_telemetry:ScenarioTelemetry,)->int:returnself.execute_demo(scenario,lib_telemetry)# Implement execute_demo() with validation and exception handling.
The example shows the interface; execute_demo() is supplied by the
demo repository.
The run() signature does not include a krkn_config argument.
get_scenario_types()
Return a list of strings that can appear beneath kraken.chaos_scenarios.
One class can serve multiple types. Reusing a type already claimed by another
plugin causes the conflicting registration to fail and is reported in the
factory logs.
run()
run_uuid identifies the Krkn run.
scenario is the path to the individual scenario configuration file,
not its parsed contents. Read and validate that file in the plugin.
lib_telemetry is the initialized KrknTelemetryOpenshift wrapper.
Its Kubernetes and OpenShift helpers use the run’s cluster credentials.
scenario_telemetry is the telemetry record for this scenario execution.
The engine manages its timing and exit status; use it for supported
scenario-specific telemetry.
Return integer 0 on success and 1 on failure. Handle exceptions within the
plugin’s execution path so they do not escape run(). Log enough context to
diagnose a failure without dumping sensitive HTTP headers or credentials.
The engine finds the type returned by get_scenario_types() and passes
scenarios/dev_demo.yaml to run(). Relative paths are resolved from the
Krkn process’s working directory. Discovery imports plugin modules even when
their types are not selected; do not perform experiments at import time.
Prefer these helpers to creating a second Kubernetes client and loading another
kubeconfig. Reusable Kubernetes or OpenShift operations belong in krkn-lib;
scenario-specific sequencing belongs in the plugin. See
Using and extending krkn-lib.
Before packaging
Check naming, unique type registration, configuration validation, failure
handling, and cleanup behavior. A Python syntax check alone does not verify
discovery or cluster access. Use the
testing guide to cover those stages.
4 - Scenario Image Contract
Define discoverable scenario images with OCI labels and krknctl input descriptors.
Krknctl and Krkn Operator discover scenario definitions from container image
metadata. To add a custom scenario, publish an image with the expected labels
and configure its registry in the client. You do not need to add the scenario
to krknctl’s source code when the existing input types cover its parameters.
krknctl.is_a_scenario identifies scenario images when set to "true".
krknctl.title and krknctl.description describe the scenario.
krknctl.input_fields contains a JSON array of field descriptors.
krknctl.kubeconfig_path appears in the companion image as metadata.
The clients used in this lab do not read it to configure the kubeconfig mount.
Set the runtime mount path independently.
krknctl.privileged declares a privilege requirement. It is not an RBAC
grant; actual permissions depend on the client, runtime, and target credentials.
krknctl.has_rollback describes rollback capability. Set it to match
implemented behavior, not to indicate an intended future feature.
The base image can also supply global input labels such as
krknctl.input_fields.global. Keep scenario-specific fields in the scenario
label rather than copying every framework setting into it.
Image labels are build artifacts. Changing a JSON file on disk does not change
the published label until the scenario image is rebuilt and pushed.
Input field descriptors
krknctl-input.json contains an array, not a JSON Schema object with
properties. A typical field is:
{"name":"namespace","short_description":"Namespace","description":"Kubernetes namespace whose pod names will be logged.","variable":"NAMESPACE","type":"string","required":"true","validator":"^[a-z0-9]([-a-z0-9]*[a-z0-9])?$","validation_message":"namespace must be a Kubernetes namespace name"}
For ordinary input fields, name, variable, and type are required by the
parser. short_description supplies the display label; description supplies
help text. variable must exactly match the environment variable consumed by
the image’s startup code.
The descriptor format uses strings for required, secret, and default,
including numeric and boolean defaults. Use "required": "true", not the
JSON boolean true. Omit an optional property when it is not needed.
Supported input types
string accepts text and can apply a validator regular expression with
a custom validation_message.
number validates a numeric value. Enforce application-specific integer
constraints, ranges, and units in the runtime implementation as well.
boolean represents a boolean input; it still reaches the container as
a string environment value.
enum restricts values using allowed_values and separator.
file describes a file input with a container mount_path.
file_base64 describes file content encoded for a consuming script or
application. Use this exact type name, not file64.
group supplies presentation metadata for a group of related fields.
It is metadata rather than a value injected as a normal input.
Keep defaults in the input metadata and runtime defaults consistent. Client
validation improves feedback, but direct podman run -e ... bypasses it. Validate
all values again before performing scenario actions.
Runtime compatibility
For Krkn Operator 1.1.2, provide /bin/bash and an executable
/home/krkn/run.sh. The controller invokes that script directly, mounts the
target kubeconfig, injects environment variables, and records the exit status.
Do not rely only on a custom OCI ENTRYPOINT being honored by every client.
The script can launch Krkn or another implementation that obeys the same
runtime contract. It must read the declared inputs, use the configured file
paths, and exit with a meaningful status. For the Krkn plugin path, return
0 or 1 from the plugin and preserve the engine’s process status with exec.
Kubeconfig mount path
Krkn Operator selects the container mount path from the run’s
spec.kubeconfigPath (kubeconfigPath in the API request). When omitted, it
uses the krknctl configuration default, /home/krkn/.kube/config in this lab.
Krknctl also uses its own configured kubeconfig_path for the mount destination;
the image label does not override that setting.
For a custom path, configure the client’s mount destination and make the
startup script read the same path. The demo’s script reads KUBECONFIG and
writes it into kraken.kubeconfig_path, so update that environment value or
the script’s default as well. Changing spec.kubeconfigPath moves the mount;
it does not rewrite the script or the Krkn YAML. Changing only the image label
does neither.
Verify discovery, input rendering, runtime validation, and execution separately.
An HTTP-enabled registry client can successfully read these labels while the
node runtime still fails to pull the image. See
the registry lab.
5 - Using and Extending Krkn-lib
Reuse the cluster clients provided to scenario plugins and contribute shared Kubernetes and OpenShift helpers.
krkn-lib provides the Kubernetes, OpenShift, telemetry, metrics, and data-model
building blocks used by Krkn scenarios. Prefer its existing helpers for cluster
operations so scenarios share authentication, behavior, and testable interfaces.
Use the client supplied to the plugin
Krkn passes a KrknTelemetryOpenshift instance as lib_telemetry to run().
Obtain the Kubernetes helper from it:
For OpenShift-specific operations, use lib_telemetry.get_lib_ocp() and the
methods documented for the installed library version. The wrapper has already
been initialized for the run; creating another client with a separately loaded
kubeconfig can make the plugin act on the wrong cluster.
The demo’s namespace existence check lists namespaces before listing pods.
Its target credentials therefore need both permissions. Always derive access
requirements from the operations your implementation actually performs.
krkn_lib.k8s and krkn_lib.ocp for cluster operations.
krkn_lib.telemetry for telemetry collection and integration.
krkn_lib.prometheus and krkn_lib.elastic for metrics and result storage.
krkn_lib.models for shared models.
krkn_lib.utils for common utilities.
Check the version installed in the runtime image when consulting a helper’s
signature. A newer library checkout on the host does not change the dependency
already installed in a container image.
Add reusable behavior
Place broadly reusable Kubernetes or OpenShift operations in the library.
Keep experiment sequencing, scenario configuration, and success criteria in
the plugin. For a new helper, document its arguments, return type, exceptions,
required permissions, and behavior when resources are absent.
Clone the library and install its development dependencies using the project’s
Poetry configuration:
git clone https://github.com/krkn-chaos/krkn-lib.git
cd krkn-lib
poetry install --no-interaction
Follow the repository’s tests and contribution instructions for the operation
you change. Mock isolated API behavior in unit tests, and use a disposable
cluster for tests that exercise real Kubernetes resources. See
Testing changes.
If a plugin depends on an unreleased helper, install that library build into
the custom base image before packaging the scenario. Publishing a plugin alone
cannot make a missing helper appear in the runtime.
6 - Testing Changes
Verify plugin behavior, the image contract, container startup, and operator execution as separate stages.
Test the behavior at the layer you changed, then verify the next integration
boundary. A correct Python method does not prove that its image can be
discovered or started by the operator.
The hands-on lab supplies the concrete
build and run commands. Use this page when deciding what evidence a scenario
change needs before contribution.
Plugin and engine tests
For the development scenario, cover:
A valid configuration and an existing namespace, including one with no pods.
A missing namespace and a Kubernetes API failure.
Invalid wait values: negative, above 60, fractional, boolean, or missing.
Missing or invalid namespace values and unreadable or malformed YAML.
The expected return code and useful error context for each failure.
Mock the sleep and cluster helper in unit tests so they remain fast and do not
depend on credentials. Verify externally meaningful behavior such as whether
pods are listed after a successful namespace check, rather than reproducing
the implementation’s internal steps in the test.
Krkn’s top-level scenario tests live under tests/, while plugin-factory tests
also live under krkn/tests/. After installing the dependencies required by the
checked-out version, run the relevant test module from the repository root.
The existing factory test is:
Add a focused test module for your scenario and run it explicitly before the
broader suite. See Adding tests for
test structure and functional CI integration.
Container and metadata checks
Rebuild the custom base after changing plugin Python or dependencies. Rebuild
the scenario image after changing labels, startup scripts, or templates.
Inspect the resulting image, rather than only the source Dockerfile:
Run the container against a disposable target and exercise both success and
failure. Use the operator’s startup convention in the test: execute
/home/krkn/run.sh directly under UID 1001. Check the mounted kubeconfig,
generated scenario YAML, effective kraken.kubeconfig_path, and exit status.
The local test
handles the internal Kind API address and the Podman Machine mount location.
Operator integration
Verify these checkpoints in order:
The registry lists the scenario tag and its label JSON can be read.
The console shows the expected fields and maps them to the intended
environment variables.
The Pod is scheduled in the management cluster and its image is pulled.
The startup script runs with the operator’s user and file permissions.
The plugin accesses the selected target and produces the expected result.
Capture the resolved image digest when reporting the test. Pushing a new tag
value does not change the image reference of an existing Pod. Start a new run
and inspect its digest after a rebuild.
For a private HTTP registry, test operator API access and node-runtime access
separately. A successful listing does not validate containerd’s transport or
the kubelet’s pull credentials. The
troubleshooting guide
shows the relevant evidence for each failure.
Functional CI infrastructure
The small demo needs only a target with namespace and pod read permissions.
The full repository test suites can require additional nodes, workloads,
Prometheus, Elasticsearch, or platform-specific resources. Use the fixtures
and setup in the checked-out repository’s CI documentation for the tests you
select.
Krkn’s shell functional runner uses CI/tests/functional_tests, individual
scripts under CI/tests/, and CI/run.sh. It writes results to
CI/results.markdown and logs under CI/out/. Follow the
current CI instructions
and adding tests guide
when extending that suite.
Monitoring installation references
These services support other Krkn test suites and the metrics and long-term
storage user journeys. Neither is required for the custom scenario lab, which
disables telemetry export and resiliency scoring. Install them separately in
an environment appropriate to those tests; this is not part of provisioning
the lab’s target.
Prometheus
For a Kubernetes test environment, use the maintained
kube-prometheus-stack Helm installation instructions.
Select a chart version compatible with your Kubernetes version and the test
requirements, then configure the tests with a reachable Prometheus endpoint.
A workstation port-forward is suitable for a test process on that workstation;
localhost in a scenario container refers to the container itself.
Elasticsearch
Use an Elasticsearch version supported by the library and tests you are
running. Elastic documents
installing ECK with Helm
and deploying and accessing an Elasticsearch cluster.
Follow both steps: installing the operator alone does not create an
Elasticsearch instance. Supply the resulting endpoint, credentials, and TLS
configuration to the selected test suite.
Use these maintained installation references instead of copying old chart
values or image tags across incompatible releases.
Library changes
Test new krkn-lib helpers in the library repository first, using the test
environment appropriate to their API behavior. Then install the changed
library into the custom Krkn base image and rerun the affected scenario.
This second check catches dependency-version mismatches and different
authentication or permission requirements. Updating the host’s virtual
environment does not update a previously built container image.
Community image packaging
When contributing to krkn-hub, follow its template generation and Compose build
workflow. Build the affected scenario first, inspect its labels, and run the
same container and operator checks above. A collection-wide build is not a
prerequisite for testing a standalone custom scenario.
Use the client’s private registry support to discover and exercise the
published scenario. You should not need to edit krknctl’s embedded registry
configuration merely to add an image that uses existing input types.
When changing the type system or client implementation itself, add tests in
the corresponding krknctl packages and rebuild the client. Validate the
image contract against both CLI and
console behavior when the change affects their shared metadata.
7 - Adding Tests to Krkn
Add focused unit tests and functional coverage to the Krkn test suites.
This guide covers how to add both unit tests and functional tests to the krkn project. Tests are essential for ensuring code quality and preventing regressions.
Unit Tests
Unit tests in krkn are located in the tests/ directory and use Python’s unittest framework with comprehensive mocking to avoid requiring external dependencies like cloud providers or Kubernetes clusters.
Creating a Unit Test
1. File Location and Naming
Place your test file in the tests/ directory with the naming pattern test_<feature>.py:
#!/usr/bin/env python3"""
Test suite for <Feature Name>
IMPORTANT: These tests use comprehensive mocking and do NOT require any external
infrastructure, cloud credentials, or Kubernetes cluster. All API calls are mocked.
Test Coverage:
- Feature 1: Description
- Feature 2: Description
Usage:
# Run all tests
python -m unittest tests.test_<your_feature> -v
# Run with coverage
python -m coverage run -a -m unittest tests/test_<your_feature>.py -v
Assisted By: Claude Code
"""importunittestfromunittest.mockimportMagicMock,patch,Mock# Import the classes you're testingfromkrkn.scenario_plugins.<module>importYourClassclassTestYourFeature(unittest.TestCase):"""Test cases for YourClass"""defsetUp(self):"""Set up test fixtures before each test"""# Mock environment variables if neededself.env_patcher=patch.dict('os.environ',{'API_KEY':'test-api-key','API_URL':'https://test.example.com'})self.env_patcher.start()# Mock external dependenciesself.mock_client=MagicMock()# Create instance to testself.instance=YourClass()deftearDown(self):"""Clean up after each test"""self.env_patcher.stop()deftest_success_scenario(self):"""Test successful operation"""# Arrange: Set up test dataexpected_result="success"# Act: Call the method being testedresult=self.instance.your_method()# Assert: Verify the resultself.assertEqual(result,expected_result)deftest_failure_scenario(self):"""Test failure handling"""# Arrange: Set up failure conditionself.mock_client.some_method.side_effect=Exception("API Error")# Act & Assert: Verify exception is handledwithself.assertRaises(Exception):self.instance.your_method()if__name__=='__main__':unittest.main()
3. Best Practices for Unit Tests
Use Comprehensive Mocking: Mock all external dependencies (cloud APIs, Kubernetes, file I/O)
Add IMPORTANT Note: Include a note in the docstring that tests don’t require credentials
Document Test Coverage: List what scenarios each test covers
Organize Tests by Category: Use section comments like # ==================== Core Tests ====================
Test Edge Cases: Include tests for timeouts, missing parameters, API exceptions
Use Descriptive Names: Test names should clearly describe what they test
4. Running Unit Tests
# Run all unit testspython -m unittest discover -s tests -v
# Run specific test filepython -m unittest tests.test_your_feature -v
# Run with coveragepython -m coverage run -a -m unittest discover -s tests -v
python -m coverage report
Functional Tests
Functional tests in krkn are integration tests that run complete chaos scenarios against a real Kubernetes cluster (typically KinD in CI). They are located in the CI/tests/ directory.
Understanding the Functional Test Structure
CI/
├── run.sh # Main test runner
├── run_test.sh # Individual test executor
├── config/
│ ├── common_test_config.yaml # Base configuration template
│ └── <scenario>_config.yaml # Generated configs per scenario
├── tests/
│ ├── common.sh # Common helper functions
│ ├── functional_tests # List of tests to run
│ └── test_*.sh # Individual test scripts
└── out/
└── <test_name>.out # Test output logs
Adding a New Functional Test
Step 1: Create the Test Script
Create a new test script in CI/tests/ following the naming pattern test_<scenario>.sh:
#!/bin/bash
set -xeEo pipefail
source CI/tests/common.sh
trap error ERR
trap finish EXIT
function functional_test_<your_scenario> {# Set environment variables for the scenarioexportscenario_type="<scenario_type>"exportscenario_file="scenarios/kind/<scenario_file>.yml"exportpost_config=""# Generate config from template with variable substitution envsubst < CI/config/common_test_config.yaml > CI/config/<your_scenario>_config.yaml
# Optional: View the generated config cat CI/config/<your_scenario>_config.yaml
# Run kraken with coverage python3 -m coverage run -a run_kraken.py -c CI/config/<your_scenario>_config.yaml
# Success messageecho"<Your Scenario> scenario test: Success"# Optional: Verify expected state date
kubectl get pods -n <namespace> -l <label>=<value> -o yaml
}# Execute the test functionfunctional_test_<your_scenario>
Step 2: Create or Reference Scenario File
Ensure your scenario YAML file exists in scenarios/kind/:
Check cluster state: kubectl get pods --all-namespaces
Review kraken logs: Look for error messages in the output
Verify configuration: Ensure variables are properly substituted in generated config
Summary
Unit Tests: Located in tests/, use comprehensive mocking, require no external dependencies
Functional Tests: Located in CI/tests/, run against real Kubernetes, test full scenarios
Test Execution: Unit tests via unittest, functional tests via CI/run.sh
Coverage: Both test types contribute to code coverage metrics
CI Integration: All tests run automatically in GitHub Actions
8 - Publishing Through Krkn-hub
Add a scenario to the community image collection after validating its standalone image contract.
Krkn-hub maintains the community collection of packaged scenario images. Add
your scenario there when you want it distributed through that collection.
The custom scenario lab builds and
publishes its own image directly, so it does not require changes to this
repository.
Before contributing packaging, validate the Python plugin, configuration
template, metadata, and executable startup script. Use the
testing guide to cover success and
failure paths.
Add the scenario package
Create a scenario directory in
krkn-hub using a neighboring scenario
as a reference. A package normally includes:
A scenario YAML template whose configurable values come from environment
variables.
env.sh with documented runtime defaults.
An executable run.sh that prepares configuration and starts Krkn.
krknctl-input.json describing the scenario-specific inputs.
Dockerfile.template with the scenario labels and required files.
Keep the names consistent across the JSON descriptor’s variable, shell
defaults, YAML placeholders, and plugin configuration fields. The template file
name used by run.sh must match the actual file copied into the image.
The base image must already contain the Python plugin and its dependencies.
Packaging a new scenario type on a base image that predates the plugin will
produce a discoverable image that fails at engine startup.
Connect the build
Follow the build conventions in the checked-out krkn-hub revision. In the
repository’s template and Compose workflow, this includes registering the
scenario directory in build.sh and adding its service to docker-compose.yaml.
Use generated Dockerfiles in the build and confirm that the input JSON is
embedded in the final image labels.
Only add shared defaults or main-configuration variables to the root templates
when the behavior is shared across scenarios. A scenario-specific field belongs in
its own package.
See the repository’s build script
and application outages package
for the current layout. Avoid instructions that depend on fixed source line
numbers; the surrounding scripts evolve.
Complete the contribution
Build and test the selected scenario image before running a collection-wide
build. Confirm that it works through the intended clients and that runtime
permissions match the scenario’s needs. Official release publishing also
includes the project’s image signing and distribution process.
Add a scenario page under content/en/docs/scenarios/ in the website and follow
the website’s scenario marker conventions. Document inputs, target permissions,
expected disruption, recovery behavior, and the meaningful success condition.
Use the contribution guidelines for the
related code and documentation pull requests. Changes to krknctl itself are
needed when extending client behavior or the type system, rather than simply
introducing another image that uses the existing contract.
9 - Health Check Plugins
How to use and create health check plugins in Krkn to monitor services during chaos experiments
Health check plugins allow Krkn to continuously monitor the health of your services and infrastructure during chaos experiments. They run in background threads alongside the chaos scenario, detecting outages, tracking downtime duration, and collecting telemetry data.
Overview
The health check system uses a plugin architecture:
HealthCheckFactory — automatically discovers and loads all plugins from the krkn.health_checks package
AbstractHealthCheckPlugin — base class all plugins must extend
Plugins run in separate threads and write telemetry to a shared queue
The factory tracks all active plugin instances and provides lifecycle management (increment_all_iterations, stop_all)
Exit Codes
Code
Meaning
0
Success — all health checks passed
2
Critical alert detected during the run
3
Health check failure (e.g. exit_on_failure: true triggered)
Monitors HTTP endpoints by making periodic GET requests. Tracks status changes, measures downtime duration, and records telemetry for each state transition.
Configuration example:
health_checks:interval:2# seconds between checksconfig:- url:"http://my-service/health"verify_url: true # SSL certificate verification (default:true)exit_on_failure: false # exit with code 3 if endpoint goes down (default:false)- url:"https://api.example.com/status"bearer_token:"your-token"# Authorization: Bearer <token>exit_on_failure:true- url:"http://internal-service"auth:"username,password"# HTTP basic authverify_url:false
Monitors KubeVirt VirtualMachineInstance (VMI) connectivity during chaos experiments. It tracks SSH/network access to VMs, detects disconnections, and records recovery data.
Configuration example:
kubevirt_checks:interval:5config:namespace:"my-namespace"node_name:"worker-1"# optional: filter VMIs by nodeexit_on_failure:falsedisconnected_mode:false# track VMs that become unreachableonly_failures:true# only record failed checks in telemetrybatch_size:10# VMIs to check concurrently (0 = no limit)ssh_port:22ssh_user:"cloud-user"ssh_private_key:"/path/to/key"
Creating a Custom Health Check Plugin
You can add your own health check plugin by following these steps.
1. Naming Conventions
The factory enforces strict naming conventions for auto-discovery. Both the file name and class name must follow these rules:
Rule
Example
File must be in krkn/health_checks/
krkn/health_checks/
File name must end with _health_check_plugin.py
my_service_health_check_plugin.py
Class name must be CapitalCamelCase of the file name
MyServiceHealthCheckPlugin
Class name must end with HealthCheckPlugin
MyServiceHealthCheckPlugin
Class must inherit from AbstractHealthCheckPlugin
—
The snake_case-to-CamelCase conversion is automatic — my_service_health_check_plugin becomes MyServiceHealthCheckPlugin. The factory will reject your plugin if these rules are not followed.
importloggingimportqueueimporttimefromtypingimportAnyfromkrkn.health_checks.abstract_health_check_pluginimportAbstractHealthCheckPluginclassMyServiceHealthCheckPlugin(AbstractHealthCheckPlugin):"""
Health check plugin that monitors MyService during chaos experiments.
"""def__init__(self,health_check_type:str="my_service_health_check",iterations:int=1,**kwargs):super().__init__(health_check_type)self.iterations=iterationsself.current_iterations=0defget_health_check_types(self)->list[str]:"""
Returns the internal type identifiers for this plugin.
One plugin can handle multiple type strings, but they must be
unique across all plugins.
"""return["my_service_health_check"]defget_config_key(self)->str:"""
Returns the top-level config.yaml key this plugin reads from.
The factory maps this key to the plugin so run_kraken.py discovers
and starts it automatically — no code changes needed there.
Must be unique across all plugins.
"""return"my_service_checks"defincrement_iterations(self)->None:"""
Called by the main run loop after each chaos iteration.
Increment your counter so the plugin knows when to stop.
"""self.current_iterations+=1defrun_health_check(self,config:dict[str,Any],telemetry_queue:queue.Queue,)->None:"""
Main health check loop. Runs in a background thread.
- Check `self._stop_event.is_set()` to support cooperative shutdown
- Check `self.current_iterations < self.iterations` to stop after the run
- Put telemetry results into `telemetry_queue`
- Set `self.ret_value = 3` to signal health check failure
"""ifnotconfigornotconfig.get("config"):logging.info("my_service_health_check config not defined, skipping")returninterval=config.get("interval",5)exit_on_failure=config.get("config",{}).get("exit_on_failure",False)endpoint=config.get("config",{}).get("endpoint")telemetry_results=[]whileself.current_iterations<self.iterationsandnotself._stop_event.is_set():healthy=self._check_my_service(endpoint)ifnothealthy:logging.warning(f"MyService at {endpoint} is unhealthy")ifexit_on_failureandself.ret_value==0:self.ret_value=3# Collect telemetry (structure depends on your needs)telemetry_results.append({"endpoint":endpoint,"status":healthy,})time.sleep(interval)# Always put results into the queue when donetelemetry_queue.put(telemetry_results)def_check_my_service(self,endpoint:str)->bool:"""Check if the service is healthy. Returns True if healthy."""try:# Your service-specific health check logic herereturnTrueexceptExceptionase:logging.error(f"Health check failed: {e}")returnFalse
3. Configure in config.yaml
Add a section using the key returned by get_config_key(). The factory discovers this mapping at startup — no changes to run_kraken.py are needed:
Each plugin owns its own top-level config key. Multiple plugins can be active simultaneously, each reading from their own section:
health_checks:# read by HttpHealthCheckPlugininterval:2config:- url:"http://frontend/health"my_service_checks:# read by MyServiceHealthCheckPlugininterval:5config:endpoint:"http://my-service:8080"
4. AbstractHealthCheckPlugin API Reference
Your plugin inherits the following from AbstractHealthCheckPlugin:
Member
Type
Description
self._stop_event
threading.Event
Set when the main loop requests shutdown. Check self._stop_event.is_set() in your loop.
self.ret_value
int
Return code. 0 = success, 3 = health check failure.
stop()
method
Called by the factory to signal your loop to exit. Do not override — check _stop_event instead.
get_return_value()
method
Returns self.ret_value. Used by the main loop to detect failures.
set_return_value(value)
method
Sets self.ret_value.
Methods you must implement:
Method
Description
run_health_check(config, telemetry_queue)
Main health check loop, runs in a background thread
get_health_check_types()
Returns list of internal type identifier strings for this plugin
get_config_key()
Returns the top-level config.yaml key this plugin reads from (must be unique)
increment_iterations()
Increments your iteration counter when called by the factory
5. Factory Auto-Discovery
The HealthCheckFactory uses pkgutil.walk_packages to scan the krkn.health_checks package at startup. Any file ending in _health_check_plugin.py that contains a class following the naming conventions will be automatically loaded. No registration step is needed beyond placing the file in the right directory.
You can verify your plugin was loaded by checking the factory’s loaded_plugins dict:
If your plugin fails to load (naming violation, import error, duplicate type), it will appear in factory.failed_plugins as a list of (module_name, class_name, error_message) tuples.
Questions?
For questions or guidance, reach out on the Kubernetes Slack in the #krkn channel.
