This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Build a Custom Scenario

Build, package, publish, and run a small custom Krkn plugin with Podman, Kind, and Krkn Operator.

Build a scenario that waits, lists pods in a namespace, and returns a clear result. Start with a plugin skeleton in its own repository, add two methods, package the plugin in a Krkn image, and expose it to Krkn Operator through a private registry running inside Kind.

Get the lab files: krkn-chaos/krkn-dev-demo. The commands below pin revision bf2ea8d, including the executable-script and containerd configuration fixes used in this walkthrough.

This lab focuses on the integration contract. The scenario reads Kubernetes resources and does not disrupt workloads. You provide the target cluster and register it with the operator separately; this lab creates only the Kind management cluster that hosts the operator and registry.

What you will build

  • DevDemoScenarioPlugin, discovered as dev_demo_scenarios.
  • A custom Krkn base image containing that plugin.
  • A scenario image exposing wait and namespace through OCI labels.
  • An authenticated registry, installed with Helm inside Kind.
  • An operator run whose logs show the development banner and target pod names.

The plugin is built directly into the custom base image. There is no krkn-hub checkout in this workflow, and the image metadata supplies the client-facing parameter definitions.

Before you begin

Install Git, Podman, Kind, kubectl, Helm 3, jq, envsubst from GNU gettext, OpenSSL, and an htpasswd implementation supporting bcrypt (-B). Have Python 3 available for the optional source syntax check. On macOS, start Podman Machine and use a rootful Podman connection for this Kind lab.

You also need:

  • Internet access to clone the source and pull the published images and charts.
  • A running cluster and kubeconfig for the local container test. If it is a Kind cluster, know its name from kind get clusters.
  • Free host ports 5001 for the registry and 3000 for the console forward.
  • An operator target whose Kubernetes API is reachable from the management cluster’s scenario Pods. Registering that target is outside this lab.

The examples use Kind node image kindest/node:v1.32.2 with containerd 2, Docker Registry chart 3.0.0, and Krkn Operator chart 1.1.2. These are the lab’s version choices, not a claim that they are the latest releases. The companion repository is pinned below so the permission and registry fixes demonstrated here are included.

Prepare the workspace

Run these commands in one shell and keep its exported variables for the whole lab. Paths are absolute, so later commands do not depend on the current working directory. Start from /tmp in case a previous shell was left inside a removed checkout.

cd /tmp

export KRKN_DIR=/tmp/krkn
export KRKN_DEV_PLUGIN_DIR="$KRKN_DIR/krkn/scenario_plugins/dev_demo"
export KRKN_BASE_IMAGE=krkn-dev-demo:base
export KRKN_SCENARIO_IMAGE=krkn-dev-demo:scenario
export KIND_EXPERIMENTAL_PROVIDER=podman
export KRKN_DEMO_REVISION=bf2ea8d920de5a6e5d8ea26a8d705ff600bc6684

git clone https://github.com/krkn-chaos/krkn.git "$KRKN_DIR"
git clone https://github.com/krkn-chaos/krkn-dev-demo.git "$KRKN_DEV_PLUGIN_DIR"
git -C "$KRKN_DEV_PLUGIN_DIR" checkout "$KRKN_DEMO_REVISION"

Use an unused KRKN_DIR. If /tmp/krkn already contains work, choose another absolute path before cloning. Do not remove an existing checkout to make the command succeed.

The pinned plugin repository is a separate checkout nested inside Krkn’s plugin directory. It contains:

dev_demo/
├── __init__.py
├── dev_demo_scenario_plugin.py  # helper implemented; abstract methods left for you
├── scenario.yaml               # example plugin input
├── scenario.yaml.template      # environment-to-YAML template
├── env.sh                      # container defaults
├── run.sh                      # render configuration and start Krkn
├── krknctl-input.json           # user-facing input descriptors
├── Dockerfile.base             # add the plugin to a published Krkn runtime
├── Dockerfile.template         # package the scenario contract
├── kind-config.yml             # registry port and containerd hosts directory
└── registry-values.yml         # authenticated registry Helm values

The cloned Krkn source lets you inspect its interface and loader. The base image build copies the plugin into the published Krkn runtime; it does not rebuild the rest of the cloned engine. Pin the published base image as well when turning this workshop into a repeatable release pipeline.

Follow the lab

  1. Implement the plugin.
  2. Package and test the image.
  3. Publish and run with Krkn Operator.

Keep Troubleshooting open while working. It covers the failures encountered while developing this demo, including successful discovery followed by an HTTPS image pull error.

1 - Implement the Plugin

Complete the abstract interface and connect a scenario YAML file to krkn-lib operations.

Starting point: complete the workspace setup. Result: a concrete plugin class that Krkn can discover and execute.

Read the prepared behavior

Open $KRKN_DEV_PLUGIN_DIR/dev_demo_scenario_plugin.py. Its DevDemoScenarioPlugin class already implements execute_demo(scenario, lib_telemetry). The method:

  1. Reads the YAML file identified by scenario.
  2. Requires wait to be an integer from 0 to 60, and namespace to be a nonempty string. A YAML boolean is not accepted as an integer.
  3. Sleeps for the requested duration.
  4. Obtains the Kubernetes helper from lib_telemetry and checks namespace existence with list_namespaces().
  5. Lists pods with list_pods(namespace) and logs a distinctive banner.
  6. Returns 0 on success and 1 on a missing namespace or another failure.

The input file is intentionally small:

wait: 2
namespace: default

The helper uses the cluster client already initialized by Krkn:

            kubernetes = lib_telemetry.get_lib_kubernetes()
            if namespace not in kubernetes.list_namespaces():
                logging.error(
                    "Development scenario: namespace '%s' does not exist", namespace
                )
                return 1

            pods = kubernetes.list_pods(namespace)

Checking the namespace list avoids intentionally triggering the library’s verbose read_namespace_status error path for a missing namespace. This demo therefore requires permission to list namespaces as well as pods. An existing namespace with no pods is a successful result and is logged as (none).

Implement the two abstract methods

Ensure these imports are present once at the top of the file:

from krkn_lib.models.telemetry import ScenarioTelemetry
from krkn_lib.telemetry.ocp import KrknTelemetryOpenshift

Paste the following methods inside DevDemoScenarioPlugin, at the same indentation level as execute_demo(). The four-space class indentation is included for copying:

    def get_scenario_types(self) -> list[str]:
        return ["dev_demo_scenarios"]

    def run(
        self,
        run_uuid: str,
        scenario: str,
        lib_telemetry: KrknTelemetryOpenshift,
        scenario_telemetry: ScenarioTelemetry,
    ) -> int:
        return self.execute_demo(scenario, lib_telemetry)

get_scenario_types() supplies the configuration key that selects this class. run() delegates to the helper, which contains the validation and exception handling. The framework supplies the run identifier and telemetry record even though this small helper does not need to use them directly.

The class remains abstract until both methods are implemented. Avoid renaming the package to dev_demo_plugin or dev_demo_scenario: those directory names violate the factory’s naming rules. The module is dev_demo_scenario_plugin.py and its class is DevDemoScenarioPlugin.

Understand the configuration binding

The main Krkn configuration selects the type and its scenario file:

kraken:
  chaos_scenarios:
    - dev_demo_scenarios:
        - scenarios/dev_demo.yaml

The engine calls run(scenario="scenarios/dev_demo.yaml", ...). The plugin opens that file and reads wait and namespace. The scenario type, file path, and field names serve different purposes; changing one does not automatically change the others.

For a source-based Krkn run, copy the sample and derive a dedicated main config:

cp "$KRKN_DEV_PLUGIN_DIR/scenario.yaml" "$KRKN_DIR/scenarios/dev_demo.yaml"
cp "$KRKN_DIR/config/config.yaml" "$KRKN_DIR/config/dev_demo_config.yaml"

Set the kraken.chaos_scenarios section in dev_demo_config.yaml to the example above. The next part packages this same binding in the image’s startup script, so a source-based run is optional.

Check before building

python3 -m py_compile "$KRKN_DEV_PLUGIN_DIR/dev_demo_scenario_plugin.py"

The plugin keeps a nonzero result meaningful: an invalid input or missing namespace must not be converted into success just to keep the container running. For a real chaos action, add restoration and rollback behavior appropriate to the resources it changes.

Next: Package and test the image.

2 - Package and Test the Image

Build a custom Krkn base and a standalone scenario image, then verify its metadata and runtime behavior.

Starting point: the plugin methods are implemented. Result: a locally tested image that clients can discover through its labels.

Build the custom Krkn base

Dockerfile.base overlays the local Python package on the published runtime:

FROM quay.io/krkn-chaos/krkn:latest

COPY --chown=krkn:krkn __init__.py \
    /home/krkn/kraken/krkn/scenario_plugins/dev_demo/__init__.py
COPY --chown=krkn:krkn dev_demo_scenario_plugin.py \
    /home/krkn/kraken/krkn/scenario_plugins/dev_demo/dev_demo_scenario_plugin.py
podman build \
  -f "$KRKN_DEV_PLUGIN_DIR/Dockerfile.base" \
  -t "$KRKN_BASE_IMAGE" \
  "$KRKN_DEV_PLUGIN_DIR"

This build context is the plugin repository. Using Krkn’s upstream container Dockerfile unchanged would clone Krkn from GitHub during the build and omit the local plugin. If you change the core engine or krkn-lib, build a runtime that includes those changes explicitly.

Inspect the scenario contract

The image has two user inputs in krknctl-input.json:

[
  {
    "name": "wait",
    "short_description": "Wait (seconds)",
    "description": "Seconds to wait before listing pods in the namespace (0 to 60).",
    "variable": "WAIT_SECONDS",
    "type": "number",
    "required": "false"
  },
  {
    "name": "namespace",
    "short_description": "Namespace",
    "description": "Kubernetes namespace whose pod names will be logged.",
    "variable": "NAMESPACE",
    "type": "string",
    "required": "true",
    "validator": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$",
    "validation_message": "namespace must be a Kubernetes namespace name"
  }
]

name identifies the input presented by a client; variable is the environment variable injected into the container. The string values "true" and "false" match krknctl’s field descriptor format. The plugin still enforces its own integer range: a generic number field alone does not restrict input to an integer from 0 to 60.

Dockerfile.template embeds the JSON in krknctl.input_fields, together with the other labels included in the companion image:

LABEL krknctl.is_a_scenario="true"
LABEL krknctl.privileged="false"
LABEL krknctl.has_rollback="false"
LABEL krknctl.kubeconfig_path="/home/krkn/.kube/config"
LABEL krknctl.title="Dev Demo"
LABEL krknctl.description="Waits, then logs pod names from one Kubernetes namespace."
LABEL krknctl.input_fields='$KRKNCTL_INPUT'

The same Dockerfile copies the startup script with an explicit executable bit:

COPY --chmod=0755 --chown=krkn:krkn run.sh /home/krkn/run.sh
ENTRYPOINT ["/bin/bash", "/home/krkn/run.sh"]

This permission is required because the operator invokes /home/krkn/run.sh directly. A local entrypoint that passes the script to bash can conceal a missing executable bit.

Follow the runtime configuration

env.sh supplies defaults of 2 seconds and namespace default for direct container execution. The console still requires the namespace input declared in the metadata.

scenario.yaml.template contains:

wait: ${WAIT_SECONDS}
namespace: ${NAMESPACE}

run.sh changes to /home/krkn/kraken and uses an explicit substitution list:

envsubst '$WAIT_SECONDS $NAMESPACE' \
  < scenarios/scenario.yaml.template > scenarios/dev_demo.yaml

It derives config/dev_demo_config.yaml from the base configuration, sets kraken.kubeconfig_path from KUBECONFIG, and selects only dev_demo_scenarios. For this focused demo, telemetry export, event backup, reports, resiliency scoring, and the inter-scenario stabilization delay are disabled. These are demo settings, not a requirement for custom plugins.

Finally it hands control to Krkn while preserving the process exit status:

exec ./containers/entrypoint.sh --config=config/dev_demo_config.yaml

Build and inspect the scenario image

Compact the JSON, expand only its build-time placeholder, and select the custom base image in the generated Dockerfile. The runtime placeholders in the YAML and shell files remain untouched.

export KRKNCTL_INPUT="$(jq -c . "$KRKN_DEV_PLUGIN_DIR/krknctl-input.json")"
envsubst '${KRKNCTL_INPUT}' \
  < "$KRKN_DEV_PLUGIN_DIR/Dockerfile.template" |
  sed "1s|^FROM .*|FROM $KRKN_BASE_IMAGE|" \
  > "$KRKN_DEV_PLUGIN_DIR/Dockerfile"

podman build \
  -f "$KRKN_DEV_PLUGIN_DIR/Dockerfile" \
  -t "$KRKN_SCENARIO_IMAGE" \
  "$KRKN_DEV_PLUGIN_DIR"

podman image inspect "$KRKN_SCENARIO_IMAGE" |
  jq '.[0].Config.Labels["krknctl.input_fields"] | fromjson'
podman run --rm --user 1001:1001 --entrypoint /bin/bash \
  "$KRKN_SCENARIO_IMAGE" -c 'test -x /home/krkn/run.sh'

Run against an existing Kind cluster

Set KRKN_TEST_CLUSTER to a running cluster you supply. This test does not create or register an operator target.

Kind’s host kubeconfig normally uses a loopback address. A container on the kind network needs the internal API address instead. Save that kubeconfig under your home directory so Podman Machine on macOS can mount it. The temporary directory is private; the mounted file is readable by container UID 1001.

export KRKN_TEST_CLUSTER=kind
export KRKN_TEST_NAMESPACE=default

(
  set -e
  mkdir -p "$HOME/.kube"
  KRKN_TEST_CONFIG_DIR="$(mktemp -d "$HOME/.kube/krkn-dev-demo.XXXXXX")"
  trap 'rm -rf "$KRKN_TEST_CONFIG_DIR"' EXIT
  kind get kubeconfig --internal --name "$KRKN_TEST_CLUSTER" \
    > "$KRKN_TEST_CONFIG_DIR/config"
  chmod 644 "$KRKN_TEST_CONFIG_DIR/config"

  podman run --rm --user 1001:1001 --network kind \
    -e WAIT_SECONDS=2 \
    -e NAMESPACE="$KRKN_TEST_NAMESPACE" \
    -e KUBECONFIG=/home/krkn/.kube/config \
    -v "$KRKN_TEST_CONFIG_DIR/config:/home/krkn/.kube/config:ro" \
    --entrypoint /bin/bash \
    "$KRKN_SCENARIO_IMAGE" -c 'exec /home/krkn/run.sh'
)

On a Linux host with SELinux enforcing, use :ro,Z for this dedicated temporary kubeconfig mount if a label denial prevents access.

For a non-Kind target, use a kubeconfig whose API endpoint is reachable from the container. Replace kind get kubeconfig ... with kubectl --kubeconfig "$KUBECONFIG" config view --raw --flatten --minify and omit --network kind. Flattening embeds certificate files; an external credential plugin still needs to be available in the image.

Check both outcomes

For an existing namespace, expect container exit status 0 and a log block:

##### DEVELOPMENT SCENARIO #####
Namespace: default
Pods:
  (none)
##### END DEVELOPMENT SCENARIO #####

Pod names replace (none) when the namespace contains pods. Set KRKN_TEST_NAMESPACE=dev-demo-missing-namespace to a namespace you have confirmed does not exist and repeat the test. Expect a nonzero exit status and a concise missing-namespace error. Do not create that namespace as part of this failure-path test.

Next: Publish and run with Krkn Operator.

3 - Publish and Run with Krkn Operator

Host an authenticated registry inside Kind, configure containerd, and run the custom scenario through Krkn Operator.

Starting point: the scenario image passes its local tests. Result: the operator discovers the image, renders its two inputs, and runs it against a target you have already registered.

The management cluster hosts the registry, operator, and scenario Pods. The target’s kubeconfig is mounted into each scenario Pod. You do not need to place the private registry on the target or configure the target’s runtime to pull this scenario image.

Podman pushes to localhost port 5001, which maps to the Kind registry NodePort 30500. Inside Kind, the operator reads metadata and containerd pulls image layers through separate HTTP connections to the stable node hostname. The scenario Pod then uses a mounted kubeconfig to reach the target API.

One registry, two client configurations. Select the diagram to view it at full size.

1. Create the management cluster

Use a dedicated name so an existing Kind cluster is preserved:

export KIND_EXPERIMENTAL_PROVIDER=podman
export KRKN_MGMT_CLUSTER=krkn-dev-demo-mgmt
export KRKN_MGMT_CONTEXT="kind-$KRKN_MGMT_CLUSTER"
export KRKN_OPERATOR_NAMESPACE=krkn-operator-system

kind create cluster --name "$KRKN_MGMT_CLUSTER" \
  --config "$KRKN_DEV_PLUGIN_DIR/kind-config.yml"
kubectl --context "$KRKN_MGMT_CONTEXT" get nodes

The repository’s kind-config.yml contains:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
  - |-
    [plugins."io.containerd.cri.v1.images".registry]
      config_path = "/etc/containerd/certs.d"
nodes:
  - role: control-plane
    image: kindest/node:v1.32.2
    extraPortMappings:
      - containerPort: 30500
        hostPort: 5001
        listenAddress: "127.0.0.1"
        protocol: TCP

The port mapping is created with the node container. Editing this file does not change an existing cluster. The containerd plugin key above is for the containerd 2 runtime in this node image.

2. Install the authenticated registry

Keep generated credentials outside the checkout. Run the following in the same shell so the registry credentials remain available for the push and UI setup:

export KRKN_DEMO_STATE_DIR="$(mktemp -d /tmp/krkn-dev-demo.XXXXXX)"
chmod 700 "$KRKN_DEMO_STATE_DIR"
export REGISTRY_USER=demo
export REGISTRY_PASSWORD="$(openssl rand -hex 16)"
htpasswd -Bbn "$REGISTRY_USER" "$REGISTRY_PASSWORD" \
  > "$KRKN_DEMO_STATE_DIR/registry.htpasswd"
chmod 600 "$KRKN_DEMO_STATE_DIR/registry.htpasswd"

helm repo add twuni https://twuni.github.io/docker-registry.helm
helm repo update twuni
helm upgrade --install demo-registry twuni/docker-registry \
  --version 3.0.0 \
  --kube-context "$KRKN_MGMT_CONTEXT" \
  --namespace registry --create-namespace \
  --values "$KRKN_DEV_PLUGIN_DIR/registry-values.yml" \
  --set-file "secrets.htpasswd=$KRKN_DEMO_STATE_DIR/registry.htpasswd" \
  --wait --timeout 5m

The Twuni chart deploys Docker Distribution. The demo values select NodePort 30500, service port 5000, and persistence.enabled: false. Storage is disposable: replacing the registry Pod or deleting the cluster can remove the pushed images. Use TLS and durable storage for a registry outside this local lab.

Verify authentication through the host port:

curl --fail --silent --show-error \
  -u "$REGISTRY_USER:$REGISTRY_PASSWORD" http://localhost:5001/v2/

Expect {}. An unauthenticated request should receive HTTP 401.

3. Configure image pulls on the management node

Use the node’s name on the Podman kind network as the registry address inside the cluster. This avoids embedding an IP that can change after a Podman Machine restart. The host continues to push through localhost:5001.

export KRKN_MGMT_NODE="$(kind get nodes --name "$KRKN_MGMT_CLUSTER" | head -n 1)"
export KRKN_REGISTRY_URL="$KRKN_MGMT_NODE:30500"
printf 'Registry URL for the operator: %s\n' "$KRKN_REGISTRY_URL"

for node in $(kind get nodes --name "$KRKN_MGMT_CLUSTER"); do
  registry_dir="/etc/containerd/certs.d/$KRKN_REGISTRY_URL"
  podman exec "$node" mkdir -p "$registry_dir"
  printf 'server = "http://%s"\n[host."http://%s"]\n  capabilities = ["pull", "resolve"]\n' \
    "$KRKN_REGISTRY_URL" "$KRKN_REGISTRY_URL" |
    podman exec -i "$node" sh -c "cat > '$registry_dir/hosts.toml'"
done

The fresh cluster already has config_path enabled. Changes to the hosts files do not require a containerd restart. For a cluster created with an older config, follow Repair an existing node before retrying a pull.

The registry URL, hosts.toml directory name, and operator pull secret must use the same host and port. The operator’s Allow Insecure HTTP option affects its registry API requests; this node configuration controls the actual pull. Skip TLS Verification only changes certificate checks and does not enable HTTP. See the containerd registry documentation.

Check that cluster Pods can resolve and reach the chosen hostname:

kubectl --context "$KRKN_MGMT_CONTEXT" -n registry run dev-demo-registry-probe \
  --image=curlimages/curl:8.9.1 --restart=Never --attach=true --rm \
  --command -- curl --silent --show-error --output /dev/null \
  --write-out 'HTTP %{http_code}\n' "http://$KRKN_REGISTRY_URL/v2/"

Expect HTTP 401: this probe intentionally sends no credentials. If hostname resolution fails, fix the Kind/Podman network before registering the URL.

4. Push the scenario tag

export KRKN_PRIVATE_SCENARIO=localhost:5001/demo/krkn-hub:dev-demo
podman tag "$KRKN_SCENARIO_IMAGE" "$KRKN_PRIVATE_SCENARIO"
printf '%s' "$REGISTRY_PASSWORD" | podman login \
  --tls-verify=false --username "$REGISTRY_USER" --password-stdin localhost:5001
podman push --tls-verify=false "$KRKN_PRIVATE_SCENARIO"

curl --fail --silent --show-error -u "$REGISTRY_USER:$REGISTRY_PASSWORD" \
  http://localhost:5001/v2/demo/krkn-hub/tags/list

The response should list dev-demo. The repository is demo/krkn-hub; the tag is the scenario identity. The operator accesses the same content as $KRKN_REGISTRY_URL/demo/krkn-hub:dev-demo, using the internal address.

5. Install Krkn Operator

helm upgrade --install krkn-operator \
  oci://quay.io/krkn-chaos/charts/krkn-operator \
  --version 1.1.2 \
  --kube-context "$KRKN_MGMT_CONTEXT" \
  --namespace "$KRKN_OPERATOR_NAMESPACE" --create-namespace \
  --wait --timeout 10m

kubectl --context "$KRKN_MGMT_CONTEXT" \
  -n "$KRKN_OPERATOR_NAMESPACE" get pods,services

In another terminal, start the console forward using explicit values so it does not depend on exports from the first shell:

kubectl --context kind-krkn-dev-demo-mgmt -n krkn-operator-system \
  port-forward service/krkn-operator-console 3000:3000

Open http://localhost:3000, create the first administrator account, and sign in. Keep the forward running. If you changed the cluster or namespace names, use those names in the second terminal as well.

6. Register the image source

In Settings → Private Registries, add:

  • Name: dev-demo-registry.
  • Registry URL: the value of $KRKN_REGISTRY_URL, without http://.
  • Scenario Repository: demo/krkn-hub.
  • Authentication Type: Username & Password.
  • Username and password: the values of $REGISTRY_USER and $REGISTRY_PASSWORD, not their variable names.
  • Allow Insecure HTTP: enabled.
  • Skip TLS Verification: disabled; this local registry serves plain HTTP.
  • Available to All Users: enabled for this dedicated demo installation.

For the default cluster name, the URL is krkn-dev-demo-mgmt-control-plane:30500. The operator stores the credentials in a Kubernetes pull secret. If you edit the URL later, use the registry form so the stored URL and authentication host stay aligned.

The new custom image is unsigned. On the same settings tab, disable Image signature verification for this dedicated lab installation. This setting is operator-wide. Normal deployments should require signatures that the operator trusts; adding an arbitrary signature is not sufficient. For the official scenario collection, see mirroring signed images.

7. Run against your registered target

Provide a target through your usual operator setup. Its API must be reachable from a scenario Pod in the management cluster, and its credentials must allow listing namespaces and pods. This lab does not create or register the target.

In Run Scenario:

  1. Select your registered target.
  2. Choose the private registry dev-demo-registry.
  3. Select dev-demo (display title Dev Demo).
  4. Set Wait (seconds) to 2 and Namespace to an existing namespace.
  5. Review the configuration and start the run.

In Jobs, inspect the resolved image, status, and logs. Expect the ##### DEVELOPMENT SCENARIO ##### banner, the selected namespace’s pod list, and a successful result. An empty existing namespace is also successful.

Inspect the Kubernetes resources separately:

kubectl --context "$KRKN_MGMT_CONTEXT" \
  -n "$KRKN_OPERATOR_NAMESPACE" get krknscenarioruns,pods

For the selected krkn-job-... Pod, use kubectl describe pod to inspect pull events and kubectl logs to inspect plugin output. The current controller creates Pods directly; kubectl get jobs can legitimately return no resources.

Run again with a namespace you have confirmed is absent. Expect the plugin to return 1 with a concise error. That failure proves the input and exit status reached the plugin; it is different from a Pod that never started.

Iterate on the scenario

After editing plugin Python, rebuild the base and scenario images. For a startup-script, template, or metadata change, rebuild the scenario image. Repeat the tag and push commands, then start a new run and inspect its resolved digest. Existing Pods retain the digest they were created with.

Continue with Troubleshooting or Testing changes.

4 - Troubleshooting the Demo

Diagnose discovery, registry transport, image startup, kubeconfig, and plugin failures at the layer that owns them.

First determine whether the scenario was discovered, whether its image was pulled, and whether its process started. These are separate checkpoints. Kubernetes events explain scheduling and pull failures; container logs explain startup and plugin failures.

The scenario is visible, but the pull attempts HTTPS

Typical event:

http: server gave HTTP response to HTTPS client

The operator’s registry client successfully used HTTP to list metadata. Kubelet then asked containerd to pull the image, and containerd defaulted to HTTPS. Krknctl’s insecure setting does not configure the node runtime.

Check all three items on the management node where the scenario Pod runs:

  1. The effective containerd registry config_path is /etc/containerd/certs.d.
  2. /etc/containerd/certs.d/$KRKN_REGISTRY_URL/hosts.toml exists and uses http:// for both the server and host entry.
  3. The registry host and port exactly match the Pod’s image reference.
podman exec "$KRKN_MGMT_NODE" containerd --version
podman exec "$KRKN_MGMT_NODE" cat /etc/containerd/config.toml
podman exec "$KRKN_MGMT_NODE" \
  cat "/etc/containerd/certs.d/$KRKN_REGISTRY_URL/hosts.toml"

The lab uses this containerd 2 section:

[plugins."io.containerd.cri.v1.images".registry]
  config_path = "/etc/containerd/certs.d"

Containerd 1 uses plugins."io.containerd.grpc.v1.cri".registry instead. Match the runtime you actually run; copying the older plugin key into a containerd 2 configuration can leave the intended registry configuration inactive.

Repair an existing node

If the containerd 2 registry section above is absent, append it once and restart containerd on the affected demo node:

printf '\n[plugins."io.containerd.cri.v1.images".registry]\n  config_path = "/etc/containerd/certs.d"\n' |
  podman exec -i "$KRKN_MGMT_NODE" sh -c 'cat >> /etc/containerd/config.toml'
podman exec "$KRKN_MGMT_NODE" systemctl restart containerd

If the section already exists with a different value, edit that section rather than appending a duplicate TOML table. Then write the registry hosts file. Hosts-file edits are read dynamically once config_path is active.

A Pod already in ImagePullBackOff may take several minutes to retry. During the lab, restarting kubelet on the dedicated node cleared that wait:

podman exec "$KRKN_MGMT_NODE" systemctl restart kubelet

Waiting for the normal retry is also valid. Confirm a new Pulled event; an old HTTPS failure remains in the event history after a successful recovery.

See containerd registry configuration and registry host files.

The pull reaches HTTP but returns 401

Transport is now working; authentication still needs checking. Verify the saved registry credentials and the Pod’s imagePullSecrets. The authentication key in the secret must match the registry host and port in the image reference. A login to localhost:5001 in Podman does not automatically configure a Kubernetes pull secret for the node hostname.

Update credentials through Settings → Private Registries. Do not put passwords in hosts.toml or paste decoded pull secrets into logs. The operator supplies authentication through the pull secret; the node configuration supplies the transport.

/home/krkn/run.sh: Permission denied

The operator invokes /home/krkn/run.sh directly. A script with mode 0644 can work when passed to bash locally and still fail in the operator Pod.

Use this Dockerfile instruction, then rebuild and push the scenario image:

COPY --chmod=0755 --chown=krkn:krkn run.sh /home/krkn/run.sh

Verify the built artifact as the operator’s user:

podman run --rm --user 1001:1001 --entrypoint /bin/bash \
  "$KRKN_SCENARIO_IMAGE" -c 'test -x /home/krkn/run.sh'

Start a new run and inspect its digest. Fixing the file on the host does not change an already built image, and pushing a new tag value does not rewrite an existing Pod’s image reference.

Krkn looks for /root/.kube/config

Setting KUBECONFIG does not override an explicit kraken.kubeconfig_path in the Krkn YAML configuration. The demo’s startup script sets that configuration key from KUBECONFIG and defaults it to /home/krkn/.kube/config.

Align the mounted path, the environment variable, and the main Krkn configuration. In the operator, the mount destination comes from the run’s spec.kubeconfigPath or the configured default, /home/krkn/.kube/config in this lab. The image’s krknctl.kubeconfig_path label does not control the mount. For a custom path, update the run configuration and the startup script’s KUBECONFIG value together. Running the container as root can change a home-directory fallback, which is why the demo uses an explicit path and tests startup as UID 1001.

The kubeconfig works on the host but not in the container

127.0.0.1 inside a container refers to that container. For a Kind target, generate the kubeconfig with kind get kubeconfig --internal --name ... and attach the test container to the kind network. For another target, use an API endpoint reachable from the container and include the required certificates.

If the host API port changed after a Podman Machine restart, start the stopped Kind node and refresh the host kubeconfig:

podman start "$KRKN_MGMT_NODE"
kind export kubeconfig --name "$KRKN_MGMT_CLUSTER"

Use the stable node hostname for the registry URL. If an old registry entry contains a numeric IP, update its URL in the console and write a matching hosts.toml. Do not change the target’s kubeconfig just to repair a registry URL.

The plugin is not discovered

Inspect Krkn’s factory logs. Verify the dev_demo package and __init__.py, the dev_demo_scenario_plugin.py module, the DevDemoScenarioPlugin class, and both abstract methods. The configuration type must be dev_demo_scenarios and must not collide with another plugin.

If source edits appear to have no effect, rebuild the base image containing the Python plugin before rebuilding the scenario image. The upstream Krkn Dockerfile clones its own source and does not automatically include your local plugin checkout.

The image is pushed but missing from scenario selection

Check the selected registry, repository (demo/krkn-hub), and tag (dev-demo). Inspect the image for krknctl.is_a_scenario="true" and valid JSON in krknctl.input_fields. A repository path is not a scenario name.

An unsigned custom image can also be blocked by signature verification. The lab’s override is configured on the operator, not in the plugin or image label. Keep it scoped to the dedicated lab installation.

The process starts and reports a missing namespace

This is the demo’s expected failure path: image discovery, pull, startup, and configuration injection have already succeeded. Check the selected target and namespace. Namespace existence is evaluated in the target cluster, not necessarily in the management cluster hosting the Pod.

For an existing namespace, verify that the target credentials allow listing namespaces and pods. An existing namespace with no pods should return success.