Krkn Operator

• 1: Installation
• 2: Administration
• 2.1: Cluster Management
• 2.2: User Management
• 2.3: Registry Management
• 2.4: Provider Configuration
• 2.5: Elasticsearch Management
• 3: Usage
• 3.1: Jobs
• 3.2: Cluster Terminal
• 3.3: Run Scenarios
• 3.4: Chaos Studio
• 3.5: File Management
• 4: Releases & Roadmap
• 5: Feedback
• 6: OCM/ACM Compatibility

Krkn Operator is a Kubernetes-native platform to deploy, configure and orchestrate Chaos Engineering experiments through a web console. Instead of manually executing scenarios, the Operator provides a complete platform to manage users, clusters, permissions and experiment execution from a single place.

---

Multi-Cluster Chaos Testing

Multi-cluster execution is a core capability of the platform. Every chaos scenario can target one or more clusters simultaneously, regardless of how those clusters were registered:

• Manual targets — clusters registered directly by an administrator through the web console
• OCM/ACM discovery — clusters discovered automatically through Open Cluster Management or Red Hat Advanced Cluster Management

Both methods produce the same unified cluster pool. When executing a scenario, users select any combination of available targets and the platform runs the experiment in parallel across all of them. This enables cross-environment resilience testing (dev, staging, production), regional failover validation, and multi-tenant chaos campaigns — all from a single execution.

INFO

Multi-cluster is always available. OCM/ACM integration is optional and adds automatic cluster discovery, but multi-cluster execution works identically with manually registered targets. No external cluster management platform is required.

---

Personas

The platform defines two roles with different access levels.

Admin

Full access to the platform. Administrators configure the infrastructure that users operate on.

• Register and remove target clusters
• Create groups with cluster access and permissions
• Create users and assign them to groups
• Configure private registries and their visibility
• Configure target providers (e.g. ACM integration)
• All operational features available to users

User

Operational access scoped by group membership. Every user must belong to a group — the group determines which clusters, registries and features are accessible.

• View and manage jobs (based on View/Cancel permissions)
• Use the cluster terminal (requires Run permission)
• Run scenarios on assigned clusters (requires Run permission)
• Design workflows in Chaos Studio (requires Run permission)
• Upload and manage files (scoped by group visibility)

INFO

Users only see jobs from their own group. A cluster-wide indicator shows the names of running scenarios across all groups, but no details.

---

Permission Model

        Admin
          │
    Creates Groups
          │
    ┌─────┴──────┐
    │             │
  Group A      Group B
    │             │
  Clusters     Clusters
  Permissions  Permissions
  (View,Run,   (View)
   Cancel)
    │             │
  Users        Users

Permissions are assigned at the group level and determine what users can do on clusters accessible to their group.

View

View

Allows users to see jobs and their execution results. Users with this permission can inspect single run outcomes, console logs, graph run nodes, and Resiliency Scores — but cannot execute, cancel or remove anything.

Applies to: Jobs

Run

Run

Allows users to execute chaos experiments on the clusters assigned to their group. This includes running single scenarios, designing and executing Chaos Studio workflows, and accessing the cluster terminal for read-only inspection.

Applies to: Run Scenarios · Chaos Studio · Cluster Terminal

Cancel

Cancel

Allows users to cancel running scenarios and to remove scenario runs and their execution history. Without this permission, jobs remain visible but cannot be cancelled or removed.

Applies to: Jobs

---

Key Capabilities

Capability	Description
Cluster Management	Register and manage target Kubernetes clusters
User & Group Management	Organize users through groups with granular permissions
Private Registries	Configure private container registries with group-based visibility
Chaos Studio	Design reusable visual workflows with serial and parallel execution
Multi-cluster Execution	Run experiments on one or more clusters simultaneously
Resiliency Score	Measure application resilience using PromQL-based metrics
Jobs	Monitor experiment progress and inspect execution logs
Cluster Terminal	Explore managed clusters using read-only kubectl and oc commands
File Management	Store reusable configuration files and PromQL queries
Provider Configuration	Configure target providers (ACM/OCM integration)

---

Compatibility

• Kubernetes
• OpenShift
• Open Cluster Management (OCM)
• Red Hat Advanced Cluster Management (ACM)

See the Compatibility Matrix for tested ACM/OCM versions and platform details.

---

Next Steps

• Installation — Deploy Krkn Operator with Helm
• Administration — Configure clusters, users, registries and providers
• Usage — Run scenarios, use Chaos Studio, manage jobs
• Compatibility Matrix — Supported platforms and ACM/OCM versions

1 - Installation

Installation Admin

Deploy Krkn Operator on Kubernetes or OpenShift using Helm.

Prerequisites

• Kubernetes 1.19+ or OpenShift 4.x
• Helm 3.0+

Quick Start

Latest Version: loading…

helm install krkn-operator oci://quay.io/krkn-chaos/charts/krkn-operator \
  --version <VERSION> \
  --namespace krkn-operator-system \
  --create-namespace

Verify:

kubectl get pods -n krkn-operator-system -l app.kubernetes.io/name=krkn-operator

Access the console (local):

kubectl port-forward -n krkn-operator-system svc/krkn-operator-console 3000:3000

Production Installation

Choose the method that matches your environment:

Install with a custom values.yaml:

helm install krkn-operator oci://quay.io/krkn-chaos/charts/krkn-operator \
  --version <VERSION> \
  --namespace krkn-operator-system \
  --create-namespace \
  -f values.yaml

Kubernetes with Gateway API

console:
  enabled: true
  gateway:
    enabled: true
    gatewayName: krkn-gateway
    hostname: krkn.example.com
    path: /
    pathType: PathPrefix

Kubernetes with Ingress

console:
  enabled: true
  ingress:
    enabled: true
    className: nginx
    hostname: krkn.example.com
    tls:
      - secretName: krkn-tls
        hosts:
          - krkn.example.com

OpenShift with Routes

console:
  enabled: true
  route:
    enabled: true
    hostname: krkn.apps.cluster.example.com
    tls:
      enabled: true
      termination: edge
      insecureEdgeTerminationPolicy: Redirect

OCM/ACM Integration

Enable automatic cluster discovery through Open Cluster Management (OCM) or Red Hat Advanced Cluster Management (ACM):

See the OCM/ACM Compatibility page for tested versions.

helm install krkn-operator oci://quay.io/krkn-chaos/charts/krkn-operator \
  --version <VERSION> \
  --namespace krkn-operator-system \
  --create-namespace \
  --set acm.enabled=true

Upgrade

helm upgrade krkn-operator oci://quay.io/krkn-chaos/charts/krkn-operator \
  --version <VERSION> \
  --namespace krkn-operator-system \
  -f values.yaml

Uninstall

helm uninstall krkn-operator --namespace krkn-operator-system

kubectl delete crds -l app.kubernetes.io/name=krkn-operator

Complete values.yaml Reference

namespaceOverride: ""

operator:
  image: quay.io/krkn-chaos/krkn-operator:latest
  pullPolicy: IfNotPresent
  enabled: true
  replicaCount: 1
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi
  dataProvider:
    resources:
      requests:
        cpu: 50m
        memory: 64Mi
      limits:
        cpu: 200m
        memory: 256Mi
  service:
    type: ClusterIP
    port: 8080
    grpcPort: 50051
  logging:
    level: info
    format: json
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  nodeSelector: {}
  tolerations: []
  affinity: {}
  extraEnv: []

dataProvider:
  image: quay.io/krkn-chaos/data-provider:latest

acm:
  enabled: false
  image: quay.io/krkn-chaos/krkn-operator-acm:latest
  replicaCount: 1
  config:
    secretName: "application-manager"
  service:
    port: 8080
    grpcPort: 50051
  logging:
    level: info
    format: json
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 200m
      memory: 256Mi
  nodeSelector: {}
  tolerations: []
  affinity: {}

console:
  enabled: true
  image: quay.io/krkn-chaos/console:latest
  replicaCount: 1
  service:
    type: ClusterIP
    port: 3000
    nodePort: null
  ingress:
    enabled: false
    className: nginx
    hostname: krkn.example.com
    annotations: {}
    tls: []
  gateway:
    enabled: false
    gatewayName: krkn-gateway
    gatewayNamespace: ""
    sectionName: ""
    hostname: krkn.example.com
    path: /
    pathType: PathPrefix
    annotations: {}
  route:
    enabled: false
    hostname: ""
    tls:
      termination: edge
  resources:
    requests:
      cpu: 50m
      memory: 64Mi
    limits:
      cpu: 200m
      memory: 256Mi
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  nodeSelector: {}
  tolerations: []
  affinity: {}

pullSecrets: []

auth:
  jwtSecret: ""
  jwtExpiryHours: 24

rbac:
  create: true

serviceAccount:
  create: true
  name: ""
  annotations: {}

crds:
  keep: true

monitoring:
  enabled: false
  service:
    port: 8443
  serviceMonitor:
    enabled: false
    interval: 30s

networkPolicy:
  enabled: false
  ingress: []
  egress: []

updateStrategy:
  type: RollingUpdate

podDisruptionBudget:
  enabled: false
  minAvailable: 1

commonLabels: {}
commonAnnotations: {}
nameOverride: ""
fullnameOverride: ""

2 - Administration

Administration Admin

Administrators configure the platform infrastructure that users operate on. This includes registering clusters, managing users and groups, setting up private registries and configuring target providers.

Permission Model

The Admin creates groups that define what users can do on the platform. Each group specifies:

• Which clusters are accessible
• Which permissions are granted (View, Run, Cancel)
• Which registries are visible

Users inherit all permissions from their assigned group.

Admin creates Group
      │
      ├── Assigns Clusters (target-1, target-2, ...)
      ├── Assigns Permissions (View, Run, Cancel)
      └── Assigns Registry Visibility
            │
            └── Users in this group inherit everything

Admin Features

2.1 - Cluster Management

Cluster Management Admin

Register target Kubernetes clusters that will be available for chaos scenario execution. The operator runs on a control plane cluster and executes scenarios against registered targets — it never runs chaos against itself.

Operations

Once registered, clusters become available for assignment to groups through User Management.

2.2 - User Management

User Management Admin

Organize users through groups that define cluster access and permissions. The group is the central unit of the permission model — users inherit all capabilities from their assigned group.

Groups

A group defines what its members can access and do on the platform.

Groups also receive permission flags that control what operations their members can perform:

Users

Users are created and assigned to a group during creation.

Permission Cascade

Group "SRE Team"
  ├── Clusters: production-us, production-eu
  ├── Permissions: View, Run
  └── Members: alice, bob
        │
        ├── alice → can run scenarios on production-us, production-eu
        └── bob   → can run scenarios on production-us, production-eu

2.3 - Registry Management

Registry Management Admin

Configure private container registries for chaos scenario images and control which groups can access them. By default, scenarios are pulled from the public Quay.io registry. Private registries enable air-gapped deployments and custom scenario images.

Creating a Private Registry

Configure a private registry by providing its connection settings (URL, credentials, TLS configuration).

Visibility

Each registry has a visibility setting that controls who can use it:

Impact on Scenario Selection

When a user selects a private registry during scenario execution, only the scenarios that have been mirrored to that registry will be available. Scenarios not present in the private registry will not appear in the selection list.

2.4 - Provider Configuration

Provider Configuration Admin

Configure target providers that integrate with external cluster management platforms. The provider configuration interface adapts dynamically based on the selected provider.

Supported Providers

Cloud Provider Configuration

Some chaos scenarios interact directly with cloud infrastructure and require cloud provider credentials (AWS, GCP, Azure, OpenStack, etc.) to perform node-level operations. The following scenarios are not currently available in Krkn Operator until cloud provider configuration support is added:

2.5 - Elasticsearch Management

Elasticsearch Management Admin

Administrators can configure and save Elasticsearch endpoints that users can reference when executing chaos scenarios. This enables centralized observability configuration without requiring users to manually enter connection details.

Overview

Elasticsearch integration allows krkn scenarios to index execution metrics, logs, and telemetry data for analysis and historical tracking. Instead of requiring each user to configure Elasticsearch details manually, admins can:

• Pre-configure Elasticsearch endpoints with connection details
• Save named configurations for reuse across all scenarios
• Control which Elasticsearch endpoints are available platform-wide

Users can then select from the saved Elasticsearch configurations when setting global parameters during scenario execution.

Adding an Elasticsearch Endpoint

1. Navigate to Administration > Elasticsearch Management
2. Click Add Elasticsearch
3. Configure the following details:

4. Click Save to add the configuration to the saved list

Managing Saved Configurations

View Saved Elasticsearch Endpoints

The Elasticsearch Management page displays all saved configurations:

• Name: Friendly identifier shown to users
• URL: Elasticsearch endpoint
• Index: Target index pattern
• Actions: Edit or Delete options

Edit an Endpoint

1. Click the Edit icon next to the saved configuration
2. Modify any field (Name, URL, Index, credentials, SSL verification)
3. Click Save to update

Delete an Endpoint

1. Click the Delete icon next to the saved configuration
2. Confirm the deletion

How Users Select Elasticsearch

When users configure global parameters for a scenario or workflow, they can:

1. Expand the Global parameters section
2. Locate the Elasticsearch parameter
3. Click the dropdown to view all saved Elasticsearch configurations
4. Select an Elasticsearch endpoint by name (e.g., “Production ES”)

The selected configuration automatically applies the admin-defined connection details (URL, index, credentials) to the scenario execution.

Users cannot:

• Add new Elasticsearch endpoints from the scenario configuration interface
• Edit existing saved configurations
• View sensitive details like passwords (only admins can see/edit credentials)

Users can:

• Choose from the dropdown list of saved Elasticsearch endpoints
• Opt not to use Elasticsearch (by leaving the parameter unset)

Best Practices

Naming Conventions

Use descriptive names that indicate the purpose or environment:

• ✅ Production Observability
• ✅ Staging Metrics
• ✅ Dev Team Shared ES
• ❌ ES1, Elasticsearch, Test

Security

• Use dedicated credentials for krkn with limited write permissions to specific indices
• Enable SSL verification for production endpoints
• Restrict index patterns to prevent accidental data overwrites (e.g., krkn-* not *)

Index Patterns

Use time-based or scenario-based index naming:

• krkn-chaos-{YYYY-MM-DD} — daily indices for easy retention management
• krkn-scenarios-* — scenario-specific index pattern
• krkn-{environment}-* — environment-specific indices

Verification

After adding an Elasticsearch endpoint, verify it works:

1. Run a simple chaos scenario with the Run Scenarios feature
2. Select the saved Elasticsearch configuration in Global parameters
3. Execute the scenario
4. Check the Elasticsearch index for new documents:

   curl -X GET "https://elasticsearch.example.com:9200/krkn-chaos-*/_search?pretty" \
     -u username:password
   

Successful indexing confirms the configuration is correct and accessible from the krkn-operator platform.

3 - Usage

Usage

The operational features of Krkn Operator are available to all users, scoped by the permissions assigned to their group. This section covers the day-to-day workflow: running scenarios, designing workflows, monitoring jobs and managing files.

Permission Summary

Features

• Jobs — Monitor and inspect scenario execution results
• Cluster Terminal — Explore target clusters with read-only commands
• Run Scenarios — Execute single chaos scenarios
• Chaos Studio — Design visual workflows with serial and parallel execution
• File Management — Upload and manage configuration files

3.1 - Jobs

Jobs View Cancel

The Jobs list is the home screen of the platform. It displays all scenario executions for your group, with real-time status updates and access to logs and results.

Run Types

Single Run

A single scenario execution on one or more target clusters. For each run you can inspect:

• Execution outcome (success, failure)
• Console logs of the krkn pod that executed the scenario

Replay & Edit

Single runs can be replayed to re-execute the same scenario with the same configuration. You can also edit the details before replaying to modify:

• Target cluster(s)
• Scenario parameters
• Configuration values
• Run Name

This allows you to quickly iterate on chaos experiments or run the same test across different environments.

Graph Run (Chaos Studio)

A workflow execution created through Chaos Studio. The execution is displayed as a graph where each node represents an individual krkn scenario. For each node you can inspect the outcome and logs independently.

Graph runs can include a Resiliency Score — a calculated metric based on PromQL queries executed during the workflow. This score measures how well the target application withstood the chaos experiment.

Visibility & Permissions

• Only jobs from your own group are displayed
• A cluster-wide tip shows the names of scenarios currently running across all groups (no details)
• Cancelling or removing jobs requires Cancel permission

3.2 - Cluster Terminal

Cluster Terminal Run

A built-in terminal for exploring target clusters using read-only oc and kubectl commands. Useful for inspecting cluster state before, during or after chaos experiments.

Details

• Read-only: only read commands are available — no modifications to the cluster are possible
• Supported commands: oc and kubectl
• Target selection: only clusters accessible to your group are available
• Requires Run permission

3.3 - Run Scenarios

Run Scenarios Run

Execute a chaos scenario on one or more target clusters through a guided step-by-step wizard. All scenarios can run simultaneously across multiple clusters.

Execution Flow

1. Select Target Clusters

Choose one or more clusters to run the scenario on. All selected clusters execute the scenario simultaneously.

2. Select Registry

Choose the container registry for scenario images:

3. Select Scenario

Browse and select a chaos scenario from the chosen registry.

4. Configure Parameters

Each scenario defines its own parameter set, divided into three categories:

Elasticsearch in Global Parameters

When configuring Global parameters, you can select a saved Elasticsearch endpoint to index scenario metrics and telemetry:

1. Expand the Global parameters section
2. Locate the Elasticsearch dropdown
3. Select from available saved configurations (configured by your admin)

Selecting an Elasticsearch configuration automatically applies connection details (URL, index, credentials) without requiring manual input.

5. Mount Files

Optionally attach configuration files previously uploaded through File Management. Files are available via a select dropdown.

6. Preview

Review a summary of all configured parameters before execution.

7. Run

Launch the scenario. You are redirected to the Jobs page where you can monitor the execution in real time.

3.4 - Chaos Studio

Chaos Studio Run

A visual drag-and-drop editor for designing complex chaos workflows. Build graphs of scenarios that execute in series or parallel, and measure application resilience with the Resiliency Score.

Creating and Saving Workflows

Creating a New Workflow

1. Navigate to Chaos Studio from the main navigation
2. Click the “New Workflow” button to start a blank canvas
3. Give your workflow a meaningful name that describes its purpose

Adding Nodes

1. Click the “Add Node” button or drag from the node palette
2. Position nodes on the canvas by dragging them
3. Configure each node by clicking on it and selecting:
   • Registry: Choose your scenario source
   • Scenario: Select the chaos scenario to run
   • Parameters: Set mandatory and optional parameters
   • Files: Mount any required configuration files

Connecting Nodes

• Click and drag from a node’s output port to another node’s input port to create dependencies
• Serial execution: connect nodes in a chain (A → B → C)
• Parallel execution: connect one node to multiple targets (A → B and A → C)

Saving Your Workflow

1. Click the “Save” button in the top toolbar
2. The workflow is automatically saved to your workspace
3. Saved workflows appear in the Chaos Studio workflow list
4. You can edit saved workflows by selecting them from the list

Video Walkthrough

Visual Editor

• Nodes represent individual krkn scenarios
• Arrows define execution order — connect nodes to create dependencies
• Serial execution: chain nodes sequentially with single arrows
• Parallel execution: branch from one node to multiple targets
• Drag and drop to arrange the workflow visually

Node Configuration

Each node is configured exactly like a single scenario: select a registry, choose a scenario, set mandatory/optional/global parameters, and mount files.

Global Parameters in Workflows

When configuring global parameters for workflow nodes:

• Elasticsearch: Select from saved admin-configured endpoints to index metrics across the entire workflow
• Prometheus: Configure Prometheus integration for metric collection
• Cerberus: Enable cluster health monitoring during scenario execution

Clone: duplicate a configured node to repeat the same scenario multiple times within the workflow.

Validation

The editor validates the workflow before execution:

• Configured nodes: green arrows and connections — ready to execute
• Unconfigured nodes: red arrows and red node border — must be configured before the workflow can run

Only fully configured nodes can be connected and executed.

Resiliency Score

Enable the Resiliency Score on a workflow to measure application resilience:

1. Enable the resiliency score option on the workflow
2. Mount a PromQL file — a special file containing PromQL queries uploaded via File Management
3. Queries execute during the workflow and their results contribute to the final score
4. View the score in the Jobs page alongside the graph run results

The PromQL file defines which metrics to evaluate (latency, error rate, throughput, etc.) and their expected thresholds. The score reflects how closely the application maintained its expected behavior under chaos.

3.5 - File Management

File Management

Upload and manage text-based configuration files that can be used across scenario executions and Chaos Studio workflows.

Uploading Files

Files are created by pasting content into the text input box — there is no file upload button.

Where Files Are Used

4 - Releases & Roadmap

This document outlines the Krkn Operator release strategy, roadmap, and how the community prioritizes features, bug fixes, and security updates.

Roadmap

Release Cadence

Krkn Operator aims to deliver feature releases every 6–8 weeks. These releases may include new features, enhancements, usability improvements, and other roadmap items driven by community feedback and contributor activity.

The release cadence is intended to balance rapid delivery of value to users while maintaining stability and quality across supported environments.

Feature Requests and Roadmap

Community feedback plays a key role in shaping the future of Krkn Operator.

Feature requests submitted through GitHub issues, enhancement proposals, community discussions, and feedback forms are reviewed regularly and considered as part of the project roadmap.

Features are prioritized based on:

• Community interest
• User impact
• Project goals
• Contributor capacity

Planned enhancements and major initiatives will be communicated through project documentation, GitHub, community meetings, and release notes.

Bug Fixes and Security Updates

Bug fixes and security updates are prioritized independently of the feature release cadence.

Critical security vulnerabilities and high-impact issues may be addressed through patch or minor releases outside the normal release cycle when necessary.

Community contributions for bug fixes, testing, and validation are always welcome.

Release Notes

Each release includes documentation for:

• New features and enhancements
• Bug fixes
• Security updates
• Compatibility updates
• Breaking changes and deprecations
• Upgrade or migration guidance, when applicable

Any compatibility changes, breaking changes, or upgrade considerations will be documented as part of the release notes to help users plan upgrades and understand changes between releases.

Getting Involved

The Krkn community welcomes feedback, feature requests, bug reports, testing, and code contributions. Community participation helps shape the roadmap and improve the overall user experience.

For feature requests or bug reports, please open an issue in the Krkn-operator GitHub repository or participate in community discussions.

5 - Feedback

Share Feedback on Krkn Operator

We are collecting feedback from users trying Krkn Operator through OCM/ACM, upstream Krkn Operator, or community environments.

Use this form to share adoption stories, use cases, feature requests, bugs, install issues, documentation feedback, adoption blockers, or interest in demos and office hours.

For bugs, feature requests, install issues, and documentation gaps, the recommended path is to use the Krkn Operator GitHub repository. If GitHub is not preferred due to security or sharing concerns, you can submit feedback through this form instead.

6 - OCM/ACM Compatibility

OCM/ACM Compatibility

Krkn Operator supports multi-cluster chaos testing on Kubernetes and OpenShift. Cluster discovery can be manual or automated through ACM/OCM integration.

Platform Support

ACM Compatibility

The following versions of Red Hat Advanced Cluster Management have been tested and validated with Krkn Operator:

OCM Compatibility

The following versions of Open Cluster Management have been tested and validated:

How ACM Integration Works

When ACM is enabled (--set acm.enabled=true at install time), Krkn Operator deploys a dedicated component that connects to the ACM hub and automatically discovers managed clusters. Discovered clusters appear in the same cluster pool as manually registered targets — users select from both without distinction.

See Provider Configuration for setup details and Installation for the Helm flag.

Reference

• Open Cluster Management (OCM)
• OCM Releases
• Red Hat ACM Documentation