ETCD Split Brain Scenarios

1: ETCD Split Brain Scenarios using Krkn
2: ETCD Split Brain Scenario using Krkn-Hub
3: ETCD Split Brain Scenario using Krknctl

This scenario isolates an etcd node by blocking its network traffic. This action forces an etcd leader re-election. Once the scenario concludes, the cluster should temporarily exhibit a split-brain condition, with two etcd leaders active simultaneously. This is particularly useful for testing the etcd cluster’s resilience under such a challenging state.

DANGER

This scenario carries a significant risk: it might break the cluster API, making it impossible to automatically revert the applied network rules. The iptables rules will be printed to the console, allowing for manual reversal via a shell on the affected node. This scenario is best suited for disposable clusters and should be used at your own risk.

1 - ETCD Split Brain Scenarios using Krkn

This scenario isolates an etcd node by blocking its network traffic. This action forces an etcd leader re-election. Once the scenario concludes, the cluster should temporarily exhibit a split-brain condition, with two etcd leaders active simultaneously. This is particularly useful for testing the etcd cluster’s resilience under such a challenging state.

To run

Sample scenario config

- id: node_network_filter
  wait_duration: 0
  test_duration: 60
  label_selector: ''
  service_account: ''
  namespace: 'default'
  instance_count: 1
  execution: parallel
  ingress: false
  egress: true
  target: '<NODE_NAME>'
  interfaces: []
  ports: [2379, 2380]
  taints: []
  protocols:
    - tcp
  image: quay.io/krkn-chaos/krkn-network-chaos:latest

How to Use Plugin Name

Add the plugin name to the list of chaos_scenarios section in the config/config.yaml file

kraken:
    kubeconfig_path: ~/.kube/config                     # Path to kubeconfig
    .. 
    chaos_scenarios:
        - network_chaos_ng_scenarios:
            - scenarios/<scenario_name>.yaml

Run

python run_kraken.py --config config/config.yaml

DANGER

This scenario carries a significant risk: it might break the cluster API, making it impossible to automatically revert the applied network rules. The iptables rules will be printed to the console, allowing for manual reversal via a shell on the affected node. This scenario is best suited for disposable clusters and should be used at your own risk.

2 - ETCD Split Brain Scenario using Krkn-Hub

podman run -v ~/.kube/config:/home/krkn/.kube/config:z -e TEST_DURATION="60" -e INGRESS="false" -e EGRESS="true" -e PROTOCOLS="tcp" -e PORTS="2379,2380" -e NODE_NAME="kind-control-plane" quay.io/krkn-chaos/krkn-hub:node-network-filter

3 - ETCD Split Brain Scenario using Krknctl

krknctl run node-network-filter \
 --chaos-duration 60 \
 --node-name <node_name> \
 --ingress false \
 --egress true \
 --protocols tcp \
 --ports 2379,2380