Testing a distributed system is difficult
- Distributed systems are more and more complex nowadays:
- Faults can happen anytime, anywhere, in any ways
- Writing tests and debugging is hard:
- Deterministic test is very hard and impossible to cover all faults
- But, No crash, No data loss, No wrong results
Chaos Engieering to the rescue
- Chaos engineering is about breaking things in a controlled environment and through well-planned experiments in order to build confidence in your application to withstand turbulent conditions.
- Chaos engineering is NOT about breaking things randomly without a perpose.
- Program Cycle:
- Improve -> Steady State -> Hypothesis -> Run Experiment -> Verify -> Improve…
Why Chaos Mesh
On Kubernetes:
- More application clusters (40+)
- More nodes on each cluster
- More target objects may fail, e.g. Container / Pod / Network / Disk / System Clock / Kernel / etc.
We need more Chaos experiments. However, managing and scheduling many chaos experiments is a hug pain!
In Docker:
- The environment is different from the physical nodes
- Tools like tc / iptables / fuse / bcc can’t be used directly
- Containers on the same node cannot affect each other
Chaos scope must be customizable and manageable for containers.
What is Chaos Mesh
- A Cloud-Native Chaos Engineering platform on Kubernetes environments
- Started out as the internal platform to test TiDB
- Provides fault injection methods into the container, Pod, network, system I/O, kernel, etc.
- Chaos Mesh’s Mission:
- Make Chaos Engineering easy
Deep into Chaos Mesh
Architecture
- Chaos Dashboard:
- Manage and monitor chaos experiment
- Chaos Controller Manager:
- Schedule and controle component
- Workflow engine
- Chaos Daemon:
- Executive component on kubernetes node
- Chaosd:
- Executive component on non-kubernetes node
CustomResourceDeifnitions
- PodChaos, NetworkChaos, …
- Examples
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
name: pod-kill-example
spec:
action: pod-kill
mode: one
selector:
labelSelectors:
"app.kubernetes.io/component": "tikv"
apiVersion: chaos-mesh.org/v1alph1
kind: Schedule
metadata:
name: schedule-pod-kill-example
spec:
schedule: "@every 5m"
type: "PodChaos"
historyLimit: 5
concurrencyPolicy: Forbid
podChaos:
action: "pod-kill"
mode: one
selector:
labelSelectors:
"app.kubernetes.io/component": "tikv"
Workflow Engine
- Three parts of the a workflow:
- Workflow Name
- Entry, the entry of the whole workflow
- Template array
- Five different types of templates:
- Serial
- Parallel
- Chaos
- Suspend
- Task
- Serial, Parallel, Task allow other nodes to be referenced as child nodes
Selectors
- Namespace, Label, Expression, Annotation, Field, PodPhase, Node Selector
- Node, Pod list
Authorization
- Authorization mechanism based on Kubernetes RBAC permission policies
개인 의견
- 강연 자체가 깔끔했다. 어떻게 동작하는지 아키텍쳐 그림만 봐도 이해가 가능할정도로 설명하고 있다.
- 기능 자체가 kubernetes 틱 하다. 그러면서도 web ui를 잘 제공한다고 생각한다. 개인적으로는 argo cd 같은 느낌이 들었다. dashboard 가 좋아서 시각화가 잘되고, 시각화가 잘되니까 머리속으로 어떻게 해야하는 구나가 잘 떠오르는 것 같다.
- demo가 엄청 좋다. 공식 홈페이지에서 직접 해볼수 있도록 제공하는데, 어떻게 설치하는 지, 어떻게 설정하는지를 잘 보여준다.
- 아직 documentation 을 읽어보지는 않았지만, 이 정도만 해도 어떻게 사용하는지 감을 잡고 자잘한것 정도만 찾아보는 수준일것 같다.
- 생각한게 거의 다 있어서 좋았다. 강연 흐름이 들으면 궁금해할만한 순서로 잘 구성되어 있다.
- 예전에 Litmus를 찍먹 해본적이 있었는데 튜토리얼 경험은 이게 더 좋은것 같다. 물론 litmus chaos 가 지금은 더 널리쓰여서 써야한다면 litmus를 쓰겠지만 말이다. 개인적으로 기대가 되는 프로젝트다 2022 2월에서야 인큐베이팅 됬으니까 아직은 발전을 더 지켜봐야할듯 하다.