Chaos testing for Data Engineering

Data pipeline resiliency testing is often delegated to human (or agentic AI) reasoning through time/token consuming and error-prone "What if?" sessions. What if an automated tool existed?

1. Context

Ensuring data pipelines behave as expected requires thinking beyond the software happy path and analyzing those scenarios in which things go wrong and crashes happen.

In order to ensure data pipeline resiliency, the pipeline design should take care of:

Idempotent operations: running the pipeline multiple times over the same input always produces the same output as running it once.
Recovery capabilities: the pipeline can recover from crashes at any step, preventing or fixing eventual inconsistent states.

Moreover, dealing with multi-step pipelines (multiple write steps) requires ensuring that the pipeline is able to fulfill such properties at every step.

1.1. State of the art

Nowadays, there is a lack of resources for automating this kind of test.

Existing tools such as Netflix's Chaos Monkey act at the machine/network level with random fault injections, but they don't provide the granularity and determinism required to reproduce such tests.

Some other rule-based tools provide linting functionalities that alert the user when the code contains a set of potential non-idempotent operations, but they don't verify the actual outcome at execution time.

Whenever there is a need to analyze an unknown codebase, fix a bug, or implement a new feature, developers are forced to constantly question the codebase through "What if?" sessions.
Such sessions are meant to understand whether the pipeline is still resilient or a potential regression is going to be introduced in the codebase.

1.2. What developers need

A useful tool for developers and AI Agents should provide the following features:

development and automation: the tool should be usable as a standalone utility in the development workflow (e.g., analysis and debugging) or as an automation (e.g., within automated tests run in CI/CD pipelines).
Packed as a library: the tool should be available as a library, importable and usable within the codebase test suites.
black-box testing: the user job (data pipeline) should not be aware of the tool; the user doesn't need to adapt the job to make it aware of the library.

2. Problem formalization

2.1. Assumptions

Testing a data pipeline's resiliency first requires defining what to test.
By potentially having to deal with any kind of data pipeline, the testing tool must be capable of identifying those pipeline actions that may introduce resiliency issues.

Assuming that the only externally observable side effects relevant to the tested execution are data materialization operations intercepted by the tool, the storage data write operations can be used as fault points.
Write operations are assumed to be executed sequentially (one starts after the previous one completed).

---
config:
  block:
    padding: 0
---
block-beta
  columns 9
  block:job:9
    columns 9
    space:4 title["<b style='display:inline-block;padding:10px 0'>Compute</b>"] space:4
    space w1["<span style='color:#ffffff;display:inline-block;padding:8px 20px'>Write 1</span>"] space w2["<span style='color:#ffffff;display:inline-block;padding:8px 20px'>Write 2</span>"] space w3["<span style='color:#ffffff;display:inline-block;padding:8px 20px'>Write …</span>"] space wn["<span style='color:#ffffff;display:inline-block;padding:8px 20px'>Write N</span>"] space
    space:9
  end
  block:gap:9
    columns 1
    gapfill["<span style='display:inline-block;height:40px'></span>"]
  end
  block:storage:9
    columns 9
    space p1[" "] space p2[" "] st["<b style='display:inline-block;padding:16px 0'>Storage</b>"] p3[" "] space p4[" "] space
  end
  w1 --> w2
  w2 --> w3
  w3 --> wn
  w1 --> p1
  w2 --> p2
  w3 --> p3
  wn --> p4
  classDef write fill:#14181c,stroke:#14181c,color:#ffffff
  classDef ghost fill:transparent,stroke:transparent
  class w1,w2,w3,wn write
  class title,p1,p2,p3,p4,st,gap,gapfill ghost
  style job fill:#f7fbff,stroke:#5f6163
  style storage fill:#ffffff,stroke:#5f6163

Fig 1. A typical data pipelines made of multiple write steps

Considering a pipeline composed of \(N\) data writes.
The testing tool must test the pipeline's capability of recovering from one or many partial executions, where each execution can be interrupted before a specific data write.

Such a process, named a test sequence, is built as follows:

Start from a clean state \(S\).
Perform one or more crash runs.
Every next run starts from the previous state left by the previous run (no state clean-up).
In the end, a happy run \(H\) is performed (no crashes)
The output is verified through a validation \(V\). If the pipeline is resilient, the final output is the same as the one generated by a single happy run.

For example, running the test sequence:

\(S\) → Write 2 → Write 4 → \(H\) → \(V\)

means:

Start the test sequence from a clean state \(S\)
Run and crash before Write 2
Run and crash before Write 4, starting from the previous stale state
Run and perform all the writes without crashes (\(H\))
Check the output through validation \(V\)

Note: Crashing before Write 1 is never considered a valid crash run, since it means performing no writes at all, which is the same as performing no execution at all. Starting each test sequence from a clean state S makes test sequences independent.

2.2. Formal definition

Given \(N\) data writes, the set of possible crash points is defined as \(W = \{w_i | i\in \{2, ..., N\}\}\), where \(w_i\) is a crash right before executing Data write \(i\).

Hence, \(|W| = N-1\) is the number of possible crash points.

A Test Sequence \(Seq j\) is made of a sequence of job runs:

A series of job crash runs built on an ordered subset of \(W\)
A final job run \(H\) (happy run) without crashes.

By choosing \(k\), the desired number of intermediate crashes within a test sequence, the number of possible test sequences is \(\binom{N - 1}{k}, 0 \leq k \leq |W|\)

The total number of possible test sequences within a chaos test is then \(\sum_{k=0}^{N-1}\binom{N-1}{k} = 2^{N-1}\) (the sum of test sequences across every possible \(k\))

Note: the number of test sequences executed by a full chaos testing run grows exponentially with the number of data writes performed by the user job.
However, considering the resources required to perform such a number of job executions in a testing environment (local or small remote clusters, sample data, etc.) and the expected average number of data writes performed by a job, such an approach doesn't represent an obstacle.

2.3. A concrete example

Given \(N = 4\) (number of data writes), \(W=\{w_2, w_3, w_4\}\) is the set of crash points. Hence, there are \(2^{4-1} = 8\) possible test sequences \(Seq j\):

---
config:
  block:
    padding: 1
  themeVariables:
    mainBkg: transparent
    nodeBorder: transparent
  themeCSS: ".edge-thickness-normal.edge-pattern-solid{stroke:#b85450 !important;stroke-width:1.25px !important;stroke-linecap:round !important}.edge-thickness-thick{stroke:#82b366 !important;stroke-width:1.25px !important}.edge-pattern-dotted{stroke:#5f6163 !important;stroke-width:1.25px !important;stroke-dasharray:none !important;stroke-linecap:square !important}[id$='-pointEnd'] .arrowMarkerPath{fill:#82b366 !important;stroke:#82b366 !important;transform-box:fill-box;transform-origin:100% 50%;transform:scale(0.6)}[id$='-pointStart'] .arrowMarkerPath{fill:#5f6163 !important;stroke:#5f6163 !important;transform-box:fill-box;transform-origin:0 50%;transform:scale(0.6)}foreignObject{overflow:visible !important}foreignObject div{font-size:11px !important}[id$='-c0'] rect.composite{transform-box:fill-box;transform-origin:100% 0;transform:translateY(13px) scale(0.76892,0.85217);vector-effect:non-scaling-stroke}[id$='-c1'] rect.composite{transform-box:fill-box;transform-origin:100% 0;transform:translateY(13px) scale(0.76892,0.90116);vector-effect:non-scaling-stroke}[id$='-c2'] rect.composite{transform-box:fill-box;transform-origin:100% 0;transform:translateY(13px) scale(0.76892,0.92576);vector-effect:non-scaling-stroke}[id$='-c3'] rect.composite{transform-box:fill-box;transform-origin:100% 0;transform:translateY(13px) scale(0.76892,0.87313);vector-effect:non-scaling-stroke}"
---
block-beta
  columns 1
  block:head
    columns 13
    space:8 hpad[" "]:4 tlab["<span style='display:inline-block;width:33px;height:17px;line-height:17px;text-align:center'><i>t</i></span>"]
    space:3 tstart(("<span></span>")) tw1(("<span style='padding:0 4px'>$$w_1$$</span>")) space tw2(("<span style='padding:0 4px'>$$w_2$$</span>")) space tw3(("<span style='padding:0 4px'>$$w_3$$</span>")) space tw4(("<span style='padding:0 4px'>$$w_4$$</span>")) space tend(("<span></span>"))
  end
  block:c0
    columns 13
    space:13
    space:6 c0t["<b style='white-space:nowrap;margin:0 -0.2px 0 0.2px'>0 crashes</b>"]:4 space:3
    space s1r0b(("<span></span>")) s1r0t[" "] space:10
    q1["<span style='display:inline-block;width:33px;height:17px;line-height:17px;text-align:center;vertical-align:top;margin:0px -8px 0px 8px'>Seq 1</span>"] space s1r1l["<i style='padding:0 2px'>H</i>"] s1r1s(("<span></span>")) space:8 s1r1e(("<span></span>"))
    space s1r2b(("<span></span>")) s1r2t[" "] space:10
    space:13
  end
  block:c1
    columns 13
    space:13
    space:6 c1t["<b style='white-space:nowrap;margin:0 -0.2px 0 0.2px'>1 crash</b>"]:4 space:3
    q2["<span style='display:inline-block;width:33px;height:17px;line-height:17px;text-align:center;vertical-align:top;margin:9px -8px -9px 8px'>Seq 2</span>"] s2r0b(("<span></span>")) s2r0l["<i style='padding:0 2px'>$$w_2$$</i>"] s2r0s(("<span></span>")) space:2 s2r0x(("<span style='font-size:14px;color:#e03131'>✸</span>")) space:6
    space s2r1b(("<span></span>")) s2r1l["<i style='padding:0 2px'>H</i>"] s2r1s(("<span></span>")) space:8 s2r1e(("<span></span>"))
    q3["<span style='display:inline-block;width:33px;height:17px;line-height:17px;text-align:center;vertical-align:top;margin:9px -8px -9px 8px'>Seq 3</span>"] s3r0b(("<span></span>")) s3r0l["<i style='padding:0 2px'>$$w_3$$</i>"] s3r0s(("<span></span>")) space:4 s3r0x(("<span style='font-size:14px;color:#e03131'>✸</span>")) space:4
    space s3r1b(("<span></span>")) s3r1l["<i style='padding:0 2px'>H</i>"] s3r1s(("<span></span>")) space:8 s3r1e(("<span></span>"))
    q4["<span style='display:inline-block;width:33px;height:17px;line-height:17px;text-align:center;vertical-align:top;margin:9px -8px -9px 8px'>Seq 4</span>"] s4r0b(("<span></span>")) s4r0l["<i style='padding:0 2px'>$$w_4$$</i>"] s4r0s(("<span></span>")) space:6 s4r0x(("<span style='font-size:14px;color:#e03131'>✸</span>")) space:2
    space s4r1b(("<span></span>")) s4r1l["<i style='padding:0 2px'>H</i>"] s4r1s(("<span></span>")) space:8 s4r1e(("<span></span>"))
    space:13
  end
  block:c2
    columns 13
    space:13
    space:6 c2t["<b style='white-space:nowrap;margin:0 -0.2px 0 0.2px'>2 crashes</b>"]:4 space:3
    space s5r0b(("<span></span>")) s5r0l["<i style='padding:0 2px'>$$w_2$$</i>"] s5r0s(("<span></span>")) space:2 s5r0x(("<span style='font-size:14px;color:#e03131'>✸</span>")) space:6
    q5["<span style='display:inline-block;width:33px;height:17px;line-height:17px;text-align:center;vertical-align:top;margin:0px -8px 0px 8px'>Seq 5</span>"] space s5r1l["<i style='padding:0 2px'>$$w_3$$</i>"] s5r1s(("<span></span>")) space:4 s5r1x(("<span style='font-size:14px;color:#e03131'>✸</span>")) space:4
    space s5r2b(("<span></span>")) s5r2l["<i style='padding:0 2px'>H</i>"] s5r2s(("<span></span>")) space:8 s5r2e(("<span></span>"))
    space s6r0b(("<span></span>")) s6r0l["<i style='padding:0 2px'>$$w_3$$</i>"] s6r0s(("<span></span>")) space:4 s6r0x(("<span style='font-size:14px;color:#e03131'>✸</span>")) space:4
    q6["<span style='display:inline-block;width:33px;height:17px;line-height:17px;text-align:center;vertical-align:top;margin:0px -8px 0px 8px'>Seq 6</span>"] space s6r1l["<i style='padding:0 2px'>$$w_4$$</i>"] s6r1s(("<span></span>")) space:6 s6r1x(("<span style='font-size:14px;color:#e03131'>✸</span>")) space:2
    space s6r2b(("<span></span>")) s6r2l["<i style='padding:0 2px'>H</i>"] s6r2s(("<span></span>")) space:8 s6r2e(("<span></span>"))
    space s7r0b(("<span></span>")) s7r0l["<i style='padding:0 2px'>$$w_2$$</i>"] s7r0s(("<span></span>")) space:2 s7r0x(("<span style='font-size:14px;color:#e03131'>✸</span>")) space:6
    q7["<span style='display:inline-block;width:33px;height:17px;line-height:17px;text-align:center;vertical-align:top;margin:0px -8px 0px 8px'>Seq 7</span>"] space s7r1l["<i style='padding:0 2px'>$$w_4$$</i>"] s7r1s(("<span></span>")) space:6 s7r1x(("<span style='font-size:14px;color:#e03131'>✸</span>")) space:2
    space s7r2b(("<span></span>")) s7r2l["<i style='padding:0 2px'>H</i>"] s7r2s(("<span></span>")) space:8 s7r2e(("<span></span>"))
    space:13
  end
  block:c3
    columns 13
    space:13
    space:6 c3t["<b style='white-space:nowrap;margin:0 -0.2px 0 0.2px'>3 crashes</b>"]:4 space:3
    space s8r0b(("<span></span>")) s8r0l["<i style='padding:0 2px'>$$w_2$$</i>"] s8r0s(("<span></span>")) space:2 s8r0x(("<span style='font-size:14px;color:#e03131'>✸</span>")) space:6
    q8["<span style='display:inline-block;width:33px;height:17px;line-height:17px;text-align:center;vertical-align:top;margin:9px -8px -9px 8px'>Seq 8</span>"] space s8r1l["<i style='padding:0 2px'>$$w_3$$</i>"] s8r1s(("<span></span>")) space:4 s8r1x(("<span style='font-size:14px;color:#e03131'>✸</span>")) space:4
    space space s8r2l["<i style='padding:0 2px'>$$w_4$$</i>"] s8r2s(("<span></span>")) space:6 s8r2x(("<span style='font-size:14px;color:#e03131'>✸</span>")) space:2
    space s8r3b(("<span></span>")) s8r3l["<i style='padding:0 2px'>H</i>"] s8r3s(("<span></span>")) space:8 s8r3e(("<span></span>"))
    space:13
  end
  tstart -.- tw1
  tw1 -.- tw2
  tw2 -.- tw3
  tw3 -.- tw4
  tend <-.- tw4
  s1r0b -.- s1r0t
  s1r1s ==> s1r1e
  s1r2b -.- s1r2t
  s1r0b -.- s1r2b
  s2r0b -.- s2r0l
  s2r0s --- s2r0x
  s2r1b -.- s2r1l
  s2r1s ==> s2r1e
  s2r0x --- s2r1s
  s2r0b -.- s2r1b
  s3r0b -.- s3r0l
  s3r0s --- s3r0x
  s3r1b -.- s3r1l
  s3r1s ==> s3r1e
  s3r0x --- s3r1s
  s3r0b -.- s3r1b
  s4r0b -.- s4r0l
  s4r0s --- s4r0x
  s4r1b -.- s4r1l
  s4r1s ==> s4r1e
  s4r0x --- s4r1s
  s4r0b -.- s4r1b
  s5r0b -.- s5r0l
  s5r0s --- s5r0x
  s5r1s --- s5r1x
  s5r0x --- s5r1s
  s5r2b -.- s5r2l
  s5r2s ==> s5r2e
  s5r1x --- s5r2s
  s5r0b -.- s5r2b
  s6r0b -.- s6r0l
  s6r0s --- s6r0x
  s6r1s --- s6r1x
  s6r0x --- s6r1s
  s6r2b -.- s6r2l
  s6r2s ==> s6r2e
  s6r1x --- s6r2s
  s6r0b -.- s6r2b
  s7r0b -.- s7r0l
  s7r0s --- s7r0x
  s7r1s --- s7r1x
  s7r0x --- s7r1s
  s7r2b -.- s7r2l
  s7r2s ==> s7r2e
  s7r1x --- s7r2s
  s7r0b -.- s7r2b
  s8r0b -.- s8r0l
  s8r0s --- s8r0x
  s8r1s --- s8r1x
  s8r0x --- s8r1s
  s8r2s --- s8r2x
  s8r1x --- s8r2s
  s8r3b -.- s8r3l
  s8r3s ==> s8r3e
  s8r2x --- s8r3s
  s8r0b -.- s8r3b
  style head fill:transparent,stroke:transparent
  style c0 fill:#f7fbff,stroke:#5f6163
  style c1 fill:#f7fbff,stroke:#5f6163
  style c2 fill:#f7fbff,stroke:#5f6163
  style c3 fill:#f7fbff,stroke:#5f6163

Fig 2. Chaos test execution's timeline.

Zero-crash sequences: sequences executed with no crashes in between
- S → H → V : the happy run. The pipeline completes all its data writes successfully. This is where developer tests usually stop.
One-crash sequences: sequences executed with one crash in between
- S → Write 2 → H → V
- S → Write 3 → H → V
- S → Write 4 → H → V
Two-crash sequences: sequences executed with two crashes in between
- S → Write 2 → Write 3 → H → V
- S → Write 2 → Write 4 → H → V
- S → Write 3 → Write 4 → H → V
Three-crash sequences: sequences executed by crashing at every meaningful crash point
- S → Write 2 → Write 3 → Write 4 → H → V

3. Implementing the library

There are dozens of query engines in the Data Engineering field. In order to build a chaos testing tool, such a tool must sit between the job business logic and the actual execution.

There are at least two possible approaches to inject controlled failures, and their feasibility depends on how extendable the engine is:

Query optimizer level: The tool can inject failures into the query plan through custom rules that intercept data write operators.
Client API level: The user job is built through an API (e.g., DataFrames or plain SQL queries). The tool can wrap such an API with a thin "fault layer" that decides whether the call must produce a crash or act as a pass-through towards the original query engine API call.

Both of them can heavily depends on the chosen query engine.

4. Chaos testing for Apache Spark

Apache Spark is one of the most famous distributed data processing engines in the Big Data Engineering industry. For this reason, it's the perfect candidate to be chaos tested.

Started as a Scala-only framework (the language used to build Spark itself), today it's also usable with the first choice language in the market, Python, through PySpark (mostly made of Scala bindings).

4.1. Tested tech stack

pyspark >= 3.5.x is the chosen framework "flavor". The execution is started from the user's local machine with two execution modes:

Local cluster: the user spins up a Spark cluster on their own machine
Remote cluster: the user uses a remote Databricks cluster through Spark Connect protocol

4.2. Fault injection approaches

When it comes to extensions useful for chaos testing purposes, the framework offers various options:

Query optimizer level: Spark already provides Catalyst extensions entry point through SparkSessionExtensions.
Event-listening level: Spark provides a way to intercept events emitted by the Spark scheduler through SparkListener.
Client API level: The engine is used by users through a high user-friendly framework which let them build data pipelines. Users can choose or mix two different API types (DataFrames or SQL).

However, the first two approaches have a few main drawbacks, which made the Client API level the chosen approach to build the chaos testing library:

The query optimizer approach needs to work around internal representations that can be subjected to frequent changes. Custom Catalyst rules cannot be turned on/off programmatically; hence, crash injection could be tricky to implement. Moreover, such an approach requires treating the execution differently based on the execution mode (e.g., Spark Connect moved the Driver to the server-side).
Being an async process, the event-listening approach doesn't allow you to intercept data writes before they happen, which goes against the library’s main goal.

The following table summarizes the pros and cons of each approach, where colors mean:

Green: considered positive.
Yellow: considered a potential trade-off at feature/ implementation level (e.g., could be subject to some pitfalls).
Red: considered a relevant roadblock (cannot be implemented is subject to several pitfalls).

	Query Optimizer	Event-listening	Client API
Client languages (out of scope)	Any (Python, Scala, Java)	Any (Python, Scala, Java)	Python
Interception point	Engine	Engine	User code (Job business logic)
Official hook	Yes (Spark session extensions API) {green/70}	Yes (Spark listener API) {green/70}	No (monkey-patches PySpark internal modules and classes) {yellow/70}
Crash before the write	Yes (during query planning), with some pitfalls {yellow/70}	No (the event arrives after the write has started) {red/70}	Yes (synchronous and deterministic in the Python process, before Spark receives the write) {green/70}
Stability of the hooked surface	Low (internal plan representations, e.g. physical operators, change often) {yellow/70}	Official hooks {green/70}	Stable client API, but the implementation depends on PySpark internals {yellow/70}
Spark Connect support	Needs a different approach: protobuf plan handling plus a client-server interception mechanism {red/70}	Can't be registered from the client; server-side only {red/70}	Same as local Spark, no extra configuration {green/70}
Remote cluster requirements	Extra JARs and cluster configuration; shared clusters add restrictions {yellow/70}	Extra JARs and cluster configuration {green/70}	None {green/70}
Python context (e.g. call site of a write)	No {yellow/70}	No {yellow/70}	Yes {green/70}
One mechanism for DataFrame and SQL writes	Yes (both become query plans) {green/70}	Yes (both become Spark executions) {green/70}	No (DataFrame API calls and SQL statements are detected differently) {yellow/70}

4.3. System design

The library is implemented by monkey-patching the PySpark framework: the chaos testing library patches the SparkSession object before it gets used by the user job.
Such patches are dynamic, and they allow the library to inject controlled failures during user job executions. The following diagram shows the library workflow:

---
config:
  block:
    padding: 0
  themeVariables:
    mainBkg: transparent
    nodeBorder: transparent
  themeCSS: "foreignObject{overflow:visible !important}.edgeLabel span.edgeLabel,.edgeLabel p{display:block !important;margin:0 !important;padding:0 !important;background:transparent !important}[id$='-bgchaos'] rect.composite{transform-box:fill-box;transform-origin:50% 0;transform:scale(1,9);vector-effect:non-scaling-stroke;rx:5px;ry:0.5556px}"
---
block-beta
  columns 33
  block:bgchaos:33
    columns 1
    bgchaosx["<span style='display:block;width:660px;height:44px;line-height:44px;text-align:center;font-weight:600;font-size:15px;color:rgb(20,24,28)'>Spark Chaos Testing</span>"]
  end
  space
  block:left:20
    columns 20
    space:20
    space space space space space space space space py(("<span style='display:block;width:24px;height:24px;margin:0 -2.0px;background:url(data:image/svg+xml;base64,PHN2ZyB2aWV3Qm94PSIxMi44MSAxMi4yNSAxMDIuMzggMTAzLjUxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjxsaW5lYXJHcmFkaWVudCBpZD0icHl0aG9uLWEiIGdyYWRpZW50VW5pdHM9InVzZXJTcGFjZU9uVXNlIiB4MT0iNzAuMjUyIiB5MT0iMTIzNy40NzYiIHgyPSIxNzAuNjU5IiB5Mj0iMTE1MS4wODkiIGdyYWRpZW50VHJhbnNmb3JtPSJtYXRyaXgoLjU2MyAwIDAgLS41NjggLTI5LjIxNSA3MDcuODE3KSI+PHN0b3Agb2Zmc2V0PSIwIiBzdG9wLWNvbG9yPSIjNUE5RkQ0Ii8+PHN0b3Agb2Zmc2V0PSIxIiBzdG9wLWNvbG9yPSIjMzA2OTk4Ii8+PC9saW5lYXJHcmFkaWVudD48bGluZWFyR3JhZGllbnQgaWQ9InB5dGhvbi1iIiBncmFkaWVudFVuaXRzPSJ1c2VyU3BhY2VPblVzZSIgeDE9IjIwOS40NzQiIHkxPSIxMDk4LjgxMSIgeDI9IjE3My42MiIgeTI9IjExNDkuNTM3IiBncmFkaWVudFRyYW5zZm9ybT0ibWF0cml4KC41NjMgMCAwIC0uNTY4IC0yOS4yMTUgNzA3LjgxNykiPjxzdG9wIG9mZnNldD0iMCIgc3RvcC1jb2xvcj0iI0ZGRDQzQiIvPjxzdG9wIG9mZnNldD0iMSIgc3RvcC1jb2xvcj0iI0ZGRTg3MyIvPjwvbGluZWFyR3JhZGllbnQ+PHBhdGggZmlsbD0idXJsKCNweXRob24tYSkiIGQ9Ik02My4zOTEgMS45ODhjLTQuMjIyLjAyLTguMjUyLjM3OS0xMS44IDEuMDA3LTEwLjQ1IDEuODQ2LTEyLjM0NiA1LjcxLTEyLjM0NiAxMi44Mzd2OS40MTFoMjQuNjkzdjMuMTM3SDI5Ljk3N2MtNy4xNzYgMC0xMy40NiA0LjMxMy0xNS40MjYgMTIuNTIxLTIuMjY4IDkuNDA1LTIuMzY4IDE1LjI3NSAwIDI1LjA5NiAxLjc1NSA3LjMxMSA1Ljk0NyAxMi41MTkgMTMuMTI0IDEyLjUxOWg4LjQ5MVY2Ny4yMzRjMC04LjE1MSA3LjA1MS0xNS4zNCAxNS40MjYtMTUuMzRoMjQuNjY1YzYuODY2IDAgMTIuMzQ2LTUuNjU0IDEyLjM0Ni0xMi41NDhWMTUuODMzYzAtNi42OTMtNS42NDYtMTEuNzIwLTEyLjM0Ni0xMi44MzctNC4yNDQtLjcwNi04LjY0NS0xLjAyNy0xMi44NjYtMS4wMDh6TTUwLjAzNyA5LjU1N2MyLjU1IDAgNC42MzQgMi4xMTcgNC42MzQgNC43MjEgMCAyLjU5My0yLjA4MyA0LjY5LTQuNjM0IDQuNjktMi41NiAwLTQuNjMzLTIuMDk3LTQuNjMzLTQuNjktLjAwMS0yLjYwNCAyLjA3My00LjcyMSA0LjYzMy00LjcyMXoiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgMTAuMjYpIi8+PHBhdGggZmlsbD0idXJsKCNweXRob24tYikiIGQ9Ik05MS42ODIgMjguMzh2MTAuOTY2YzAgOC41LTcuMjA4IDE1LjY1NS0xNS40MjYgMTUuNjU1SDUxLjU5MWMtNi43NTYgMC0xMi4zNDYgNS43ODMtMTIuMzQ2IDEyLjU0OXYyMy41MTVjMCA2LjY5MSA1LjgxOCAxMC42MjggMTIuMzQ2IDEyLjU0NyA3LjgxNiAyLjI5NyAxNS4zMTIgMi43MTMgMjQuNjY1IDAgNi4yMTYtMS44MDEgMTIuMzQ2LTUuNDIzIDEyLjM0Ni0xMi41NDd2LTkuNDEySDYzLjkzOHYtMy4xMzhoMzcuMDEyYzcuMTc2IDAgOS44NTItNS4wMDUgMTIuMzQ4LTEyLjUxOSAyLjU3OC03LjczNSAyLjQ2Ny0xNS4xNzQgMC0yNS4wOTYtMS43NzQtNy4xNDUtNS4xNjEtMTIuNTIxLTEyLjM0OC0xMi41MjFoLTkuMjY4ek03Ny44MDkgODcuOTI3YzIuNTYxIDAgNC42MzQgMi4wOTcgNC42MzQgNC42OTIgMCAyLjYwMi0yLjA3NCA0LjcxOS00LjYzNCA0LjcxOS0yLjU1IDAtNC42MzMtMi4xMTctNC42MzMtNC43MTkgMC0yLjU5NSAyLjA4My00LjY5MiA0LjYzMy00LjY5MnoiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgMTAuMjYpIi8+PC9zdmc+Cg==) center/contain no-repeat'></span>")) space:11
    k2(("<span style='display:block;width:1px;height:1px'></span>")) space space space space rl["<span style='display:block;width:20px;height:42px;'></span>"] space space runner["<span style='display:block;width:140px;height:42px;box-sizing:border-box;margin:0 -60px;line-height:42px;text-align:center;background:rgb(20,24,28);border-radius:5px;color:rgb(255,255,255);font-weight:500'>Runner</span>"] space space rr["<span style='display:block;width:20px;height:42px;'></span>"] space:8
    space space space space space space space space inject(("<span style='display:block;width:20px;height:20px;line-height:20px;font-size:24px;color:rgb(224,49,49);text-align:center'>✸</span>")) il["<span style='display:block;width:140px;height:42px;box-sizing:border-box;margin-right:-120px;padding:5px 0 0 8px;line-height:16px;text-align:left;font-size:13px'>Inject failure<br>(Monkey Patch) <span style='display:inline-block;min-width:16px;height:16px;line-height:16px;border-radius:999px;background:rgb(232,236,239);color:rgb(20,24,28);font-size:10.5px;font-weight:600;font-style:normal;text-align:center;vertical-align:middle'>1</span></span>"] space:10
    space:20
    space space space space space space space al["<span style='display:block;width:140px;height:42px;box-sizing:border-box;line-height:42px;text-align:right;padding-right:10px;font-size:13px;margin-left:-120px'>Spark Client API</span>"] api(("<span style='display:block;width:33px;height:28px;margin:0 -6.5px;background:url(data:image/svg+xml;base64,PHN2ZyB2aWV3Qm94PSIyNTYuOTUgLTMuMTQgMTU5LjE2IDEzNS4wOSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj48ZyB0cmFuc2Zvcm09Im1hdHJpeCgxLjMzIDAgMCAxLjMzIDAgNC4wNWUtNikiPjxwYXRoIGQ9Im0yODQgODEuNWMtLjI2OS0uNTc1LS4zOS0uODY0LS41MzUtMS4xNC0zLjg5LTcuNC03Ljc3LTE0LjgtMTEuNy0yMi4yLS4zOTQtLjc0Mi0uMzQ0LTEuMTguMTk1LTEuODIgNi4xOS03LjIzIDEyLjMtMTQuNSAxOC41LTIxLjcuMjE5LS4yNTQuNDE0LS41MjMuNDk2LS45NzYtMS44LjQ2OS0zLjYuOTMzLTUuNDEgMS40MS03LjQ3IDEuOTgtMTUgMy45NS0yMi40IDUuOTYtLjY5NS4xODctMS4wMS0uMDE2LTEuMzYtLjU5NC00LjI0LTcuMDktOC41MS0xNC4yLTEyLjgtMjEuMi0uMjE5LS4zNjctLjQ1OC0uNzIzLS45MjYtMS4wMy0uMzQ0IDEuODktLjY5MiAzLjc3LTEuMDMgNS42Ni0xLjE5IDYuNjctMi4zOCAxMy4zLTMuNTYgMjAtLjEyOS43MTktLjMwNSAxLjQ0LS4zNjcgMi4xNi0uMDU5LjY4OC0uNDE0Ljk0Mi0xLjA0IDEuMTQtOC44MSAyLjc3LTE3LjYgNS41Ni0yNi40IDguMzUtLjM4Ni4xMjEtLjc2NS4yODEtMS4xOC42NTYgNy4yIDIuODYgMTQuNCA1LjcyIDIxLjcgOC42My0uMjY2LjIxMS0uNDQyLjM3NS0uNjM3LjUtNC41IDIuOTEtOS4wMSA1LjgyLTEzLjUgOC43NC0uNTM5LjM1Mi0uOTY1LjQwMi0xLjU3LjEyOS01LjM4LTIuNDItMTAuOC00LjgtMTYuMi03LjItMi40Mi0xLjA4LTQuNTktMi41MS02LjI4LTQuNTktMy44My00LjY4LTMuMDctMTAgMi4wMy0xMy4zIDEuNjctMS4wNyAzLjU3LTEuODYgNS40Ni0yLjQ4IDguNjMtMi44MSAxNy4zLTUuNTIgMjUuOS04LjI0LjcyNi0uMjMxIDEuMDYtLjU1NSAxLjItMS4zNCAxLjE2LTYuNjggMi4zNS0xMy40IDMuNTYtMjAgLjY0NC0zLjU3Ljk4NC03LjIyIDIuNzItMTAuNS42NjUtMS4yNiAxLjQ2LTIuNDkgMi40Mi0zLjU0IDMuNDQtMy44MiA4LjIzLTMuOTYgMTEuOS0uMzI4IDEuMjMgMS4yMyAyLjI4IDIuNjcgMy4xOSA0LjE2IDQgNi41MyA3Ljk0IDEzLjEgMTEuOSAxOS43LjQ2NS43NzcuODgzLjkzMyAxLjc0LjcwNyA5LjY1LTIuNTkgMTkuMy01LjE0IDI5LTcuNjkgMi0uNTI3IDQuMDEtLjcxOCA2LjA2LS4zMzUgNC40NS44MzIgNi40IDQuMjIgNC44OSA4LjUzLS42ODMgMS45Ni0xLjg3IDMuNjEtMy4yIDUuMTgtNi43NCA3LjkzLTEzLjUgMTUuOS0yMC4yIDIzLjgtLjU1MS42NDgtLjU2MiAxLjExLS4xOCAxLjg0IDQuMDQgNy42MSA4LjA0IDE1LjIgMTIuMSAyMi45Ljk2MSAxLjgyIDEuNyAzLjcyIDEuNzIgNS44MS4wNDcgNC43Ni0zLjQ0IDguNjYtOC4xNyA5LjM2LTIuNjUuMzg3LTUuMTEtLjE3OS03LjYtLjk0NS02LjA1LTEuODctMTIuMS0zLjcxLTE4LjItNS41NC0uNTYzLS4xNjgtLjc3OC0uMzkxLS44NzktLjk5Ni0uNjk2LTQuMjctMS40Ni04LjUzLTIuMi0xMi44LS4wMTktLjExNy4wMTYtLjI0Mi4wMzYtLjUwNCA2LjkxIDEuOTEgMTMuOCAzLjc5IDIwLjkgNS43NiIgc3R5bGU9ImZpbGwtcnVsZTpldmVub2RkO2ZpbGw6I2UyNWExYyIvPjwvZz48L3N2Zz4K) center/contain no-repeat'></span>")) space:11
    space:20
    k1(("<span style='display:block;width:1px;height:1px'></span>")) space space space space space space space crash(("<span style='display:block;width:20px;height:20px;line-height:20px;font-size:24px;color:rgb(224,49,49);text-align:center'>✸</span>")) cl["<span style='display:block;width:80px;height:42px;box-sizing:border-box;line-height:42px;text-align:left;padding-left:8px;font-size:13px;margin-right:-60px'>Crash</span>"] space:10
  end
  block:user:11
    columns 11
    space:11
    space space space space space ut["<span style='display:block;width:220px;height:14px;margin:0 -100px;line-height:14px;text-align:center;font-weight:600;font-size:13px'>User code</span>"] space space space space space
    space:11
    rt["<span style='display:block;width:20px;height:14px;'></span>"] space space space space space space space space space space
    space space clean["<span style='display:block;width:140px;height:40px;box-sizing:border-box;margin:-13px -120px -13px 0;line-height:38px;text-align:center;background:rgb(245,245,245);border:1px solid rgb(102,102,102);border-radius:5px;color:rgb(20,24,28);font-weight:500'>Clean</span>"] space space space space space space space space
    space:11
    space:11
    space:11
    space space validate["<span style='display:block;width:140px;height:40px;box-sizing:border-box;margin:-13px -120px -13px 0;line-height:38px;text-align:center;background:rgb(245,245,245);border:1px solid rgb(102,102,102);border-radius:5px;color:rgb(20,24,28);font-weight:500'>Validate</span>"] space space space space space space space space
    space:11
    space:11
    space:11
    space space job["<span style='display:block;width:140px;height:40px;box-sizing:border-box;margin:-13px -120px -13px 0;line-height:38px;text-align:center;background:rgb(20,24,28);border:1px solid rgb(20,24,28);border-radius:5px;color:rgb(255,255,255);font-weight:500'>Job</span>"] space space space space space space space space
    space:11
    space:11
    space:11
  end
  space
  block:gap:33
    columns 1
    gapx["<span style='display:block;width:660px;height:46px;'></span>"]
  end
  block:bgstore:33
    columns 33
    space space space space space space space space space sx["<span style='display:block;width:20px;height:48px;'></span>"] space space space space space space sl["<span style='display:block;width:160px;height:48px;margin:0 -70.0px 0 -70.0px;line-height:48px;text-align:center;font-weight:600;font-size:15px;color:rgb(20,24,28)'>Storage</span>"] space space space space space space space space space space space space space space space space
  end
  runner --- inject
  inject --> api
  rr -- "<span style='display:block;width:max-content'><span style='display:flex;align-items:center;justify-content:center;height:26px;margin-bottom:-2px'><span style='display:inline-block;min-width:16px;height:16px;line-height:16px;border-radius:999px;background:rgb(232,236,239);color:rgb(20,24,28);font-size:10.5px;font-weight:600;font-style:normal;text-align:center;vertical-align:middle'>2</span><span style='margin-left:5px;color:rgb(232,89,12);font-size:26px;line-height:26px'>⇄</span></span><span style='display:block;width:max-content;margin:0 auto;padding:0 7px;line-height:16px;background:rgb(247,251,255)'><span style='font-style:italic;color:rgb(90,101,112);font-size:12.5px'>Run</span></span><span style='display:block;height:24px'></span></span>" --> rt
  job -- "<span style='display:block;width:max-content;margin:0 auto;padding:0 7px;line-height:16px;background:rgb(247,251,255)'><span style='font-style:italic;color:rgb(90,101,112);font-size:12.5px'>Write</span> <span style='display:inline-block;min-width:16px;height:16px;line-height:16px;border-radius:999px;background:rgb(232,236,239);color:rgb(20,24,28);font-size:10.5px;font-weight:600;font-style:normal;text-align:center;vertical-align:middle'>3</span></span>" --> api
  api -- "<span style='display:block;width:max-content;margin:0 auto;padding:7px 0;line-height:16px;background:rgb(247,251,255)'><span style='font-style:italic;color:rgb(90,101,112);font-size:12.5px'>Write</span> <span style='display:inline-block;min-width:16px;height:16px;line-height:16px;border-radius:999px;background:rgb(232,236,239);color:rgb(20,24,28);font-size:10.5px;font-weight:600;font-style:normal;text-align:center;vertical-align:middle'>3</span></span>" --- crash
  crash --> sx
  crash -.- k1
  k1 -- "<span style='display:block;width:max-content;margin:0 auto;padding:7px 0;line-height:16px;background:rgb(247,251,255)'><span style='font-style:italic;color:rgb(90,101,112);font-size:12.5px'>Catch</span></span>" -.- k2
  k2 -.-> rl
  style bgchaos fill:#f7fbff,stroke:#5f6163
  style bgstore fill:#ffffff,stroke:#5f6163
  style user fill:#ffffff,stroke:#5f6163
  classDef bare fill:transparent,stroke:transparent
  class left,gap bare

Fig 3. Chaos testing library's software architecture

In particular:

The chaos testing library is used by invoking the Runner . The Runner gets the SparkSession and wraps it with a thin injection layer that decides whether the API call should produce a crash or behave as a pass-through towards the original Spark API.
The Runner starts the chaos testing routine (running test sequences) by using three user-provided parameters:
- Job: the user Spark job which must be chaos tested
- Clean : a callback function that restores the system to a clean state \(S\) (e.g., set data inputs used by the job). It's called at the beginning of each test sequence.
- Validate: a callback function that runs after each test sequence, once the happy run has been executed. The purpose of such a callback is testing the job output and ensure its resiliency to failures. For example, such a function can be used as a given-expected test over output data produced by the job.
Within each execution, the user job blindly interact with the SparkSession and invokes data write actions through the Spark API. Based on the current chaos round, such API calls produce a controlled crash (caught by the chaos library) or an actual data write.

4.4. Constraints

The library has the following constraints:

No multithreading: there is just one simple Python process running the chaos testing. The chaos testing library executes the user job within the same process and catches crashes. Data writes are assumed to be performed sequentially.
The library works best with "straight-line" jobs with business logic that doesn't alter the execution path across crash runs (e.g., checkpointing or rollback mechanisms through control flows).
The first test sequence of the chaos test (S -> H -> V) is exploited to analyze the job and determine the chaos plan. If crash runs performed during next test sequences deviate from the chaos plan, the library could not be able to ensure the plan execution. Even though the library tries its best to manage some of such scenarios with some mechanism, it may not be suitable for convoluted business logics.
The user job doesn't catch crashes with try-catch logics.
The user job can be executed within a single session by using the same SparkSession.
No streaming support. However, single stream scenarios where the job writes to arbitrary data sink through "batch" API (e.g., writing to a table within a foreachBatch ) could be already supported out of the box or relatively easy to implement.
The set of intercept API is predetermined and unknown writes don't get recognized:
- DataFrame API: the library patches common Spark classes such as DataFrameWriter and DataFrameWriterV2 , which catch almost any possible batch write (files, tables, JDBC, etc.)
- Spark SQL: writing actions are intercepted by parsing SQL queries through sqlglot (the only dependency of the chaos testing library). Intercepted actions are DML such as INSERT, UPDATE, DELETE, etc.

5. Testing Spark pipelines

The chaos testing is invoked by calling the method run_chaos , which takes the three arguments mentioned in the previous section

run_chaos(job=job_main, clean=clean_callback, validate=validate_callback)

The method, after the chaos test ends, returns a report, which summarizes the test outcome, the executed test sequences, and other useful information for root cause analysis.

5.1. Testing a resilient pipeline

Suppose to have a simple pipeline made of three data writes which generate a sequence of numbers, multiplies them by a value and filters the result:

---
config:
  block:
    padding: 0
  themeCSS: "foreignObject{overflow:visible !important}.edgeLabel span.edgeLabel,.edgeLabel p{display:block !important;margin:0 !important;padding:0 !important;background:transparent !important}[id$='-storage'] rect.composite{transform-box:fill-box;transform-origin:50% 100%;transform:scale(1,1.08446);vector-effect:non-scaling-stroke}[id$='-job'] rect.composite{transform-box:fill-box;transform-origin:50% 0;transform:scale(1,1.0001);vector-effect:non-scaling-stroke}"
---
block-beta
  columns 7
  block:job:7
    columns 7
    space:7
    title["<b style='display:block;width:110px;height:40px;line-height:40px;text-align:center'>Compute</b>"] w1["<span style='display:block;width:110px;height:40px;line-height:40px;text-align:center;color:rgb(255,255,255)'>Write 1</span>"] space w2["<span style='display:block;width:110px;height:40px;line-height:40px;text-align:center;color:rgb(255,255,255)'>Write 2</span>"] space w3["<span style='display:block;width:110px;height:40px;line-height:40px;text-align:center;color:rgb(255,255,255)'>Write 3</span>"] space
    space:7
  end
  block:gap:7
    columns 1
    gapfill["<span style='display:block;width:770px;height:84px'></span>"]
  end
  block:anchors:7
    columns 7
    space a1["<span style='display:block;width:110px;height:1px'></span>"] space a2["<span style='display:block;width:110px;height:1px'></span>"] space a3["<span style='display:block;width:110px;height:1px'></span>"] space
  end
  block:storage:7
    columns 7
    st["<b style='display:block;width:110px;height:296px;line-height:296px;text-align:center'>Storage</b>"] t1["<div style='box-sizing:border-box;width:110px;height:296px'><table style='width:110px;margin:0;border-collapse:separate;border-spacing:0;border:var(--hairline) solid var(--border);border-radius:var(--radius-lg)'><thead><tr><th style='padding:2px 16px;line-height:20px;position:static;border-radius:calc(var(--radius-lg) - var(--hairline)) calc(var(--radius-lg) - var(--hairline)) 0 0'>value</th></tr></thead><tbody><tr><td style='padding:2px 16px;line-height:20px'>0</td></tr><tr><td style='padding:2px 16px;line-height:20px'>1</td></tr><tr><td style='padding:2px 16px;line-height:20px'>2</td></tr><tr><td style='padding:2px 16px;line-height:20px'>3</td></tr><tr><td style='padding:2px 16px;line-height:20px'>4</td></tr><tr><td style='padding:2px 16px;line-height:20px'>5</td></tr><tr><td style='padding:2px 16px;line-height:20px'>6</td></tr><tr><td style='padding:2px 16px;line-height:20px'>7</td></tr><tr><td style='padding:2px 16px;line-height:20px'>8</td></tr><tr><td style='padding:2px 16px;line-height:20px'>9</td></tr></tbody></table></div>"] space t2["<div style='box-sizing:border-box;width:110px;height:296px'><table style='width:110px;margin:0;border-collapse:separate;border-spacing:0;border:var(--hairline) solid var(--border);border-radius:var(--radius-lg)'><thead><tr><th style='padding:2px 16px;line-height:20px;position:static;border-radius:calc(var(--radius-lg) - var(--hairline)) calc(var(--radius-lg) - var(--hairline)) 0 0'>value</th></tr></thead><tbody><tr><td style='padding:2px 16px;line-height:20px'>0</td></tr><tr><td style='padding:2px 16px;line-height:20px'>2</td></tr><tr><td style='padding:2px 16px;line-height:20px'>4</td></tr><tr><td style='padding:2px 16px;line-height:20px'>6</td></tr><tr><td style='padding:2px 16px;line-height:20px'>8</td></tr><tr><td style='padding:2px 16px;line-height:20px'>10</td></tr><tr><td style='padding:2px 16px;line-height:20px'>12</td></tr><tr><td style='padding:2px 16px;line-height:20px'>14</td></tr><tr><td style='padding:2px 16px;line-height:20px'>16</td></tr><tr><td style='padding:2px 16px;line-height:20px'>18</td></tr></tbody></table></div>"] space t3["<div style='box-sizing:border-box;width:110px;height:296px'><table style='width:110px;margin:0;border-collapse:separate;border-spacing:0;border:var(--hairline) solid var(--border);border-radius:var(--radius-lg)'><thead><tr><th style='padding:2px 16px;line-height:20px;position:static;border-radius:calc(var(--radius-lg) - var(--hairline)) calc(var(--radius-lg) - var(--hairline)) 0 0'>value</th></tr></thead><tbody><tr><td style='padding:2px 16px;line-height:20px'>12</td></tr><tr><td style='padding:2px 16px;line-height:20px'>14</td></tr><tr><td style='padding:2px 16px;line-height:20px'>16</td></tr><tr><td style='padding:2px 16px;line-height:20px'>18</td></tr></tbody></table></div>"] space
  end
  w1 --> w2
  w2 --> w3
  w1 -- "<span style='display:block;width:max-content;margin:0 auto'><span style='display:block;height:36px'></span><span style='display:block;padding:0 7px;line-height:16px;background:var(--bg);font-style:italic;color:rgb(90,101,112);font-size:12.5px'>0...9 numbers</span></span>" --> a1
  t1 --> w2
  w2 -- "<span style='display:block;width:max-content;margin:0 auto'><span style='display:block;height:36px'></span><span style='display:block;padding:0 7px;line-height:16px;background:var(--bg);font-style:italic;color:rgb(90,101,112);font-size:12.5px'>numbers * 2</span></span>" --> a2
  t2 --> w3
  w3 -- "<span style='display:block;width:max-content;margin:0 auto'><span style='display:block;height:36px'></span><span style='display:block;padding:0 7px;line-height:16px;background:var(--bg);font-style:italic;color:rgb(90,101,112);font-size:12.5px'>numbers > 10</span></span>" --> a3
  classDef write fill:#14181c,stroke:#14181c,color:#ffffff
  classDef ghost fill:transparent,stroke:transparent
  class w1,w2,w3 write
  class title,st,t1,t2,t3,a1,a2,a3,anchors,gap,gapfill ghost
  style job fill:#f7fbff,stroke:#5f6163
  style storage fill:#ffffff,stroke:#5f6163

Fig 4. A three write steps pipeline

Such a pipeline can be implemented as follows:

def job() -> None: 
    spark = SparkSession.builder.getOrCreate()

    # Step 1: numbers 0..9.
    numbers = spark.range(10)
    numbers.write.mode("overwrite").parquet(num_path)

    # Step 2: read step 1 and double each number.
    doubled = spark.read.parquet(num_path).select((col("id") * 2).alias("value"))
    doubled.write.mode("overwrite").parquet(doubled_path)

    # Step 3: read step 2 and keep only the values above 10.
    big = spark.read.parquet(doubled_path).filter(col("value") > 10)
    big.write.mode("overwrite").parquet(big_path)

Then, running the chaos test requires few lines:

def test_job(spark: SparkSession) -> None:
    def clean() -> None:
        """
        Resets the state to "S" (clean state)
        """
        for path in (num_path, doubled_path, big_path):
            shutil.rmtree(path, ignore_errors=True)

    def validate() -> None:
        """
        The validation step "V", executed after each test sequence
        """
        results = sorted(r.value for r in spark.read.parquet(big_path).collect())
        assert results == [12, 14, 16, 18]

    report = run_chaos(job=job, clean=clean, validate=validate)
    print(report)

    assert report.outcome is Outcome.SUCCESS

The library returns the following output:

Summary

Property	Value
Result	SUCCESS {green/70}
Data writes found (N)	3
Test sequences	4 (4 passed, 0 failed)
Duration	6.1s

The chaos test completes with SUCCESS: The job is considered resilient.
The library correctly detected all three Parquet writes performed in the user job
Since there are \(N=3\) data writes, the number of possible test sequences is \(2^{3-1}=4\)

Detected writes

#	API	Target	Call site
Write 1	`DataFrameWriter.parquet`	`.../numbers`	`.../test_job.py:6`
Write 2	`DataFrameWriter.parquet`	`.../doubled`	`.../test_job.py:10`
Write 3	`DataFrameWriter.parquet`	`.../big`	`.../test_job.py:14`

Since every data write stores outputs as Parquet, the job uses the DataFrameWriter.parquet Spark API
The "Target" column shows the output path of such data writes.
The "Call site" column shows the code line in which the user job performs such data writes.

Executed sequences

#	Result	Sequence	Runs
1	PASS {green/70}	S -> H -> V	COMPLETED (H)
2	PASS {green/70}	S -> Write 2 -> H -> V	CRASHED (w2), COMPLETED (H)
3	PASS {green/70}	S -> Write 3 -> H -> V	CRASHED (w3), COMPLETED (H)
4	PASS {green/70}	S -> Write 2 -> Write 3 -> H -> V	CRASHED(w2), CRASHED (w3), COMPLETED (H)

5.2. Testing a not resilient pipeline

If line 10 (Write 2) in the job changes from

doubled.write.mode("overwrite").parquet(doubled_path)

doubled.write.mode("append").parquet(doubled_path)

The chaos test should mark the job as not resilient, since there is a non-idempotent operation that generates duplicates across crash runs.

Running the chaos test again returns the expected output:

Summary

Property	Value
Result	FAILURE {red/70}
Data writes found (N)	3
Test sequences	4 (2 passed, 2 failed)
Duration	6.4s

Detected writes

#	API	Target	Call site
Write 1	`DataFrameWriter.parquet`	`.../numbers`	`.../test_job.py:6`
Write 2	`DataFrameWriter.parquet`	`.../doubled`	`.../test_job.py:10`
Write 3	`DataFrameWriter.parquet`	`.../big`	`.../test_job.py:14`

Executed sequences

#	Result	Sequence	Runs
1	PASS {green/70}	S -> H -> V	COMPLETED (H)
2	PASS {green/70}	S -> Write 2 -> H -> V	CRASHED (w2), COMPLETED (H)
3	FAIL {red/70}	S -> Write 3 -> H -> V	CRASHED (w3), COMPLETED (H)
4	FAIL {red/70}	S -> Write 2 -> Write 3 -> H -> V	CRASHED(w2), CRASHED (w3), COMPLETED (H)

In particular:

Test sequence #1: the happy run. The test ends as PASS, since there is no crash at all.
Test sequence #2: the test ends as PASS, since the job crashes before performing the non-idempotent write. Hence, a new job run (H) operates on data generated by Write 1 (which is idempotent).
Test sequence #3: The test ends as FAIL, since the job crashes when Write 2 was already performed (non-idempotent write). A new job run (H) operates on data previously generated by Write 2 and generate duplicates. The final output ("big") contains 8 records instead of 4.
Test sequence #4: Same as #3. The crash before Write 2 doesn't interfere, since the non-idempotent write didn't happen at that time. The final output ("big") contains 8 records instead of 4.

6. Wrap up

Pipeline resilience testing shouldn't rely on manual "What if?" sessions.

By automating deterministic crash injections and recovery verifications, developers can turn those questions into repeatable tests, run during development or as part of CI/CD pipelines.

Moreover, such an approach becomes particularly useful when working with AI agents. Instead of relying on an agent's reasoning alone to assess the impact of code changes, developers can let it run automated resilience tests and use the result to identify regressions, validate fixes, and verify recovery behavior.

Notes

alternative-approach

Alternative approaches

An alternative approach that may be worth further exploration could be shifting the fault injection to the storage/catalog level rather than managing it at the engine level (e.g., making a write crash by dropping its target table).

Such an approach would eliminate the need for engine-specific implementations, but it would require different crash strategies based on the sink type.

mixed-api

Please don't.