Beyond the happy path: Building resilient data pipelines

Happy path thinking is a recurring behavior adopted by humans to simplify things. Often, such behavior leads to fragile systems built on oversimplified models which fails in real-world conditions.

NOTE This article is the first part of Beyond the happy path: Building resilient data pipelines series.

The (happy) ideal scenario

In software and data modeling, a happy path (or happy flow) describes the default execution path where no errors or exceptional scenarios occur.

Relatedly, happy-path thinking is human behavior that leads us to think and operate exclusively under the happy path.
Mostly derived from the primary human need to simplify things, solutions get built based on an oversimplified model where there is just one possible scenario: everything goes fine forever.

Under this optimistic scenario, the system operates flawlessly:

The system is human-proof: business requirements never change once they’ve been discussed. Moreover, pipelines are always bug-free and human mistakes never happen during the whole system life-cycle.
Pipelines never fail: pipelines always run on schedule and complete successfully during the whole system lifecycle.
Source data are always well-formed and received on time.

The (unhappy) real scenario

The reality is less forgiving: every system will fail; it’s only a matter of time and how it gracefully recovers.
When non-functional requirements are overlooked, they inevitably turn into major pain points:

Humans do human things: humans can change their mind over business requirements by bringing new rules onto the table.
Human mistakes can be mitigated, but they can (and they will) happen in different circumstances (running a wrong query, re-executing a pipeline in an unexpected way, discovering a bug after months of tests and “happy” runs).
Pipelines can fail at any step, potentially leaving behind incomplete states.

Consequences

Ignoring such events can cause technical, operational, and organizational issues:

Data quality degradation: duplicate records, inconsistent or wrong outputs.
Operational fragility: pipelines become hard to recover after a failure, forcing manual intervention, cascading issues and platform mistrust.
Inability to evolve: adapting to new business requirements becomes hard, sometimes even impossible.
Increased cost and waste: wasted time in manual fixes and ad-hoc patches, waste of compute resources.
Loss of trust: stakeholders lose confidence in data, shadow systems are born, every new change is felt as “risky”.

Managing the unhappy scenario

Maintainability first

Unless justified by a concrete requirement (e.g., SLA and costs), prefer the simplest architecture that meets the actual need.
The following table compares a typical low and high complexity pipeline:

	Low complexity pipeline	High complexity pipeline
Examples	Batch, functional and recomputational logic	Streaming or complex incremental logic Complex data models
Mental & Operational model	Re-computational: No states No lookback (past) dependencies	Incremental: State management Lookback (past) dependencies
Recovery	You just re-run the pipeline and whole data get fixed	Must take care of: Run dependencies Already in-place data

When requirements become a real technical constraint, introduce complexity only as required:

Start with a full re-computation
Move to incremental processing when the data volume or latency requirements justify it. Possibly, introduce re-computation only for affected data.
Introduce streaming when the business requirements actually require continuous processing.

Fault tolerance

Well-engineered systems are designed to fail gracefully by incorporating automated error handling, rollbacks, and recovery mechanisms. In such systems, “unexpected paths” are an integral part of the design process.

Human fault tolerance

A resilient, well-engineered data pipeline should be able to:

Accommodate evolving business requirements (e.g., changes in business logic).
Tolerate human errors (e.g., manual corrections, pipeline re-execution in an unexpected way).
Enable easy bug fixing.

In such cases, the engineering team needs to adopt backfilling strategies to reprocess raw data and replace old outputs with new ones.

Backfilling
The practice of reprocessing historical input data to populate or replace previously generated outputs.
When properly used, it allows teams to recover data gaps, apply corrected business logic, or fixing bugs.

Implementing backfilling functionality requires the system to provide certain key properties:

Property	Question	Description
Idempotency	Can I safely run it again?	Running a pipeline multiple times over the same input always produces the same output as running it once.
Reproducibility	Can I regenerate the same result?	The pipeline is able to regenerate outputs from old input data by storing historical data in its rawest form.
Replayability	Can I rerun it only on the affected input?	The pipeline is able to run over a specific subset of input data

System fault tolerance

Since every system can fail, a well-engineered data pipeline should be designed with a pessimistic (realistic) approach, handling such failing events by design:

The pipeline must be able to recover from crash events at any step.
The pipeline must be able to prevent or recover from inconsistent states.
The pipeline must not make assumptions on run frequency and be able to handle N-runs worth of pending unprocessed data.
The pipeline must cover corner cases, such as empty input scenarios.

In such cases, the engineering team needs to implement Recoverability capabilities:

Property	Question	Description
Recoverability	Can the pipeline recover from failure events?	The pipeline is able to recover from crash events at any step, preventing or fixing eventual inconsistent states.

Wrap-up

The happy path is useful to understand how a system is expected to work, but not enough to design one that can survive in production.
Failures, human mistakes, and changing requirements are inevitable.
A resilient data pipeline must therefore be designed to process data successfully, even when things go wrong. This leads to two complementary dimensions of fault tolerance:

Human fault tolerance: dealing with evolving requirements, human mistakes, and bugs.
System fault tolerance: dealing with crashes, partial executions, and pending unprocessed data.

The next two articles in this series will explore these two dimensions and the patterns that accomplish resiliency.

Notes

complexity-requirements

Complexity should be driven by requirements, not by architectural ego.

complexity-comparison

This comparison is intended as a guideline, not a strict rule: there can be batch jobs that are more complex than streaming jobs.