Data Ownership: The single writer principle in Big Data analytics

Context

You have been asked to implement a data system capable of collecting data from dozens of different data sources and harmonizing them into a table with a single common data model used to perform data analysis.

Once requirements are clear, you open up draw.io and start drawing architectural rectangles.
Suppose that, given the heterogeneous set of data source and their corresponding extraction logic complexity, you honor the single responsibility principle and split the system into several independent ingestors.

Chances are you might find yourself thinking about the WRITE path: how to manage the writing process in the common data model table (Fig. 1).

Who should write into it?

The more you deep dive into the problem, the more questions arise:

Who owns the data? Who is responsible for data correctness? Who is responsible for changing it? Who do you call when data is wrong?

All these questions can be grouped into a common topic: Data Ownership.

    
    flowchart LR
    A@{ icon: "adk:file", label: "Source 1" }
    B@{ icon: "adk:database", label: "Source 2" }
    C@{ icon: "adk:queue", label: "Source N" }


    D["Ingestor 1"]:::adkJob
    E["Ingestor 2"]:::adkJob
    F["Ingestor N"]:::adkJob
    
    G@{ icon: "adk:table", label: "Target table", pos: "r" }
    
    A e1@--> |read| D
    B e2@--> |read| E
    C e3@--> |read| F
    
    D e4@--> |should write?| G
    E e5@--> |should write?| G
    F e6@--> |should write?| G
    
    
    
    class e1,e2,e3,e4,e5,e6 adkHot
    
  

A good data flow requires clean ownership

A crystal-clear data flow goes hand in hand with clear ownership boundaries.

You should always be able to tell where data comes from, where it goes, and who is responsible for the state at each step.
When ownership boundaries are explicit, data can flow between components without ending up into a coordination problem. Moreover, as a good side effect, having a clear data flow usually implies better:

• Software quality: by keeping components cohesive and loosely coupled
• Activities breakdown and planning: by splitting the system into independent and parallelizable development tasks.
• System resilience: by making failures easier to understand, recover, and replay

Conversely, when such boundaries are blurry, the data flow becomes harder to reason about, and every additional writer becomes another potential source of contention and potential conflicts.

Dealing with distributed systems

Even if you don't think about data ownership, you still need to deal with technical constraints: Contention and Coordination have always been a big challenge in distributed systems.

Even though ACID transactions landed in the analytical world with modern table formats (Delta Lake, Iceberg and Hudi), they don't behave as transactions you would find in the operational world.

In the analytical world, transactions often operate under OCC, which properly fits the low data contention of an analytical environment. Given that assumption, abusing such a property by using it as a safety net could end up with recurring failures in production.

Data ownership and single writer principle

Since the WRITE path poses so many challenges, there must be some guiding principles which can tackle such recurring issues.

An important principle that should be within every engineer's swiss army knife is what Zhamak Dehghani et al. define as Data Ownership within the book "Software Architecture: The Hard Parts".

The principle states that data ownership must be defined for every table involved in the data flow:

The general rule of thumb for data ownership is that the service that performs write operations to a table is the owner of that table.

Within the book, authors cover many data ownership scenarios (from the straightforward ones to most complex ones) and offers useful criteria to assign proper ownerships.

In particular, the straightforward scenario (single table ownership) is what this article is about.
Moreover, such approach resembles Martin Thompson's Single Writer Principle which highlights the benefits of designing systems that remove or minimize data contention.

Applying the principle

A general rule of thumb to tackle write path issues is trying to model any data flow using single table ownership.
Whenever multiple writers need to write data into the same target table, the single table ownership can be enforced with two approaches: Staging and Queueing.

Staging

Writers who wants to write into the target table, write their data into their own staging area (usually plain parquet files or intermediate tables). Then, a new additional "N+1" writer uniquely takes the ownership of the target table.
This writer is responsible for reading data from the N staging areas and writing it such to the target table.

    
    flowchart LR
    A@{ icon: "adk:file", label: "Source 1" }
    B@{ icon: "adk:database", label: "Source 2" }
    C@{ icon: "adk:queue", label: "Source N" }
    
    D["Ingestor 1"]:::adkJob
    E["Ingestor 2"]:::adkJob
    F["Ingestor N"]:::adkJob

    G@{ icon: "adk:table", label: "Staging 1" }
    H@{ icon: "adk:table", label: "Staging 2" }
    I@{ icon: "adk:table", label: "Staging N" }  
     
    A e1@--> |read| D
    B e2@--> |read| E
    C e3@--> |read| F
    D e4@--> |write| G
    E e5@--> |write| H
    F e6@--> |write| I

    L["Ingestor N+1"]:::adkJob

    G e7@--> |read| L
    H e8@--> |read| L
    I e9@--> |read| L

    M@{ icon: "adk:table", label: "Target table" }
    L e10@--> |write| M
    
    class e1,e2,e3,e4,e5,e6,e7,e8,e9,e10 adkHot
    
  

Queueing

In this case, writers write their data into a shared message queue. Then, a new additional "N+1" writer uniquely takes the ownership of the target table, reads from the queue and writes data into the table.

    
    flowchart LR
    A@{ icon: "adk:file", label: "Source 1" }
    B@{ icon: "adk:database", label: "Source 2" }
    C@{ icon: "adk:queue", label: "Source N" }
    
    D["Ingestor 1"]:::adkJob
    E["Ingestor 2"]:::adkJob
    F["Ingestor N"]:::adkJob

    G@{ icon: "adk:queue", label: "Queue" } 
     
    A e1@--> |read| D
    B e2@--> |read| E
    C e3@--> |read| F
    D e4@--> |write| G
    E e5@--> |write| G
    F e6@--> |write| G

    L["Ingestor N+1"]:::adkJob

    G e7@--> |read| L

    H@{ icon: "adk:table", label: "Target table" }
    L e8@--> |write| H
    
    class e1,e2,e3,e4,e5,e6,e7,e8 adkHot
    
  

Now show me the cons

All the pros have already been mentioned through the whole article.
As always, when you obtain something, you are losing something else.
Here are the main drawbacks which you may encounter:

• Performance issues: intermediate steps can slow down the end-to-end process.
• More complexity: the system is now made of additional intermediate components such as staging areas and queues, which you now need to take care of (more source code, more corner case handling, more infrastructure burden). Moreover, if not mitigated, the complexity shift to the latest "N+1" writer, which now needs to deal with N different datasets and handle them properly during the write phase.
• Write coupling: The N writers cannot write data directly to the target table, and they are all forced to depend on a single latest "N+1" writer.

However, such drawbacks may or may not represent a major issue depending on the use case.

Fallback approaches

There may be times when cons outweigh the pros or constraints don't let you enforce the single writer.
In such case, concurrency issues can be eliminated by proper disjoined writes when the technology allows it (e.g. Delta Lake disjoined partitioned writes).

Takeaways

Data ownership is one of those topics that almost every data architect has to grapple with time and time again.
The single writer principle offers a simple and reusable architectural pattern to deal with ownership and data contention issues.

Notes

requirements