artenis.alija
ende
Data Pipelines · Python · Architecture6 min read10 December 2024

Data Pipeline Design Patterns That Hold

The patterns that separate pipelines that work from pipelines that work reliably — idempotency, observability, graceful degradation, and schema evolution.

Most data pipelines work on day one. The interesting question is how they behave on day 90, when the source changed its API, when a batch run was interrupted halfway through, when someone accidentally ran the pipeline twice. The patterns that matter are the boring ones.

Idempotency first. Every pipeline step should produce the same output given the same input, regardless of how many times it runs. This means content-based deduplication (hash the record, store the hash), not timestamp-based. Timestamps are fragile — they change, they're timezone-ambiguous, they lead to subtle double-ingestion bugs.

Observability is the second pillar. A pipeline with no metrics is an unmaintainable pipeline. I instrument every step with: records in, records out, records skipped (with reason), duration, and last successful run time. This goes to a Postgres table first, then to a dashboard if the client wants one.

Schema evolution is where pipelines go brittle. I use Pydantic models for all inter-stage data contracts and version them explicitly. When upstream schema changes, the model migration is a PR — reviewable, testable, rollbackable. Never parse raw JSON beyond the ingestion boundary.

Work with me

Want this working in your business?

I build the automations I write about: AI agents, n8n workflows and integrations that save hours every week. Tell me what is slowing you down and I’ll show you what can be automated, free.

Or join the monthly newsletter and get my 6 n8n workflow templates free

One email at the start of each month. Nothing more.

Monthly newsletter: one email a month, never more, with free tools and occasional offers of my services. I may also send the odd update on WhatsApp; reply STOP to opt out. Unsubscribe with one click.

Citation

Artenis Alija. "Data Pipeline Design Patterns That Hold Under Pressure." 2024. https://artenisalija.com/blog/data-pipeline-design-patterns/

https://artenisalija.com/blog/data-pipeline-design-patterns/
Free monthly newsletter

One email a month with an AI automation you can use

Sent at the start of each month, never more often. Real workflows I build for clients, new free tools first, and the n8n templates behind them. Sign up now and get my 6 n8n workflow templates straight away.

Monthly newsletter: one email a month, never more, with free tools and occasional offers of my services. I may also send the odd update on WhatsApp; reply STOP to opt out. Unsubscribe with one click.