A new way to think about connected work. Explore the RelayOS demo ↗
Data

Data quality: a practical automation guide

Reject incomplete rows before warehouse ingestion. Learn how to define the event, map data, test edge cases and document a maintainable workflow.

RelayOS Editorial · · Demo content

Start with the decision

Before automating data quality, describe the decision the receiving team needs to make. Reject incomplete rows before warehouse ingestion is specific enough to guide the design, but it still needs an owner and a stopping condition. An automation that moves every available field is harder to review than a small contract with a clear purpose.

Write a compact event contract

In this example PostgreSQL owns the initial event and BigQuery owns the follow-up. Start with primary key, changed fields and version. Add the event identifier, event time, schema version and intended action. Distinguish a value that is unknown from a value that was intentionally cleared.

Make duplicate delivery harmless

Many event systems can deliver the same update more than once. Use the source event ID or a documented composite key to recognize a replay. Store the outcome after the destination confirms the write. If a network timeout leaves the result unknown, inspect the destination before creating another record.

Keep humans in the exception path

Treat a validation failure as a request for review. An operator should see which record failed, why it failed and whether retrying can help. Do not make someone search a raw log to understand the next action. Provide a link to the source and mask any fields that are unnecessary for troubleshooting.

Test the boundaries

For data quality, the minimum sample should include an ordinary update, a repeated event, a missing required value, an unknown owner and a destination timeout. Compare the expected final state with what BigQuery actually contains. Record the result in the rollout checklist.

Document the operating agreement

Name the process owner, connection owner and escalation contact. Explain how to pause the workflow, rotate credentials and replay an event. Set a review date for the mapping so a change to the PostgreSQL schema does not silently break the destination. This is general engineering guidance for a demonstration product; validate vendor-specific behavior against current documentation.